2024-04-25T01:32:09.125 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T01:32:09.126 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337 --verbose -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/orig.config.yaml 2024-04-25T01:32:09.192 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T01:32:09.277 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672337' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI80cQANYn6auBWie0SBScKiuFzz2PmEvH93W8Y3q9Y7kQRaX1BHdFKsP5WQRswML5W8CikniUufyDC6nWaa/+w= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJMoSnR1gKobgS4De0dX/OFqvNlR9inQTGyhB68FiOaRsGqt0GFifQg46k+8BsuFkoQzbZiEJHkLWaEyjXAwDo4= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBASZS/7rqPR1H3ItqxETJ0kvC81fX9Mm6etnf1mW6n1/+Z/IM03qWCnxnHPrRnRMF3A4xxKveguR+jT0D5SQQ8w= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T01:32:09.277 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-25T01:32:09.278 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-25T01:32:09.278 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T01:32:09.280 INFO:teuthology.task.internal:Checking packages... 2024-04-25T01:32:09.303 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-25T01:32:09.303 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T01:32:09.303 INFO:teuthology.packaging:ref: None 2024-04-25T01:32:09.303 INFO:teuthology.packaging:tag: None 2024-04-25T01:32:09.303 INFO:teuthology.packaging:branch: squid 2024-04-25T01:32:09.303 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:32:09.304 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-25T01:32:09.619 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-25T01:32:09.621 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T01:32:09.628 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T01:32:09.628 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T01:32:09.635 INFO:teuthology.task.internal:Saving configuration 2024-04-25T01:32:09.659 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T01:32:09.666 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T01:32:09.687 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:27:15.861733', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI80cQANYn6auBWie0SBScKiuFzz2PmEvH93W8Y3q9Y7kQRaX1BHdFKsP5WQRswML5W8CikniUufyDC6nWaa/+w='} 2024-04-25T01:32:09.708 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:27:15.864535', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJMoSnR1gKobgS4De0dX/OFqvNlR9inQTGyhB68FiOaRsGqt0GFifQg46k+8BsuFkoQzbZiEJHkLWaEyjXAwDo4='} 2024-04-25T01:32:09.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:27:15.863231', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBASZS/7rqPR1H3ItqxETJ0kvC81fX9Mm6etnf1mW6n1/+Z/IM03qWCnxnHPrRnRMF3A4xxKveguR+jT0D5SQQ8w='} 2024-04-25T01:32:09.732 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T01:32:09.738 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T01:32:09.738 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T01:32:09.738 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T01:32:09.738 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T01:32:09.834 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc8d51c4160>, signals=[15]) 2024-04-25T01:32:09.835 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T01:32:09.841 INFO:teuthology.task.internal:Opening connections... 2024-04-25T01:32:09.841 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T01:32:09.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:32:09.918 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-04-25T01:32:09.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:32:10.002 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-04-25T01:32:10.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:32:10.078 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T01:32:10.085 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-04-25T01:32:10.102 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-04-25T01:32:10.102 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-04-25T01:32:10.159 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:32:10.160 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:32:10.161 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-04-25T01:32:10.177 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-04-25T01:32:10.194 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-04-25T01:32:10.195 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-04-25T01:32:10.252 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:32:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:32:10.254 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-04-25T01:32:10.277 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-04-25T01:32:10.296 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-04-25T01:32:10.296 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T01:32:10.353 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-04-25T01:32:10.354 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T01:32:10.354 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T01:32:10.354 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T01:32:10.354 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-04-25T01:32:10.377 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T01:32:10.388 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T01:32:10.393 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T01:32:10.393 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:32:10.396 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:32:10.398 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-04-25T01:32:10.414 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T01:32:10.420 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T01:32:10.421 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:32:10.453 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:32:10.455 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-04-25T01:32:10.542 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T01:32:10.615 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T01:32:10.629 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-04-25T01:32:10.629 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-25T01:32:10.629 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-25T01:32:10.629 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:32:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:32:10.646 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-25T01:32:10.906 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T01:32:10.906 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-25T01:32:10.906 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-04-25T01:32:10.922 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-437.el9.x86_64 2024-04-25T01:32:10.923 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T01:32:10.923 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-25T01:32:10.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum upgrade -y linux-firmware 2024-04-25T01:32:13.361 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-04-25T01:32:16.329 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-04-25T01:32:24.387 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.2 MB 00:04 2024-04-25T01:32:26.130 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-25T01:32:29.550 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 21 MB 00:03 2024-04-25T01:32:34.698 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-25T01:32:36.645 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T01:32:36.646 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2024-04-25T01:32:36.646 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T01:32:36.766 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-25T01:32:36.767 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T01:32:36.915 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-25T01:32:36.916 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-25T01:32:36.918 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi102.front.sepia.ceph.com to complete... 2024-04-25T01:32:49.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25 01:32:49 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-25T01:32:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:FINISHED --2024-04-25 01:32:49-- 2024-04-25T01:32:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:Total wall clock time: 12s 2024-04-25T01:32:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:Downloaded: 1 files, 341M in 12s (28.8 MB/s) 2024-04-25T01:32:49.050 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-25T01:32:49.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-25T01:32:51.167 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying... ######################################## 2024-04-25T01:32:51.298 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing... ######################################## 2024-04-25T01:32:51.401 INFO:teuthology.orchestra.run.smithi102.stdout:Updating / installing... 2024-04-25T01:32:59.307 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-25T01:33:38.723 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-25T01:33:38.724 DEBUG:teuthology.orchestra.run.smithi102:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-25T01:33:38.797 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-25T01:33:38.798 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-25T01:33:38.799 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-25T01:33:38.800 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-25T01:33:38.801 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-25T01:33:38.802 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-25T01:33:38.803 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-25T01:33:38.804 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-25T01:33:38.805 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-25T01:33:38.806 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-25T01:33:38.807 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-25T01:33:38.808 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-25T01:33:38.809 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-25T01:33:38.810 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-25T01:33:38.811 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-25T01:33:38.812 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-25T01:33:38.813 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-25T01:33:38.814 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-25T01:33:38.815 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-25T01:33:38.816 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-25T01:33:38.817 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-25T01:33:38.818 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-25T01:33:38.819 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-25T01:33:38.820 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-25T01:33:38.821 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-25T01:33:38.822 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-25T01:33:38.823 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-25T01:33:38.824 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-25T01:33:38.825 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-25T01:33:38.826 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-25T01:33:38.827 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-25T01:33:38.828 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-25T01:33:38.829 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-25T01:33:38.830 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-25T01:33:38.831 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-25T01:33:38.832 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-25T01:33:38.833 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-25T01:33:38.834 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-25T01:33:38.835 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-25T01:33:38.836 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-25T01:33:38.837 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-25T01:33:38.838 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-25T01:33:38.839 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-25T01:33:38.840 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-25T01:33:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-25T01:33:38.842 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-25T01:33:38.843 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-25T01:33:38.844 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-25T01:33:38.845 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-25T01:33:38.846 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-25T01:33:38.847 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-25T01:33:38.848 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-25T01:33:38.849 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-25T01:33:38.850 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-25T01:33:38.851 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-25T01:33:38.852 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-25T01:33:38.853 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-25T01:33:38.854 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-25T01:33:38.855 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-25T01:33:38.856 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-25T01:33:38.857 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-25T01:33:38.858 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-25T01:33:38.859 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-25T01:33:38.860 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-25T01:33:38.861 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-25T01:33:38.862 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-25T01:33:38.863 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-25T01:33:38.864 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-25T01:33:38.865 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-25T01:33:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-25T01:33:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-25T01:33:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-25T01:33:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-25T01:33:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-25T01:33:38.871 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-25T01:33:38.872 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-25T01:33:38.873 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-25T01:33:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-25T01:33:38.875 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-25T01:33:38.876 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-25T01:33:38.877 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-25T01:33:38.878 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-25T01:33:38.879 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-25T01:33:38.880 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-25T01:33:38.881 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-25T01:33:38.882 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-25T01:33:38.883 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-25T01:33:38.884 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-25T01:33:38.885 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-25T01:33:38.886 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-25T01:33:38.887 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-25T01:33:38.888 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-25T01:33:38.889 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-25T01:33:38.890 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-25T01:33:38.891 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-25T01:33:38.892 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-25T01:33:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-25T01:33:38.894 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-25T01:33:38.895 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-25T01:33:38.896 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-25T01:33:38.897 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-25T01:33:38.898 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-25T01:33:38.899 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-25T01:33:38.900 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-25T01:33:38.901 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-25T01:33:38.902 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-25T01:33:38.903 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-25T01:33:38.904 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-25T01:33:38.905 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-25T01:33:38.906 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-25T01:33:38.907 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-25T01:33:38.908 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-25T01:33:38.909 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-25T01:33:38.910 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-25T01:33:38.911 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-25T01:33:38.912 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-25T01:33:38.912 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-25T01:33:38.912 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-25T01:33:38.912 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-25T01:33:38.912 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-25T01:33:38.912 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-25T01:33:38.913 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-25T01:33:38.913 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-25T01:33:38.914 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-25T01:33:38.915 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-25T01:33:38.916 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-25T01:33:38.917 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-25T01:33:38.918 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-25T01:33:38.919 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-25T01:33:38.920 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-25T01:33:38.921 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-25T01:33:38.922 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-25T01:33:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-25T01:33:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-25T01:33:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-25T01:33:38.926 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-25T01:33:38.927 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-25T01:33:38.928 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-25T01:33:38.929 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-25T01:33:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-25T01:33:38.931 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-25T01:33:38.932 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-25T01:33:38.933 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-25T01:33:38.934 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-25T01:33:38.935 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-25T01:33:38.936 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-25T01:33:38.937 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-25T01:33:38.938 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-25T01:33:38.939 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-25T01:33:38.939 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T01:33:38.943 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-25T01:33:38.943 DEBUG:teuthology.orchestra.run.smithi102:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-25T01:33:38.965 INFO:teuthology.orchestra.run.smithi102.stdout:postinstall scriptlet (using /bin/sh): 2024-04-25T01:33:38.965 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T01:33:38.965 INFO:teuthology.orchestra.run.smithi102.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T01:33:38.965 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-04-25T01:33:38.965 INFO:teuthology.orchestra.run.smithi102.stdout:for file in vmlinuz System.map config; do 2024-04-25T01:33:38.965 INFO:teuthology.orchestra.run.smithi102.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-25T01:33:38.965 INFO:teuthology.orchestra.run.smithi102.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout: fi 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout:done 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-25T01:33:38.966 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-04-25T01:33:38.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-04-25T01:33:39.039 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-04-25T01:33:39.039 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-04-25T01:33:39.039 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-04-25T01:33:39.039 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 68.el9 2024-04-25T01:33:39.039 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-04-25T01:33:39.039 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-25T01:33:39.039 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-04-25T01:33:39.039 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8264254 2024-04-25T01:33:39.039 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-25T01:33:39.040 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-04-25T01:33:39.041 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:33:39.041 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-04-25T01:33:39.041 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-25T01:33:39.042 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 6.9.0-rc3-g806ec36b2112 2024-04-25T01:33:39.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-25T01:33:39.619 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-04-25T01:33:41.047 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-25T01:33:41.078 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-04-25T01:33:41.080 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-04-25T01:33:41.105 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.ydYXtmzJpN 2024-04-25T01:33:41.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ydYXtmzJpN 2024-04-25T01:33:41.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.ydYXtmzJpN 2024-04-25T01:33:41.318 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.ydYXtmzJpN is 6KB 2024-04-25T01:33:41.367 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.ydYXtmzJpN 2024-04-25T01:33:41.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-04-25T01:33:41.444 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-25T01:33:41.444 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-25T01:33:41.444 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-25T01:33:41.444 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-25T01:33:41.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-25T01:33:41.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-04-25T01:34:11.593 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-25T01:34:11.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T01:34:11.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:34:29.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-04-25T01:34:38.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T01:34:38.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:34:41.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-04-25T01:34:53.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T01:34:53.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:35:25.837 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-04-25T01:35:26.406 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T01:35:26.407 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-25T01:35:26.407 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-04-25T01:35:26.443 INFO:teuthology.orchestra.run.smithi102.stdout:6.9.0-rc3-g806ec36b2112 2024-04-25T01:35:26.443 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-25T01:35:26.443 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T01:35:26.443 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-25T01:35:27.444 DEBUG:teuthology.parallel:result is None 2024-04-25T01:35:27.444 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T01:35:27.451 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T01:35:27.451 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:35:27.459 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:35:27.461 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T01:35:27.503 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T01:35:27.509 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T01:35:27.520 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T01:35:27.520 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:35:27.527 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:35:27.529 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T01:35:27.644 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T01:35:27.651 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T01:35:27.651 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:35:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:35:27.668 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T01:35:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:35:27.684 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:35:27.689 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:35:27.710 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T01:35:27.738 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:35:27.764 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:35:27.770 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:35:27.807 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:35:27.818 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:35:27.877 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T01:35:27.883 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T01:35:27.890 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T01:35:27.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:35:27.895 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:35:27.898 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T01:35:27.957 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T01:35:27.966 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T01:35:27.966 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:35:27.971 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:35:27.973 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T01:35:28.004 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:35:28.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:35:28.206 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:35:28.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:35:28.357 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:35:28.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:35:28.447 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:35:28.492 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:35:28.540 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:35:28.623 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:35:28.665 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:35:28.665 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:35:28.731 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:35:28.776 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T01:35:28.826 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:35:28.908 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T01:35:28.951 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:35:28.952 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T01:35:29.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-04-25T01:35:29.022 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-04-25T01:35:29.025 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-04-25T01:35:29.063 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:35:29.096 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:35:29.123 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T01:35:29.517 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T01:35:29.525 INFO:teuthology.task.internal:Starting timer... 2024-04-25T01:35:29.525 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T01:35:29.535 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T01:35:29.543 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T01:35:29.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-04-25T01:35:29.676 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-04-25T01:35:29.681 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-04-25T01:35:29.743 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-04-25T01:35:29.746 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-04-25T01:35:29.802 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-04-25T01:35:29.804 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T01:35:29.804 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-04-25T01:35:29.857 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-04-25T01:35:29.857 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-04-25T01:35:29.893 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-04-25T01:35:29.893 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-04-25T01:35:29.917 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-04-25T01:35:29.917 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-04-25T01:35:29.917 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:35:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:35:29.984 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:35:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:35:30.010 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T01:35:30.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:35:30.038 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T01:35:30.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:35:30.213 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:35:30.237 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-04-25T01:35:30.311 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T01:35:30.321 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T01:35:30.337 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T01:35:30.338 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi110.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T01:40:29.611 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-04-25T01:40:29.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T01:40:29.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:40:29.719 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-04-25T01:40:29.872 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T01:40:29.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-25T01:40:29.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:40:29.950 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-04-25T01:40:30.002 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-04-25T01:40:30.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-04-25T01:40:30.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T01:40:30.079 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-04-25T01:40:30.157 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-04-25T01:40:30.158 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T01:40:30.169 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T01:40:30.169 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:40:30.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:40:30.178 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:40:30.178 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:40:30.180 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T01:40:30.180 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T01:40:30.209 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:40:30.223 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:40:30.225 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:40:30.238 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:40:30.248 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-04-25T01:40:30.258 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-04-25T01:40:30.261 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-04-25T01:40:30.263 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T01:40:30.271 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-04-25T01:40:30.272 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:40:30.284 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:40:30.285 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:40:30.297 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:40:30.306 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T01:40:30.341 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:40:30.345 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:40:30.345 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-04-25T01:40:30.345 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.345 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.345 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.346 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.349 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:40:30.352 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:40:30.353 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-04-25T01:40:30.353 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.353 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.353 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.353 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.377 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-04-25T01:40:30.409 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-04-25T01:40:30.447 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T01:40:30.488 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T01:40:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-04-25T01:40:30.619 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T01:40:30.619 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-04-25T01:40:30.619 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.619 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.619 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.619 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T01:40:30.621 INFO:teuthology.run_tasks:Running task install... 2024-04-25T01:40:30.631 DEBUG:teuthology.task.install:project ceph 2024-04-25T01:40:30.631 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-25T01:40:30.631 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-25T01:40:30.631 INFO:teuthology.task.install:Using flavor: default 2024-04-25T01:40:30.637 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T01:40:30.637 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T01:40:30.637 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:40:30.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:40:30.640 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:40:30.640 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:40:30.641 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-25T01:40:30.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:40:30.799 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:40:30.799 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:40:30.803 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:40:30.803 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:40:30.804 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-25T01:40:30.805 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-25T01:40:30.971 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:40:30.972 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:40:30.972 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:40:30.979 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:40:30.979 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:40:30.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:40:30.986 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T01:40:30.986 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:40:30.986 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T01:40:30.997 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:40:30.997 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:40:31.011 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:40:31.012 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:40:31.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:40:31.071 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T01:40:31.071 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T01:40:31.078 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:40:31.135 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:40:31.152 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:40:31.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T01:40:31.198 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-04-25T01:40:31.199 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-04-25T01:40:31.215 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-04-25T01:40:31.217 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-04-25T01:40:31.310 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T01:40:31.385 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-04-25T01:40:31.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-04-25T01:40:31.425 INFO:teuthology.orchestra.run.smithi193.stdout:48 files removed 2024-04-25T01:40:31.440 INFO:teuthology.orchestra.run.smithi110.stdout:48 files removed 2024-04-25T01:40:31.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:40:31.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:40:31.706 INFO:teuthology.orchestra.run.smithi102.stdout:48 files removed 2024-04-25T01:40:31.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T01:40:32.115 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 207 kB/s | 83 kB 00:00 2024-04-25T01:40:32.190 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 179 kB/s | 83 kB 00:00 2024-04-25T01:40:32.541 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-04-25T01:40:32.607 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-04-25T01:40:32.610 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 197 kB/s | 83 kB 00:00 2024-04-25T01:40:32.966 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-04-25T01:40:33.016 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-04-25T01:40:33.211 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 23 kB/s | 11 kB 00:00 2024-04-25T01:40:33.558 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-25T01:40:33.820 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-04-25T01:40:34.758 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-04-25T01:40:35.959 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-04-25T01:40:36.567 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-25T01:40:38.251 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-04-25T01:40:40.493 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 6.2 MB 00:00 2024-04-25T01:40:40.569 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-25T01:40:41.836 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 19 MB 00:02 2024-04-25T01:40:41.952 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-04-25T01:40:42.644 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 16 kB 00:00 2024-04-25T01:40:43.645 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-25T01:40:43.896 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-25T01:40:48.646 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.2 MB 00:02 2024-04-25T01:40:48.679 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-04-25T01:40:48.963 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-25T01:40:50.538 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-04-25T01:40:50.771 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:40:50.771 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:40:50.782 INFO:teuthology.orchestra.run.smithi193.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:40:50.783 INFO:teuthology.orchestra.run.smithi193.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:40:50.783 INFO:teuthology.orchestra.run.smithi193.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:40:50.784 INFO:teuthology.orchestra.run.smithi193.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:40:50.786 INFO:teuthology.orchestra.run.smithi193.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:40:50.788 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:40:50.789 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:40:50.848 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-04-25T01:40:50.861 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-04-25T01:40:50.861 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-04-25T01:40:50.861 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-04-25T01:40:50.861 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:40:50.862 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:40:50.863 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:40:50.864 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:40:50.865 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:40:50.866 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:40:50.867 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:40:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:40:50.869 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:40:50.870 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:40:50.870 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:40:50.870 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:40:50.870 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:40:50.870 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:40:50.870 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:40:50.870 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:40:50.870 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:40:50.870 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:40:50.871 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:40:50.872 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:40:50.872 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:40:50.872 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:40:50.872 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:40:50.872 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:40:50.872 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:40:50.872 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:40:50.872 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:40:50.872 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:40:50.873 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:40:50.873 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:40:50.873 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:40:50.873 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:40:50.873 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:40:50.873 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:40:50.873 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:40:50.873 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:40:50.873 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:40:50.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:40:50.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:40:50.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:40:50.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:40:50.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:40:50.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:40:50.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:40:50.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:40:50.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-04-25T01:40:50.875 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-04-25T01:40:50.876 INFO:teuthology.orchestra.run.smithi193.stdout:Install 148 Packages 2024-04-25T01:40:50.876 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 4 Packages 2024-04-25T01:40:50.876 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:40:50.876 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 220 M 2024-04-25T01:40:50.876 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-04-25T01:40:51.064 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:40:51.064 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:40:51.075 INFO:teuthology.orchestra.run.smithi110.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:40:51.076 INFO:teuthology.orchestra.run.smithi110.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:40:51.076 INFO:teuthology.orchestra.run.smithi110.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:40:51.077 INFO:teuthology.orchestra.run.smithi110.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:40:51.079 INFO:teuthology.orchestra.run.smithi110.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:40:51.081 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:40:51.082 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:40:51.143 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-04-25T01:40:51.158 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-04-25T01:40:51.158 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-04-25T01:40:51.158 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:40:51.159 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:40:51.160 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:40:51.161 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:40:51.162 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:40:51.163 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:40:51.164 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:40:51.165 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:40:51.166 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:40:51.167 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:40:51.168 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:40:51.169 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout:Installing weak dependencies: 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout:Install 148 Packages 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 4 Packages 2024-04-25T01:40:51.170 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:40:51.171 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 220 M 2024-04-25T01:40:51.171 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-04-25T01:40:51.871 INFO:teuthology.orchestra.run.smithi193.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-04-25T01:40:52.105 INFO:teuthology.orchestra.run.smithi193.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 3.7 MB/s | 890 kB 00:00 2024-04-25T01:40:52.180 INFO:teuthology.orchestra.run.smithi193.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.9 MB/s | 145 kB 00:00 2024-04-25T01:40:52.189 INFO:teuthology.orchestra.run.smithi110.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-04-25T01:40:52.331 INFO:teuthology.orchestra.run.smithi193.stdout:(4/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 8.5 MB/s | 5.4 MB 00:00 2024-04-25T01:40:52.564 INFO:teuthology.orchestra.run.smithi193.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-04-25T01:40:52.573 INFO:teuthology.orchestra.run.smithi110.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 2.3 MB/s | 890 kB 00:00 2024-04-25T01:40:52.614 INFO:teuthology.orchestra.run.smithi193.stdout:(6/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 5.4 MB/s | 2.4 MB 00:00 2024-04-25T01:40:52.640 INFO:teuthology.orchestra.run.smithi110.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.1 MB/s | 145 kB 00:00 2024-04-25T01:40:53.332 INFO:teuthology.orchestra.run.smithi110.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 3.4 MB/s | 2.4 MB 00:00 2024-04-25T01:40:53.573 INFO:teuthology.orchestra.run.smithi193.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 4.7 MB/s | 4.7 MB 00:01 2024-04-25T01:40:53.749 INFO:teuthology.orchestra.run.smithi110.stdout:(5/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 3.0 MB/s | 5.4 MB 00:01 2024-04-25T01:40:53.790 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-04-25T01:40:53.899 INFO:teuthology.orchestra.run.smithi110.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-04-25T01:40:54.340 INFO:teuthology.orchestra.run.smithi193.stdout:(8/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 9.7 MB/s | 17 MB 00:01 2024-04-25T01:40:54.407 INFO:teuthology.orchestra.run.smithi193.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 379 kB/s | 25 kB 00:00 2024-04-25T01:40:55.308 INFO:teuthology.orchestra.run.smithi110.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-04-25T01:40:55.690 INFO:teuthology.orchestra.run.smithi193.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 4.8 MB/s | 10 MB 00:02 2024-04-25T01:40:55.907 INFO:teuthology.orchestra.run.smithi193.stdout:(11/152): ceph-common-19.0.0-2445.gd2c45d97.el9 5.0 MB/s | 21 MB 00:04 2024-04-25T01:40:55.940 INFO:teuthology.orchestra.run.smithi193.stdout:(12/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 888 kB/s | 222 kB 00:00 2024-04-25T01:40:55.966 INFO:teuthology.orchestra.run.smithi193.stdout:(13/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 464 kB/s | 27 kB 00:00 2024-04-25T01:40:55.991 INFO:teuthology.orchestra.run.smithi193.stdout:(14/152): libcephfs-devel-19.0.0-2445.gd2c45d97 630 kB/s | 32 kB 00:00 2024-04-25T01:40:56.050 INFO:teuthology.orchestra.run.smithi193.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.7 MB/s | 163 kB 00:00 2024-04-25T01:40:56.142 INFO:teuthology.orchestra.run.smithi193.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.4 MB/s | 127 kB 00:00 2024-04-25T01:40:56.184 INFO:teuthology.orchestra.run.smithi193.stdout:(17/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 3.3 MB/s | 745 kB 00:00 2024-04-25T01:40:56.310 INFO:teuthology.orchestra.run.smithi193.stdout:(18/152): libradosstriper1-19.0.0-2445.gd2c45d9 2.9 MB/s | 501 kB 00:00 2024-04-25T01:40:56.410 INFO:teuthology.orchestra.run.smithi193.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 452 kB/s | 45 kB 00:00 2024-04-25T01:40:56.502 INFO:teuthology.orchestra.run.smithi193.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.4 MB/s | 133 kB 00:00 2024-04-25T01:40:56.594 INFO:teuthology.orchestra.run.smithi193.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.7 MB/s | 162 kB 00:00 2024-04-25T01:40:56.702 INFO:teuthology.orchestra.run.smithi193.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 2.9 MB/s | 321 kB 00:00 2024-04-25T01:40:56.811 INFO:teuthology.orchestra.run.smithi193.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 2.7 MB/s | 299 kB 00:00 2024-04-25T01:40:56.911 INFO:teuthology.orchestra.run.smithi193.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 995 kB/s | 99 kB 00:00 2024-04-25T01:40:57.011 INFO:teuthology.orchestra.run.smithi193.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 854 kB/s | 85 kB 00:00 2024-04-25T01:40:57.637 INFO:teuthology.orchestra.run.smithi193.stdout:(26/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 3.7 MB/s | 5.3 MB 00:01 2024-04-25T01:40:57.712 INFO:teuthology.orchestra.run.smithi193.stdout:(27/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 4.4 MB/s | 3.1 MB 00:00 2024-04-25T01:40:57.737 INFO:teuthology.orchestra.run.smithi193.stdout:(28/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-25T01:40:57.779 INFO:teuthology.orchestra.run.smithi193.stdout:(29/152): ceph-grafana-dashboards-19.0.0-2445.g 475 kB/s | 32 kB 00:00 2024-04-25T01:40:57.813 INFO:teuthology.orchestra.run.smithi193.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.8 MB/s | 141 kB 00:00 2024-04-25T01:40:58.016 INFO:teuthology.orchestra.run.smithi110.stdout:(8/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el9 3.8 MB/s | 10 MB 00:02 2024-04-25T01:40:58.174 INFO:teuthology.orchestra.run.smithi110.stdout:(9/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x86 3.9 MB/s | 17 MB 00:04 2024-04-25T01:40:58.191 INFO:teuthology.orchestra.run.smithi110.stdout:(10/152): ceph-selinux-19.0.0-2445.gd2c45d97.el 144 kB/s | 25 kB 00:00 2024-04-25T01:40:58.283 INFO:teuthology.orchestra.run.smithi110.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 2.4 MB/s | 222 kB 00:00 2024-04-25T01:40:58.326 INFO:teuthology.orchestra.run.smithi110.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 640 kB/s | 27 kB 00:00 2024-04-25T01:40:58.368 INFO:teuthology.orchestra.run.smithi110.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 764 kB/s | 32 kB 00:00 2024-04-25T01:40:58.510 INFO:teuthology.orchestra.run.smithi110.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 5.2 MB/s | 745 kB 00:00 2024-04-25T01:40:58.568 INFO:teuthology.orchestra.run.smithi110.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.7 MB/s | 163 kB 00:00 2024-04-25T01:40:58.619 INFO:teuthology.orchestra.run.smithi110.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 2.5 MB/s | 127 kB 00:00 2024-04-25T01:40:58.697 INFO:teuthology.orchestra.run.smithi193.stdout:(31/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 11 MB/s | 48 MB 00:04 2024-04-25T01:40:58.719 INFO:teuthology.orchestra.run.smithi110.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 4.9 MB/s | 501 kB 00:00 2024-04-25T01:40:58.763 INFO:teuthology.orchestra.run.smithi193.stdout:(32/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 3.6 MB/s | 3.5 MB 00:00 2024-04-25T01:40:58.789 INFO:teuthology.orchestra.run.smithi193.stdout:(33/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.7 MB/s | 249 kB 00:00 2024-04-25T01:40:58.822 INFO:teuthology.orchestra.run.smithi193.stdout:(34/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 846 kB/s | 49 kB 00:00 2024-04-25T01:40:58.847 INFO:teuthology.orchestra.run.smithi193.stdout:(35/152): ceph-prometheus-alerts-19.0.0-2445.gd 263 kB/s | 15 kB 00:00 2024-04-25T01:40:58.939 INFO:teuthology.orchestra.run.smithi193.stdout:(36/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 2.2 MB/s | 264 kB 00:00 2024-04-25T01:40:58.981 INFO:teuthology.orchestra.run.smithi193.stdout:(37/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 5.6 MB/s | 762 kB 00:00 2024-04-25T01:40:59.006 INFO:teuthology.orchestra.run.smithi193.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 273 kB/s | 18 kB 00:00 2024-04-25T01:40:59.115 INFO:teuthology.orchestra.run.smithi193.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 601 kB/s | 80 kB 00:00 2024-04-25T01:40:59.199 INFO:teuthology.orchestra.run.smithi193.stdout:(40/152): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-04-25T01:40:59.226 INFO:teuthology.orchestra.run.smithi193.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 195 kB 00:00 2024-04-25T01:40:59.260 INFO:teuthology.orchestra.run.smithi193.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-04-25T01:40:59.310 INFO:teuthology.orchestra.run.smithi193.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-25T01:40:59.352 INFO:teuthology.orchestra.run.smithi193.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.2 MB 00:00 2024-04-25T01:40:59.377 INFO:teuthology.orchestra.run.smithi193.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-25T01:40:59.402 INFO:teuthology.orchestra.run.smithi193.stdout:(46/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-04-25T01:40:59.428 INFO:teuthology.orchestra.run.smithi193.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-25T01:40:59.453 INFO:teuthology.orchestra.run.smithi193.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-25T01:40:59.469 INFO:teuthology.orchestra.run.smithi110.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 7.1 MB/s | 5.3 MB 00:00 2024-04-25T01:40:59.637 INFO:teuthology.orchestra.run.smithi193.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 583 kB/s | 107 kB 00:00 2024-04-25T01:40:59.678 INFO:teuthology.orchestra.run.smithi110.stdout:(19/152): ceph-common-19.0.0-2445.gd2c45d97.el9 2.7 MB/s | 21 MB 00:07 2024-04-25T01:40:59.687 INFO:teuthology.orchestra.run.smithi193.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 689 kB 00:00 2024-04-25T01:40:59.703 INFO:teuthology.orchestra.run.smithi110.stdout:(20/152): python3-ceph-argparse-19.0.0-2445.gd2 193 kB/s | 45 kB 00:00 2024-04-25T01:40:59.704 INFO:teuthology.orchestra.run.smithi193.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 321 kB/s | 21 kB 00:00 2024-04-25T01:40:59.745 INFO:teuthology.orchestra.run.smithi110.stdout:(21/152): python3-ceph-common-19.0.0-2445.gd2c4 1.9 MB/s | 133 kB 00:00 2024-04-25T01:40:59.746 INFO:teuthology.orchestra.run.smithi193.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-04-25T01:40:59.748 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-04-25T01:40:59.763 INFO:teuthology.orchestra.run.smithi193.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-04-25T01:40:59.770 INFO:teuthology.orchestra.run.smithi110.stdout:(22/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.4 MB/s | 162 kB 00:00 2024-04-25T01:40:59.805 INFO:teuthology.orchestra.run.smithi193.stdout:(54/152): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 943 kB 00:00 2024-04-25T01:40:59.821 INFO:teuthology.orchestra.run.smithi110.stdout:(23/152): python3-rados-19.0.0-2445.gd2c45d97.e 4.2 MB/s | 321 kB 00:00 2024-04-25T01:40:59.902 INFO:teuthology.orchestra.run.smithi193.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-04-25T01:40:59.902 INFO:teuthology.orchestra.run.smithi193.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-25T01:40:59.902 INFO:teuthology.orchestra.run.smithi110.stdout:(24/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 2.9 MB/s | 299 kB 00:00 2024-04-25T01:40:59.902 INFO:teuthology.orchestra.run.smithi110.stdout:(25/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-25T01:40:59.905 INFO:teuthology.orchestra.run.smithi193.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 904 kB/s | 45 kB 00:00 2024-04-25T01:40:59.922 INFO:teuthology.orchestra.run.smithi110.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-25T01:40:59.931 INFO:teuthology.orchestra.run.smithi193.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-04-25T01:40:59.981 INFO:teuthology.orchestra.run.smithi110.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-25T01:40:59.998 INFO:teuthology.orchestra.run.smithi193.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-25T01:41:00.032 INFO:teuthology.orchestra.run.smithi193.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-04-25T01:41:00.032 INFO:teuthology.orchestra.run.smithi110.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 637 kB/s | 32 kB 00:00 2024-04-25T01:41:00.056 INFO:teuthology.orchestra.run.smithi193.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 437 kB/s | 25 kB 00:00 2024-04-25T01:41:00.082 INFO:teuthology.orchestra.run.smithi193.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-04-25T01:41:00.091 INFO:teuthology.orchestra.run.smithi110.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 2.4 MB/s | 141 kB 00:00 2024-04-25T01:41:00.107 INFO:teuthology.orchestra.run.smithi193.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-04-25T01:41:00.132 INFO:teuthology.orchestra.run.smithi193.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-25T01:41:00.166 INFO:teuthology.orchestra.run.smithi193.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-25T01:41:00.191 INFO:teuthology.orchestra.run.smithi193.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 420 kB/s | 35 kB 00:00 2024-04-25T01:41:00.358 INFO:teuthology.orchestra.run.smithi193.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-25T01:41:00.381 INFO:teuthology.orchestra.run.smithi110.stdout:(30/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 6.4 MB/s | 3.1 MB 00:00 2024-04-25T01:41:00.475 INFO:teuthology.orchestra.run.smithi193.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-04-25T01:41:00.517 INFO:teuthology.orchestra.run.smithi193.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 5.6 MB/s | 911 kB 00:00 2024-04-25T01:41:00.540 INFO:teuthology.orchestra.run.smithi110.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 7.8 MB/s | 3.5 MB 00:00 2024-04-25T01:41:00.542 INFO:teuthology.orchestra.run.smithi193.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-04-25T01:41:00.590 INFO:teuthology.orchestra.run.smithi110.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 4.9 MB/s | 249 kB 00:00 2024-04-25T01:41:00.601 INFO:teuthology.orchestra.run.smithi193.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-25T01:41:00.640 INFO:teuthology.orchestra.run.smithi110.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 992 kB/s | 49 kB 00:00 2024-04-25T01:41:00.668 INFO:teuthology.orchestra.run.smithi193.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-25T01:41:00.691 INFO:teuthology.orchestra.run.smithi110.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 306 kB/s | 15 kB 00:00 2024-04-25T01:41:00.701 INFO:teuthology.orchestra.run.smithi193.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.3 MB/s | 1.0 MB 00:00 2024-04-25T01:41:00.758 INFO:teuthology.orchestra.run.smithi110.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 3.9 MB/s | 264 kB 00:00 2024-04-25T01:41:00.769 INFO:teuthology.orchestra.run.smithi193.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-04-25T01:41:00.836 INFO:teuthology.orchestra.run.smithi193.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-04-25T01:41:00.883 INFO:teuthology.orchestra.run.smithi110.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 6.0 MB/s | 762 kB 00:00 2024-04-25T01:41:00.902 INFO:teuthology.orchestra.run.smithi193.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 384 kB/s | 26 kB 00:00 2024-04-25T01:41:00.933 INFO:teuthology.orchestra.run.smithi110.stdout:(37/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 365 kB/s | 18 kB 00:00 2024-04-25T01:41:01.061 INFO:teuthology.orchestra.run.smithi193.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 121 kB/s | 19 kB 00:00 2024-04-25T01:41:01.137 INFO:teuthology.orchestra.run.smithi193.stdout:(78/152): postgresql-server-13.14-1.el9.x86_64. 9.3 MB/s | 5.8 MB 00:00 2024-04-25T01:41:01.161 INFO:teuthology.orchestra.run.smithi193.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-25T01:41:01.237 INFO:teuthology.orchestra.run.smithi193.stdout:(80/152): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-04-25T01:41:01.262 INFO:teuthology.orchestra.run.smithi193.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 277 kB/s | 35 kB 00:00 2024-04-25T01:41:01.313 INFO:teuthology.orchestra.run.smithi193.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-25T01:41:01.338 INFO:teuthology.orchestra.run.smithi193.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-04-25T01:41:01.350 INFO:teuthology.orchestra.run.smithi110.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 7.6 MB/s | 7.4 MB 00:00 2024-04-25T01:41:01.375 INFO:teuthology.orchestra.run.smithi110.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 182 kB/s | 80 kB 00:00 2024-04-25T01:41:01.380 INFO:teuthology.orchestra.run.smithi193.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-25T01:41:01.405 INFO:teuthology.orchestra.run.smithi193.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 634 kB/s | 42 kB 00:00 2024-04-25T01:41:01.480 INFO:teuthology.orchestra.run.smithi193.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-25T01:41:01.505 INFO:teuthology.orchestra.run.smithi193.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-04-25T01:41:01.539 INFO:teuthology.orchestra.run.smithi193.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-25T01:41:01.556 INFO:teuthology.orchestra.run.smithi193.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T01:41:01.581 INFO:teuthology.orchestra.run.smithi193.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-04-25T01:41:01.598 INFO:teuthology.orchestra.run.smithi193.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 913 kB/s | 54 kB 00:00 2024-04-25T01:41:01.632 INFO:teuthology.orchestra.run.smithi193.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-04-25T01:41:01.649 INFO:teuthology.orchestra.run.smithi193.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 882 kB/s | 44 kB 00:00 2024-04-25T01:41:01.699 INFO:teuthology.orchestra.run.smithi193.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-04-25T01:41:01.716 INFO:teuthology.orchestra.run.smithi193.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 538 kB/s | 36 kB 00:00 2024-04-25T01:41:01.750 INFO:teuthology.orchestra.run.smithi193.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-04-25T01:41:01.783 INFO:teuthology.orchestra.run.smithi193.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-25T01:41:01.792 INFO:teuthology.orchestra.run.smithi110.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-04-25T01:41:01.800 INFO:teuthology.orchestra.run.smithi193.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T01:41:01.842 INFO:teuthology.orchestra.run.smithi193.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-25T01:41:01.869 INFO:teuthology.orchestra.run.smithi193.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 952 kB/s | 64 kB 00:00 2024-04-25T01:41:01.876 INFO:teuthology.orchestra.run.smithi110.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 390 kB/s | 195 kB 00:00 2024-04-25T01:41:01.910 INFO:teuthology.orchestra.run.smithi110.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-04-25T01:41:01.917 INFO:teuthology.orchestra.run.smithi193.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 596 kB/s | 45 kB 00:00 2024-04-25T01:41:01.943 INFO:teuthology.orchestra.run.smithi193.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2024-04-25T01:41:02.026 INFO:teuthology.orchestra.run.smithi110.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 913 kB/s | 106 kB 00:00 2024-04-25T01:41:02.150 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:41:02.151 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T01:41:02.152 INFO:teuthology.orchestra.run.smithi110.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-04-25T01:41:02.172 INFO:teuthology.orchestra.run.smithi102.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T01:41:02.173 INFO:teuthology.orchestra.run.smithi102.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T01:41:02.175 INFO:teuthology.orchestra.run.smithi102.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T01:41:02.176 INFO:teuthology.orchestra.run.smithi102.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T01:41:02.178 INFO:teuthology.orchestra.run.smithi102.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T01:41:02.182 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T01:41:02.183 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T01:41:02.253 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-04-25T01:41:02.274 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-04-25T01:41:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-04-25T01:41:02.274 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-04-25T01:41:02.274 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-04-25T01:41:02.274 INFO:teuthology.orchestra.run.smithi102.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-25T01:41:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T01:41:02.276 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-25T01:41:02.277 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-25T01:41:02.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T01:41:02.279 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T01:41:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T01:41:02.281 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T01:41:02.282 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T01:41:02.283 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T01:41:02.284 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T01:41:02.285 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout:======================================================================================= 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout:Install 148 Packages 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 4 Packages 2024-04-25T01:41:02.286 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:41:02.287 INFO:teuthology.orchestra.run.smithi110.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-04-25T01:41:02.287 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 220 M 2024-04-25T01:41:02.287 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-04-25T01:41:02.319 INFO:teuthology.orchestra.run.smithi110.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 2.8 MB/s | 1.2 MB 00:00 2024-04-25T01:41:02.369 INFO:teuthology.orchestra.run.smithi193.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-25T01:41:02.403 INFO:teuthology.orchestra.run.smithi110.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 215 kB 00:00 2024-04-25T01:41:02.527 INFO:teuthology.orchestra.run.smithi193.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.4 MB/s | 863 kB 00:00 2024-04-25T01:41:02.660 INFO:teuthology.orchestra.run.smithi193.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 764 kB/s | 548 kB 00:00 2024-04-25T01:41:02.802 INFO:teuthology.orchestra.run.smithi193.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-04-25T01:41:02.869 INFO:teuthology.orchestra.run.smithi193.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 617 kB/s | 309 kB 00:00 2024-04-25T01:41:02.911 INFO:teuthology.orchestra.run.smithi193.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-25T01:41:02.928 INFO:teuthology.orchestra.run.smithi110.stdout:(48/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 10 MB/s | 48 MB 00:04 2024-04-25T01:41:02.995 INFO:teuthology.orchestra.run.smithi193.stdout:(109/152): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-25T01:41:03.020 INFO:teuthology.orchestra.run.smithi193.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 323 kB/s | 49 kB 00:00 2024-04-25T01:41:03.071 INFO:teuthology.orchestra.run.smithi193.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-25T01:41:03.078 INFO:teuthology.orchestra.run.smithi110.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 908 kB/s | 689 kB 00:00 2024-04-25T01:41:03.096 INFO:teuthology.orchestra.run.smithi193.stdout:(112/152): libunwind-1.6.2-1.el9.x86_64.rpm 366 kB/s | 67 kB 00:00 2024-04-25T01:41:03.163 INFO:teuthology.orchestra.run.smithi193.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-04-25T01:41:03.170 INFO:teuthology.orchestra.run.smithi110.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 443 kB/s | 107 kB 00:00 2024-04-25T01:41:03.196 INFO:teuthology.orchestra.run.smithi193.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 434 kB/s | 43 kB 00:00 2024-04-25T01:41:03.221 INFO:teuthology.orchestra.run.smithi110.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-04-25T01:41:03.221 INFO:teuthology.orchestra.run.smithi193.stdout:(115/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-04-25T01:41:03.246 INFO:teuthology.orchestra.run.smithi110.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 128 kB/s | 21 kB 00:00 2024-04-25T01:41:03.247 INFO:teuthology.orchestra.run.smithi193.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 385 kB/s | 32 kB 00:00 2024-04-25T01:41:03.280 INFO:teuthology.orchestra.run.smithi193.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-04-25T01:41:03.314 INFO:teuthology.orchestra.run.smithi193.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-25T01:41:03.339 INFO:teuthology.orchestra.run.smithi193.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-04-25T01:41:03.398 INFO:teuthology.orchestra.run.smithi193.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-25T01:41:03.413 INFO:teuthology.orchestra.run.smithi110.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 170 kB/s | 33 kB 00:00 2024-04-25T01:41:03.431 INFO:teuthology.orchestra.run.smithi193.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-04-25T01:41:03.532 INFO:teuthology.orchestra.run.smithi193.stdout:(122/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.2 MB/s | 303 kB 00:00 2024-04-25T01:41:03.557 INFO:teuthology.orchestra.run.smithi193.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-25T01:41:03.571 INFO:teuthology.orchestra.run.smithi110.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 115 kB/s | 18 kB 00:00 2024-04-25T01:41:03.600 INFO:teuthology.orchestra.run.smithi102.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-04-25T01:41:03.630 INFO:teuthology.orchestra.run.smithi110.stdout:(55/152): flex-2.6.4-9.el9.x86_64.rpm 682 kB/s | 313 kB 00:00 2024-04-25T01:41:03.632 INFO:teuthology.orchestra.run.smithi193.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-04-25T01:41:03.649 INFO:teuthology.orchestra.run.smithi193.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-04-25T01:41:03.666 INFO:teuthology.orchestra.run.smithi193.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-25T01:41:03.708 INFO:teuthology.orchestra.run.smithi193.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-04-25T01:41:03.714 INFO:teuthology.orchestra.run.smithi110.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 78 kB/s | 11 kB 00:00 2024-04-25T01:41:03.725 INFO:teuthology.orchestra.run.smithi193.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-25T01:41:03.742 INFO:teuthology.orchestra.run.smithi193.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-25T01:41:03.817 INFO:teuthology.orchestra.run.smithi193.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-04-25T01:41:03.851 INFO:teuthology.orchestra.run.smithi193.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-04-25T01:41:03.872 INFO:teuthology.orchestra.run.smithi110.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 663 kB/s | 160 kB 00:00 2024-04-25T01:41:03.876 INFO:teuthology.orchestra.run.smithi193.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 589 kB/s | 79 kB 00:00 2024-04-25T01:41:03.910 INFO:teuthology.orchestra.run.smithi193.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 625 kB/s | 58 kB 00:00 2024-04-25T01:41:03.914 INFO:teuthology.orchestra.run.smithi110.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 226 kB/s | 45 kB 00:00 2024-04-25T01:41:03.959 INFO:teuthology.orchestra.run.smithi102.stdout:(2/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 7.2 MB/s | 5.4 MB 00:00 2024-04-25T01:41:03.960 INFO:teuthology.orchestra.run.smithi193.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-25T01:41:03.977 INFO:teuthology.orchestra.run.smithi193.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-04-25T01:41:04.010 INFO:teuthology.orchestra.run.smithi193.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 896 kB/s | 90 kB 00:00 2024-04-25T01:41:04.034 INFO:teuthology.orchestra.run.smithi102.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 1.9 MB/s | 145 kB 00:00 2024-04-25T01:41:04.052 INFO:teuthology.orchestra.run.smithi193.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-04-25T01:41:04.078 INFO:teuthology.orchestra.run.smithi193.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-04-25T01:41:04.111 INFO:teuthology.orchestra.run.smithi193.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-25T01:41:04.145 INFO:teuthology.orchestra.run.smithi193.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-04-25T01:41:04.179 INFO:teuthology.orchestra.run.smithi193.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-04-25T01:41:04.204 INFO:teuthology.orchestra.run.smithi193.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-04-25T01:41:04.256 INFO:teuthology.orchestra.run.smithi110.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 1.7 MB/s | 582 kB 00:00 2024-04-25T01:41:04.271 INFO:teuthology.orchestra.run.smithi193.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-04-25T01:41:04.304 INFO:teuthology.orchestra.run.smithi193.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 714 kB/s | 90 kB 00:00 2024-04-25T01:41:04.318 INFO:teuthology.orchestra.run.smithi102.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 8.4 MB/s | 2.4 MB 00:00 2024-04-25T01:41:04.330 INFO:teuthology.orchestra.run.smithi193.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-25T01:41:04.363 INFO:teuthology.orchestra.run.smithi193.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-04-25T01:41:04.422 INFO:teuthology.orchestra.run.smithi193.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-25T01:41:04.464 INFO:teuthology.orchestra.run.smithi193.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T01:41:04.473 INFO:teuthology.orchestra.run.smithi110.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 117 kB/s | 25 kB 00:00 2024-04-25T01:41:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:(5/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 8.2 MB/s | 1.5 MB 00:00 2024-04-25T01:41:04.540 INFO:teuthology.orchestra.run.smithi110.stdout:(61/152): librdkafka-1.6.1-102.el9.x86_64.rpm 992 kB/s | 662 kB 00:00 2024-04-25T01:41:04.551 INFO:teuthology.orchestra.run.smithi102.stdout:(6/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 937 kB/s | 890 kB 00:00 2024-04-25T01:41:04.664 INFO:teuthology.orchestra.run.smithi193.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 6.7 MB/s | 1.3 MB 00:00 2024-04-25T01:41:04.673 INFO:teuthology.orchestra.run.smithi110.stdout:(62/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.1 MB/s | 3.0 MB 00:01 2024-04-25T01:41:04.699 INFO:teuthology.orchestra.run.smithi110.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 210 kB/s | 47 kB 00:00 2024-04-25T01:41:04.764 INFO:teuthology.orchestra.run.smithi193.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 24 MB/s | 2.4 MB 00:00 2024-04-25T01:41:04.832 INFO:teuthology.orchestra.run.smithi110.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-04-25T01:41:04.858 INFO:teuthology.orchestra.run.smithi110.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 222 kB/s | 35 kB 00:00 2024-04-25T01:41:04.925 INFO:teuthology.orchestra.run.smithi110.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 752 kB/s | 188 kB 00:00 2024-04-25T01:41:04.943 INFO:teuthology.orchestra.run.smithi102.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 11 MB/s | 4.7 MB 00:00 2024-04-25T01:41:05.210 INFO:teuthology.orchestra.run.smithi102.stdout:(8/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 11 MB/s | 21 MB 00:01 2024-04-25T01:41:05.248 INFO:teuthology.orchestra.run.smithi193.stdout:(151/152): librados2-19.0.0-2445.gd2c45d97.el9. 3.8 MB/s | 3.4 MB 00:00 2024-04-25T01:41:05.269 INFO:teuthology.orchestra.run.smithi102.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 439 kB/s | 25 kB 00:00 2024-04-25T01:41:05.333 INFO:teuthology.orchestra.run.smithi110.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-04-25T01:41:05.525 INFO:teuthology.orchestra.run.smithi110.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 6.6 MB/s | 4.4 MB 00:00 2024-04-25T01:41:05.673 INFO:teuthology.orchestra.run.smithi193.stdout:(152/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 2.5 MB/s | 3.1 MB 00:01 2024-04-25T01:41:05.679 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:41:05.679 INFO:teuthology.orchestra.run.smithi193.stdout:Total 15 MB/s | 220 MB 00:14 2024-04-25T01:41:05.759 INFO:teuthology.orchestra.run.smithi110.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 2.1 MB/s | 911 kB 00:00 2024-04-25T01:41:05.809 INFO:teuthology.orchestra.run.smithi110.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 470 kB/s | 133 kB 00:00 2024-04-25T01:41:05.869 INFO:teuthology.orchestra.run.smithi102.stdout:(10/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 11 MB/s | 10 MB 00:00 2024-04-25T01:41:05.928 INFO:teuthology.orchestra.run.smithi102.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 3.8 MB/s | 222 kB 00:00 2024-04-25T01:41:05.995 INFO:teuthology.orchestra.run.smithi102.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 410 kB/s | 27 kB 00:00 2024-04-25T01:41:06.062 INFO:teuthology.orchestra.run.smithi102.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 483 kB/s | 32 kB 00:00 2024-04-25T01:41:06.173 INFO:teuthology.orchestra.run.smithi102.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 6.6 MB/s | 745 kB 00:00 2024-04-25T01:41:06.184 INFO:teuthology.orchestra.run.smithi110.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 3.6 MB/s | 4.7 MB 00:01 2024-04-25T01:41:06.246 INFO:teuthology.orchestra.run.smithi102.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.2 MB/s | 163 kB 00:00 2024-04-25T01:41:06.321 INFO:teuthology.orchestra.run.smithi102.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.7 MB/s | 127 kB 00:00 2024-04-25T01:41:06.368 INFO:teuthology.orchestra.run.smithi110.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-04-25T01:41:06.401 INFO:teuthology.orchestra.run.smithi110.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 403 kB/s | 87 kB 00:00 2024-04-25T01:41:06.421 INFO:teuthology.orchestra.run.smithi102.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 4.9 MB/s | 501 kB 00:00 2024-04-25T01:41:06.685 INFO:teuthology.orchestra.run.smithi110.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 862 kB/s | 244 kB 00:00 2024-04-25T01:41:06.789 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-04-25T01:41:06.932 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-04-25T01:41:06.932 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-04-25T01:41:06.952 INFO:teuthology.orchestra.run.smithi110.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 930 kB/s | 248 kB 00:00 2024-04-25T01:41:06.963 INFO:teuthology.orchestra.run.smithi102.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 9.8 MB/s | 5.3 MB 00:00 2024-04-25T01:41:07.047 INFO:teuthology.orchestra.run.smithi102.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 547 kB/s | 45 kB 00:00 2024-04-25T01:41:07.094 INFO:teuthology.orchestra.run.smithi110.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 181 kB/s | 26 kB 00:00 2024-04-25T01:41:07.113 INFO:teuthology.orchestra.run.smithi102.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 2.0 MB/s | 133 kB 00:00 2024-04-25T01:41:07.189 INFO:teuthology.orchestra.run.smithi102.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 2.1 MB/s | 162 kB 00:00 2024-04-25T01:41:07.278 INFO:teuthology.orchestra.run.smithi110.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 104 kB/s | 19 kB 00:00 2024-04-25T01:41:07.281 INFO:teuthology.orchestra.run.smithi102.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 3.5 MB/s | 321 kB 00:00 2024-04-25T01:41:07.365 INFO:teuthology.orchestra.run.smithi102.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.5 MB/s | 299 kB 00:00 2024-04-25T01:41:07.440 INFO:teuthology.orchestra.run.smithi102.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-25T01:41:07.515 INFO:teuthology.orchestra.run.smithi102.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-25T01:41:07.537 INFO:teuthology.orchestra.run.smithi110.stdout:(78/152): python3-babel-2.9.1-2.el9.noarch.rpm 5.1 MB/s | 6.0 MB 00:01 2024-04-25T01:41:07.562 INFO:teuthology.orchestra.run.smithi110.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 606 kB/s | 172 kB 00:00 2024-04-25T01:41:07.695 INFO:teuthology.orchestra.run.smithi110.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 219 kB/s | 35 kB 00:00 2024-04-25T01:41:07.883 INFO:teuthology.orchestra.run.smithi102.stdout:(26/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 5.0 MB/s | 17 MB 00:03 2024-04-25T01:41:07.946 INFO:teuthology.orchestra.run.smithi110.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 250 kB 00:00 2024-04-25T01:41:08.024 INFO:teuthology.orchestra.run.smithi102.stdout:(27/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 6.1 MB/s | 3.1 MB 00:00 2024-04-25T01:41:08.058 INFO:teuthology.orchestra.run.smithi102.stdout:(28/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 983 kB/s | 171 kB 00:00 2024-04-25T01:41:08.083 INFO:teuthology.orchestra.run.smithi102.stdout:(29/152): ceph-grafana-dashboards-19.0.0-2445.g 550 kB/s | 32 kB 00:00 2024-04-25T01:41:08.125 INFO:teuthology.orchestra.run.smithi102.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 2.1 MB/s | 141 kB 00:00 2024-04-25T01:41:08.238 INFO:teuthology.orchestra.run.smithi110.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 762 kB/s | 222 kB 00:00 2024-04-25T01:41:08.429 INFO:teuthology.orchestra.run.smithi110.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 403 kB/s | 77 kB 00:00 2024-04-25T01:41:08.459 INFO:teuthology.orchestra.run.smithi102.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 9.4 MB/s | 3.5 MB 00:00 2024-04-25T01:41:08.542 INFO:teuthology.orchestra.run.smithi102.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.9 MB/s | 249 kB 00:00 2024-04-25T01:41:08.605 INFO:teuthology.orchestra.run.smithi110.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 241 kB/s | 42 kB 00:00 2024-04-25T01:41:08.618 INFO:teuthology.orchestra.run.smithi102.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 665 kB/s | 49 kB 00:00 2024-04-25T01:41:08.687 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-04-25T01:41:08.688 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-04-25T01:41:08.710 INFO:teuthology.orchestra.run.smithi102.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 169 kB/s | 15 kB 00:00 2024-04-25T01:41:08.713 INFO:teuthology.orchestra.run.smithi110.stdout:(85/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-04-25T01:41:08.868 INFO:teuthology.orchestra.run.smithi102.stdout:(35/152): ceph-mgr-diskprediction-local-19.0.0- 10 MB/s | 7.4 MB 00:00 2024-04-25T01:41:08.889 INFO:teuthology.orchestra.run.smithi110.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 949 kB/s | 269 kB 00:00 2024-04-25T01:41:08.902 INFO:teuthology.orchestra.run.smithi102.stdout:(36/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 264 kB 00:00 2024-04-25T01:41:08.956 INFO:teuthology.orchestra.run.smithi110.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 656 kB/s | 159 kB 00:00 2024-04-25T01:41:08.961 INFO:teuthology.orchestra.run.smithi102.stdout:(37/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 8.2 MB/s | 762 kB 00:00 2024-04-25T01:41:08.986 INFO:teuthology.orchestra.run.smithi102.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 220 kB/s | 18 kB 00:00 2024-04-25T01:41:09.131 INFO:teuthology.orchestra.run.smithi110.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 291 kB/s | 51 kB 00:00 2024-04-25T01:41:09.156 INFO:teuthology.orchestra.run.smithi110.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.0 MB/s | 279 kB 00:00 2024-04-25T01:41:09.390 INFO:teuthology.orchestra.run.smithi110.stdout:(90/152): postgresql-server-13.14-1.el9.x86_64. 1.6 MB/s | 5.8 MB 00:03 2024-04-25T01:41:09.415 INFO:teuthology.orchestra.run.smithi110.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 189 kB/s | 54 kB 00:00 2024-04-25T01:41:09.528 INFO:teuthology.orchestra.run.smithi102.stdout:(39/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 11 MB/s | 48 MB 00:04 2024-04-25T01:41:09.565 INFO:teuthology.orchestra.run.smithi110.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 238 kB/s | 42 kB 00:00 2024-04-25T01:41:09.616 INFO:teuthology.orchestra.run.smithi110.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 221 kB/s | 44 kB 00:00 2024-04-25T01:41:09.620 INFO:teuthology.orchestra.run.smithi102.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 122 kB/s | 80 kB 00:00 2024-04-25T01:41:09.687 INFO:teuthology.orchestra.run.smithi102.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-25T01:41:09.720 INFO:teuthology.orchestra.run.smithi102.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-04-25T01:41:09.758 INFO:teuthology.orchestra.run.smithi110.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 253 kB/s | 36 kB 00:00 2024-04-25T01:41:09.771 INFO:teuthology.orchestra.run.smithi102.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-04-25T01:41:09.820 INFO:teuthology.orchestra.run.smithi102.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-04-25T01:41:09.855 INFO:teuthology.orchestra.run.smithi102.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-25T01:41:09.866 INFO:teuthology.orchestra.run.smithi110.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 734 kB/s | 221 kB 00:00 2024-04-25T01:41:09.888 INFO:teuthology.orchestra.run.smithi102.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-25T01:41:09.922 INFO:teuthology.orchestra.run.smithi102.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-04-25T01:41:09.958 INFO:teuthology.orchestra.run.smithi110.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 237 kB/s | 47 kB 00:00 2024-04-25T01:41:09.980 INFO:teuthology.orchestra.run.smithi102.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 4.2 MB/s | 1.2 MB 00:00 2024-04-25T01:41:10.031 INFO:teuthology.orchestra.run.smithi102.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 4.8 MB/s | 689 kB 00:00 2024-04-25T01:41:10.056 INFO:teuthology.orchestra.run.smithi102.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-04-25T01:41:10.081 INFO:teuthology.orchestra.run.smithi102.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 435 kB/s | 21 kB 00:00 2024-04-25T01:41:10.115 INFO:teuthology.orchestra.run.smithi102.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-04-25T01:41:10.134 INFO:teuthology.orchestra.run.smithi110.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 305 kB 00:00 2024-04-25T01:41:10.140 INFO:teuthology.orchestra.run.smithi102.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 568 kB/s | 33 kB 00:00 2024-04-25T01:41:10.159 INFO:teuthology.orchestra.run.smithi110.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 289 kB/s | 58 kB 00:00 2024-04-25T01:41:10.208 INFO:teuthology.orchestra.run.smithi102.stdout:(54/152): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 943 kB 00:00 2024-04-25T01:41:10.232 INFO:teuthology.orchestra.run.smithi102.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 201 kB/s | 18 kB 00:00 2024-04-25T01:41:10.249 INFO:teuthology.orchestra.run.smithi102.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 275 kB/s | 11 kB 00:00 2024-04-25T01:41:10.283 INFO:teuthology.orchestra.run.smithi102.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 160 kB 00:00 2024-04-25T01:41:10.300 INFO:teuthology.orchestra.run.smithi102.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 911 kB/s | 45 kB 00:00 2024-04-25T01:41:10.309 INFO:teuthology.orchestra.run.smithi110.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 535 kB/s | 94 kB 00:00 2024-04-25T01:41:10.343 INFO:teuthology.orchestra.run.smithi110.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 346 kB/s | 64 kB 00:00 2024-04-25T01:41:10.358 INFO:teuthology.orchestra.run.smithi102.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-04-25T01:41:10.408 INFO:teuthology.orchestra.run.smithi102.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.3 MB/s | 582 kB 00:00 2024-04-25T01:41:10.434 INFO:teuthology.orchestra.run.smithi102.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 341 kB/s | 25 kB 00:00 2024-04-25T01:41:10.460 INFO:teuthology.orchestra.run.smithi102.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 47 kB 00:00 2024-04-25T01:41:10.484 INFO:teuthology.orchestra.run.smithi110.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 256 kB/s | 45 kB 00:00 2024-04-25T01:41:10.493 INFO:teuthology.orchestra.run.smithi102.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-25T01:41:10.510 INFO:teuthology.orchestra.run.smithi110.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-04-25T01:41:10.556 INFO:teuthology.orchestra.run.smithi102.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.9 MB/s | 188 kB 00:00 2024-04-25T01:41:10.585 INFO:teuthology.orchestra.run.smithi102.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 387 kB/s | 35 kB 00:00 2024-04-25T01:41:10.661 INFO:teuthology.orchestra.run.smithi102.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.5 MB/s | 3.0 MB 00:00 2024-04-25T01:41:10.786 INFO:teuthology.orchestra.run.smithi102.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-25T01:41:10.810 INFO:teuthology.orchestra.run.smithi110.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.6 MB/s | 863 kB 00:00 2024-04-25T01:41:10.852 INFO:teuthology.orchestra.run.smithi110.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-25T01:41:10.911 INFO:teuthology.orchestra.run.smithi110.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-04-25T01:41:10.926 INFO:teuthology.orchestra.run.smithi102.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-25T01:41:11.020 INFO:teuthology.orchestra.run.smithi102.stdout:(69/152): openblas-openmp-0.3.21-2.el9.x86_64.r 10 MB/s | 4.7 MB 00:00 2024-04-25T01:41:11.053 INFO:teuthology.orchestra.run.smithi102.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-04-25T01:41:11.103 INFO:teuthology.orchestra.run.smithi110.stdout:(106/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.3 MB/s | 16 MB 00:01 2024-04-25T01:41:11.112 INFO:teuthology.orchestra.run.smithi102.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 2.7 MB/s | 911 kB 00:00 2024-04-25T01:41:11.179 INFO:teuthology.orchestra.run.smithi102.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-04-25T01:41:11.204 INFO:teuthology.orchestra.run.smithi102.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 961 kB/s | 87 kB 00:00 2024-04-25T01:41:11.245 INFO:teuthology.orchestra.run.smithi110.stdout:(107/152): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-25T01:41:11.263 INFO:teuthology.orchestra.run.smithi102.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.2 MB/s | 244 kB 00:00 2024-04-25T01:41:11.271 INFO:teuthology.orchestra.run.smithi110.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 156 kB/s | 26 kB 00:00 2024-04-25T01:41:11.304 INFO:teuthology.orchestra.run.smithi110.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 830 kB/s | 49 kB 00:00 2024-04-25T01:41:11.321 INFO:teuthology.orchestra.run.smithi102.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-25T01:41:11.329 INFO:teuthology.orchestra.run.smithi110.stdout:(110/152): gperftools-libs-2.9.1-2.el9.x86_64.r 596 kB/s | 309 kB 00:00 2024-04-25T01:41:11.363 INFO:teuthology.orchestra.run.smithi110.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-25T01:41:11.372 INFO:teuthology.orchestra.run.smithi102.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 521 kB/s | 26 kB 00:00 2024-04-25T01:41:11.388 INFO:teuthology.orchestra.run.smithi110.stdout:(112/152): libunwind-1.6.2-1.el9.x86_64.rpm 572 kB/s | 67 kB 00:00 2024-04-25T01:41:11.438 INFO:teuthology.orchestra.run.smithi110.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-04-25T01:41:11.463 INFO:teuthology.orchestra.run.smithi110.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 578 kB/s | 43 kB 00:00 2024-04-25T01:41:11.480 INFO:teuthology.orchestra.run.smithi102.stdout:(77/152): postgresql-server-13.14-1.el9.x86_64. 13 MB/s | 5.8 MB 00:00 2024-04-25T01:41:11.489 INFO:teuthology.orchestra.run.smithi110.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-04-25T01:41:11.505 INFO:teuthology.orchestra.run.smithi102.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 144 kB/s | 19 kB 00:00 2024-04-25T01:41:11.514 INFO:teuthology.orchestra.run.smithi110.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-04-25T01:41:11.539 INFO:teuthology.orchestra.run.smithi102.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-25T01:41:11.548 INFO:teuthology.orchestra.run.smithi110.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-25T01:41:11.565 INFO:teuthology.orchestra.run.smithi102.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 602 kB/s | 35 kB 00:00 2024-04-25T01:41:11.615 INFO:teuthology.orchestra.run.smithi110.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-04-25T01:41:11.665 INFO:teuthology.orchestra.run.smithi110.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 705 kB/s | 35 kB 00:00 2024-04-25T01:41:11.690 INFO:teuthology.orchestra.run.smithi110.stdout:(120/152): python3-cherrypy-18.6.1-2.el9.noarch 2.0 MB/s | 358 kB 00:00 2024-04-25T01:41:11.707 INFO:teuthology.orchestra.run.smithi102.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-25T01:41:11.724 INFO:teuthology.orchestra.run.smithi110.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-04-25T01:41:11.740 INFO:teuthology.orchestra.run.smithi102.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 250 kB 00:00 2024-04-25T01:41:11.774 INFO:teuthology.orchestra.run.smithi102.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-04-25T01:41:11.782 INFO:teuthology.orchestra.run.smithi110.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-04-25T01:41:11.799 INFO:teuthology.orchestra.run.smithi102.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-25T01:41:11.824 INFO:teuthology.orchestra.run.smithi102.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 858 kB/s | 42 kB 00:00 2024-04-25T01:41:11.841 INFO:teuthology.orchestra.run.smithi110.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-25T01:41:11.858 INFO:teuthology.orchestra.run.smithi102.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-04-25T01:41:11.883 INFO:teuthology.orchestra.run.smithi102.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-25T01:41:11.891 INFO:teuthology.orchestra.run.smithi110.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-04-25T01:41:11.917 INFO:teuthology.orchestra.run.smithi110.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-04-25T01:41:11.917 INFO:teuthology.orchestra.run.smithi102.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-25T01:41:11.942 INFO:teuthology.orchestra.run.smithi102.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 882 kB/s | 51 kB 00:00 2024-04-25T01:41:11.942 INFO:teuthology.orchestra.run.smithi110.stdout:(126/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-04-25T01:41:11.967 INFO:teuthology.orchestra.run.smithi110.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-04-25T01:41:11.968 INFO:teuthology.orchestra.run.smithi102.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-04-25T01:41:11.992 INFO:teuthology.orchestra.run.smithi110.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-25T01:41:12.018 INFO:teuthology.orchestra.run.smithi110.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-04-25T01:41:12.043 INFO:teuthology.orchestra.run.smithi102.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 559 kB/s | 42 kB 00:00 2024-04-25T01:41:12.151 INFO:teuthology.orchestra.run.smithi102.stdout:(92/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.3 MB/s | 5.1 MB 00:00 2024-04-25T01:41:12.185 INFO:teuthology.orchestra.run.smithi102.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 313 kB/s | 44 kB 00:00 2024-04-25T01:41:12.219 INFO:teuthology.orchestra.run.smithi102.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-04-25T01:41:12.244 INFO:teuthology.orchestra.run.smithi102.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 616 kB/s | 36 kB 00:00 2024-04-25T01:41:12.353 INFO:teuthology.orchestra.run.smithi102.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 357 kB/s | 47 kB 00:00 2024-04-25T01:41:12.404 INFO:teuthology.orchestra.run.smithi102.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 305 kB 00:00 2024-04-25T01:41:12.429 INFO:teuthology.orchestra.run.smithi102.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 774 kB/s | 58 kB 00:00 2024-04-25T01:41:12.454 INFO:teuthology.orchestra.run.smithi102.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.9 MB/s | 94 kB 00:00 2024-04-25T01:41:12.479 INFO:teuthology.orchestra.run.smithi102.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-04-25T01:41:12.504 INFO:teuthology.orchestra.run.smithi102.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-04-25T01:41:12.580 INFO:teuthology.orchestra.run.smithi102.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 225 kB/s | 22 kB 00:00 2024-04-25T01:41:12.698 INFO:teuthology.orchestra.run.smithi102.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-04-25T01:41:12.906 INFO:teuthology.orchestra.run.smithi102.stdout:(104/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-04-25T01:41:13.147 INFO:teuthology.orchestra.run.smithi102.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 81 kB/s | 19 kB 00:00 2024-04-25T01:41:13.306 INFO:teuthology.orchestra.run.smithi102.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 756 kB/s | 548 kB 00:00 2024-04-25T01:41:13.373 INFO:teuthology.orchestra.run.smithi102.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 457 kB/s | 309 kB 00:00 2024-04-25T01:41:13.407 INFO:teuthology.orchestra.run.smithi102.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 261 kB/s | 26 kB 00:00 2024-04-25T01:41:13.490 INFO:teuthology.orchestra.run.smithi102.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 417 kB/s | 49 kB 00:00 2024-04-25T01:41:13.515 INFO:teuthology.orchestra.run.smithi102.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 624 kB/s | 67 kB 00:00 2024-04-25T01:41:13.599 INFO:teuthology.orchestra.run.smithi102.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-04-25T01:41:13.708 INFO:teuthology.orchestra.run.smithi102.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-04-25T01:41:13.850 INFO:teuthology.orchestra.run.smithi102.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-04-25T01:41:13.875 INFO:teuthology.orchestra.run.smithi102.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 262 kB/s | 43 kB 00:00 2024-04-25T01:41:13.967 INFO:teuthology.orchestra.run.smithi102.stdout:(115/152): libarrow-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 4.4 MB 00:00 2024-04-25T01:41:13.992 INFO:teuthology.orchestra.run.smithi102.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 129 kB/s | 15 kB 00:00 2024-04-25T01:41:14.017 INFO:teuthology.orchestra.run.smithi102.stdout:(117/152): python3-cachetools-4.2.4-1.el9.noarc 193 kB/s | 32 kB 00:00 2024-04-25T01:41:14.068 INFO:teuthology.orchestra.run.smithi102.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-25T01:41:14.110 INFO:teuthology.orchestra.run.smithi102.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-25T01:41:14.143 INFO:teuthology.orchestra.run.smithi102.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-25T01:41:14.211 INFO:teuthology.orchestra.run.smithi102.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 248 kB/s | 35 kB 00:00 2024-04-25T01:41:14.244 INFO:teuthology.orchestra.run.smithi102.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.5 MB/s | 206 kB 00:00 2024-04-25T01:41:14.319 INFO:teuthology.orchestra.run.smithi102.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-25T01:41:14.403 INFO:teuthology.orchestra.run.smithi102.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 7.9 MB/s | 2.0 MB 00:00 2024-04-25T01:41:14.430 INFO:teuthology.orchestra.run.smithi102.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-04-25T01:41:14.453 INFO:teuthology.orchestra.run.smithi102.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 133 kB/s | 18 kB 00:00 2024-04-25T01:41:14.504 INFO:teuthology.orchestra.run.smithi102.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 232 kB/s | 23 kB 00:00 2024-04-25T01:41:14.562 INFO:teuthology.orchestra.run.smithi102.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 146 kB/s | 19 kB 00:00 2024-04-25T01:41:14.587 INFO:teuthology.orchestra.run.smithi102.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 147 kB/s | 20 kB 00:00 2024-04-25T01:41:14.646 INFO:teuthology.orchestra.run.smithi102.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-04-25T01:41:14.671 INFO:teuthology.orchestra.run.smithi102.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 428 kB/s | 46 kB 00:00 2024-04-25T01:41:14.696 INFO:teuthology.orchestra.run.smithi102.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 729 kB/s | 79 kB 00:00 2024-04-25T01:41:14.747 INFO:teuthology.orchestra.run.smithi102.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 578 kB/s | 58 kB 00:00 2024-04-25T01:41:14.781 INFO:teuthology.orchestra.run.smithi102.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-04-25T01:41:14.806 INFO:teuthology.orchestra.run.smithi102.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-25T01:41:14.848 INFO:teuthology.orchestra.run.smithi102.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 901 kB/s | 90 kB 00:00 2024-04-25T01:41:14.866 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-04-25T01:41:14.881 INFO:teuthology.orchestra.run.smithi102.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 283 kB/s | 28 kB 00:00 2024-04-25T01:41:14.925 INFO:teuthology.orchestra.run.smithi102.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 260 kB/s | 31 kB 00:00 2024-04-25T01:41:14.957 INFO:teuthology.orchestra.run.smithi102.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-25T01:41:14.977 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:41:14.982 INFO:teuthology.orchestra.run.smithi102.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 592 kB/s | 59 kB 00:00 2024-04-25T01:41:15.015 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:41:15.024 INFO:teuthology.orchestra.run.smithi102.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 364 kB/s | 36 kB 00:00 2024-04-25T01:41:15.066 INFO:teuthology.orchestra.run.smithi102.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 471 kB/s | 51 kB 00:00 2024-04-25T01:41:15.068 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:41:15.099 INFO:teuthology.orchestra.run.smithi102.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-25T01:41:15.103 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:41:15.125 INFO:teuthology.orchestra.run.smithi102.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 897 kB/s | 90 kB 00:00 2024-04-25T01:41:15.128 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:41:15.171 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:41:15.175 INFO:teuthology.orchestra.run.smithi102.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.9 MB/s | 427 kB 00:00 2024-04-25T01:41:15.200 INFO:teuthology.orchestra.run.smithi102.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-04-25T01:41:15.234 INFO:teuthology.orchestra.run.smithi102.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-25T01:41:15.553 INFO:teuthology.orchestra.run.smithi102.stdout:(148/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 9.9 MB/s | 3.1 MB 00:00 2024-04-25T01:41:15.557 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:41:15.603 INFO:teuthology.orchestra.run.smithi102.stdout:(149/152): thrift-0.15.0-2.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2024-04-25T01:41:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:41:15.735 INFO:teuthology.orchestra.run.smithi102.stdout:(150/152): librados2-19.0.0-2445.gd2c45d97.el9. 6.4 MB/s | 3.4 MB 00:00 2024-04-25T01:41:15.751 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:41:15.782 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:41:15.804 INFO:teuthology.orchestra.run.smithi102.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 5.4 MB/s | 1.3 MB 00:00 2024-04-25T01:41:15.855 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:41:15.885 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:41:15.913 INFO:teuthology.orchestra.run.smithi102.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 7.8 MB/s | 2.4 MB 00:00 2024-04-25T01:41:15.918 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:41:15.919 INFO:teuthology.orchestra.run.smithi102.stdout:Total 16 MB/s | 220 MB 00:13 2024-04-25T01:41:16.037 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:41:16.090 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:41:16.119 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:41:16.213 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:41:16.239 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:41:16.282 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:41:16.303 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:41:16.326 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:41:16.528 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:41:16.561 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:41:16.667 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:41:16.763 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:41:16.794 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:41:16.837 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:41:16.910 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:41:16.952 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:41:16.997 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:41:17.028 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:41:17.056 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:41:17.126 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:41:17.216 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:41:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:41:17.293 INFO:teuthology.orchestra.run.smithi110.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 8.7 kB/s | 46 kB 00:05 2024-04-25T01:41:17.306 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:41:17.318 INFO:teuthology.orchestra.run.smithi110.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 15 kB/s | 79 kB 00:05 2024-04-25T01:41:17.352 INFO:teuthology.orchestra.run.smithi110.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 194 kB/s | 1.0 MB 00:05 2024-04-25T01:41:17.410 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:41:17.447 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:41:17.490 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:41:17.519 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:41:17.588 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:41:17.641 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:41:17.694 INFO:teuthology.orchestra.run.smithi110.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 144 kB/s | 58 kB 00:00 2024-04-25T01:41:17.719 INFO:teuthology.orchestra.run.smithi110.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 45 kB/s | 16 kB 00:00 2024-04-25T01:41:17.725 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:41:17.811 INFO:teuthology.orchestra.run.smithi110.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 768 kB/s | 90 kB 00:00 2024-04-25T01:41:17.836 INFO:teuthology.orchestra.run.smithi110.stdout:(136/152): python3-pecan-1.4.2-3.el9.noarch.rpm 526 kB/s | 272 kB 00:00 2024-04-25T01:41:17.858 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:41:17.861 INFO:teuthology.orchestra.run.smithi110.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 198 kB/s | 28 kB 00:00 2024-04-25T01:41:17.904 INFO:teuthology.orchestra.run.smithi110.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-04-25T01:41:17.911 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:41:17.929 INFO:teuthology.orchestra.run.smithi110.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-04-25T01:41:18.046 INFO:teuthology.orchestra.run.smithi110.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 252 kB/s | 36 kB 00:00 2024-04-25T01:41:18.180 INFO:teuthology.orchestra.run.smithi110.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 550 kB/s | 188 kB 00:00 2024-04-25T01:41:18.205 INFO:teuthology.orchestra.run.smithi110.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 185 kB/s | 51 kB 00:00 2024-04-25T01:41:18.230 INFO:teuthology.orchestra.run.smithi110.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-04-25T01:41:18.272 INFO:teuthology.orchestra.run.smithi110.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 6.3 MB/s | 427 kB 00:00 2024-04-25T01:41:18.297 INFO:teuthology.orchestra.run.smithi110.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2024-04-25T01:41:18.314 INFO:teuthology.orchestra.run.smithi110.stdout:(146/152): python3-websocket-client-1.2.3-2.el9 824 kB/s | 90 kB 00:00 2024-04-25T01:41:18.339 INFO:teuthology.orchestra.run.smithi110.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-25T01:41:18.381 INFO:teuthology.orchestra.run.smithi110.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-04-25T01:41:18.439 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:41:18.523 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:41:18.552 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:41:18.594 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:41:18.632 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:41:18.665 INFO:teuthology.orchestra.run.smithi110.stdout:(149/152): librados2-19.0.0-2445.gd2c45d97.el9. 9.7 MB/s | 3.4 MB 00:00 2024-04-25T01:41:18.674 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:41:18.771 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-04-25T01:41:18.798 INFO:teuthology.orchestra.run.smithi110.stdout:(150/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 6.8 MB/s | 3.1 MB 00:00 2024-04-25T01:41:18.826 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:41:18.902 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:41:18.957 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:41:19.010 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-04-25T01:41:19.010 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-04-25T01:41:19.030 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:41:19.061 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:41:19.098 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:41:19.144 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:41:19.180 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:41:19.223 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:41:19.274 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:41:19.290 INFO:teuthology.orchestra.run.smithi110.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 1.5 MB/s | 1.3 MB 00:00 2024-04-25T01:41:19.418 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:41:19.470 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:41:19.501 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:41:19.542 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:41:19.636 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:41:19.676 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:41:19.715 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:41:19.741 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:41:19.766 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:41:19.841 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:41:19.907 INFO:teuthology.orchestra.run.smithi110.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.9 MB/s | 2.4 MB 00:01 2024-04-25T01:41:19.910 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:41:19.910 INFO:teuthology.orchestra.run.smithi110.stdout:Total 7.7 MB/s | 220 MB 00:28 2024-04-25T01:41:19.970 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:41:19.998 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:41:20.026 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:41:20.060 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:41:20.099 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:41:20.131 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:41:20.161 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:41:20.203 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:41:20.245 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:41:20.286 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:41:20.321 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:41:20.408 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:41:20.661 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:41:20.750 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:41:20.798 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:41:20.906 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:41:20.968 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:41:21.015 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-04-25T01:41:21.156 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-04-25T01:41:21.156 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-04-25T01:41:21.587 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-04-25T01:41:21.588 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-04-25T01:41:21.693 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:41:21.731 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:41:21.764 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:41:21.865 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:41:21.924 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:41:22.163 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:41:22.250 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:41:22.428 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:41:22.456 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:41:22.533 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:41:22.924 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-04-25T01:41:22.925 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-04-25T01:41:23.183 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:41:23.296 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:41:24.508 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:41:24.593 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:41:24.621 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:41:24.654 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:41:24.964 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:41:25.008 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:41:25.099 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:41:25.130 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:41:25.166 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:41:25.636 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:41:25.678 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:41:25.749 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:41:25.788 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:41:28.040 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:41:28.101 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:41:28.372 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-04-25T01:41:28.512 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:41:28.613 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:41:28.643 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:41:28.688 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:41:28.740 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:41:28.785 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:41:28.785 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:41:28.785 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:41:28.785 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:28.801 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:41:28.814 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:41:28.864 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:41:28.963 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-04-25T01:41:28.983 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:41:29.045 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T01:41:29.091 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T01:41:29.145 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T01:41:29.187 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T01:41:29.213 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T01:41:29.255 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T01:41:29.438 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:41:29.584 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T01:41:29.592 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:41:29.705 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:41:29.774 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:41:29.804 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:41:29.829 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:41:29.899 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:41:29.931 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:41:29.987 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:41:29.990 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:41:30.028 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:41:30.176 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:41:30.226 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:41:30.271 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:41:30.308 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T01:41:30.365 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:41:30.375 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T01:41:30.387 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:41:30.400 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:41:30.433 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:41:30.455 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:41:30.475 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:41:30.530 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:41:30.568 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:41:30.638 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:41:30.671 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:41:30.681 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-25T01:41:30.711 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:41:30.778 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:41:30.786 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T01:41:30.872 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:41:30.912 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:41:30.946 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:41:31.011 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:41:31.046 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:41:31.147 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:41:31.147 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:41:31.147 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:41:31.150 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:41:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T01:41:31.218 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:41:31.272 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:41:31.303 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:41:31.338 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:41:31.350 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T01:41:31.413 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:41:31.449 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:41:31.492 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:41:31.499 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T01:41:31.521 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:41:31.554 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T01:41:31.589 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:41:31.603 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T01:41:31.636 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:41:31.709 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T01:41:31.711 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:41:31.782 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T01:41:31.850 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:41:31.856 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T01:41:31.904 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T01:41:31.904 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:41:31.943 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T01:41:32.062 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T01:41:32.134 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T01:41:32.179 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T01:41:32.238 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T01:41:32.346 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T01:41:32.390 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-25T01:41:32.428 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:41:32.443 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-25T01:41:32.482 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-25T01:41:32.508 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:41:32.538 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:41:32.580 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:41:32.619 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:41:32.619 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T01:41:32.660 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:41:32.684 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T01:41:32.784 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T01:41:32.817 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:41:32.872 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:41:32.926 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:41:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T01:41:32.999 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:41:33.031 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:41:33.045 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T01:41:33.067 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:41:33.114 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:41:33.150 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:41:33.193 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:41:33.234 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:41:33.370 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:41:33.415 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:41:33.446 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:41:33.487 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:41:33.581 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:41:33.621 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:41:33.660 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:41:33.686 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:41:33.711 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:41:33.738 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T01:41:33.785 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:41:33.912 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:41:33.944 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:41:33.963 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:41:33.998 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:41:34.037 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:41:34.046 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T01:41:34.068 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:41:34.087 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T01:41:34.098 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:41:34.133 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:41:34.134 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T01:41:34.166 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:41:34.183 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-25T01:41:34.208 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:41:34.242 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:41:34.257 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T01:41:34.329 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:41:34.406 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:41:34.437 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:41:34.484 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:41:34.583 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:41:34.594 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T01:41:34.630 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:41:34.650 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T01:41:34.734 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T01:41:34.847 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T01:41:34.898 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T01:41:34.938 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T01:41:35.005 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T01:41:35.046 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T01:41:35.107 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T01:41:35.173 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T01:41:35.347 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:41:35.371 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T01:41:35.395 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:41:35.427 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T01:41:35.428 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:41:35.483 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T01:41:35.526 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:41:35.550 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T01:41:35.578 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:41:35.665 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:41:35.689 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T01:41:35.697 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:41:35.731 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T01:41:35.779 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T01:41:35.838 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T01:41:35.875 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:41:35.888 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T01:41:35.912 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:41:35.998 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:41:36.109 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T01:41:36.291 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T01:41:36.336 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T01:41:36.374 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T01:41:36.420 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T01:41:36.479 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T01:41:36.530 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T01:41:36.597 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T01:41:36.638 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:41:36.643 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T01:41:36.693 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T01:41:36.751 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:41:36.769 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T01:41:36.818 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T01:41:36.960 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T01:41:37.020 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T01:41:37.065 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T01:41:37.135 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T01:41:37.291 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T01:41:37.362 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T01:41:37.974 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:41:38.049 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:41:38.077 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:41:38.110 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:41:38.320 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T01:41:38.387 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T01:41:38.421 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T01:41:38.426 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:41:38.482 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:41:38.550 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T01:41:38.573 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:41:38.603 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:41:38.623 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T01:41:38.639 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:41:38.723 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T01:41:38.771 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T01:41:39.005 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T01:41:39.035 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T01:41:39.112 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:41:39.143 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:41:39.171 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T01:41:39.214 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:41:39.236 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:41:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:41:39.976 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-04-25T01:41:39.977 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-04-25T01:41:39.977 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-04-25T01:41:39.977 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-04-25T01:41:39.977 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-04-25T01:41:39.977 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-04-25T01:41:39.977 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-04-25T01:41:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:40.007 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T01:41:40.148 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:41:40.158 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T01:41:40.164 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:41:40.165 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:40.165 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:41:40.165 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:41.528 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:41:41.567 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:41:41.678 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:41:41.703 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:41:41.704 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:41.704 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:41:41.704 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:41:41.704 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:41:41.704 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:41.762 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T01:41:41.859 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T01:41:41.903 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T01:41:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:41:41.944 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T01:41:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:41:41.959 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:41.959 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:41:41.960 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:41:41.960 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:41:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:42.105 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:41:42.153 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:41:42.245 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:41:42.245 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:41:42.245 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:41:42.246 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:41:42.280 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:41:42.352 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T01:41:42.396 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:41:42.404 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:41:42.427 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:41:42.427 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:42.427 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:41:42.427 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:41:42.427 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:41:42.427 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:42.518 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-25T01:41:42.542 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:41:42.553 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-25T01:41:42.591 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:41:42.610 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T01:41:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:41:42.833 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:41:42.879 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:41:42.966 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:41:43.055 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:41:43.145 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T01:41:43.174 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:41:43.194 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:41:43.238 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:41:43.249 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-25T01:41:43.276 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-25T01:41:43.278 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:41:43.306 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:41:43.697 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:41:43.731 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:41:43.761 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:41:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:41:43.861 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:41:44.893 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:41:44.929 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:41:45.888 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:41:45.913 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:41:46.014 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:41:46.183 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:41:46.205 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:41:46.215 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:41:46.238 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:41:46.238 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:46.238 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:41:46.238 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:41:46.238 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:41:46.239 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:46.248 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:41:46.290 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:41:46.302 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:41:46.398 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:41:46.447 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:41:46.882 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-25T01:41:46.927 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:41:47.088 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-25T01:41:47.088 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T01:41:47.088 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T01:41:47.089 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:41:47.128 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:41:47.341 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:41:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:41:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:41:47.391 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:47.391 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:41:47.391 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:41:47.391 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:41:47.391 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:47.436 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:41:47.457 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:41:47.457 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:47.458 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:41:47.458 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:47.503 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:41:47.525 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:41:47.525 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:47.525 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:41:47.525 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:41:47.526 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:41:47.526 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:47.828 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:41:47.851 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:41:47.851 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:47.851 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:41:47.851 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:41:47.851 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:41:47.852 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:41:52.374 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:41:52.415 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:41:52.456 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:41:52.488 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:41:52.515 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:41:52.704 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:41:52.753 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:41:52.782 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:41:53.374 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:41:53.413 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:41:53.433 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:41:53.605 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:41:53.605 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-04-25T01:41:53.605 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-04-25T01:41:53.606 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-04-25T01:41:53.606 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-04-25T01:41:53.606 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-04-25T01:41:53.606 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-04-25T01:41:53.606 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-04-25T01:41:53.606 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:41:53.767 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:41:53.783 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:41:53.783 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:53.783 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:41:53.783 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:41:54.263 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:41:54.351 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:41:54.391 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:41:54.473 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:41:54.505 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:41:54.505 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:41:54.589 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:41:54.621 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:41:54.622 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:41:54.711 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:41:54.779 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:41:55.270 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:41:55.295 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:41:55.296 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:55.296 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:41:55.296 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:41:55.296 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:41:55.296 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:41:55.530 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:41:55.552 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:41:55.552 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:55.553 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:41:55.553 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:41:55.553 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:41:55.553 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:41:55.998 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:41:56.021 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:41:56.021 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:56.021 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:41:56.021 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:41:56.021 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:41:56.021 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:41:56.120 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:41:56.169 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:41:56.223 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:41:56.412 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:41:56.465 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:41:56.543 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:41:56.632 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:41:56.763 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:41:56.799 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:41:56.831 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:41:56.859 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:41:57.207 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:41:57.233 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:41:57.264 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:41:57.319 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:41:57.348 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:41:58.384 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:41:58.424 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:41:59.393 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:41:59.416 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:41:59.518 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:41:59.680 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:41:59.710 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:41:59.733 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:41:59.733 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:41:59.734 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:41:59.734 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:41:59.734 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:41:59.734 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:41:59.777 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:41:59.788 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:41:59.885 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:41:59.934 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:42:00.375 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-25T01:42:00.375 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-04-25T01:42:00.375 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-04-25T01:42:00.375 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-04-25T01:42:00.375 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-04-25T01:42:00.375 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-04-25T01:42:00.375 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-04-25T01:42:00.375 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-04-25T01:42:00.375 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:00.604 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:42:00.650 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-25T01:42:00.651 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:00.651 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T01:42:00.651 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:00.691 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:42:00.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:42:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:42:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:42:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:42:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:42:00.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:42:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:42:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:42:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:42:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:42:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:42:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:42:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:42:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:42:00.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:42:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:42:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:42:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:42:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:42:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:42:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:42:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:42:00.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:42:00.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:42:00.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:42:00.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:42:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:42:00.704 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:42:00.826 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:42:00.855 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:42:00.879 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:42:00.879 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:00.879 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:42:00.879 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:42:00.879 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:42:00.879 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:42:00.923 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:42:00.945 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:42:00.945 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:00.945 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:42:00.945 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:42:00.990 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:42:01.012 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:42:01.013 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:01.013 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:42:01.013 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:42:01.013 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:42:01.013 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:42:01.324 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:42:01.347 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:42:01.347 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:01.347 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:42:01.347 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:42:01.347 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:42:01.347 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:42:02.292 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:42:02.378 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-25T01:42:02.378 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:02.378 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T01:42:02.378 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:42:02.379 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T01:42:02.379 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:02.678 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:42:02.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:42:02.694 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.695 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:02.696 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.697 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:42:02.698 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:42:02.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:42:02.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:42:02.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:42:02.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:42:02.704 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:42:02.705 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:42:02.706 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:42:02.706 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-04-25T01:42:02.750 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-25T01:42:02.750 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:02.750 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T01:42:02.750 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:42:02.750 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T01:42:02.750 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:03.089 DEBUG:teuthology.parallel:result is None 2024-04-25T01:42:03.255 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:42:03.324 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-25T01:42:03.324 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:03.324 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T01:42:03.324 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:42:03.324 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T01:42:03.325 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:03.455 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T01:42:03.515 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T01:42:03.583 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T01:42:03.823 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T01:42:03.885 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T01:42:03.971 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T01:42:04.079 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:42:04.291 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:42:04.339 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T01:42:04.383 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T01:42:04.417 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T01:42:05.099 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:42:05.145 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-25T01:42:05.200 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-25T01:42:05.340 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-25T01:42:05.372 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:42:05.853 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:42:05.894 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:42:05.935 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:42:05.967 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:42:05.995 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:42:06.174 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:42:06.216 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:42:06.254 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:42:06.677 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-25T01:42:06.724 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:42:06.842 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:42:06.877 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:42:06.897 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:42:07.724 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:42:07.800 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:42:07.820 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-25T01:42:07.849 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:42:07.857 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:42:07.932 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:42:07.963 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:42:07.963 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:42:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-25T01:42:08.030 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:42:08.055 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:42:08.055 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:42:08.161 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:42:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-25T01:42:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:42:08.228 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-25T01:42:08.315 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:08.315 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T01:42:08.315 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:42:08.315 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T01:42:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:42:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-25T01:42:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-25T01:42:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-25T01:42:09.600 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T01:42:09.638 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:42:09.716 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-25T01:42:09.716 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:09.716 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T01:42:09.716 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:42:09.716 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T01:42:09.717 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:09.764 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:42:09.847 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-25T01:42:09.847 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:09.847 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T01:42:09.847 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:09.923 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:42:10.026 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-25T01:42:10.026 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:10.026 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T01:42:10.026 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:42:10.027 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T01:42:10.027 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:10.415 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:42:10.500 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-25T01:42:10.500 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T01:42:10.500 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T01:42:10.500 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:42:10.501 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T01:42:10.501 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:42:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:42:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:42:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:42:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:42:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:42:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:42:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:42:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:42:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:42:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:42:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:42:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:42:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:42:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:42:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:42:14.053 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:42:14.064 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:42:14.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:42:14.066 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:42:14.067 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:42:14.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:42:14.069 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:42:14.070 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:42:14.071 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:42:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:42:15.445 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-25T01:42:15.490 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-25T01:42:15.576 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-25T01:42:15.620 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T01:42:15.666 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T01:42:16.009 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-25T01:42:16.068 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T01:42:16.113 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:42:16.114 INFO:teuthology.orchestra.run.smithi110.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.115 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:42:16.116 INFO:teuthology.orchestra.run.smithi110.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.117 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:42:16.118 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:42:16.119 INFO:teuthology.orchestra.run.smithi110.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:42:16.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:42:16.121 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:42:16.122 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:42:16.123 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:42:16.124 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:42:16.125 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:42:16.126 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-25T01:42:16.126 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:42:16.126 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-04-25T01:42:16.126 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-04-25T01:42:16.516 DEBUG:teuthology.parallel:result is None 2024-04-25T01:42:17.074 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:42:17.127 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:42:17.194 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T01:42:18.526 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-25T01:42:18.676 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T01:42:18.767 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T01:42:18.945 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T01:42:19.015 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T01:42:19.015 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:42:19.172 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T01:42:19.214 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:42:19.214 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:42:19.421 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T01:42:19.564 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:19.853 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:19.854 INFO:teuthology.orchestra.run.smithi102.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-25T01:42:19.854 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:25.439 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-25T01:42:25.439 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-25T01:42:25.440 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-25T01:42:25.441 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-25T01:42:25.453 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-25T01:42:25.453 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-25T01:42:25.454 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T01:42:25.455 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T01:42:25.456 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T01:42:25.457 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T01:42:25.458 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T01:42:25.459 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T01:42:25.460 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T01:42:25.461 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T01:42:25.462 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T01:42:25.462 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T01:42:25.462 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T01:42:25.462 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T01:42:25.462 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T01:42:25.462 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T01:42:25.462 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T01:42:25.462 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T01:42:25.462 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T01:42:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T01:42:25.464 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T01:42:25.464 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T01:42:25.464 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-25T01:42:25.464 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T01:42:25.464 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-25T01:42:25.464 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T01:42:25.464 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T01:42:25.464 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T01:42:25.464 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T01:42:27.915 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T01:42:27.915 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:27.915 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-04-25T01:42:27.915 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.916 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.917 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T01:42:27.918 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T01:42:27.919 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T01:42:27.920 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.921 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T01:42:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T01:42:27.923 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T01:42:27.924 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T01:42:27.925 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T01:42:27.926 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.927 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.927 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-25T01:42:27.927 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T01:42:27.927 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T01:42:27.927 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T01:42:27.927 INFO:teuthology.orchestra.run.smithi102.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T01:42:27.927 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T01:42:27.927 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T01:42:27.927 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T01:42:27.928 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:42:27.928 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-04-25T01:42:28.556 DEBUG:teuthology.parallel:result is None 2024-04-25T01:42:28.557 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:42:28.840 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:42:28.896 INFO:teuthology.orchestra.run.smithi102.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:42:28.896 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:42:28.896 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:42:28.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:42:29.036 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:42:29.062 INFO:teuthology.orchestra.run.smithi110.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:42:29.062 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:42:29.062 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:42:29.064 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:42:29.277 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T01:42:29.303 INFO:teuthology.orchestra.run.smithi193.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-25T01:42:29.303 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-25T01:42:29.303 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-25T01:42:29.305 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T01:42:29.305 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:42:29.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:42:29.393 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:42:29.393 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:42:29.423 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:42:29.424 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T01:42:29.455 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T01:42:29.455 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:42:29.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:42:29.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:42:29.643 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:42:29.643 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:42:29.671 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:42:29.737 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:42:29.737 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T01:42:29.767 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T01:42:29.833 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T01:42:29.833 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:42:29.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:42:29.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:42:30.007 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:42:30.007 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:42:30.031 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:42:30.095 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:42:30.095 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T01:42:30.120 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T01:42:30.183 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T01:42:30.184 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:42:30.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:42:30.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:42:30.348 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:42:30.348 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:42:30.372 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:42:30.435 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:42:30.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T01:42:30.458 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T01:42:30.522 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T01:42:30.624 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'subvols': {'create': 2}} 2024-04-25T01:42:30.624 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:42:30.625 INFO:tasks.cephadm:Cluster fsid is 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:42:30.625 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T01:42:30.625 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.102', 'mon.b': '172.21.15.110', 'mon.c': '172.21.15.193'} 2024-04-25T01:42:30.625 INFO:tasks.cephadm:First mon is mon.a on smithi102 2024-04-25T01:42:30.625 INFO:tasks.cephadm:First mgr is x 2024-04-25T01:42:30.625 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T01:42:30.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2024-04-25T01:42:30.691 DEBUG:teuthology.orchestra.run.smithi110:> sudo hostname $(hostname -s) 2024-04-25T01:42:30.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-04-25T01:42:30.738 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T01:42:30.738 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:42:30.952 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54987/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 22:37:46.422067', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54981/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2445-gd2c45d97', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2445.gd2c45d97'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 17:00:00.872117', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-25T01:42:31.168 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 d2c45d977142e024a59b68a8f1dce8f0b9840551 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:42:31.170 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm 2024-04-25T01:42:31.170 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:42:31.848 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:42 /home/ubuntu/cephtest/cephadm 2024-04-25T01:42:31.848 DEBUG:teuthology.orchestra.run.smithi110:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:42:32.271 INFO:teuthology.orchestra.run.smithi110.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:42 /home/ubuntu/cephtest/cephadm 2024-04-25T01:42:32.271 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T01:42:32.794 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r--. 1 ubuntu ubuntu 781409 Apr 25 01:42 /home/ubuntu/cephtest/cephadm 2024-04-25T01:42:32.795 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:42:32.835 DEBUG:teuthology.orchestra.run.smithi110:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:42:32.854 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T01:42:32.888 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 on all hosts... 2024-04-25T01:42:32.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:42:32.892 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:42:32.896 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 pull 2024-04-25T01:42:34.408 INFO:teuthology.orchestra.run.smithi110.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:42:34.455 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:42:35.212 INFO:teuthology.orchestra.run.smithi102.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:43:01.622 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-25T01:43:01.622 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:43:01.622 INFO:teuthology.orchestra.run.smithi110.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:43:01.622 INFO:teuthology.orchestra.run.smithi110.stdout: "repo_digests": [ 2024-04-25T01:43:01.622 INFO:teuthology.orchestra.run.smithi110.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:43:01.622 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-04-25T01:43:01.623 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-25T01:43:08.538 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-25T01:43:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:43:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:43:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-04-25T01:43:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:43:08.538 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-04-25T01:43:08.538 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-25T01:43:12.872 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:43:12.873 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T01:43:12.873 INFO:teuthology.orchestra.run.smithi102.stdout: "image_id": "f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e", 2024-04-25T01:43:12.873 INFO:teuthology.orchestra.run.smithi102.stdout: "repo_digests": [ 2024-04-25T01:43:12.873 INFO:teuthology.orchestra.run.smithi102.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901" 2024-04-25T01:43:12.873 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T01:43:12.873 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:43:12.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2024-04-25T01:43:12.999 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph 2024-04-25T01:43:13.030 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-04-25T01:43:13.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2024-04-25T01:43:13.146 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /etc/ceph 2024-04-25T01:43:13.173 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-04-25T01:43:13.202 INFO:tasks.cephadm:Writing seed config... 2024-04-25T01:43:13.203 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T01:43:13.203 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T01:43:13.203 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T01:43:13.204 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-25T01:43:13.205 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T01:43:13.206 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T01:43:13.207 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T01:43:13.207 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-25T01:43:13.207 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:43:13.207 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T01:43:13.253 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 145ca8d0-02a5-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T01:43:13.253 DEBUG:teuthology.orchestra.run.smithi102:mon.a> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T01:43:13.300 DEBUG:teuthology.orchestra.run.smithi102:mgr.x> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T01:43:13.346 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T01:43:13.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 -v bootstrap --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.102 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:43:13.612 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-04-25T01:43:13.612 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551', '-v', 'bootstrap', '--fsid', '145ca8d0-02a5-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.102', '--skip-admin-label'] 2024-04-25T01:43:13.613 INFO:teuthology.orchestra.run.smithi102.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T01:43:13.613 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying podman|docker is present... 2024-04-25T01:43:13.667 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:43:13.668 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying lvm2 is present... 2024-04-25T01:43:13.668 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying time synchronization is in place... 2024-04-25T01:43:13.684 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:43:13.684 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:43:13.699 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:43:13.699 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-25T01:43:13.716 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-04-25T01:43:13.731 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-04-25T01:43:13.731 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:43:13.731 INFO:teuthology.orchestra.run.smithi102.stdout:Repeating the final host check... 2024-04-25T01:43:13.781 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T01:43:13.781 INFO:teuthology.orchestra.run.smithi102.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T01:43:13.781 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl is present 2024-04-25T01:43:13.781 INFO:teuthology.orchestra.run.smithi102.stdout:lvcreate is present 2024-04-25T01:43:13.798 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T01:43:13.798 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T01:43:13.812 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T01:43:13.813 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-25T01:43:13.829 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-04-25T01:43:13.845 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-04-25T01:43:13.845 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-04-25T01:43:13.845 INFO:teuthology.orchestra.run.smithi102.stdout:Host looks OK 2024-04-25T01:43:13.845 INFO:teuthology.orchestra.run.smithi102.stdout:Cluster fsid: 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:13.845 INFO:teuthology.orchestra.run.smithi102.stdout:Acquiring lock 140340032775072 on /run/cephadm/145ca8d0-02a5-11ef-bc93-c7b262605968.lock 2024-04-25T01:43:13.846 INFO:teuthology.orchestra.run.smithi102.stdout:Lock 140340032775072 acquired on /run/cephadm/145ca8d0-02a5-11ef-bc93-c7b262605968.lock 2024-04-25T01:43:13.846 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 3300 ... 2024-04-25T01:43:13.846 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 6789 ... 2024-04-25T01:43:13.846 INFO:teuthology.orchestra.run.smithi102.stdout:Base mon IP(s) is [172.21.15.102:3300, 172.21.15.102:6789], mon addrv is [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-04-25T01:43:13.855 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.102 metric 100 2024-04-25T01:43:13.855 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.102 metric 100 2024-04-25T01:43:13.862 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T01:43:13.862 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-25T01:43:13.862 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-04-25T01:43:13.869 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T01:43:13.869 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T01:43:13.869 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:43:13.869 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-25T01:43:13.869 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-04-25T01:43:13.869 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T01:43:13.870 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-04-25T01:43:13.870 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-04-25T01:43:13.870 INFO:teuthology.orchestra.run.smithi102.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T01:43:13.871 INFO:teuthology.orchestra.run.smithi102.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T01:43:13.872 INFO:teuthology.orchestra.run.smithi102.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:43:23.759 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:43:23.759 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551... 2024-04-25T01:43:23.759 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T01:43:23.760 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:58a99251f41e8a467bafa0ec849428709568906ca732f10972304f37056be6c5 2024-04-25T01:43:23.760 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T01:43:23.760 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying config sha256:f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e 2024-04-25T01:43:23.760 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T01:43:25.242 INFO:teuthology.orchestra.run.smithi102.stdout:ceph: stdout ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:43:25.242 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph version: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-25T01:43:25.243 INFO:teuthology.orchestra.run.smithi102.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T01:43:26.507 INFO:teuthology.orchestra.run.smithi102.stdout:stat: stdout 167 167 2024-04-25T01:43:26.508 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial keys... 2024-04-25T01:43:27.608 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQA+tSlm9B8pOBAAzRQ5h8DB495iluSXehUtJQ== 2024-04-25T01:43:28.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQBAtSlmEqWdBhAAGFIC7Xu/4ROBgP/PutPksQ== 2024-04-25T01:43:29.975 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQBBtSlmPDwFERAAC5ZJFhsaIbsABnsb0lFVEA== 2024-04-25T01:43:29.977 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial monmap... 2024-04-25T01:43:31.095 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:43:31.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T01:43:31.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:31.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:43:31.096 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool for a [v2:172.21.15.102:3300,v1:172.21.15.102:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T01:43:31.096 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = quincy 2024-04-25T01:43:31.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: set fsid to 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:31.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T01:43:31.096 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:43:31.096 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mon... 2024-04-25T01:43:32.282 INFO:teuthology.orchestra.run.smithi102.stdout:create mon.a on 2024-04-25T01:43:32.663 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T01:43:32.907 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T01:43:33.200 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968.target. 2024-04-25T01:43:33.200 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968.target. 2024-04-25T01:43:33.601 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.a 2024-04-25T01:43:33.601 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T01:43:33.874 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968.target.wants/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@.service. 2024-04-25T01:43:34.565 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:34 smithi102 podman[27407]: 2024-04-25 01:43:34.193995048 +0000 UTC m=+0.037098169 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:43:34.565 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:34 smithi102 podman[27407]: 2024-04-25 01:43:34.574721476 +0000 UTC m=+0.417824620 container init 06363e825aa9924ff17751e78a3f807b09bbcb1a7e4a582f56fe281b8696ff59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T01:43:34.718 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:43:34.718 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-04-25T01:43:34.734 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:43:34.735 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-25T01:43:34.735 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-04-25T01:43:34.735 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:43:34.737 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon to start... 2024-04-25T01:43:34.737 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon... 2024-04-25T01:43:34.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:34 smithi102 podman[27407]: 2024-04-25 01:43:34.58510665 +0000 UTC m=+0.428209776 container start 06363e825aa9924ff17751e78a3f807b09bbcb1a7e4a582f56fe281b8696ff59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-25T01:43:34.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:34 smithi102 ceph-mon[27426]: mkfs 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:34.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:34 smithi102 ceph-mon[27426]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:43:34.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:34 smithi102 bash[27407]: 06363e825aa9924ff17751e78a3f807b09bbcb1a7e4a582f56fe281b8696ff59 2024-04-25T01:43:34.890 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:34 smithi102 systemd[1]: Started Ceph mon.a for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: monmap epoch 1 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: last_changed 2024-04-25T01:43:30.536087+0000 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: min_mon_release 19 (squid) 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: election_strategy: 1 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: fsmap 2024-04-25T01:43:35.846 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:43:35.847 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: mgrmap e1: no daemons active 2024-04-25T01:43:35.847 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:35 smithi102 ceph-mon[27426]: from='client.? 172.21.15.102:0/3242163090' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T01:43:36.121 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T01:43:36.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout id: 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:36.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T01:43:36.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:36.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout services: 2024-04-25T01:43:36.122 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.828986s) 2024-04-25T01:43:36.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T01:43:36.123 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T01:43:36.128 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:36.128 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout data: 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.397+0000 7ff851200700 1 Processor -- start 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.398+0000 7ff851200700 1 -- start start 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.398+0000 7ff851200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.398+0000 7ff851200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff84c108600 con 0x7ff84c107c10 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.399+0000 7ff84b400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.399+0000 7ff84b400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36766/0 (socket says 172.21.15.102:36766) 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.399+0000 7ff84b400700 1 -- 172.21.15.102:0/3434530992 learned_addr learned my addr 172.21.15.102:0/3434530992 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:36.129 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.399+0000 7ff84b400700 1 -- 172.21.15.102:0/3434530992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84c108e10 con 0x7ff84c107c10 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.400+0000 7ff84b400700 1 --2- 172.21.15.102:0/3434530992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c108030 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff83c0098a0 tx=0x7ff83c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7f40a36edec7e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.400+0000 7ff84a000700 1 -- 172.21.15.102:0/3434530992 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff83c004160 con 0x7ff84c107c10 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.400+0000 7ff84a000700 1 -- 172.21.15.102:0/3434530992 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff83c004300 con 0x7ff84c107c10 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.400+0000 7ff84a000700 1 -- 172.21.15.102:0/3434530992 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff83c0036a0 con 0x7ff84c107c10 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.401+0000 7ff851200700 1 -- 172.21.15.102:0/3434530992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 msgr2=0x7ff84c108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.401+0000 7ff851200700 1 --2- 172.21.15.102:0/3434530992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c108030 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff83c0098a0 tx=0x7ff83c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.401+0000 7ff851200700 1 -- 172.21.15.102:0/3434530992 shutdown_connections 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.401+0000 7ff851200700 1 --2- 172.21.15.102:0/3434530992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c108030 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.401+0000 7ff851200700 1 -- 172.21.15.102:0/3434530992 >> 172.21.15.102:0/3434530992 conn(0x7ff84c1031d0 msgr2=0x7ff84c1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.401+0000 7ff851200700 1 -- 172.21.15.102:0/3434530992 shutdown_connections 2024-04-25T01:43:36.130 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.401+0000 7ff851200700 1 -- 172.21.15.102:0/3434530992 wait complete. 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.402+0000 7ff851200700 1 Processor -- start 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.402+0000 7ff851200700 1 -- start start 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.402+0000 7ff851200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c19d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.402+0000 7ff851200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff84c109340 con 0x7ff84c107c10 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.403+0000 7ff84b400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c19d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.403+0000 7ff84b400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c19d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36774/0 (socket says 172.21.15.102:36774) 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.403+0000 7ff84b400700 1 -- 172.21.15.102:0/3242163090 learned_addr learned my addr 172.21.15.102:0/3242163090 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.403+0000 7ff84b400700 1 -- 172.21.15.102:0/3242163090 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84c19da70 con 0x7ff84c107c10 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.403+0000 7ff84b400700 1 --2- 172.21.15.102:0/3242163090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c19d530 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff83c000c00 tx=0x7ff83c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:36.131 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.404+0000 7ff843e00700 1 -- 172.21.15.102:0/3242163090 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff83c003db0 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.404+0000 7ff843e00700 1 -- 172.21.15.102:0/3242163090 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff83c011630 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.404+0000 7ff851200700 1 -- 172.21.15.102:0/3242163090 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff84c19dd00 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.404+0000 7ff843e00700 1 -- 172.21.15.102:0/3242163090 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff83c011950 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.404+0000 7ff851200700 1 -- 172.21.15.102:0/3242163090 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff84c1a09f0 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.404+0000 7ff843e00700 1 -- 172.21.15.102:0/3242163090 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff83c011af0 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.404+0000 7ff843e00700 1 -- 172.21.15.102:0/3242163090 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff83c01c450 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.405+0000 7ff851200700 1 -- 172.21.15.102:0/3242163090 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff84c069390 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.408+0000 7ff843e00700 1 -- 172.21.15.102:0/3242163090 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff83c032020 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.487+0000 7ff851200700 1 -- 172.21.15.102:0/3242163090 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7ff84c1a0ee0 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.488+0000 7ff843e00700 1 -- 172.21.15.102:0/3242163090 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7ff83c004260 con 0x7ff84c107c10 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.489+0000 7ff851200700 1 -- 172.21.15.102:0/3242163090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 msgr2=0x7ff84c19d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.489+0000 7ff851200700 1 --2- 172.21.15.102:0/3242163090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c19d530 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff83c000c00 tx=0x7ff83c004750 comp rx=0 tx=0).stop 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.489+0000 7ff851200700 1 -- 172.21.15.102:0/3242163090 shutdown_connections 2024-04-25T01:43:36.132 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.489+0000 7ff851200700 1 --2- 172.21.15.102:0/3242163090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff84c107c10 0x7ff84c19d530 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:36.133 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.489+0000 7ff851200700 1 -- 172.21.15.102:0/3242163090 >> 172.21.15.102:0/3242163090 conn(0x7ff84c1031d0 msgr2=0x7ff84c07d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:36.133 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.490+0000 7ff851200700 1 -- 172.21.15.102:0/3242163090 shutdown_connections 2024-04-25T01:43:36.133 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:35.490+0000 7ff851200700 1 -- 172.21.15.102:0/3242163090 wait complete. 2024-04-25T01:43:36.133 INFO:teuthology.orchestra.run.smithi102.stdout:mon is available 2024-04-25T01:43:36.133 INFO:teuthology.orchestra.run.smithi102.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T01:43:37.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:36 smithi102 ceph-mon[27426]: from='client.? 172.21.15.102:0/3390948249' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:43:37.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:36 smithi102 ceph-mon[27426]: from='client.? 172.21.15.102:0/3390948249' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T01:43:37.511 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:37.512 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:43:37.512 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:37.519 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-04-25T01:43:37.519 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:43:37.519 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:43:37.519 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:43:37.519 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.738+0000 7f25b8a00700 1 Processor -- start 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.738+0000 7f25b8a00700 1 -- start start 2024-04-25T01:43:37.520 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.739+0000 7f25b8a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b410a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.739+0000 7f25b8a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410a7a0 con 0x7f25b4109e40 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.739+0000 7f25b3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b410a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.739+0000 7f25b3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b410a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36780/0 (socket says 172.21.15.102:36780) 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.739+0000 7f25b3400700 1 -- 172.21.15.102:0/2703194181 learned_addr learned my addr 172.21.15.102:0/2703194181 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.740+0000 7f25b3400700 1 -- 172.21.15.102:0/2703194181 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b410afa0 con 0x7f25b4109e40 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.740+0000 7f25b3400700 1 --2- 172.21.15.102:0/2703194181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b410a260 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f25a40098a0 tx=0x7f25a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41948a31f7dc1c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.740+0000 7f25b2000700 1 -- 172.21.15.102:0/2703194181 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25a4004160 con 0x7f25b4109e40 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.740+0000 7f25b2000700 1 -- 172.21.15.102:0/2703194181 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a4004300 con 0x7f25b4109e40 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.740+0000 7f25b2000700 1 -- 172.21.15.102:0/2703194181 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25a40036a0 con 0x7f25b4109e40 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.741+0000 7f25b8a00700 1 -- 172.21.15.102:0/2703194181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 msgr2=0x7f25b410a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.741+0000 7f25b8a00700 1 --2- 172.21.15.102:0/2703194181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b410a260 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f25a40098a0 tx=0x7f25a4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.741+0000 7f25b8a00700 1 -- 172.21.15.102:0/2703194181 shutdown_connections 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.741+0000 7f25b8a00700 1 --2- 172.21.15.102:0/2703194181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b410a260 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:37.521 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.741+0000 7f25b8a00700 1 -- 172.21.15.102:0/2703194181 >> 172.21.15.102:0/2703194181 conn(0x7f25b407d240 msgr2=0x7f25b407d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.741+0000 7f25b8a00700 1 -- 172.21.15.102:0/2703194181 shutdown_connections 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.742+0000 7f25b8a00700 1 -- 172.21.15.102:0/2703194181 wait complete. 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.743+0000 7f25b8a00700 1 Processor -- start 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.743+0000 7f25b8a00700 1 -- start start 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.743+0000 7f25b8a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b419d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.743+0000 7f25b8a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410b470 con 0x7f25b4109e40 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.744+0000 7f25b3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b419d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.744+0000 7f25b3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b419d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36788/0 (socket says 172.21.15.102:36788) 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.744+0000 7f25b3400700 1 -- 172.21.15.102:0/3390948249 learned_addr learned my addr 172.21.15.102:0/3390948249 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.744+0000 7f25b3400700 1 -- 172.21.15.102:0/3390948249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b419da40 con 0x7f25b4109e40 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.744+0000 7f25b3400700 1 --2- 172.21.15.102:0/3390948249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b419d500 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f25a4000c00 tx=0x7f25a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:37.522 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.745+0000 7f25abe00700 1 -- 172.21.15.102:0/3390948249 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25a4003db0 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.745+0000 7f25abe00700 1 -- 172.21.15.102:0/3390948249 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a4011630 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.745+0000 7f25abe00700 1 -- 172.21.15.102:0/3390948249 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25a4011950 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.745+0000 7f25b8a00700 1 -- 172.21.15.102:0/3390948249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b419dcd0 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.745+0000 7f25b8a00700 1 -- 172.21.15.102:0/3390948249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b41a09c0 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.745+0000 7f25abe00700 1 -- 172.21.15.102:0/3390948249 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f25a401a460 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.745+0000 7f25abe00700 1 -- 172.21.15.102:0/3390948249 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f25a401b780 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.746+0000 7f25b8a00700 1 -- 172.21.15.102:0/3390948249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b4069390 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.749+0000 7f25abe00700 1 -- 172.21.15.102:0/3390948249 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f25a401ba20 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.826+0000 7f25b8a00700 1 -- 172.21.15.102:0/3390948249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f25b402de30 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.833+0000 7f25abe00700 1 -- 172.21.15.102:0/3390948249 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7f25a401bc00 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.834+0000 7f25abe00700 1 -- 172.21.15.102:0/3390948249 <== mon.0 v2:172.21.15.102:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f25a401bde0 con 0x7f25b4109e40 2024-04-25T01:43:37.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.835+0000 7f25b8a00700 1 -- 172.21.15.102:0/3390948249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 msgr2=0x7f25b419d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:37.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.835+0000 7f25b8a00700 1 --2- 172.21.15.102:0/3390948249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b419d500 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f25a4000c00 tx=0x7f25a4004750 comp rx=0 tx=0).stop 2024-04-25T01:43:37.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.835+0000 7f25b8a00700 1 -- 172.21.15.102:0/3390948249 shutdown_connections 2024-04-25T01:43:37.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.835+0000 7f25b8a00700 1 --2- 172.21.15.102:0/3390948249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25b4109e40 0x7f25b419d500 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:37.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.835+0000 7f25b8a00700 1 -- 172.21.15.102:0/3390948249 >> 172.21.15.102:0/3390948249 conn(0x7f25b407d240 msgr2=0x7f25b4193fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:37.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.835+0000 7f25b8a00700 1 -- 172.21.15.102:0/3390948249 shutdown_connections 2024-04-25T01:43:37.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:36.835+0000 7f25b8a00700 1 -- 172.21.15.102:0/3390948249 wait complete. 2024-04-25T01:43:37.525 INFO:teuthology.orchestra.run.smithi102.stdout:Generating new minimal ceph.conf... 2024-04-25T01:43:38.413 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:38 smithi102 ceph-mon[27426]: from='client.? 172.21.15.102:0/3386948364' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:43:38.866 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.155+0000 7f2504a00700 1 Processor -- start 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.155+0000 7f2504a00700 1 -- start start 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.156+0000 7f2504a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f2500111060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.156+0000 7f2504a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25001115a0 con 0x7f250010ec70 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.156+0000 7f24fde00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f2500111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.156+0000 7f24fde00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f2500111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36798/0 (socket says 172.21.15.102:36798) 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.156+0000 7f24fde00700 1 -- 172.21.15.102:0/1765505308 learned_addr learned my addr 172.21.15.102:0/1765505308 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.156+0000 7f24fde00700 1 -- 172.21.15.102:0/1765505308 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2500111720 con 0x7f250010ec70 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.157+0000 7f24fde00700 1 --2- 172.21.15.102:0/1765505308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f2500111060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24f00098a0 tx=0x7f24f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48da5451c8436562 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.157+0000 7f24fca00700 1 -- 172.21.15.102:0/1765505308 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f0004160 con 0x7f250010ec70 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.157+0000 7f24fca00700 1 -- 172.21.15.102:0/1765505308 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f24f0004300 con 0x7f250010ec70 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.157+0000 7f24fca00700 1 -- 172.21.15.102:0/1765505308 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f0011b50 con 0x7f250010ec70 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.158+0000 7f2504a00700 1 -- 172.21.15.102:0/1765505308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 msgr2=0x7f2500111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:38.867 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.158+0000 7f2504a00700 1 --2- 172.21.15.102:0/1765505308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f2500111060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f24f00098a0 tx=0x7f24f0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.158+0000 7f2504a00700 1 -- 172.21.15.102:0/1765505308 shutdown_connections 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.158+0000 7f2504a00700 1 --2- 172.21.15.102:0/1765505308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f2500111060 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.158+0000 7f2504a00700 1 -- 172.21.15.102:0/1765505308 >> 172.21.15.102:0/1765505308 conn(0x7f250007d3c0 msgr2=0x7f250007f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.158+0000 7f2504a00700 1 -- 172.21.15.102:0/1765505308 shutdown_connections 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.158+0000 7f2504a00700 1 -- 172.21.15.102:0/1765505308 wait complete. 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.159+0000 7f2504a00700 1 Processor -- start 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.159+0000 7f2504a00700 1 -- start start 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.160+0000 7f2504a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f25001a2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.160+0000 7f2504a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2500112800 con 0x7f250010ec70 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.160+0000 7f24fde00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f25001a2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.160+0000 7f24fde00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f25001a2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36810/0 (socket says 172.21.15.102:36810) 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.160+0000 7f24fde00700 1 -- 172.21.15.102:0/3386948364 learned_addr learned my addr 172.21.15.102:0/3386948364 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:38.868 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.160+0000 7f24fde00700 1 -- 172.21.15.102:0/3386948364 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25001a2960 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.161+0000 7f24fde00700 1 --2- 172.21.15.102:0/3386948364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f25001a2420 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f24f0011b60 tx=0x7f24f0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.161+0000 7f24f6a00700 1 -- 172.21.15.102:0/3386948364 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f0003b20 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.161+0000 7f24f6a00700 1 -- 172.21.15.102:0/3386948364 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f24f0003cc0 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.161+0000 7f24f6a00700 1 -- 172.21.15.102:0/3386948364 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24f0024840 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.161+0000 7f2504a00700 1 -- 172.21.15.102:0/3386948364 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25001a2bf0 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.161+0000 7f2504a00700 1 -- 172.21.15.102:0/3386948364 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25001a3710 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.162+0000 7f24f6a00700 1 -- 172.21.15.102:0/3386948364 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f24f0022020 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.162+0000 7f24f6a00700 1 -- 172.21.15.102:0/3386948364 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f24f0011e00 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.163+0000 7f2504a00700 1 -- 172.21.15.102:0/3386948364 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2500063890 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.166+0000 7f24f6a00700 1 -- 172.21.15.102:0/3386948364 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f24f0043dd0 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.241+0000 7f2504a00700 1 -- 172.21.15.102:0/3386948364 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f25001a3b50 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.242+0000 7f24f6a00700 1 -- 172.21.15.102:0/3386948364 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f24f0032020 con 0x7f250010ec70 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.243+0000 7f2504a00700 1 -- 172.21.15.102:0/3386948364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 msgr2=0x7f25001a2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:38.869 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.243+0000 7f2504a00700 1 --2- 172.21.15.102:0/3386948364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f25001a2420 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f24f0011b60 tx=0x7f24f0011c40 comp rx=0 tx=0).stop 2024-04-25T01:43:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.243+0000 7f2504a00700 1 -- 172.21.15.102:0/3386948364 shutdown_connections 2024-04-25T01:43:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.243+0000 7f2504a00700 1 --2- 172.21.15.102:0/3386948364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f250010ec70 0x7f25001a2420 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.243+0000 7f2504a00700 1 -- 172.21.15.102:0/3386948364 >> 172.21.15.102:0/3386948364 conn(0x7f250007d3c0 msgr2=0x7f250007de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.244+0000 7f2504a00700 1 -- 172.21.15.102:0/3386948364 shutdown_connections 2024-04-25T01:43:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:38.244+0000 7f2504a00700 1 -- 172.21.15.102:0/3386948364 wait complete. 2024-04-25T01:43:38.870 INFO:teuthology.orchestra.run.smithi102.stdout:Restarting the monitor... 2024-04-25T01:43:39.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:38 smithi102 systemd[1]: Stopping Ceph mon.a for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:43:39.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a[27421]: 2024-04-25T01:43:39.338+0000 7f290d600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T01:43:39.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a[27421]: 2024-04-25T01:43:39.338+0000 7f290d600700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T01:43:39.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:39 smithi102 podman[27673]: 2024-04-25 01:43:39.471740507 +0000 UTC m=+0.254605544 container died 06363e825aa9924ff17751e78a3f807b09bbcb1a7e4a582f56fe281b8696ff59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True) 2024-04-25T01:43:40.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:40 smithi102 podman[27673]: 2024-04-25 01:43:40.315019086 +0000 UTC m=+1.097884152 container cleanup 06363e825aa9924ff17751e78a3f807b09bbcb1a7e4a582f56fe281b8696ff59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_CLEAN=True) 2024-04-25T01:43:40.311 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:40 smithi102 bash[27673]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a 2024-04-25T01:43:40.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:40 smithi102 podman[27686]: 2024-04-25 01:43:40.474119356 +0000 UTC m=+0.988298457 container remove 06363e825aa9924ff17751e78a3f807b09bbcb1a7e4a582f56fe281b8696ff59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T01:43:40.974 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:40 smithi102 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T01:43:40.974 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:40 smithi102 systemd[1]: Stopped Ceph mon.a for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:43:40.974 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:40 smithi102 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.886s CPU time. 2024-04-25T01:43:41.337 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:40 smithi102 systemd[1]: Starting Ceph mon.a for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:43:41.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 podman[27776]: 2024-04-25 01:43:41.251665259 +0000 UTC m=+0.031924071 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:43:41.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 podman[27776]: 2024-04-25 01:43:41.558356678 +0000 UTC m=+0.338615519 container create 7e663022620a73e032edbc75ede968d589e57846b0437e3a481d4a7dfccba870 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T01:43:41.902 INFO:teuthology.orchestra.run.smithi102.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T01:43:41.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 podman[27776]: 2024-04-25 01:43:41.803238565 +0000 UTC m=+0.583497413 container init 7e663022620a73e032edbc75ede968d589e57846b0437e3a481d4a7dfccba870 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T01:43:41.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 podman[27776]: 2024-04-25 01:43:41.814104364 +0000 UTC m=+0.594363176 container start 7e663022620a73e032edbc75ede968d589e57846b0437e3a481d4a7dfccba870 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:43:41.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T01:43:41.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev), process ceph-mon, pid 2 2024-04-25T01:43:41.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: pidfile_write: ignore empty --pid-file 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: load: jerasure load: lrc 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: RocksDB version: 7.9.2 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Git sha 0 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Compile date 2024-04-24 16:41:23 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: DB SUMMARY 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: DB Session ID: VDNVENBPWJTGRWFOYWCF 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: CURRENT file: CURRENT 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106146 ; 2024-04-25T01:43:41.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.error_if_exists: 0 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.create_if_missing: 0 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.paranoid_checks: 1 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.env: 0x558a53dd6c60 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.info_log: 0x558a80a71ae0 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T01:43:41.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.statistics: (nil) 2024-04-25T01:43:41.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.use_fsync: 0 2024-04-25T01:43:41.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_log_file_size: 0 2024-04-25T01:43:41.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T01:43:41.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T01:43:41.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.allow_fallocate: 1 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.use_direct_reads: 0 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.db_log_dir: 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.wal_dir: 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T01:43:41.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.write_buffer_manager: 0x558a80a441e0 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T01:43:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.unordered_write: 0 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.row_cache: None 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.wal_filter: None 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.two_write_queues: 0 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.wal_compression: 0 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.atomic_flush: 0 2024-04-25T01:43:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.log_readahead_size: 0 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_background_jobs: 2 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_background_compactions: -1 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_subcompactions: 1 2024-04-25T01:43:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_open_files: -1 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_background_flushes: -1 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Compression algorithms supported: 2024-04-25T01:43:41.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: kZSTD supported: 0 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: kXpressCompression supported: 0 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: kZlibCompression supported: 1 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: kSnappyCompression supported: 1 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: kLZ4Compression supported: 1 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: kBZip2Compression supported: 0 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T01:43:41.933 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.merge_operator: 2024-04-25T01:43:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_filter: None 2024-04-25T01:43:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T01:43:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T01:43:41.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558a80a71be0) 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_type: 0 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_index_type: 0 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_shortening: 1 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: checksum: 4 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: no_block_cache: 0 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache: 0x558a80a75090 2024-04-25T01:43:41.935 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_options: 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: capacity : 536870912 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_shard_bits : 4 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: strict_capacity_limit : 0 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_compressed: (nil) 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: persistent_cache: (nil) 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size: 4096 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size_deviation: 10 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_restart_interval: 16 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_block_restart_interval: 1 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: metadata_block_size: 4096 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: partition_filters: 0 2024-04-25T01:43:41.936 INFO:journalctl@ceph.mon.a.smithi102.stdout: use_delta_encoding: 1 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: filter_policy: bloomfilter 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: whole_key_filtering: 1 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: verify_compression: 0 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: format_version: 5 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_index_compression: 1 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_align: 0 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_auto_readahead_size: 262144 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: prepopulate_block_cache: 0 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: initial_auto_readahead_size: 8192 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression: NoCompression 2024-04-25T01:43:41.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.num_levels: 7 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T01:43:41.938 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T01:43:41.939 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T01:43:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T01:43:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T01:43:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T01:43:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T01:43:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T01:43:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T01:43:41.940 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T01:43:41.941 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.table_properties_collectors: 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.inplace_update_support: 0 2024-04-25T01:43:41.942 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.bloom_locality: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.max_successive_merges: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.ttl: 2592000 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T01:43:41.943 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.enable_blob_files: false 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.min_blob_size: 0 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: acd93909-fde1-43ac-b259-ee8028627d63 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714009421867171, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T01:43:41.944 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714009421868928, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101070, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 332, "table_properties": {"data_size": 98981, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14516, "raw_average_key_size": 49, "raw_value_size": 91165, "raw_average_value_size": 313, "num_data_blocks": 12, "num_entries": 291, "num_filter_entries": 291, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714009421, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "acd93909-fde1-43ac-b259-ee8028627d63", "db_session_id": "VDNVENBPWJTGRWFOYWCF", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714009421869072, "job": 1, "event": "recovery_finished"} 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558a80b66000 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: DB pointer 0x558a80b50000 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** DB Stats ** 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:43:41.945 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: L0 2/0 100.54 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 86.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: Sum 2/0 100.54 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 86.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 86.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T01:43:41.946 INFO:journalctl@ceph.mon.a.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 86.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative compaction: 0.00 GB write, 15.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval compaction: 0.00 GB write, 15.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T01:43:41.947 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache BinnedLRUCache@0x558a80a75090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: starting mon.a rank 0 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mon.a@-1(???) e1 preinit fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mon.a@-1(???).mds e1 new map 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mon.a@-1(???).mds e1 print_map 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout: e1 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout: legacy client fscid: -1 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout: No filesystems configured 2024-04-25T01:43:41.948 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 bash[27776]: 7e663022620a73e032edbc75ede968d589e57846b0437e3a481d4a7dfccba870 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 systemd[1]: Started Ceph mon.a for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: monmap epoch 1 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: last_changed 2024-04-25T01:43:30.536087+0000 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: min_mon_release 19 (squid) 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: election_strategy: 1 2024-04-25T01:43:41.949 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:43:41.950 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: fsmap 2024-04-25T01:43:41.950 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T01:43:41.950 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:41 smithi102 ceph-mon[27796]: mgrmap e1: no daemons active 2024-04-25T01:43:43.208 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.453+0000 7fde95e00700 1 Processor -- start 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.453+0000 7fde95e00700 1 -- start start 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.453+0000 7fde95e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9007c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.453+0000 7fde95e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde9007c7a0 con 0x7fde9007dd60 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.454+0000 7fde8ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9007c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.454+0000 7fde8ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9007c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36826/0 (socket says 172.21.15.102:36826) 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.454+0000 7fde8ea00700 1 -- 172.21.15.102:0/2534306813 learned_addr learned my addr 172.21.15.102:0/2534306813 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.455+0000 7fde8ea00700 1 -- 172.21.15.102:0/2534306813 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9007c920 con 0x7fde9007dd60 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.455+0000 7fde8ea00700 1 --2- 172.21.15.102:0/2534306813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9007c260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fde800098a0 tx=0x7fde80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2df7037eb069164 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.455+0000 7fde8d600700 1 -- 172.21.15.102:0/2534306813 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde80004160 con 0x7fde9007dd60 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.455+0000 7fde8d600700 1 -- 172.21.15.102:0/2534306813 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fde80004300 con 0x7fde9007dd60 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.456+0000 7fde8d600700 1 -- 172.21.15.102:0/2534306813 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde80011b50 con 0x7fde9007dd60 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.456+0000 7fde95e00700 1 -- 172.21.15.102:0/2534306813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 msgr2=0x7fde9007c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.456+0000 7fde95e00700 1 --2- 172.21.15.102:0/2534306813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9007c260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fde800098a0 tx=0x7fde80009bb0 comp rx=0 tx=0).stop 2024-04-25T01:43:43.209 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.456+0000 7fde95e00700 1 -- 172.21.15.102:0/2534306813 shutdown_connections 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.456+0000 7fde95e00700 1 --2- 172.21.15.102:0/2534306813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9007c260 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.456+0000 7fde95e00700 1 -- 172.21.15.102:0/2534306813 >> 172.21.15.102:0/2534306813 conn(0x7fde90103420 msgr2=0x7fde90105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.456+0000 7fde95e00700 1 -- 172.21.15.102:0/2534306813 shutdown_connections 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.457+0000 7fde95e00700 1 -- 172.21.15.102:0/2534306813 wait complete. 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.458+0000 7fde95e00700 1 Processor -- start 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.458+0000 7fde95e00700 1 -- start start 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.458+0000 7fde95e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.458+0000 7fde95e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde9010e3f0 con 0x7fde9007dd60 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.459+0000 7fde8ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.459+0000 7fde8ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9019e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36840/0 (socket says 172.21.15.102:36840) 2024-04-25T01:43:43.210 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.459+0000 7fde8ea00700 1 -- 172.21.15.102:0/3578296771 learned_addr learned my addr 172.21.15.102:0/3578296771 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:43.211 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.459+0000 7fde8ea00700 1 -- 172.21.15.102:0/3578296771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9019e5a0 con 0x7fde9007dd60 2024-04-25T01:43:43.211 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.459+0000 7fde8ea00700 1 --2- 172.21.15.102:0/3578296771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9019e060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fde80011b60 tx=0x7fde80011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.460+0000 7fde94a00700 1 -- 172.21.15.102:0/3578296771 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde80003b20 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.460+0000 7fde94a00700 1 -- 172.21.15.102:0/3578296771 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fde80003cc0 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.460+0000 7fde94a00700 1 -- 172.21.15.102:0/3578296771 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde80024840 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.460+0000 7fde95e00700 1 -- 172.21.15.102:0/3578296771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9019e830 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.460+0000 7fde95e00700 1 -- 172.21.15.102:0/3578296771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9019f350 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.460+0000 7fde94a00700 1 -- 172.21.15.102:0/3578296771 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fde80022020 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.460+0000 7fde94a00700 1 -- 172.21.15.102:0/3578296771 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fde80011e00 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.462+0000 7fde95e00700 1 -- 172.21.15.102:0/3578296771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde90063890 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.465+0000 7fde94a00700 1 -- 172.21.15.102:0/3578296771 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fde80043dd0 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.560+0000 7fde95e00700 1 -- 172.21.15.102:0/3578296771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fde9019f7d0 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.565+0000 7fde94a00700 1 -- 172.21.15.102:0/3578296771 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fde80032020 con 0x7fde9007dd60 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.566+0000 7fde95e00700 1 -- 172.21.15.102:0/3578296771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 msgr2=0x7fde9019e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:43.213 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.566+0000 7fde95e00700 1 --2- 172.21.15.102:0/3578296771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9019e060 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fde80011b60 tx=0x7fde80011c40 comp rx=0 tx=0).stop 2024-04-25T01:43:43.214 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.566+0000 7fde95e00700 1 -- 172.21.15.102:0/3578296771 shutdown_connections 2024-04-25T01:43:43.214 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.566+0000 7fde95e00700 1 --2- 172.21.15.102:0/3578296771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde9007dd60 0x7fde9019e060 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:43.214 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.566+0000 7fde95e00700 1 -- 172.21.15.102:0/3578296771 >> 172.21.15.102:0/3578296771 conn(0x7fde90103420 msgr2=0x7fde90103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:43.214 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.566+0000 7fde95e00700 1 -- 172.21.15.102:0/3578296771 shutdown_connections 2024-04-25T01:43:43.214 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:42.566+0000 7fde95e00700 1 -- 172.21.15.102:0/3578296771 wait complete. 2024-04-25T01:43:43.214 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T01:43:43.239 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:43:43.240 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mgr... 2024-04-25T01:43:43.241 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T01:43:43.242 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T01:43:43.707 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.x 2024-04-25T01:43:43.707 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T01:43:43.913 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:43 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3578296771' entity='client.admin' 2024-04-25T01:43:43.970 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968.target.wants/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@.service. 2024-04-25T01:43:44.352 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:44 smithi102 systemd[1]: Starting Ceph mgr.x for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:43:44.621 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:44 smithi102 podman[28023]: 2024-04-25 01:43:44.266103365 +0000 UTC m=+0.030313587 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T01:43:44.621 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:44 smithi102 podman[28023]: 2024-04-25 01:43:44.402236064 +0000 UTC m=+0.166446322 container create 5b9d2d220f12f87c864763b3123735ccc0f3f14ef7ba2321fe2ac2fd72b719a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9) 2024-04-25T01:43:44.621 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:44 smithi102 podman[28023]: 2024-04-25 01:43:44.630617988 +0000 UTC m=+0.394828253 container init 5b9d2d220f12f87c864763b3123735ccc0f3f14ef7ba2321fe2ac2fd72b719a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T01:43:44.760 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:43:44.760 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-04-25T01:43:44.777 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:43:44.777 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-25T01:43:44.777 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-04-25T01:43:44.777 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T01:43:44.806 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:43:44.806 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-04-25T01:43:44.827 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:43:44.827 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-25T01:43:44.827 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-04-25T01:43:44.827 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T01:43:44.827 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr to start... 2024-04-25T01:43:44.827 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr... 2024-04-25T01:43:44.914 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:44 smithi102 podman[28023]: 2024-04-25 01:43:44.640201905 +0000 UTC m=+0.404412153 container start 5b9d2d220f12f87c864763b3123735ccc0f3f14ef7ba2321fe2ac2fd72b719a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T01:43:44.914 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:44 smithi102 bash[28023]: 5b9d2d220f12f87c864763b3123735ccc0f3f14ef7ba2321fe2ac2fd72b719a8 2024-04-25T01:43:44.914 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:44 smithi102 systemd[1]: Started Ceph mgr.x for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:43:44.914 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:44 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:44.869+0000 7f4b7ef1e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:43:45.246 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:45 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:45.119+0000 7f4b7ef1e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:43:45.246 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:45 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:45.255+0000 7f4b7ef1e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:43:45.913 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:45 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1363679672' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:43:45.913 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:45 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:45.674+0000 7f4b7ef1e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:43:45.913 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:45 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:45.788+0000 7f4b7ef1e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:43:46.306 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:46.308 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:43:46.309 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:46.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-25T01:43:46.310 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:43:46.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:46.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:43:46.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:43:46.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:46.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:43:46.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:46.313 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:43:46.314 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:43:46.315 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:43:34.661710+0000", 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:43:46.316 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.490+0000 7f7bc5600700 1 Processor -- start 2024-04-25T01:43:46.321 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.491+0000 7f7bc5600700 1 -- start start 2024-04-25T01:43:46.321 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.491+0000 7f7bc5600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc007c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:46.321 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.491+0000 7f7bc5600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc007c7c0 con 0x7f7bc007dd80 2024-04-25T01:43:46.321 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.492+0000 7f7bbea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc007c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:46.321 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.492+0000 7f7bbea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc007c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52616/0 (socket says 172.21.15.102:52616) 2024-04-25T01:43:46.321 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.492+0000 7f7bbea00700 1 -- 172.21.15.102:0/2118579228 learned_addr learned my addr 172.21.15.102:0/2118579228 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:46.321 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.492+0000 7f7bbea00700 1 -- 172.21.15.102:0/2118579228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc007c940 con 0x7f7bc007dd80 2024-04-25T01:43:46.321 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.493+0000 7f7bbea00700 1 --2- 172.21.15.102:0/2118579228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc007c280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00098a0 tx=0x7f7bb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb19de99928e15a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:46.321 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.493+0000 7f7bbd600700 1 -- 172.21.15.102:0/2118579228 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bb0004160 con 0x7f7bc007dd80 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.493+0000 7f7bbd600700 1 -- 172.21.15.102:0/2118579228 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7bb0004300 con 0x7f7bc007dd80 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.493+0000 7f7bbd600700 1 -- 172.21.15.102:0/2118579228 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bb0011b50 con 0x7f7bc007dd80 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.494+0000 7f7bc5600700 1 -- 172.21.15.102:0/2118579228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 msgr2=0x7f7bc007c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.494+0000 7f7bc5600700 1 --2- 172.21.15.102:0/2118579228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc007c280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00098a0 tx=0x7f7bb0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.494+0000 7f7bc5600700 1 -- 172.21.15.102:0/2118579228 shutdown_connections 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.494+0000 7f7bc5600700 1 --2- 172.21.15.102:0/2118579228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc007c280 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.494+0000 7f7bc5600700 1 -- 172.21.15.102:0/2118579228 >> 172.21.15.102:0/2118579228 conn(0x7f7bc0103420 msgr2=0x7f7bc0105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.494+0000 7f7bc5600700 1 -- 172.21.15.102:0/2118579228 shutdown_connections 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.494+0000 7f7bc5600700 1 -- 172.21.15.102:0/2118579228 wait complete. 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.495+0000 7f7bc5600700 1 Processor -- start 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.496+0000 7f7bc5600700 1 -- start start 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.496+0000 7f7bc5600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc019e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:46.322 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.496+0000 7f7bc5600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc010e3d0 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.496+0000 7f7bbea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc019e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.496+0000 7f7bbea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc019e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52628/0 (socket says 172.21.15.102:52628) 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.496+0000 7f7bbea00700 1 -- 172.21.15.102:0/1363679672 learned_addr learned my addr 172.21.15.102:0/1363679672 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.497+0000 7f7bbea00700 1 -- 172.21.15.102:0/1363679672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc019e560 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.497+0000 7f7bbea00700 1 --2- 172.21.15.102:0/1363679672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc019e020 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0011670 tx=0x7f7bb0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.497+0000 7f7bb7400700 1 -- 172.21.15.102:0/1363679672 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bb00039c0 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.498+0000 7f7bb7400700 1 -- 172.21.15.102:0/1363679672 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7bb0011e50 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.498+0000 7f7bb7400700 1 -- 172.21.15.102:0/1363679672 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7bb001a5c0 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.498+0000 7f7bc5600700 1 -- 172.21.15.102:0/1363679672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc019e7f0 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.498+0000 7f7bc5600700 1 -- 172.21.15.102:0/1363679672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bc019f310 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.499+0000 7f7bb7400700 1 -- 172.21.15.102:0/1363679672 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7bb001ac00 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.499+0000 7f7bb7400700 1 -- 172.21.15.102:0/1363679672 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7bb0019700 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.499+0000 7f7bb4c00700 1 -- 172.21.15.102:0/1363679672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bc0063890 con 0x7f7bc007dd80 2024-04-25T01:43:46.323 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.502+0000 7f7bb7400700 1 -- 172.21.15.102:0/1363679672 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7bb0003b60 con 0x7f7bc007dd80 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.581+0000 7f7bb4c00700 1 -- 172.21.15.102:0/1363679672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7bc006af00 con 0x7f7bc007dd80 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.582+0000 7f7bb7400700 1 -- 172.21.15.102:0/1363679672 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f7bb0024430 con 0x7f7bc007dd80 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.583+0000 7f7bb4c00700 1 -- 172.21.15.102:0/1363679672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 msgr2=0x7f7bc019e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.583+0000 7f7bb4c00700 1 --2- 172.21.15.102:0/1363679672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc019e020 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0011670 tx=0x7f7bb0011b60 comp rx=0 tx=0).stop 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.584+0000 7f7bb4c00700 1 -- 172.21.15.102:0/1363679672 shutdown_connections 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.584+0000 7f7bb4c00700 1 --2- 172.21.15.102:0/1363679672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc007dd80 0x7f7bc019e020 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.584+0000 7f7bb4c00700 1 -- 172.21.15.102:0/1363679672 >> 172.21.15.102:0/1363679672 conn(0x7f7bc0103420 msgr2=0x7f7bc0103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.584+0000 7f7bb4c00700 1 -- 172.21.15.102:0/1363679672 shutdown_connections 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:45.584+0000 7f7bb4c00700 1 -- 172.21.15.102:0/1363679672 wait complete. 2024-04-25T01:43:46.324 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (1/15)... 2024-04-25T01:43:47.663 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:47 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:47.375+0000 7f4b7ef1e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:43:47.663 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:47 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:47.600+0000 7f4b7ef1e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:43:48.163 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:47 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:47.810+0000 7f4b7ef1e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:43:48.477 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:48.179+0000 7f4b7ef1e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:43:48.477 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:48.299+0000 7f4b7ef1e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:43:48.805 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:48.562+0000 7f4b7ef1e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:43:49.062 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:48.817+0000 7f4b7ef1e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:43:49.413 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:49 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2034138821' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:43:49.803 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:49.622+0000 7f4b7ef1e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:43:49.817 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:49.820 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:49.821 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:43:49.822 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:43:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:43:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:43:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:43:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:43:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:43:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:43:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:43:49.824 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:43:49.825 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:43:49.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:49.831 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:43:49.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:49.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:43:49.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:43:49.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:49.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:43:49.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:43:49.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:43:49.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:43:49.832 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:43:34.661710+0000", 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:43:49.833 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.964+0000 7fd28f600700 1 Processor -- start 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.965+0000 7fd28f600700 1 -- start start 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.965+0000 7fd28f600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28807c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.965+0000 7fd28f600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd28807c7c0 con 0x7fd28807dd80 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.965+0000 7fd28cc00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28807c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.965+0000 7fd28cc00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28807c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52634/0 (socket says 172.21.15.102:52634) 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.965+0000 7fd28cc00700 1 -- 172.21.15.102:0/1093096223 learned_addr learned my addr 172.21.15.102:0/1093096223 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.966+0000 7fd28cc00700 1 -- 172.21.15.102:0/1093096223 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28807c940 con 0x7fd28807dd80 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.966+0000 7fd28cc00700 1 --2- 172.21.15.102:0/1093096223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28807c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0098a0 tx=0x7fd27c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b6a3cdc69e7ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.966+0000 7fd287400700 1 -- 172.21.15.102:0/1093096223 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd27c004160 con 0x7fd28807dd80 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.966+0000 7fd287400700 1 -- 172.21.15.102:0/1093096223 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd27c004300 con 0x7fd28807dd80 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.966+0000 7fd287400700 1 -- 172.21.15.102:0/1093096223 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd27c011b50 con 0x7fd28807dd80 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.967+0000 7fd28f600700 1 -- 172.21.15.102:0/1093096223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 msgr2=0x7fd28807c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:49.834 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.967+0000 7fd28f600700 1 --2- 172.21.15.102:0/1093096223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28807c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0098a0 tx=0x7fd27c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.967+0000 7fd28f600700 1 -- 172.21.15.102:0/1093096223 shutdown_connections 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.967+0000 7fd28f600700 1 --2- 172.21.15.102:0/1093096223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28807c280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.967+0000 7fd28f600700 1 -- 172.21.15.102:0/1093096223 >> 172.21.15.102:0/1093096223 conn(0x7fd288103420 msgr2=0x7fd288105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.968+0000 7fd28f600700 1 -- 172.21.15.102:0/1093096223 shutdown_connections 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.968+0000 7fd28f600700 1 -- 172.21.15.102:0/1093096223 wait complete. 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.968+0000 7fd28f600700 1 Processor -- start 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.969+0000 7fd28f600700 1 -- start start 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.969+0000 7fd28f600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28819e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.969+0000 7fd28f600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd28810e3d0 con 0x7fd28807dd80 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.969+0000 7fd28cc00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28819e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.969+0000 7fd28cc00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28819e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52646/0 (socket says 172.21.15.102:52646) 2024-04-25T01:43:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.969+0000 7fd28cc00700 1 -- 172.21.15.102:0/2034138821 learned_addr learned my addr 172.21.15.102:0/2034138821 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.970+0000 7fd28cc00700 1 -- 172.21.15.102:0/2034138821 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28819e560 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.970+0000 7fd28cc00700 1 --2- 172.21.15.102:0/2034138821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28819e020 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd27c011b60 tx=0x7fd27c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.970+0000 7fd285600700 1 -- 172.21.15.102:0/2034138821 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd27c003b20 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.970+0000 7fd285600700 1 -- 172.21.15.102:0/2034138821 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd27c003cc0 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.970+0000 7fd285600700 1 -- 172.21.15.102:0/2034138821 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd27c024840 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.970+0000 7fd28f600700 1 -- 172.21.15.102:0/2034138821 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28819e7f0 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.970+0000 7fd28f600700 1 -- 172.21.15.102:0/2034138821 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28819f310 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.971+0000 7fd285600700 1 -- 172.21.15.102:0/2034138821 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd27c022020 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.971+0000 7fd285600700 1 -- 172.21.15.102:0/2034138821 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd27c011e00 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.971+0000 7fd28f600700 1 -- 172.21.15.102:0/2034138821 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd288063890 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:48.974+0000 7fd285600700 1 -- 172.21.15.102:0/2034138821 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd27c043dd0 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:49.053+0000 7fd28f600700 1 -- 172.21.15.102:0/2034138821 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd28806af00 con 0x7fd28807dd80 2024-04-25T01:43:49.836 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:49.054+0000 7fd285600700 1 -- 172.21.15.102:0/2034138821 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd28806af00 con 0x7fd28807dd80 2024-04-25T01:43:49.837 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:49.055+0000 7fd28f600700 1 -- 172.21.15.102:0/2034138821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 msgr2=0x7fd28819e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:49.837 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:49.055+0000 7fd28f600700 1 --2- 172.21.15.102:0/2034138821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28819e020 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd27c011b60 tx=0x7fd27c011c40 comp rx=0 tx=0).stop 2024-04-25T01:43:49.837 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:49.056+0000 7fd28f600700 1 -- 172.21.15.102:0/2034138821 shutdown_connections 2024-04-25T01:43:49.837 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:49.056+0000 7fd28f600700 1 --2- 172.21.15.102:0/2034138821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28807dd80 0x7fd28819e020 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:49.837 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:49.056+0000 7fd28f600700 1 -- 172.21.15.102:0/2034138821 >> 172.21.15.102:0/2034138821 conn(0x7fd288103420 msgr2=0x7fd288103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:49.837 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:49.056+0000 7fd28f600700 1 -- 172.21.15.102:0/2034138821 shutdown_connections 2024-04-25T01:43:49.837 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:49.056+0000 7fd28f600700 1 -- 172.21.15.102:0/2034138821 wait complete. 2024-04-25T01:43:49.837 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (2/15)... 2024-04-25T01:43:50.413 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:50 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:50.165+0000 7f4b7ef1e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:43:50.413 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:50 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:50.289+0000 7f4b7ef1e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:43:50.913 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:50 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:50.545+0000 7f4b7ef1e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:43:51.598 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:51.176+0000 7f4b7ef1e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:43:51.598 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:51.295+0000 7f4b7ef1e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:43:51.598 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:51.606+0000 7f4b7ef1e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:43:51.907 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:51.782+0000 7f4b7ef1e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:43:51.907 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:51.916+0000 7f4b7ef1e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:43:52.163 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:52.077+0000 7f4b7ef1e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:43:52.438 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:52.208+0000 7f4b7ef1e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:43:52.913 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:52 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3397288056' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:43:53.264 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:53.264 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:43:53.264 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:43:53.264 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:43:53.264 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:43:53.264 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:43:53.264 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:43:53.265 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:53.266 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:43:53.266 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:53.266 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:43:53.266 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:43:53.266 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:43:53.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:43:53.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:43:53.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:43:53.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:53.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:43:53.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:43:53.274 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T01:43:53.275 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:43:34.661710+0000", 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:43:53.276 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.447+0000 7f2771800700 1 Processor -- start 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.448+0000 7f2771800700 1 -- start start 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.448+0000 7f2771800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c07c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.448+0000 7f2771800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f276c07c7e0 con 0x7f276c07dda0 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.448+0000 7f276aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c07c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.449+0000 7f276aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c07c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52652/0 (socket says 172.21.15.102:52652) 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.449+0000 7f276aa00700 1 -- 172.21.15.102:0/1647101240 learned_addr learned my addr 172.21.15.102:0/1647101240 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.449+0000 7f276aa00700 1 -- 172.21.15.102:0/1647101240 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f276c07c960 con 0x7f276c07dda0 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.449+0000 7f276aa00700 1 --2- 172.21.15.102:0/1647101240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c07c2a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f275c0098a0 tx=0x7f275c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce2f4ffdcfd34a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.449+0000 7f2769600700 1 -- 172.21.15.102:0/1647101240 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f275c004160 con 0x7f276c07dda0 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.450+0000 7f2769600700 1 -- 172.21.15.102:0/1647101240 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f275c004300 con 0x7f276c07dda0 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.450+0000 7f2769600700 1 -- 172.21.15.102:0/1647101240 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f275c011b50 con 0x7f276c07dda0 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.450+0000 7f2771800700 1 -- 172.21.15.102:0/1647101240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 msgr2=0x7f276c07c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.450+0000 7f2771800700 1 --2- 172.21.15.102:0/1647101240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c07c2a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f275c0098a0 tx=0x7f275c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.277 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.450+0000 7f2771800700 1 -- 172.21.15.102:0/1647101240 shutdown_connections 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.450+0000 7f2771800700 1 --2- 172.21.15.102:0/1647101240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c07c2a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.450+0000 7f2771800700 1 -- 172.21.15.102:0/1647101240 >> 172.21.15.102:0/1647101240 conn(0x7f276c103420 msgr2=0x7f276c105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.451+0000 7f2771800700 1 -- 172.21.15.102:0/1647101240 shutdown_connections 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.451+0000 7f2771800700 1 -- 172.21.15.102:0/1647101240 wait complete. 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.452+0000 7f2771800700 1 Processor -- start 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.452+0000 7f2771800700 1 -- start start 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.452+0000 7f2771800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c19e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.452+0000 7f2771800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f276c10e410 con 0x7f276c07dda0 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.452+0000 7f276aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c19e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.452+0000 7f276aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c19e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52666/0 (socket says 172.21.15.102:52666) 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.452+0000 7f276aa00700 1 -- 172.21.15.102:0/3397288056 learned_addr learned my addr 172.21.15.102:0/3397288056 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.453+0000 7f276aa00700 1 -- 172.21.15.102:0/3397288056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f276c19e580 con 0x7f276c07dda0 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.453+0000 7f276aa00700 1 --2- 172.21.15.102:0/3397288056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c19e040 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f275c011b60 tx=0x7f275c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:53.278 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.453+0000 7f2763400700 1 -- 172.21.15.102:0/3397288056 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f275c003b20 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.453+0000 7f2771800700 1 -- 172.21.15.102:0/3397288056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276c19e810 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.453+0000 7f2763400700 1 -- 172.21.15.102:0/3397288056 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f275c003cc0 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.453+0000 7f2763400700 1 -- 172.21.15.102:0/3397288056 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f275c024840 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.453+0000 7f2771800700 1 -- 172.21.15.102:0/3397288056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f276c19f2e0 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.454+0000 7f2763400700 1 -- 172.21.15.102:0/3397288056 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f275c022020 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.454+0000 7f2763400700 1 -- 172.21.15.102:0/3397288056 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f275c011e00 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.455+0000 7f2771800700 1 -- 172.21.15.102:0/3397288056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f276c063890 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.457+0000 7f2763400700 1 -- 172.21.15.102:0/3397288056 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f275c043dd0 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.547+0000 7f2771800700 1 -- 172.21.15.102:0/3397288056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f276c19f720 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.548+0000 7f2763400700 1 -- 172.21.15.102:0/3397288056 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f275c032030 con 0x7f276c07dda0 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.549+0000 7f2771800700 1 -- 172.21.15.102:0/3397288056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 msgr2=0x7f276c19e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.549+0000 7f2771800700 1 --2- 172.21.15.102:0/3397288056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c19e040 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f275c011b60 tx=0x7f275c011c40 comp rx=0 tx=0).stop 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.549+0000 7f2771800700 1 -- 172.21.15.102:0/3397288056 shutdown_connections 2024-04-25T01:43:53.279 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.549+0000 7f2771800700 1 --2- 172.21.15.102:0/3397288056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f276c07dda0 0x7f276c19e040 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:53.280 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.549+0000 7f2771800700 1 -- 172.21.15.102:0/3397288056 >> 172.21.15.102:0/3397288056 conn(0x7f276c103420 msgr2=0x7f276c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:53.280 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.550+0000 7f2771800700 1 -- 172.21.15.102:0/3397288056 shutdown_connections 2024-04-25T01:43:53.280 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:52.550+0000 7f2771800700 1 -- 172.21.15.102:0/3397288056 wait complete. 2024-04-25T01:43:53.280 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (3/15)... 2024-04-25T01:43:53.501 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:53.227+0000 7f4b7ef1e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:43:53.913 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:53.510+0000 7f4b7ef1e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: Activating manager daemon x 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: mgrmap e2: x(active, starting, since 0.0120869s) 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: Manager daemon x is now available 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:43:53.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:43:53.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' 2024-04-25T01:43:53.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' 2024-04-25T01:43:53.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:53 smithi102 ceph-mon[27796]: from='mgr.14100 172.21.15.102:0/3964142199' entity='mgr.x' 2024-04-25T01:43:55.701 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:55 smithi102 ceph-mon[27796]: mgrmap e3: x(active, since 1.02071s) 2024-04-25T01:43:56.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:56 smithi102 ceph-mon[27796]: mgrmap e4: x(active, since 2s) 2024-04-25T01:43:56.789 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:56 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1702082123' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T01:43:57.096 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T01:43:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T01:43:57.106 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T01:43:57.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T01:43:34.661710+0000", 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.872+0000 7f525be00700 1 Processor -- start 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.872+0000 7f525be00700 1 -- start start 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.872+0000 7f525be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c10a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.872+0000 7f525be00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f525c10a780 con 0x7f525c109e20 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.873+0000 7f525aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c10a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.873+0000 7f525aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c10a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46564/0 (socket says 172.21.15.102:46564) 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.873+0000 7f525aa00700 1 -- 172.21.15.102:0/3191875816 learned_addr learned my addr 172.21.15.102:0/3191875816 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.873+0000 7f525aa00700 1 -- 172.21.15.102:0/3191875816 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c10af80 con 0x7f525c109e20 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.873+0000 7f525aa00700 1 --2- 172.21.15.102:0/3191875816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c10a240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f524c0098a0 tx=0x7f524c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b862f429f0c8bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.874+0000 7f525a000700 1 -- 172.21.15.102:0/3191875816 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f524c004160 con 0x7f525c109e20 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.874+0000 7f525a000700 1 -- 172.21.15.102:0/3191875816 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f524c004300 con 0x7f525c109e20 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.874+0000 7f525a000700 1 -- 172.21.15.102:0/3191875816 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f524c011b50 con 0x7f525c109e20 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.874+0000 7f525be00700 1 -- 172.21.15.102:0/3191875816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 msgr2=0x7f525c10a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.874+0000 7f525be00700 1 --2- 172.21.15.102:0/3191875816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c10a240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f524c0098a0 tx=0x7f524c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.874+0000 7f525be00700 1 -- 172.21.15.102:0/3191875816 shutdown_connections 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.874+0000 7f525be00700 1 --2- 172.21.15.102:0/3191875816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c10a240 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.874+0000 7f525be00700 1 -- 172.21.15.102:0/3191875816 >> 172.21.15.102:0/3191875816 conn(0x7f525c07d230 msgr2=0x7f525c07d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.875+0000 7f525be00700 1 -- 172.21.15.102:0/3191875816 shutdown_connections 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.875+0000 7f525be00700 1 -- 172.21.15.102:0/3191875816 wait complete. 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.876+0000 7f525be00700 1 Processor -- start 2024-04-25T01:43:57.109 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.876+0000 7f525be00700 1 -- start start 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.876+0000 7f525be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c19dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.876+0000 7f525be00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f525c10e3c0 con 0x7f525c109e20 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.876+0000 7f525aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c19dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.876+0000 7f525aa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c19dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46574/0 (socket says 172.21.15.102:46574) 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.876+0000 7f525aa00700 1 -- 172.21.15.102:0/1702082123 learned_addr learned my addr 172.21.15.102:0/1702082123 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.877+0000 7f525aa00700 1 -- 172.21.15.102:0/1702082123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c19e320 con 0x7f525c109e20 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.877+0000 7f525aa00700 1 --2- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c19dde0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f524c011b60 tx=0x7f524c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.877+0000 7f5253400700 1 -- 172.21.15.102:0/1702082123 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f524c003970 con 0x7f525c109e20 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.877+0000 7f5253400700 1 -- 172.21.15.102:0/1702082123 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f524c003b10 con 0x7f525c109e20 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.877+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525c19e5b0 con 0x7f525c109e20 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.877+0000 7f5253400700 1 -- 172.21.15.102:0/1702082123 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f524c024610 con 0x7f525c109e20 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.877+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525c1a12a0 con 0x7f525c109e20 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.878+0000 7f5253400700 1 -- 172.21.15.102:0/1702082123 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f524c0248c0 con 0x7f525c109e20 2024-04-25T01:43:57.110 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.879+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f525c069390 con 0x7f525c109e20 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.879+0000 7f5253400700 1 --2- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5240039050 0x7f524003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.879+0000 7f5253400700 1 -- 172.21.15.102:0/1702082123 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f524c011e00 con 0x7f525c109e20 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.879+0000 7f5253e00700 1 --2- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5240039050 0x7f524003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.880+0000 7f5253e00700 1 --2- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5240039050 0x7f524003b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5244006fd0 tx=0x7f5244006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:55.886+0000 7f5253400700 1 -- 172.21.15.102:0/1702082123 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f524c0172b0 con 0x7f525c109e20 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.193+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f525c1a1470 con 0x7f525c109e20 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.194+0000 7f5253400700 1 -- 172.21.15.102:0/1702082123 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f525c1a1470 con 0x7f525c109e20 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5240039050 msgr2=0x7f524003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 --2- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5240039050 0x7f524003b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5244006fd0 tx=0x7f5244006e40 comp rx=0 tx=0).stop 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 msgr2=0x7f525c19dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 --2- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c19dde0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f524c011b60 tx=0x7f524c011c40 comp rx=0 tx=0).stop 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 shutdown_connections 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 --2- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5240039050 0x7f524003b510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 --2- 172.21.15.102:0/1702082123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f525c109e20 0x7f525c19dde0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:57.111 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 >> 172.21.15.102:0/1702082123 conn(0x7f525c07d230 msgr2=0x7f525c107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:57.112 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 shutdown_connections 2024-04-25T01:43:57.112 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:56.197+0000 7f525be00700 1 -- 172.21.15.102:0/1702082123 wait complete. 2024-04-25T01:43:57.112 INFO:teuthology.orchestra.run.smithi102.stdout:mgr is available 2024-04-25T01:43:58.163 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:57 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1975928574' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T01:43:58.493 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:58.493 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T01:43:58.493 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:43:58.493 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T01:43:58.493 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T01:43:58.493 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.676+0000 7fcaaae00700 1 Processor -- start 2024-04-25T01:43:58.494 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.676+0000 7fcaaae00700 1 -- start start 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.677+0000 7fcaaae00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa410a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.677+0000 7fcaaae00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa410a9e0 con 0x7fcaa4109ff0 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.677+0000 7fcaa3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa410a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.677+0000 7fcaa3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa410a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46582/0 (socket says 172.21.15.102:46582) 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.677+0000 7fcaa3e00700 1 -- 172.21.15.102:0/2786950579 learned_addr learned my addr 172.21.15.102:0/2786950579 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.678+0000 7fcaa3e00700 1 -- 172.21.15.102:0/2786950579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa410b1f0 con 0x7fcaa4109ff0 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.678+0000 7fcaa3e00700 1 --2- 172.21.15.102:0/2786950579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa410a410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca940098a0 tx=0x7fca94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c808b19e457d896 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.678+0000 7fcaa2a00700 1 -- 172.21.15.102:0/2786950579 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca94004160 con 0x7fcaa4109ff0 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.678+0000 7fcaa2a00700 1 -- 172.21.15.102:0/2786950579 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fca94004300 con 0x7fcaa4109ff0 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.678+0000 7fcaa2a00700 1 -- 172.21.15.102:0/2786950579 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca94011b50 con 0x7fcaa4109ff0 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.679+0000 7fcaaae00700 1 -- 172.21.15.102:0/2786950579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 msgr2=0x7fcaa410a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.679+0000 7fcaaae00700 1 --2- 172.21.15.102:0/2786950579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa410a410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fca940098a0 tx=0x7fca94009bb0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.679+0000 7fcaaae00700 1 -- 172.21.15.102:0/2786950579 shutdown_connections 2024-04-25T01:43:58.495 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.679+0000 7fcaaae00700 1 --2- 172.21.15.102:0/2786950579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa410a410 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.679+0000 7fcaaae00700 1 -- 172.21.15.102:0/2786950579 >> 172.21.15.102:0/2786950579 conn(0x7fcaa407d3c0 msgr2=0x7fcaa407d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.679+0000 7fcaaae00700 1 -- 172.21.15.102:0/2786950579 shutdown_connections 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.679+0000 7fcaaae00700 1 -- 172.21.15.102:0/2786950579 wait complete. 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.680+0000 7fcaaae00700 1 Processor -- start 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.680+0000 7fcaaae00700 1 -- start start 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.681+0000 7fcaaae00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa419e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.681+0000 7fcaaae00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa410e5a0 con 0x7fcaa4109ff0 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.681+0000 7fcaa3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa419e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.681+0000 7fcaa3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa419e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46596/0 (socket says 172.21.15.102:46596) 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.681+0000 7fcaa3e00700 1 -- 172.21.15.102:0/1975928574 learned_addr learned my addr 172.21.15.102:0/1975928574 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.681+0000 7fcaa3e00700 1 -- 172.21.15.102:0/1975928574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa419e5e0 con 0x7fcaa4109ff0 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.682+0000 7fcaa3e00700 1 --2- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa419e0a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fca94000c00 tx=0x7fca94011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.682+0000 7fcaa0c00700 1 -- 172.21.15.102:0/1975928574 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca94003b20 con 0x7fcaa4109ff0 2024-04-25T01:43:58.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.682+0000 7fcaa0c00700 1 -- 172.21.15.102:0/1975928574 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fca94003cc0 con 0x7fcaa4109ff0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.682+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa419e870 con 0x7fcaa4109ff0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.682+0000 7fcaa0c00700 1 -- 172.21.15.102:0/1975928574 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca94024910 con 0x7fcaa4109ff0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.682+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa419f390 con 0x7fcaa4109ff0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.683+0000 7fcaa0c00700 1 -- 172.21.15.102:0/1975928574 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fca94022020 con 0x7fcaa4109ff0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.683+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa4063890 con 0x7fcaa4109ff0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.684+0000 7fcaa0c00700 1 --2- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fca88039050 0x7fca8803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.684+0000 7fcaa0c00700 1 -- 172.21.15.102:0/1975928574 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fca94011e00 con 0x7fcaa4109ff0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.684+0000 7fcaa3400700 1 --2- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fca88039050 0x7fca8803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.684+0000 7fcaa3400700 1 --2- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fca88039050 0x7fca8803b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca8c006fd0 tx=0x7fca8c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.689+0000 7fcaa0c00700 1 -- 172.21.15.102:0/1975928574 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fca9401ad80 con 0x7fcaa4109ff0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.905+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fcaa406af00 con 0x7fcaa4109ff0 2024-04-25T01:43:58.497 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.906+0000 7fcaa0c00700 1 -- 172.21.15.102:0/1975928574 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fca94017ca0 con 0x7fcaa4109ff0 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.909+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fca88039050 msgr2=0x7fca8803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.909+0000 7fcaaae00700 1 --2- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fca88039050 0x7fca8803b510 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fca8c006fd0 tx=0x7fca8c006e40 comp rx=0 tx=0).stop 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.909+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 msgr2=0x7fcaa419e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.909+0000 7fcaaae00700 1 --2- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa419e0a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fca94000c00 tx=0x7fca94011c10 comp rx=0 tx=0).stop 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.909+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 shutdown_connections 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.909+0000 7fcaaae00700 1 --2- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fca88039050 0x7fca8803b510 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.909+0000 7fcaaae00700 1 --2- 172.21.15.102:0/1975928574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcaa4109ff0 0x7fcaa419e0a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.909+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 >> 172.21.15.102:0/1975928574 conn(0x7fcaa407d3c0 msgr2=0x7fcaa4107420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.910+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 shutdown_connections 2024-04-25T01:43:58.498 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:57.910+0000 7fcaaae00700 1 -- 172.21.15.102:0/1975928574 wait complete. 2024-04-25T01:43:58.499 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling cephadm module... 2024-04-25T01:43:59.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:43:59 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3594586700' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T01:43:59.664 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: ignoring --setuser ceph since I am not root 2024-04-25T01:43:59.664 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: ignoring --setgroup ceph since I am not root 2024-04-25T01:44:00.108 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:59.740+0000 7f8023aa2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:44:00.108 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:43:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:43:59.984+0000 7f8023aa2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:44:00.108 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:00.115+0000 7f8023aa2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:44:00.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.111+0000 7fb557e00700 1 Processor -- start 2024-04-25T01:44:00.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.111+0000 7fb557e00700 1 -- start start 2024-04-25T01:44:00.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.111+0000 7fb557e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb558111060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:00.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.111+0000 7fb557e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb5580824a0 con 0x7fb55810ec70 2024-04-25T01:44:00.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.112+0000 7fb556a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb558111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:00.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.112+0000 7fb556a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb558111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46608/0 (socket says 172.21.15.102:46608) 2024-04-25T01:44:00.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.112+0000 7fb556a00700 1 -- 172.21.15.102:0/770443977 learned_addr learned my addr 172.21.15.102:0/770443977 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:00.219 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.112+0000 7fb556a00700 1 -- 172.21.15.102:0/770443977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5581115a0 con 0x7fb55810ec70 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.113+0000 7fb556a00700 1 --2- 172.21.15.102:0/770443977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb558111060 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb5480098a0 tx=0x7fb548009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799860c21f9144b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.113+0000 7fb555600700 1 -- 172.21.15.102:0/770443977 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb548004160 con 0x7fb55810ec70 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.113+0000 7fb555600700 1 -- 172.21.15.102:0/770443977 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb548004300 con 0x7fb55810ec70 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.113+0000 7fb555600700 1 -- 172.21.15.102:0/770443977 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb548011b50 con 0x7fb55810ec70 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.114+0000 7fb557e00700 1 -- 172.21.15.102:0/770443977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 msgr2=0x7fb558111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.114+0000 7fb557e00700 1 --2- 172.21.15.102:0/770443977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb558111060 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb5480098a0 tx=0x7fb548009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.114+0000 7fb557e00700 1 -- 172.21.15.102:0/770443977 shutdown_connections 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.114+0000 7fb557e00700 1 --2- 172.21.15.102:0/770443977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb558111060 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.114+0000 7fb557e00700 1 -- 172.21.15.102:0/770443977 >> 172.21.15.102:0/770443977 conn(0x7fb55807d240 msgr2=0x7fb55807f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.114+0000 7fb557e00700 1 -- 172.21.15.102:0/770443977 shutdown_connections 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.114+0000 7fb557e00700 1 -- 172.21.15.102:0/770443977 wait complete. 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.115+0000 7fb557e00700 1 Processor -- start 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.115+0000 7fb557e00700 1 -- start start 2024-04-25T01:44:00.220 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.116+0000 7fb557e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb5581a2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.116+0000 7fb557e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb558112620 con 0x7fb55810ec70 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.116+0000 7fb556a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb5581a2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.116+0000 7fb556a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb5581a2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46610/0 (socket says 172.21.15.102:46610) 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.116+0000 7fb556a00700 1 -- 172.21.15.102:0/3594586700 learned_addr learned my addr 172.21.15.102:0/3594586700 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.116+0000 7fb556a00700 1 -- 172.21.15.102:0/3594586700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5581a2740 con 0x7fb55810ec70 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.117+0000 7fb556a00700 1 --2- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb5581a2200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb548011b60 tx=0x7fb548011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.117+0000 7fb54f400700 1 -- 172.21.15.102:0/3594586700 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb548003970 con 0x7fb55810ec70 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.117+0000 7fb54f400700 1 -- 172.21.15.102:0/3594586700 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb548003b10 con 0x7fb55810ec70 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.117+0000 7fb54f400700 1 -- 172.21.15.102:0/3594586700 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb548024610 con 0x7fb55810ec70 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.117+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5581a29d0 con 0x7fb55810ec70 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.117+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5581a56c0 con 0x7fb55810ec70 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.118+0000 7fb54f400700 1 -- 172.21.15.102:0/3594586700 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fb54801a9f0 con 0x7fb55810ec70 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.118+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb558069390 con 0x7fb55810ec70 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.119+0000 7fb54f400700 1 --2- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fb53c039050 0x7fb53c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:00.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.119+0000 7fb54f400700 1 -- 172.21.15.102:0/3594586700 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb548011e00 con 0x7fb55810ec70 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.119+0000 7fb556000700 1 --2- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fb53c039050 0x7fb53c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.120+0000 7fb556000700 1 --2- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fb53c039050 0x7fb53c03b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb544006fd0 tx=0x7fb544006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.124+0000 7fb54f400700 1 -- 172.21.15.102:0/3594586700 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb548016260 con 0x7fb55810ec70 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.404+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fb5581a5890 con 0x7fb55810ec70 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.459+0000 7fb54f400700 1 -- 172.21.15.102:0/3594586700 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fb5581a5890 con 0x7fb55810ec70 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.461+0000 7fb54f400700 1 -- 172.21.15.102:0/3594586700 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fb548024c40 con 0x7fb55810ec70 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.463+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fb53c039050 msgr2=0x7fb53c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.463+0000 7fb557e00700 1 --2- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fb53c039050 0x7fb53c03b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb544006fd0 tx=0x7fb544006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.463+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 msgr2=0x7fb5581a2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.463+0000 7fb557e00700 1 --2- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb5581a2200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb548011b60 tx=0x7fb548011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.464+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 shutdown_connections 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.464+0000 7fb557e00700 1 --2- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7fb53c039050 0x7fb53c03b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.464+0000 7fb557e00700 1 --2- 172.21.15.102:0/3594586700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb55810ec70 0x7fb5581a2200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:00.222 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.464+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 >> 172.21.15.102:0/3594586700 conn(0x7fb55807d240 msgr2=0x7fb55807dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:00.223 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.464+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 shutdown_connections 2024-04-25T01:44:00.223 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:43:59.464+0000 7fb557e00700 1 -- 172.21.15.102:0/3594586700 wait complete. 2024-04-25T01:44:00.622 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:00.515+0000 7f8023aa2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:44:00.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:00 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3594586700' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T01:44:00.622 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:00 smithi102 ceph-mon[27796]: mgrmap e5: x(active, since 5s) 2024-04-25T01:44:00.913 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:00.629+0000 7f8023aa2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:44:01.791 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:01 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3417228249' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:44:01.805 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:01.805 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T01:44:01.805 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:44:01.806 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:44:01.806 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:44:01.806 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:01.806 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.832+0000 7f5bcae00700 1 Processor -- start 2024-04-25T01:44:01.806 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.832+0000 7f5bcae00700 1 -- start start 2024-04-25T01:44:01.807 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.833+0000 7f5bcae00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc407c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:01.807 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.833+0000 7f5bcae00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc407c7e0 con 0x7f5bc407dda0 2024-04-25T01:44:01.807 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.833+0000 7f5bc3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc407c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:01.807 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.833+0000 7f5bc3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc407c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46642/0 (socket says 172.21.15.102:46642) 2024-04-25T01:44:01.807 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.833+0000 7f5bc3e00700 1 -- 172.21.15.102:0/421135582 learned_addr learned my addr 172.21.15.102:0/421135582 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:01.807 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.834+0000 7f5bc3e00700 1 -- 172.21.15.102:0/421135582 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc407c960 con 0x7f5bc407dda0 2024-04-25T01:44:01.808 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.834+0000 7f5bc3e00700 1 --2- 172.21.15.102:0/421135582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc407c2a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40098a0 tx=0x7f5bb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39efa9a663183e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:01.808 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.834+0000 7f5bc2a00700 1 -- 172.21.15.102:0/421135582 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4004160 con 0x7f5bc407dda0 2024-04-25T01:44:01.808 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.834+0000 7f5bc2a00700 1 -- 172.21.15.102:0/421135582 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5bb4004300 con 0x7f5bc407dda0 2024-04-25T01:44:01.808 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.834+0000 7f5bc2a00700 1 -- 172.21.15.102:0/421135582 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4011b50 con 0x7f5bc407dda0 2024-04-25T01:44:01.808 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.835+0000 7f5bcae00700 1 -- 172.21.15.102:0/421135582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 msgr2=0x7f5bc407c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:01.808 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.835+0000 7f5bcae00700 1 --2- 172.21.15.102:0/421135582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc407c2a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40098a0 tx=0x7f5bb4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:01.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.835+0000 7f5bcae00700 1 -- 172.21.15.102:0/421135582 shutdown_connections 2024-04-25T01:44:01.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.835+0000 7f5bcae00700 1 --2- 172.21.15.102:0/421135582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc407c2a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:01.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.835+0000 7f5bcae00700 1 -- 172.21.15.102:0/421135582 >> 172.21.15.102:0/421135582 conn(0x7f5bc4103420 msgr2=0x7f5bc4105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:01.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.835+0000 7f5bcae00700 1 -- 172.21.15.102:0/421135582 shutdown_connections 2024-04-25T01:44:01.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.835+0000 7f5bcae00700 1 -- 172.21.15.102:0/421135582 wait complete. 2024-04-25T01:44:01.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.836+0000 7f5bcae00700 1 Processor -- start 2024-04-25T01:44:01.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.837+0000 7f5bcae00700 1 -- start start 2024-04-25T01:44:01.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.837+0000 7f5bcae00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:01.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.837+0000 7f5bcae00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc410e410 con 0x7f5bc407dda0 2024-04-25T01:44:01.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.837+0000 7f5bc3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:01.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.837+0000 7f5bc3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46658/0 (socket says 172.21.15.102:46658) 2024-04-25T01:44:01.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.837+0000 7f5bc3e00700 1 -- 172.21.15.102:0/3417228249 learned_addr learned my addr 172.21.15.102:0/3417228249 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:01.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.838+0000 7f5bc3e00700 1 -- 172.21.15.102:0/3417228249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc419e560 con 0x7f5bc407dda0 2024-04-25T01:44:01.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.838+0000 7f5bc3e00700 1 --2- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc419e020 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4011b60 tx=0x7f5bb4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:01.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.838+0000 7f5bc0c00700 1 -- 172.21.15.102:0/3417228249 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4003b20 con 0x7f5bc407dda0 2024-04-25T01:44:01.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.838+0000 7f5bc0c00700 1 -- 172.21.15.102:0/3417228249 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5bb4003cc0 con 0x7f5bc407dda0 2024-04-25T01:44:01.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.838+0000 7f5bc0c00700 1 -- 172.21.15.102:0/3417228249 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bb4024840 con 0x7f5bc407dda0 2024-04-25T01:44:01.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.838+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc419e7f0 con 0x7f5bc407dda0 2024-04-25T01:44:01.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.838+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc419f310 con 0x7f5bc407dda0 2024-04-25T01:44:01.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.839+0000 7f5bc0c00700 1 -- 172.21.15.102:0/3417228249 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f5bb4022020 con 0x7f5bc407dda0 2024-04-25T01:44:01.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.840+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc4063890 con 0x7f5bc407dda0 2024-04-25T01:44:01.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.840+0000 7f5bc0c00700 1 --2- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5ba80390a0 0x7f5ba803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:01.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.840+0000 7f5bc0c00700 1 -- 172.21.15.102:0/3417228249 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5bb4011e00 con 0x7f5bc407dda0 2024-04-25T01:44:01.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.840+0000 7f5bc3400700 1 -- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5ba80390a0 msgr2=0x7f5ba803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2757023783 2024-04-25T01:44:01.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.840+0000 7f5bc3400700 1 --2- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5ba80390a0 0x7f5ba803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:44:01.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:00.846+0000 7f5bc0c00700 1 -- 172.21.15.102:0/3417228249 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5bb400ba30 con 0x7f5bc407dda0 2024-04-25T01:44:01.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.041+0000 7f5bc3400700 1 -- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5ba80390a0 msgr2=0x7f5ba803b560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2757023783 2024-04-25T01:44:01.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.041+0000 7f5bc3400700 1 --2- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5ba80390a0 0x7f5ba803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:44:01.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.142+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f5bc419f830 con 0x7f5bc407dda0 2024-04-25T01:44:01.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.142+0000 7f5bc0c00700 1 -- 172.21.15.102:0/3417228249 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f5bb400fdd0 con 0x7f5bc407dda0 2024-04-25T01:44:01.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5ba80390a0 msgr2=0x7f5ba803b560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:44:01.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 --2- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5ba80390a0 0x7f5ba803b560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:01.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 msgr2=0x7f5bc419e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:01.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 --2- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc419e020 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4011b60 tx=0x7f5bb4011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:01.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 shutdown_connections 2024-04-25T01:44:01.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 --2- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f5ba80390a0 0x7f5ba803b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:01.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 --2- 172.21.15.102:0/3417228249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bc407dda0 0x7f5bc419e020 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:01.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 >> 172.21.15.102:0/3417228249 conn(0x7f5bc4103420 msgr2=0x7f5bc4103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:01.814 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 shutdown_connections 2024-04-25T01:44:01.815 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:01.145+0000 7f5bcae00700 1 -- 172.21.15.102:0/3417228249 wait complete. 2024-04-25T01:44:01.815 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-04-25T01:44:01.815 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 5... 2024-04-25T01:44:02.362 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:02 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:02.228+0000 7f8023aa2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:44:02.663 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:02 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:02.458+0000 7f8023aa2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:44:03.041 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:02 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:02.673+0000 7f8023aa2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:44:03.042 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:03.049+0000 7f8023aa2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:44:03.407 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:03.168+0000 7f8023aa2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:44:03.664 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:03.414+0000 7f8023aa2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:44:03.664 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:03.661+0000 7f8023aa2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:44:04.913 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:04 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:04.425+0000 7f8023aa2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:44:05.338 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:04 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:04.963+0000 7f8023aa2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:44:05.338 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:05 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:05.088+0000 7f8023aa2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:44:05.338 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:05 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:05.344+0000 7f8023aa2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:44:06.397 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:05 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:05.976+0000 7f8023aa2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:44:06.397 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:06.094+0000 7f8023aa2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:44:06.397 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:06.404+0000 7f8023aa2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:44:06.663 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:06.579+0000 7f8023aa2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:44:06.961 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:06.701+0000 7f8023aa2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:44:06.961 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:06.849+0000 7f8023aa2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:44:06.962 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:06.968+0000 7f8023aa2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:44:08.222 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:07 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:07.953+0000 7f8023aa2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:44:08.664 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:08.229+0000 7f8023aa2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:44:08.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: Active manager daemon x restarted 2024-04-25T01:44:08.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: Activating manager daemon x 2024-04-25T01:44:08.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T01:44:08.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: mgrmap e6: x(active, starting, since 0.00909786s) 2024-04-25T01:44:08.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:44:08.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:44:08.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:44:08.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:44:08.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:44:08.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: Manager daemon x is now available 2024-04-25T01:44:08.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:08.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:08 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:09.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:09 smithi102 ceph-mon[27796]: Found migration_current of "None". Setting to last migration. 2024-04-25T01:44:09.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:09 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:09.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:09 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:09.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:09 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:44:09.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:09 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:44:09.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:09 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:09.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:09 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:09.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:09 smithi102 ceph-mon[27796]: mgrmap e7: x(active, since 1.01731s) 2024-04-25T01:44:10.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:10.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T01:44:10.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:44:10.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:10.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.517+0000 7f0565800700 1 Processor -- start 2024-04-25T01:44:10.038 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.517+0000 7f0565800700 1 -- start start 2024-04-25T01:44:10.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.518+0000 7f0565800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056007c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:10.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.518+0000 7f0565800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f056007c870 con 0x7f056007de40 2024-04-25T01:44:10.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.518+0000 7f055ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056007c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:10.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.518+0000 7f055ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056007c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46666/0 (socket says 172.21.15.102:46666) 2024-04-25T01:44:10.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.518+0000 7f055ea00700 1 -- 172.21.15.102:0/3936314929 learned_addr learned my addr 172.21.15.102:0/3936314929 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:10.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.519+0000 7f055ea00700 1 -- 172.21.15.102:0/3936314929 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056007c9f0 con 0x7f056007de40 2024-04-25T01:44:10.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.519+0000 7f055ea00700 1 --2- 172.21.15.102:0/3936314929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056007c2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f05500098a0 tx=0x7f0550009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd83b58b4dfc401f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:10.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.519+0000 7f055d600700 1 -- 172.21.15.102:0/3936314929 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0550004160 con 0x7f056007de40 2024-04-25T01:44:10.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.519+0000 7f055d600700 1 -- 172.21.15.102:0/3936314929 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0550004300 con 0x7f056007de40 2024-04-25T01:44:10.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.519+0000 7f055d600700 1 -- 172.21.15.102:0/3936314929 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0550011b50 con 0x7f056007de40 2024-04-25T01:44:10.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.520+0000 7f0565800700 1 -- 172.21.15.102:0/3936314929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 msgr2=0x7f056007c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:10.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.520+0000 7f0565800700 1 --2- 172.21.15.102:0/3936314929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056007c2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f05500098a0 tx=0x7f0550009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:10.045 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.520+0000 7f0565800700 1 -- 172.21.15.102:0/3936314929 shutdown_connections 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.520+0000 7f0565800700 1 --2- 172.21.15.102:0/3936314929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056007c2a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.520+0000 7f0565800700 1 -- 172.21.15.102:0/3936314929 >> 172.21.15.102:0/3936314929 conn(0x7f05601033e0 msgr2=0x7f0560105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.521+0000 7f0565800700 1 -- 172.21.15.102:0/3936314929 shutdown_connections 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.521+0000 7f0565800700 1 -- 172.21.15.102:0/3936314929 wait complete. 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.522+0000 7f0565800700 1 Processor -- start 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.522+0000 7f0565800700 1 -- start start 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.523+0000 7f0565800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.523+0000 7f0565800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f056010e410 con 0x7f056007de40 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.523+0000 7f055ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.523+0000 7f055ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056019e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46680/0 (socket says 172.21.15.102:46680) 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.523+0000 7f055ea00700 1 -- 172.21.15.102:0/2503414149 learned_addr learned my addr 172.21.15.102:0/2503414149 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.523+0000 7f055ea00700 1 -- 172.21.15.102:0/2503414149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056019e570 con 0x7f056007de40 2024-04-25T01:44:10.046 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.524+0000 7f055ea00700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056019e030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0550011850 tx=0x7f0550011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.524+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0550003af0 con 0x7f056007de40 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.524+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0550020070 con 0x7f056007de40 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.524+0000 7f0565800700 1 -- 172.21.15.102:0/2503414149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056019e800 con 0x7f056007de40 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.524+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05500195a0 con 0x7f056007de40 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.524+0000 7f0565800700 1 -- 172.21.15.102:0/2503414149 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05601a14f0 con 0x7f056007de40 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.525+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f05500198a0 con 0x7f056007de40 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.526+0000 7f0557400700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 0x7f054403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.526+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 --> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f054403bc20 con 0x7f0544039050 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.526+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0550011e00 con 0x7f056007de40 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.526+0000 7f055e000700 1 -- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 msgr2=0x7f054403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2757023783 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.526+0000 7f055e000700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 0x7f054403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.727+0000 7f055e000700 1 -- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 msgr2=0x7f054403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2757023783 2024-04-25T01:44:10.047 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:02.727+0000 7f055e000700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 0x7f054403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:03.128+0000 7f055e000700 1 -- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 msgr2=0x7f054403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2757023783 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:03.128+0000 7f055e000700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 0x7f054403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:03.929+0000 7f055e000700 1 -- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 msgr2=0x7f054403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2757023783 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:03.929+0000 7f055e000700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 0x7f054403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:05.531+0000 7f055e000700 1 -- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 msgr2=0x7f054403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/2757023783 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:05.531+0000 7f055e000700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 0x7f054403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:08.237+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f055002bd00 con 0x7f056007de40 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:08.238+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 msgr2=0x7f054403b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:08.238+0000 7f0557400700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 0x7f054403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.241+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f055004ebc0 con 0x7f056007de40 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.242+0000 7f0557400700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f054403c980 0x7f054403ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.242+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f054403bc20 con 0x7f054403c980 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.244+0000 7f055e000700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f054403c980 0x7f054403ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:10.048 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.247+0000 7f055e000700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f054403c980 0x7f054403ed70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f054c003a10 tx=0x7f054c009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.248+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 <== mgr.14120 v2:172.21.15.102:6800/1124203809 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f054403bc20 con 0x7f054403c980 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.260+0000 7f0565800700 1 -- 172.21.15.102:0/2503414149 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f0560069390 con 0x7f054403c980 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.261+0000 7f0557400700 1 -- 172.21.15.102:0/2503414149 <== mgr.14120 v2:172.21.15.102:6800/1124203809 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f0560069390 con 0x7f054403c980 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.261+0000 7f0565800700 1 -- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f054403c980 msgr2=0x7f054403ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.261+0000 7f0565800700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f054403c980 0x7f054403ed70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f054c003a10 tx=0x7f054c009290 comp rx=0 tx=0).stop 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.261+0000 7f0565800700 1 -- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 msgr2=0x7f056019e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.261+0000 7f0565800700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056019e030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0550011850 tx=0x7f0550011c10 comp rx=0 tx=0).stop 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.262+0000 7f0565800700 1 -- 172.21.15.102:0/2503414149 shutdown_connections 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.262+0000 7f0565800700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f054403c980 0x7f054403ed70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:10.049 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.262+0000 7f0565800700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056007de40 0x7f056019e030 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:10.050 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.262+0000 7f0565800700 1 --2- 172.21.15.102:0/2503414149 >> [v2:172.21.15.102:6800/2757023783,v1:172.21.15.102:6801/2757023783] conn(0x7f0544039050 0x7f054403b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:10.050 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.262+0000 7f0565800700 1 -- 172.21.15.102:0/2503414149 >> 172.21.15.102:0/2503414149 conn(0x7f05601033e0 msgr2=0x7f0560103f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:10.050 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.262+0000 7f0565800700 1 -- 172.21.15.102:0/2503414149 shutdown_connections 2024-04-25T01:44:10.050 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:09.262+0000 7f0565800700 1 -- 172.21.15.102:0/2503414149 wait complete. 2024-04-25T01:44:10.050 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 5 is available 2024-04-25T01:44:10.050 INFO:teuthology.orchestra.run.smithi102.stdout:Setting orchestrator backend to cephadm... 2024-04-25T01:44:10.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:10 smithi102 ceph-mon[27796]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T01:44:10.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:10 smithi102 ceph-mon[27796]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T01:44:10.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:10 smithi102 ceph-mon[27796]: [25/Apr/2024:01:44:09] ENGINE Bus STARTING 2024-04-25T01:44:10.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:10 smithi102 ceph-mon[27796]: [25/Apr/2024:01:44:09] ENGINE Serving on http://172.21.15.102:8765 2024-04-25T01:44:10.634 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:10 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:11.579 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:11 smithi102 ceph-mon[27796]: [25/Apr/2024:01:44:09] ENGINE Serving on https://172.21.15.102:7150 2024-04-25T01:44:11.579 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:11 smithi102 ceph-mon[27796]: [25/Apr/2024:01:44:09] ENGINE Bus STARTED 2024-04-25T01:44:11.579 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:11 smithi102 ceph-mon[27796]: mgrmap e8: x(active, since 2s) 2024-04-25T01:44:11.579 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:11 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:11.580 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:11 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.666+0000 7f803ee00700 1 Processor -- start 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.666+0000 7f803ee00700 1 -- start start 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.666+0000 7f803ee00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f8038111060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.667+0000 7f803ee00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f80380824a0 con 0x7f803810ec70 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.667+0000 7f803da00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f8038111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.667+0000 7f803da00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f8038111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33368/0 (socket says 172.21.15.102:33368) 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.667+0000 7f803da00700 1 -- 172.21.15.102:0/480073885 learned_addr learned my addr 172.21.15.102:0/480073885 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.668+0000 7f803da00700 1 -- 172.21.15.102:0/480073885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80381115a0 con 0x7f803810ec70 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.668+0000 7f803da00700 1 --2- 172.21.15.102:0/480073885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f8038111060 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f802c009e80 tx=0x7f802c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5490868466db572 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:11.594 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.668+0000 7f8037e00700 1 -- 172.21.15.102:0/480073885 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f802c017070 con 0x7f803810ec70 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.668+0000 7f8037e00700 1 -- 172.21.15.102:0/480073885 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f802c00b8c0 con 0x7f803810ec70 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.668+0000 7f8037e00700 1 -- 172.21.15.102:0/480073885 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f802c0045c0 con 0x7f803810ec70 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.669+0000 7f803ee00700 1 -- 172.21.15.102:0/480073885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 msgr2=0x7f8038111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.669+0000 7f803ee00700 1 --2- 172.21.15.102:0/480073885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f8038111060 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f802c009e80 tx=0x7f802c00b330 comp rx=0 tx=0).stop 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.669+0000 7f803ee00700 1 -- 172.21.15.102:0/480073885 shutdown_connections 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.669+0000 7f803ee00700 1 --2- 172.21.15.102:0/480073885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f8038111060 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.669+0000 7f803ee00700 1 -- 172.21.15.102:0/480073885 >> 172.21.15.102:0/480073885 conn(0x7f803807d240 msgr2=0x7f803807f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.669+0000 7f803ee00700 1 -- 172.21.15.102:0/480073885 shutdown_connections 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.669+0000 7f803ee00700 1 -- 172.21.15.102:0/480073885 wait complete. 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.670+0000 7f803ee00700 1 Processor -- start 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.670+0000 7f803ee00700 1 -- start start 2024-04-25T01:44:11.595 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.671+0000 7f803ee00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f80381a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.671+0000 7f803ee00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8038112620 con 0x7f803810ec70 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.671+0000 7f803da00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f80381a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.671+0000 7f803da00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f80381a21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33374/0 (socket says 172.21.15.102:33374) 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.671+0000 7f803da00700 1 -- 172.21.15.102:0/2188746255 learned_addr learned my addr 172.21.15.102:0/2188746255 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.671+0000 7f803da00700 1 -- 172.21.15.102:0/2188746255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80381a2700 con 0x7f803810ec70 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.672+0000 7f803da00700 1 --2- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f80381a21c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f802c009530 tx=0x7f802c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.672+0000 7f8036000700 1 -- 172.21.15.102:0/2188746255 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f802c020020 con 0x7f803810ec70 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.672+0000 7f8036000700 1 -- 172.21.15.102:0/2188746255 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f802c003b50 con 0x7f803810ec70 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.672+0000 7f8036000700 1 -- 172.21.15.102:0/2188746255 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f802c017040 con 0x7f803810ec70 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.672+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80381a2990 con 0x7f803810ec70 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.672+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80381a5680 con 0x7f803810ec70 2024-04-25T01:44:11.596 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.673+0000 7f8036000700 1 -- 172.21.15.102:0/2188746255 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f802c023020 con 0x7f803810ec70 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.673+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8038069390 con 0x7f803810ec70 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.674+0000 7f8036000700 1 --2- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f80200390a0 0x7f802003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.674+0000 7f8036000700 1 -- 172.21.15.102:0/2188746255 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f802c00f050 con 0x7f803810ec70 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.674+0000 7f803d000700 1 --2- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f80200390a0 0x7f802003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.675+0000 7f803d000700 1 --2- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f80200390a0 0x7f802003b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8024006fd0 tx=0x7f8024006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.679+0000 7f8036000700 1 -- 172.21.15.102:0/2188746255 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f802c02a020 con 0x7f803810ec70 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.918+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f803807e3f0 con 0x7f80200390a0 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.930+0000 7f8036000700 1 -- 172.21.15.102:0/2188746255 <== mgr.14120 v2:172.21.15.102:6800/1124203809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f803807e3f0 con 0x7f80200390a0 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.933+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f80200390a0 msgr2=0x7f802003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.933+0000 7f803ee00700 1 --2- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f80200390a0 0x7f802003b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8024006fd0 tx=0x7f8024006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.933+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 msgr2=0x7f80381a21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.933+0000 7f803ee00700 1 --2- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f80381a21c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f802c009530 tx=0x7f802c00bd50 comp rx=0 tx=0).stop 2024-04-25T01:44:11.597 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.933+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 shutdown_connections 2024-04-25T01:44:11.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.933+0000 7f803ee00700 1 --2- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f80200390a0 0x7f802003b560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:11.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.933+0000 7f803ee00700 1 --2- 172.21.15.102:0/2188746255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f803810ec70 0x7f80381a21c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:11.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.933+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 >> 172.21.15.102:0/2188746255 conn(0x7f803807d240 msgr2=0x7f803807dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:11.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.934+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 shutdown_connections 2024-04-25T01:44:11.598 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:10.934+0000 7f803ee00700 1 -- 172.21.15.102:0/2188746255 wait complete. 2024-04-25T01:44:12.461 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:12 smithi102 ceph-mon[27796]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:13.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T01:44:13.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.209+0000 7f8d49800700 1 Processor -- start 2024-04-25T01:44:13.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.209+0000 7f8d49800700 1 -- start start 2024-04-25T01:44:13.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.209+0000 7f8d49800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d44111060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:13.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.209+0000 7f8d49800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441115a0 con 0x7f8d4410ec70 2024-04-25T01:44:13.151 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.210+0000 7f8d42a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d44111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:13.156 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.210+0000 7f8d42a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d44111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33384/0 (socket says 172.21.15.102:33384) 2024-04-25T01:44:13.156 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.210+0000 7f8d42a00700 1 -- 172.21.15.102:0/2102427446 learned_addr learned my addr 172.21.15.102:0/2102427446 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:13.156 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.210+0000 7f8d42a00700 1 -- 172.21.15.102:0/2102427446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d44111720 con 0x7f8d4410ec70 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.211+0000 7f8d42a00700 1 --2- 172.21.15.102:0/2102427446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d44111060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d340098a0 tx=0x7f8d34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d113038caa53530f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.211+0000 7f8d41600700 1 -- 172.21.15.102:0/2102427446 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d34004160 con 0x7f8d4410ec70 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.211+0000 7f8d41600700 1 -- 172.21.15.102:0/2102427446 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8d34004300 con 0x7f8d4410ec70 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.211+0000 7f8d41600700 1 -- 172.21.15.102:0/2102427446 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d34011b50 con 0x7f8d4410ec70 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.212+0000 7f8d49800700 1 -- 172.21.15.102:0/2102427446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 msgr2=0x7f8d44111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.212+0000 7f8d49800700 1 --2- 172.21.15.102:0/2102427446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d44111060 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d340098a0 tx=0x7f8d34009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.212+0000 7f8d49800700 1 -- 172.21.15.102:0/2102427446 shutdown_connections 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.212+0000 7f8d49800700 1 --2- 172.21.15.102:0/2102427446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d44111060 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.212+0000 7f8d49800700 1 -- 172.21.15.102:0/2102427446 >> 172.21.15.102:0/2102427446 conn(0x7f8d4407d3e0 msgr2=0x7f8d4407f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.212+0000 7f8d49800700 1 -- 172.21.15.102:0/2102427446 shutdown_connections 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.212+0000 7f8d49800700 1 -- 172.21.15.102:0/2102427446 wait complete. 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.213+0000 7f8d49800700 1 Processor -- start 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.213+0000 7f8d49800700 1 -- start start 2024-04-25T01:44:13.157 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.213+0000 7f8d49800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d441a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.213+0000 7f8d49800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8d44112800 con 0x7f8d4410ec70 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.214+0000 7f8d42a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d441a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.214+0000 7f8d42a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d441a2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33400/0 (socket says 172.21.15.102:33400) 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.214+0000 7f8d42a00700 1 -- 172.21.15.102:0/1431600372 learned_addr learned my addr 172.21.15.102:0/1431600372 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.214+0000 7f8d42a00700 1 -- 172.21.15.102:0/1431600372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d441a28c0 con 0x7f8d4410ec70 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.214+0000 7f8d42a00700 1 --2- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d441a2380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8d34011b60 tx=0x7f8d34011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.215+0000 7f8d3b400700 1 -- 172.21.15.102:0/1431600372 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d34003970 con 0x7f8d4410ec70 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.215+0000 7f8d3b400700 1 -- 172.21.15.102:0/1431600372 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8d34003b10 con 0x7f8d4410ec70 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.215+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d441a2b50 con 0x7f8d4410ec70 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.215+0000 7f8d3b400700 1 -- 172.21.15.102:0/1431600372 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8d34024610 con 0x7f8d4410ec70 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.215+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d441a5840 con 0x7f8d4410ec70 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.216+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d44069390 con 0x7f8d4410ec70 2024-04-25T01:44:13.158 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.216+0000 7f8d3b400700 1 -- 172.21.15.102:0/1431600372 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f8d3401a9f0 con 0x7f8d4410ec70 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.216+0000 7f8d3b400700 1 --2- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f8d28039050 0x7f8d2803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.216+0000 7f8d3b400700 1 -- 172.21.15.102:0/1431600372 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8d34011e00 con 0x7f8d4410ec70 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.217+0000 7f8d42000700 1 --2- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f8d28039050 0x7f8d2803b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.217+0000 7f8d42000700 1 --2- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f8d28039050 0x7f8d2803b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8d30006fd0 tx=0x7f8d30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.222+0000 7f8d3b400700 1 -- 172.21.15.102:0/1431600372 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d34019dd0 con 0x7f8d4410ec70 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.447+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f8d4407e630 con 0x7f8d28039050 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.449+0000 7f8d3b400700 1 -- 172.21.15.102:0/1431600372 <== mgr.14120 v2:172.21.15.102:6800/1124203809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f8d4407e630 con 0x7f8d28039050 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.451+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f8d28039050 msgr2=0x7f8d2803b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.451+0000 7f8d49800700 1 --2- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f8d28039050 0x7f8d2803b510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8d30006fd0 tx=0x7f8d30006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.451+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 msgr2=0x7f8d441a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.451+0000 7f8d49800700 1 --2- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d441a2380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8d34011b60 tx=0x7f8d34011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.452+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 shutdown_connections 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.452+0000 7f8d49800700 1 --2- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f8d28039050 0x7f8d2803b510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.452+0000 7f8d49800700 1 --2- 172.21.15.102:0/1431600372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d4410ec70 0x7f8d441a2380 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:13.159 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.452+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 >> 172.21.15.102:0/1431600372 conn(0x7f8d4407d3e0 msgr2=0x7f8d4407df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:13.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.452+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 shutdown_connections 2024-04-25T01:44:13.160 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:12.452+0000 7f8d49800700 1 -- 172.21.15.102:0/1431600372 wait complete. 2024-04-25T01:44:13.160 INFO:teuthology.orchestra.run.smithi102.stdout:Generating ssh key... 2024-04-25T01:44:13.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:13 smithi102 ceph-mon[27796]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:14.415 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: Generating public/private rsa key pair. 2024-04-25T01:44:14.415 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: Your identification has been saved in /tmp/tmpklp58c8k/key. 2024-04-25T01:44:14.415 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: Your public key has been saved in /tmp/tmpklp58c8k/key.pub. 2024-04-25T01:44:14.415 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: The key fingerprint is: 2024-04-25T01:44:14.415 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: SHA256:tgjMQb1PkiIhCXJBTwc/FGHoaF+loZd7Vv235I6+OYs ceph-145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:14.415 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: The key's randomart image is: 2024-04-25T01:44:14.415 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: +---[RSA 3072]----+ 2024-04-25T01:44:14.415 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: |+o+.++*o | 2024-04-25T01:44:14.415 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: |+..+.=o . | 2024-04-25T01:44:14.416 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: | . +o.oB . | 2024-04-25T01:44:14.416 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: | +o+.O.. . . | 2024-04-25T01:44:14.416 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: | . o++ =S. . | 2024-04-25T01:44:14.416 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: | ...o+. ...| 2024-04-25T01:44:14.416 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: | .o. o..| 2024-04-25T01:44:14.416 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: | ..oo | 2024-04-25T01:44:14.416 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: | E.*=. | 2024-04-25T01:44:14.416 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: +----[SHA256]-----+ 2024-04-25T01:44:15.026 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.742+0000 7f3da9800700 1 Processor -- start 2024-04-25T01:44:15.028 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.742+0000 7f3da9800700 1 -- start start 2024-04-25T01:44:15.028 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.743+0000 7f3da9800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da410acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:15.028 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.743+0000 7f3da9800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3da407bdf0 con 0x7f3da41088d0 2024-04-25T01:44:15.028 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.743+0000 7f3da2a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da410acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.743+0000 7f3da2a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da410acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34678/0 (socket says 172.21.15.102:34678) 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.743+0000 7f3da2a00700 1 -- 172.21.15.102:0/1029251526 learned_addr learned my addr 172.21.15.102:0/1029251526 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.744+0000 7f3da2a00700 1 -- 172.21.15.102:0/1029251526 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da410b200 con 0x7f3da41088d0 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.744+0000 7f3da2a00700 1 --2- 172.21.15.102:0/1029251526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da410acc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3d940098a0 tx=0x7f3d94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe54a622d596159 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.744+0000 7f3da1600700 1 -- 172.21.15.102:0/1029251526 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d94004160 con 0x7f3da41088d0 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.744+0000 7f3da1600700 1 -- 172.21.15.102:0/1029251526 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3d94004300 con 0x7f3da41088d0 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.744+0000 7f3da1600700 1 -- 172.21.15.102:0/1029251526 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d94011b50 con 0x7f3da41088d0 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.745+0000 7f3da9800700 1 -- 172.21.15.102:0/1029251526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 msgr2=0x7f3da410acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.745+0000 7f3da9800700 1 --2- 172.21.15.102:0/1029251526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da410acc0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3d940098a0 tx=0x7f3d94009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.745+0000 7f3da9800700 1 -- 172.21.15.102:0/1029251526 shutdown_connections 2024-04-25T01:44:15.029 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.745+0000 7f3da9800700 1 --2- 172.21.15.102:0/1029251526 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da410acc0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:15.033 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.745+0000 7f3da9800700 1 -- 172.21.15.102:0/1029251526 >> 172.21.15.102:0/1029251526 conn(0x7f3da4102280 msgr2=0x7f3da41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:15.033 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.745+0000 7f3da9800700 1 -- 172.21.15.102:0/1029251526 shutdown_connections 2024-04-25T01:44:15.033 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.745+0000 7f3da9800700 1 -- 172.21.15.102:0/1029251526 wait complete. 2024-04-25T01:44:15.033 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.746+0000 7f3da9800700 1 Processor -- start 2024-04-25T01:44:15.033 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.746+0000 7f3da9800700 1 -- start start 2024-04-25T01:44:15.033 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.747+0000 7f3da9800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da419e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:15.033 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.747+0000 7f3da9800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3da410c1d0 con 0x7f3da41088d0 2024-04-25T01:44:15.033 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.747+0000 7f3da2a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da419e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.747+0000 7f3da2a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da419e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34688/0 (socket says 172.21.15.102:34688) 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.747+0000 7f3da2a00700 1 -- 172.21.15.102:0/2458724917 learned_addr learned my addr 172.21.15.102:0/2458724917 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.747+0000 7f3da2a00700 1 -- 172.21.15.102:0/2458724917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da419e5e0 con 0x7f3da41088d0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.748+0000 7f3da2a00700 1 --2- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da419e0a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d94011b60 tx=0x7f3d94011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.748+0000 7f3d9b400700 1 -- 172.21.15.102:0/2458724917 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d94003b20 con 0x7f3da41088d0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.748+0000 7f3d9b400700 1 -- 172.21.15.102:0/2458724917 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3d94003cc0 con 0x7f3da41088d0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.748+0000 7f3d9b400700 1 -- 172.21.15.102:0/2458724917 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d94024840 con 0x7f3da41088d0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.748+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da419e870 con 0x7f3da41088d0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.748+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da419f390 con 0x7f3da41088d0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.749+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3da4063890 con 0x7f3da41088d0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.749+0000 7f3d9b400700 1 -- 172.21.15.102:0/2458724917 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f3d94022020 con 0x7f3da41088d0 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.750+0000 7f3d9b400700 1 --2- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f3d880390a0 0x7f3d8803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:15.034 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.750+0000 7f3d9b400700 1 -- 172.21.15.102:0/2458724917 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3d94011e00 con 0x7f3da41088d0 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.750+0000 7f3da2000700 1 --2- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f3d880390a0 0x7f3d8803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.751+0000 7f3da2000700 1 --2- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f3d880390a0 0x7f3d8803b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3d90006fd0 tx=0x7f3d90006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.755+0000 7f3d9b400700 1 -- 172.21.15.102:0/2458724917 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3d9400b3e0 con 0x7f3da41088d0 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:13.977+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f3da4103470 con 0x7f3d880390a0 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.275+0000 7f3d9b400700 1 -- 172.21.15.102:0/2458724917 <== mgr.14120 v2:172.21.15.102:6800/1124203809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3da4103470 con 0x7f3d880390a0 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f3d880390a0 msgr2=0x7f3d8803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 --2- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f3d880390a0 0x7f3d8803b560 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3d90006fd0 tx=0x7f3d90006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 msgr2=0x7f3da419e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 --2- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da419e0a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3d94011b60 tx=0x7f3d94011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 shutdown_connections 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 --2- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f3d880390a0 0x7f3d8803b560 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 --2- 172.21.15.102:0/2458724917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3da41088d0 0x7f3da419e0a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 >> 172.21.15.102:0/2458724917 conn(0x7f3da4102280 msgr2=0x7f3da4102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:15.035 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 shutdown_connections 2024-04-25T01:44:15.036 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:14.278+0000 7f3da9800700 1 -- 172.21.15.102:0/2458724917 wait complete. 2024-04-25T01:44:15.327 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:15 smithi102 ceph-mon[27796]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:15.327 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:15 smithi102 ceph-mon[27796]: Generating ssh key... 2024-04-25T01:44:15.327 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:15 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:15.327 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:15 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:16.568 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD17LQrULNXZsl7XlVg8EDFzlzV2tnRj/lWveRrUVLJhTdy1wdceXgIrLl4X8jy88/aFkq074Mqbmkv//iOdxUDO2DnDi3AFLYlklNbrZEDh3fF1lGMsLG9xbhFgPjmhVbzt8+B/6GPznzm8zgdSLoq1n/rMAOlldteoBwMsGj4/enumFpmmu1r8pOlSZFoC7happTPbdAYR3KMWOV9zxxtcoUCm6dbWLpLEQ72NgxBv6ezfKw2UqyuY8clkhRxuXVxCN7rS5Qpy6KurcWdtdNzwOMJRda8hTPlPKghowIhpHy9apGMre8XtRtk+lYmNQvnD1g/bzwbXDLd7XoY9zMi5LIpkdEOEt4Vl8U1V8f4QQHph6e9wkFw8Nj1mwz/gmchYpjE1hRzK8e1ouIV4SZXbSkH4ZIBRlIspk6VMe515GlZIbjYZySvbx53cI/sdO5eSP3M2kw27AZpK63XMe/xVHhrMJ6AwYShS5Ygp9hLnegLitq7dpabe6Sk0dCmIvc= ceph-145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:16.568 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.632+0000 7fee0a000700 1 Processor -- start 2024-04-25T01:44:16.568 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.632+0000 7fee0a000700 1 -- start start 2024-04-25T01:44:16.568 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.632+0000 7fee0a000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0407c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:16.568 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.632+0000 7fee0a000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fee0407c7e0 con 0x7fee0407dda0 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.633+0000 7fee03400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0407c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.633+0000 7fee03400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0407c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34694/0 (socket says 172.21.15.102:34694) 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.633+0000 7fee03400700 1 -- 172.21.15.102:0/3779360686 learned_addr learned my addr 172.21.15.102:0/3779360686 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.633+0000 7fee03400700 1 -- 172.21.15.102:0/3779360686 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0407c960 con 0x7fee0407dda0 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.634+0000 7fee03400700 1 --2- 172.21.15.102:0/3779360686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0407c2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fedf40098a0 tx=0x7fedf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3586f94d0a973176 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.634+0000 7fee02a00700 1 -- 172.21.15.102:0/3779360686 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedf4004160 con 0x7fee0407dda0 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.634+0000 7fee02a00700 1 -- 172.21.15.102:0/3779360686 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fedf4004300 con 0x7fee0407dda0 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.634+0000 7fee02a00700 1 -- 172.21.15.102:0/3779360686 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedf4011b50 con 0x7fee0407dda0 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.635+0000 7fee0a000700 1 -- 172.21.15.102:0/3779360686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 msgr2=0x7fee0407c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.635+0000 7fee0a000700 1 --2- 172.21.15.102:0/3779360686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0407c2a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fedf40098a0 tx=0x7fedf4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.635+0000 7fee0a000700 1 -- 172.21.15.102:0/3779360686 shutdown_connections 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.635+0000 7fee0a000700 1 --2- 172.21.15.102:0/3779360686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0407c2a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.635+0000 7fee0a000700 1 -- 172.21.15.102:0/3779360686 >> 172.21.15.102:0/3779360686 conn(0x7fee04103420 msgr2=0x7fee04105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:16.569 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.635+0000 7fee0a000700 1 -- 172.21.15.102:0/3779360686 shutdown_connections 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.635+0000 7fee0a000700 1 -- 172.21.15.102:0/3779360686 wait complete. 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.636+0000 7fee0a000700 1 Processor -- start 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.636+0000 7fee0a000700 1 -- start start 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.637+0000 7fee0a000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.637+0000 7fee0a000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fee0410e410 con 0x7fee0407dda0 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.637+0000 7fee03400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.637+0000 7fee03400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34708/0 (socket says 172.21.15.102:34708) 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.637+0000 7fee03400700 1 -- 172.21.15.102:0/3847064982 learned_addr learned my addr 172.21.15.102:0/3847064982 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.637+0000 7fee03400700 1 -- 172.21.15.102:0/3847064982 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0419e560 con 0x7fee0407dda0 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.637+0000 7fee03400700 1 --2- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0419e020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fedf4011b60 tx=0x7fedf4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.638+0000 7fee00c00700 1 -- 172.21.15.102:0/3847064982 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedf4003b20 con 0x7fee0407dda0 2024-04-25T01:44:16.570 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.638+0000 7fee00c00700 1 -- 172.21.15.102:0/3847064982 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fedf4003cc0 con 0x7fee0407dda0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.638+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0419e7f0 con 0x7fee0407dda0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.638+0000 7fee00c00700 1 -- 172.21.15.102:0/3847064982 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedf4024840 con 0x7fee0407dda0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.638+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0419f310 con 0x7fee0407dda0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.639+0000 7fee00c00700 1 -- 172.21.15.102:0/3847064982 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fedf4022020 con 0x7fee0407dda0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.639+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee04063890 con 0x7fee0407dda0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.640+0000 7fee00c00700 1 --2- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fede80390a0 0x7fede803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.640+0000 7fee00c00700 1 -- 172.21.15.102:0/3847064982 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fedf4011e00 con 0x7fee0407dda0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.640+0000 7fedfaa00700 1 --2- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fede80390a0 0x7fede803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.641+0000 7fedfaa00700 1 --2- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fede80390a0 0x7fede803b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fedec006fd0 tx=0x7fedec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.645+0000 7fee00c00700 1 -- 172.21.15.102:0/3847064982 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fedf400b3e0 con 0x7fee0407dda0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.870+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fee041045e0 con 0x7fede80390a0 2024-04-25T01:44:16.571 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.872+0000 7fee00c00700 1 -- 172.21.15.102:0/3847064982 <== mgr.14120 v2:172.21.15.102:6800/1124203809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fee041045e0 con 0x7fede80390a0 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.875+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fede80390a0 msgr2=0x7fede803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.875+0000 7fee0a000700 1 --2- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fede80390a0 0x7fede803b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fedec006fd0 tx=0x7fedec006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.875+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 msgr2=0x7fee0419e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.875+0000 7fee0a000700 1 --2- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0419e020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fedf4011b60 tx=0x7fedf4011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.875+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 shutdown_connections 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.875+0000 7fee0a000700 1 --2- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fede80390a0 0x7fede803b560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.875+0000 7fee0a000700 1 --2- 172.21.15.102:0/3847064982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee0407dda0 0x7fee0419e020 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.875+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 >> 172.21.15.102:0/3847064982 conn(0x7fee04103420 msgr2=0x7fee04103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.875+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 shutdown_connections 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:15.876+0000 7fee0a000700 1 -- 172.21.15.102:0/3847064982 wait complete. 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T01:44:16.572 INFO:teuthology.orchestra.run.smithi102.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T01:44:16.581 INFO:teuthology.orchestra.run.smithi102.stdout:Adding host smithi102... 2024-04-25T01:44:16.898 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:16 smithi102 ceph-mon[27796]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:17.809 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:17 smithi102 ceph-mon[27796]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:19.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:19 smithi102 ceph-mon[27796]: Deploying cephadm binary to smithi102 2024-04-25T01:44:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Added host 'smithi102' with addr '172.21.15.102' 2024-04-25T01:44:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.217+0000 7f4568c00700 1 Processor -- start 2024-04-25T01:44:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.217+0000 7f4568c00700 1 -- start start 2024-04-25T01:44:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.217+0000 7f4568c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f4564111060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.217+0000 7f4568c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f45640824a0 con 0x7f456410ec70 2024-04-25T01:44:24.039 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.218+0000 7f4563400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f4564111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.218+0000 7f4563400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f4564111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34710/0 (socket says 172.21.15.102:34710) 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.218+0000 7f4563400700 1 -- 172.21.15.102:0/1397608955 learned_addr learned my addr 172.21.15.102:0/1397608955 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.218+0000 7f4563400700 1 -- 172.21.15.102:0/1397608955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45641115a0 con 0x7f456410ec70 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.218+0000 7f4563400700 1 --2- 172.21.15.102:0/1397608955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f4564111060 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f45540098a0 tx=0x7f4554009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26350b2430094ee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.219+0000 7f4562000700 1 -- 172.21.15.102:0/1397608955 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4554004160 con 0x7f456410ec70 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.219+0000 7f4562000700 1 -- 172.21.15.102:0/1397608955 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4554004300 con 0x7f456410ec70 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.219+0000 7f4562000700 1 -- 172.21.15.102:0/1397608955 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4554011b50 con 0x7f456410ec70 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.219+0000 7f4568c00700 1 -- 172.21.15.102:0/1397608955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 msgr2=0x7f4564111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.219+0000 7f4568c00700 1 --2- 172.21.15.102:0/1397608955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f4564111060 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f45540098a0 tx=0x7f4554009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.219+0000 7f4568c00700 1 -- 172.21.15.102:0/1397608955 shutdown_connections 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.219+0000 7f4568c00700 1 --2- 172.21.15.102:0/1397608955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f4564111060 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.219+0000 7f4568c00700 1 -- 172.21.15.102:0/1397608955 >> 172.21.15.102:0/1397608955 conn(0x7f456407d240 msgr2=0x7f456407f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.220+0000 7f4568c00700 1 -- 172.21.15.102:0/1397608955 shutdown_connections 2024-04-25T01:44:24.040 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.220+0000 7f4568c00700 1 -- 172.21.15.102:0/1397608955 wait complete. 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.221+0000 7f4568c00700 1 Processor -- start 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.221+0000 7f4568c00700 1 -- start start 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.221+0000 7f4568c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f45641a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.221+0000 7f4568c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4564112620 con 0x7f456410ec70 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.221+0000 7f4563400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f45641a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.221+0000 7f4563400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f45641a21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34712/0 (socket says 172.21.15.102:34712) 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.221+0000 7f4563400700 1 -- 172.21.15.102:0/2554185868 learned_addr learned my addr 172.21.15.102:0/2554185868 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.222+0000 7f4563400700 1 -- 172.21.15.102:0/2554185868 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45641a2720 con 0x7f456410ec70 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.222+0000 7f4563400700 1 --2- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f45641a21e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4554011b60 tx=0x7f4554011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.222+0000 7f455be00700 1 -- 172.21.15.102:0/2554185868 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4554003970 con 0x7f456410ec70 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.222+0000 7f455be00700 1 -- 172.21.15.102:0/2554185868 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4554003b10 con 0x7f456410ec70 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.222+0000 7f455be00700 1 -- 172.21.15.102:0/2554185868 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4554024610 con 0x7f456410ec70 2024-04-25T01:44:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.222+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45641a29b0 con 0x7f456410ec70 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.222+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45641a56a0 con 0x7f456410ec70 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.224+0000 7f455be00700 1 -- 172.21.15.102:0/2554185868 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f455401a9f0 con 0x7f456410ec70 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.223+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4564069390 con 0x7f456410ec70 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.224+0000 7f455be00700 1 --2- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f45480390a0 0x7f454803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.224+0000 7f455be00700 1 -- 172.21.15.102:0/2554185868 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4554011e00 con 0x7f456410ec70 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.224+0000 7f4562a00700 1 --2- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f45480390a0 0x7f454803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.225+0000 7f4562a00700 1 --2- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f45480390a0 0x7f454803b560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4550006fd0 tx=0x7f4550006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.230+0000 7f455be00700 1 -- 172.21.15.102:0/2554185868 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f455400b3e0 con 0x7f456410ec70 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:17.454+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}) -- 0x7f456407e410 con 0x7f45480390a0 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.322+0000 7f455be00700 1 -- 172.21.15.102:0/2554185868 <== mgr.14120 v2:172.21.15.102:6800/1124203809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f456407e410 con 0x7f45480390a0 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.326+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f45480390a0 msgr2=0x7f454803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.326+0000 7f4568c00700 1 --2- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f45480390a0 0x7f454803b560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4550006fd0 tx=0x7f4550006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:24.042 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.326+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 msgr2=0x7f45641a21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.326+0000 7f4568c00700 1 --2- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f45641a21e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4554011b60 tx=0x7f4554011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.326+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 shutdown_connections 2024-04-25T01:44:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.326+0000 7f4568c00700 1 --2- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f45480390a0 0x7f454803b560 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.326+0000 7f4568c00700 1 --2- 172.21.15.102:0/2554185868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f456410ec70 0x7f45641a21e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.326+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 >> 172.21.15.102:0/2554185868 conn(0x7f456407d240 msgr2=0x7f456407dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.327+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 shutdown_connections 2024-04-25T01:44:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:23.327+0000 7f4568c00700 1 -- 172.21.15.102:0/2554185868 wait complete. 2024-04-25T01:44:24.043 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mon service... 2024-04-25T01:44:24.623 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:24 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:24.623 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:24 smithi102 ceph-mon[27796]: Added host smithi102 2024-04-25T01:44:24.623 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:24 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:26.164 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:25 smithi102 ceph-mon[27796]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:26.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:25 smithi102 ceph-mon[27796]: Saving service mon spec with placement count:5 2024-04-25T01:44:26.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:25 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:26.336 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T01:44:26.337 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.650+0000 7f82da000700 1 Processor -- start 2024-04-25T01:44:26.337 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.651+0000 7f82da000700 1 -- start start 2024-04-25T01:44:26.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.651+0000 7f82da000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d4106a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:26.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.651+0000 7f82da000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f82d407be50 con 0x7f82d4106660 2024-04-25T01:44:26.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.651+0000 7f82d3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d4106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:26.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.651+0000 7f82d3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d4106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55238/0 (socket says 172.21.15.102:55238) 2024-04-25T01:44:26.338 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.651+0000 7f82d3400700 1 -- 172.21.15.102:0/10165494 learned_addr learned my addr 172.21.15.102:0/10165494 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:26.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.652+0000 7f82d3400700 1 -- 172.21.15.102:0/10165494 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d4106fc0 con 0x7f82d4106660 2024-04-25T01:44:26.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.652+0000 7f82d3400700 1 --2- 172.21.15.102:0/10165494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d4106a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f82c40098a0 tx=0x7f82c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59cfdb2b5376ac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:26.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.652+0000 7f82d2000700 1 -- 172.21.15.102:0/10165494 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82c4004160 con 0x7f82d4106660 2024-04-25T01:44:26.340 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.652+0000 7f82d2000700 1 -- 172.21.15.102:0/10165494 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f82c4004300 con 0x7f82d4106660 2024-04-25T01:44:26.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.652+0000 7f82d2000700 1 -- 172.21.15.102:0/10165494 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82c4011b50 con 0x7f82d4106660 2024-04-25T01:44:26.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.653+0000 7f82da000700 1 -- 172.21.15.102:0/10165494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 msgr2=0x7f82d4106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:26.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.653+0000 7f82da000700 1 --2- 172.21.15.102:0/10165494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d4106a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f82c40098a0 tx=0x7f82c4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:26.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.653+0000 7f82da000700 1 -- 172.21.15.102:0/10165494 shutdown_connections 2024-04-25T01:44:26.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.653+0000 7f82da000700 1 --2- 172.21.15.102:0/10165494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d4106a80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:26.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.653+0000 7f82da000700 1 -- 172.21.15.102:0/10165494 >> 172.21.15.102:0/10165494 conn(0x7f82d4102280 msgr2=0x7f82d41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:26.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.653+0000 7f82da000700 1 -- 172.21.15.102:0/10165494 shutdown_connections 2024-04-25T01:44:26.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.654+0000 7f82da000700 1 -- 172.21.15.102:0/10165494 wait complete. 2024-04-25T01:44:26.341 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.654+0000 7f82da000700 1 Processor -- start 2024-04-25T01:44:26.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.655+0000 7f82da000700 1 -- start start 2024-04-25T01:44:26.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.655+0000 7f82da000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d419e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:26.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.655+0000 7f82da000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f82d4108050 con 0x7f82d4106660 2024-04-25T01:44:26.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.655+0000 7f82d3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d419e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:26.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.655+0000 7f82d3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d419e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55240/0 (socket says 172.21.15.102:55240) 2024-04-25T01:44:26.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.655+0000 7f82d3400700 1 -- 172.21.15.102:0/3893577824 learned_addr learned my addr 172.21.15.102:0/3893577824 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:26.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.656+0000 7f82d3400700 1 -- 172.21.15.102:0/3893577824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d419e590 con 0x7f82d4106660 2024-04-25T01:44:26.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.656+0000 7f82d3400700 1 --2- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d419e050 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82c4011b60 tx=0x7f82c4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:26.342 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.656+0000 7f82cbe00700 1 -- 172.21.15.102:0/3893577824 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82c4003b20 con 0x7f82d4106660 2024-04-25T01:44:26.345 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.656+0000 7f82cbe00700 1 -- 172.21.15.102:0/3893577824 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f82c4003cc0 con 0x7f82d4106660 2024-04-25T01:44:26.345 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.656+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d419e820 con 0x7f82d4106660 2024-04-25T01:44:26.345 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.656+0000 7f82cbe00700 1 -- 172.21.15.102:0/3893577824 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82c4024840 con 0x7f82d4106660 2024-04-25T01:44:26.345 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.656+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d419f340 con 0x7f82d4106660 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.657+0000 7f82cbe00700 1 -- 172.21.15.102:0/3893577824 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f82c4022020 con 0x7f82d4106660 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.657+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d4063890 con 0x7f82d4106660 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.658+0000 7f82cbe00700 1 --2- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f82b80390a0 0x7f82b803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.658+0000 7f82cbe00700 1 -- 172.21.15.102:0/3893577824 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f82c4011e00 con 0x7f82d4106660 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.658+0000 7f82d2a00700 1 --2- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f82b80390a0 0x7f82b803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.659+0000 7f82d2a00700 1 --2- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f82b80390a0 0x7f82b803b560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f82c0006fd0 tx=0x7f82c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.663+0000 7f82cbe00700 1 -- 172.21.15.102:0/3893577824 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f82c401a7b0 con 0x7f82d4106660 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.913+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f82d41a0090 con 0x7f82b80390a0 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.920+0000 7f82cbe00700 1 -- 172.21.15.102:0/3893577824 <== mgr.14120 v2:172.21.15.102:6800/1124203809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f82d41a0090 con 0x7f82b80390a0 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.922+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f82b80390a0 msgr2=0x7f82b803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.922+0000 7f82da000700 1 --2- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f82b80390a0 0x7f82b803b560 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f82c0006fd0 tx=0x7f82c0006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:26.346 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.922+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 msgr2=0x7f82d419e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:26.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.922+0000 7f82da000700 1 --2- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d419e050 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82c4011b60 tx=0x7f82c4011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:26.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.922+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 shutdown_connections 2024-04-25T01:44:26.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.922+0000 7f82da000700 1 --2- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f82b80390a0 0x7f82b803b560 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:26.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.922+0000 7f82da000700 1 --2- 172.21.15.102:0/3893577824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82d4106660 0x7f82d419e050 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:26.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.922+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 >> 172.21.15.102:0/3893577824 conn(0x7f82d4102280 msgr2=0x7f82d4194b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:26.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.923+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 shutdown_connections 2024-04-25T01:44:26.347 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:24.923+0000 7f82da000700 1 -- 172.21.15.102:0/3893577824 wait complete. 2024-04-25T01:44:26.347 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mgr service... 2024-04-25T01:44:27.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:27 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:27.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:27 smithi102 ceph-mon[27796]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:27.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:27 smithi102 ceph-mon[27796]: Saving service mgr spec with placement count:2 2024-04-25T01:44:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:27 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:27 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:28.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T01:44:28.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.290+0000 7effcac00700 1 Processor -- start 2024-04-25T01:44:28.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.290+0000 7effcac00700 1 -- start start 2024-04-25T01:44:28.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.290+0000 7effcac00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc410a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:28.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.290+0000 7effcac00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7effc410a9e0 con 0x7effc4109ff0 2024-04-25T01:44:28.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.291+0000 7effc3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc410a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:28.298 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.291+0000 7effc3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc410a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55256/0 (socket says 172.21.15.102:55256) 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.291+0000 7effc3e00700 1 -- 172.21.15.102:0/1816561868 learned_addr learned my addr 172.21.15.102:0/1816561868 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.291+0000 7effc3e00700 1 -- 172.21.15.102:0/1816561868 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc410b1f0 con 0x7effc4109ff0 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.292+0000 7effc3e00700 1 --2- 172.21.15.102:0/1816561868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc410a410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7effb40098a0 tx=0x7effb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aadd07ee7f107cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.292+0000 7effc2a00700 1 -- 172.21.15.102:0/1816561868 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effb4004160 con 0x7effc4109ff0 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.292+0000 7effc2a00700 1 -- 172.21.15.102:0/1816561868 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effb4004300 con 0x7effc4109ff0 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.292+0000 7effc2a00700 1 -- 172.21.15.102:0/1816561868 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effb4011b20 con 0x7effc4109ff0 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.293+0000 7effcac00700 1 -- 172.21.15.102:0/1816561868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 msgr2=0x7effc410a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.293+0000 7effcac00700 1 --2- 172.21.15.102:0/1816561868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc410a410 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7effb40098a0 tx=0x7effb4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.293+0000 7effcac00700 1 -- 172.21.15.102:0/1816561868 shutdown_connections 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.293+0000 7effcac00700 1 --2- 172.21.15.102:0/1816561868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc410a410 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.293+0000 7effcac00700 1 -- 172.21.15.102:0/1816561868 >> 172.21.15.102:0/1816561868 conn(0x7effc407d3c0 msgr2=0x7effc407d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:28.299 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.293+0000 7effcac00700 1 -- 172.21.15.102:0/1816561868 shutdown_connections 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.293+0000 7effcac00700 1 -- 172.21.15.102:0/1816561868 wait complete. 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.294+0000 7effcac00700 1 Processor -- start 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.294+0000 7effcac00700 1 -- start start 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.295+0000 7effcac00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc419e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.295+0000 7effcac00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7effc410e5a0 con 0x7effc4109ff0 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.295+0000 7effc3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc419e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.295+0000 7effc3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc419e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55260/0 (socket says 172.21.15.102:55260) 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.295+0000 7effc3e00700 1 -- 172.21.15.102:0/575770789 learned_addr learned my addr 172.21.15.102:0/575770789 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.295+0000 7effc3e00700 1 -- 172.21.15.102:0/575770789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc419e580 con 0x7effc4109ff0 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.296+0000 7effc3e00700 1 --2- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc419e040 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7effb40117e0 tx=0x7effb4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.296+0000 7effc0c00700 1 -- 172.21.15.102:0/575770789 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effb4003bf0 con 0x7effc4109ff0 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.296+0000 7effc0c00700 1 -- 172.21.15.102:0/575770789 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effb401a910 con 0x7effc4109ff0 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.296+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc419e810 con 0x7effc4109ff0 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.296+0000 7effc0c00700 1 -- 172.21.15.102:0/575770789 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effb4024970 con 0x7effc4109ff0 2024-04-25T01:44:28.300 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.296+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc419f330 con 0x7effc4109ff0 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.298+0000 7effc0c00700 1 -- 172.21.15.102:0/575770789 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7effb4022020 con 0x7effc4109ff0 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.298+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc4063890 con 0x7effc4109ff0 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.298+0000 7effc0c00700 1 --2- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7effa80390a0 0x7effa803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.298+0000 7effc0c00700 1 -- 172.21.15.102:0/575770789 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7effb404d990 con 0x7effc4109ff0 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.299+0000 7effc3400700 1 --2- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7effa80390a0 0x7effa803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.299+0000 7effc3400700 1 --2- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7effa80390a0 0x7effa803b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7effb0006fd0 tx=0x7effb0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.304+0000 7effc0c00700 1 -- 172.21.15.102:0/575770789 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7effb402d910 con 0x7effc4109ff0 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.545+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7effc4107ad0 con 0x7effa80390a0 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.556+0000 7effc0c00700 1 -- 172.21.15.102:0/575770789 <== mgr.14120 v2:172.21.15.102:6800/1124203809 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7effc4107ad0 con 0x7effa80390a0 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7effa80390a0 msgr2=0x7effa803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 --2- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7effa80390a0 0x7effa803b560 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7effb0006fd0 tx=0x7effb0006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 msgr2=0x7effc419e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 --2- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc419e040 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7effb40117e0 tx=0x7effb4011ba0 comp rx=0 tx=0).stop 2024-04-25T01:44:28.301 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 shutdown_connections 2024-04-25T01:44:28.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 --2- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7effa80390a0 0x7effa803b560 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:28.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 --2- 172.21.15.102:0/575770789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4109ff0 0x7effc419e040 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:28.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 >> 172.21.15.102:0/575770789 conn(0x7effc407d3c0 msgr2=0x7effc41073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:28.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 shutdown_connections 2024-04-25T01:44:28.302 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:27.559+0000 7effcac00700 1 -- 172.21.15.102:0/575770789 wait complete. 2024-04-25T01:44:29.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:29 smithi102 ceph-mon[27796]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:44:29.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:29 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3889904394' entity='client.admin' 2024-04-25T01:44:30.397 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.882+0000 7f5bf0e00700 1 Processor -- start 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.882+0000 7f5bf0e00700 1 -- start start 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.883+0000 7f5bf0e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec106980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.883+0000 7f5bf0e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5bec07bdb0 con 0x7f5bec106560 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.883+0000 7f5beb400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec106980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.883+0000 7f5beb400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec106980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55262/0 (socket says 172.21.15.102:55262) 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.883+0000 7f5beb400700 1 -- 172.21.15.102:0/903347831 learned_addr learned my addr 172.21.15.102:0/903347831 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.883+0000 7f5beb400700 1 -- 172.21.15.102:0/903347831 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bec106ec0 con 0x7f5bec106560 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.884+0000 7f5beb400700 1 --2- 172.21.15.102:0/903347831 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec106980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0098a0 tx=0x7f5bdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54633a6d2bd3100 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.884+0000 7f5beaa00700 1 -- 172.21.15.102:0/903347831 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bdc004160 con 0x7f5bec106560 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.884+0000 7f5beaa00700 1 -- 172.21.15.102:0/903347831 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bdc004300 con 0x7f5bec106560 2024-04-25T01:44:30.398 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.884+0000 7f5beaa00700 1 -- 172.21.15.102:0/903347831 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bdc011b20 con 0x7f5bec106560 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.885+0000 7f5bf0e00700 1 -- 172.21.15.102:0/903347831 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 msgr2=0x7f5bec106980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.885+0000 7f5bf0e00700 1 --2- 172.21.15.102:0/903347831 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec106980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0098a0 tx=0x7f5bdc009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.885+0000 7f5bf0e00700 1 -- 172.21.15.102:0/903347831 shutdown_connections 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.885+0000 7f5bf0e00700 1 --2- 172.21.15.102:0/903347831 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec106980 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.885+0000 7f5bf0e00700 1 -- 172.21.15.102:0/903347831 >> 172.21.15.102:0/903347831 conn(0x7f5bec1020e0 msgr2=0x7f5bec104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.885+0000 7f5bf0e00700 1 -- 172.21.15.102:0/903347831 shutdown_connections 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.885+0000 7f5bf0e00700 1 -- 172.21.15.102:0/903347831 wait complete. 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.886+0000 7f5bf0e00700 1 Processor -- start 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.887+0000 7f5bf0e00700 1 -- start start 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.887+0000 7f5bf0e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.887+0000 7f5bf0e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5bec107f90 con 0x7f5bec106560 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.887+0000 7f5beb400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:30.399 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.887+0000 7f5beb400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55274/0 (socket says 172.21.15.102:55274) 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.887+0000 7f5beb400700 1 -- 172.21.15.102:0/3889904394 learned_addr learned my addr 172.21.15.102:0/3889904394 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.887+0000 7f5beb400700 1 -- 172.21.15.102:0/3889904394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bec19e2e0 con 0x7f5bec106560 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.888+0000 7f5beb400700 1 --2- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec19dda0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc011b60 tx=0x7f5bdc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.888+0000 7f5be8c00700 1 -- 172.21.15.102:0/3889904394 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bdc003b60 con 0x7f5bec106560 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.888+0000 7f5be8c00700 1 -- 172.21.15.102:0/3889904394 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bdc01a910 con 0x7f5bec106560 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.888+0000 7f5be8c00700 1 -- 172.21.15.102:0/3889904394 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5bdc024930 con 0x7f5bec106560 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.888+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bec19e570 con 0x7f5bec106560 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.888+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bec1a1220 con 0x7f5bec106560 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.889+0000 7f5be8c00700 1 -- 172.21.15.102:0/3889904394 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f5bdc022020 con 0x7f5bec106560 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.890+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bec069390 con 0x7f5bec106560 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.890+0000 7f5be8c00700 1 --2- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f5bd00390a0 0x7f5bd003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.890+0000 7f5be8c00700 1 -- 172.21.15.102:0/3889904394 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5bdc04d330 con 0x7f5bec106560 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.890+0000 7f5be3e00700 1 --2- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f5bd00390a0 0x7f5bd003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:30.400 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.891+0000 7f5be3e00700 1 --2- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f5bd00390a0 0x7f5bd003b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4006fd0 tx=0x7f5bd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:28.896+0000 7f5be8c00700 1 -- 172.21.15.102:0/3889904394 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5bdc04cc80 con 0x7f5bec106560 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.124+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f5bec02dd60 con 0x7f5bec106560 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.129+0000 7f5be8c00700 1 -- 172.21.15.102:0/3889904394 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f5bdc01a4e0 con 0x7f5bec106560 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.132+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f5bd00390a0 msgr2=0x7f5bd003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.132+0000 7f5bf0e00700 1 --2- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f5bd00390a0 0x7f5bd003b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4006fd0 tx=0x7f5bd4006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.132+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 msgr2=0x7f5bec19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.132+0000 7f5bf0e00700 1 --2- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec19dda0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc011b60 tx=0x7f5bdc011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.132+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 shutdown_connections 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.132+0000 7f5bf0e00700 1 --2- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f5bd00390a0 0x7f5bd003b560 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.132+0000 7f5bf0e00700 1 --2- 172.21.15.102:0/3889904394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bec106560 0x7f5bec19dda0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.132+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 >> 172.21.15.102:0/3889904394 conn(0x7f5bec1020e0 msgr2=0x7f5bec194980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.133+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 shutdown_connections 2024-04-25T01:44:30.401 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:29.133+0000 7f5bf0e00700 1 -- 172.21.15.102:0/3889904394 wait complete. 2024-04-25T01:44:31.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:31 smithi102 ceph-mon[27796]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:44:31.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:31 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:31.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:31 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2566489601' entity='client.admin' 2024-04-25T01:44:32.107 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.048+0000 7f1aeca00700 1 Processor -- start 2024-04-25T01:44:32.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.049+0000 7f1aeca00700 1 -- start start 2024-04-25T01:44:32.113 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.049+0000 7f1aeca00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae8108050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:32.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.049+0000 7f1aeca00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae8108620 con 0x7f1ae8107c30 2024-04-25T01:44:32.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.050+0000 7f1ae7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae8108050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:32.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.050+0000 7f1ae7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae8108050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55284/0 (socket says 172.21.15.102:55284) 2024-04-25T01:44:32.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.050+0000 7f1ae7400700 1 -- 172.21.15.102:0/1200409145 learned_addr learned my addr 172.21.15.102:0/1200409145 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:32.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.050+0000 7f1ae7400700 1 -- 172.21.15.102:0/1200409145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae8108e30 con 0x7f1ae8107c30 2024-04-25T01:44:32.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.051+0000 7f1ae7400700 1 --2- 172.21.15.102:0/1200409145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae8108050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80098a0 tx=0x7f1ad8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d01397402fa869 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:32.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.051+0000 7f1ae6000700 1 -- 172.21.15.102:0/1200409145 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ad8004160 con 0x7f1ae8107c30 2024-04-25T01:44:32.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.051+0000 7f1ae6000700 1 -- 172.21.15.102:0/1200409145 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad8004300 con 0x7f1ae8107c30 2024-04-25T01:44:32.114 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.051+0000 7f1ae6000700 1 -- 172.21.15.102:0/1200409145 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ad8011b20 con 0x7f1ae8107c30 2024-04-25T01:44:32.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.052+0000 7f1aeca00700 1 -- 172.21.15.102:0/1200409145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 msgr2=0x7f1ae8108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:32.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.052+0000 7f1aeca00700 1 --2- 172.21.15.102:0/1200409145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae8108050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80098a0 tx=0x7f1ad8009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:32.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.052+0000 7f1aeca00700 1 -- 172.21.15.102:0/1200409145 shutdown_connections 2024-04-25T01:44:32.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.052+0000 7f1aeca00700 1 --2- 172.21.15.102:0/1200409145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae8108050 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:32.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.052+0000 7f1aeca00700 1 -- 172.21.15.102:0/1200409145 >> 172.21.15.102:0/1200409145 conn(0x7f1ae81031d0 msgr2=0x7f1ae8105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:32.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.052+0000 7f1aeca00700 1 -- 172.21.15.102:0/1200409145 shutdown_connections 2024-04-25T01:44:32.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.052+0000 7f1aeca00700 1 -- 172.21.15.102:0/1200409145 wait complete. 2024-04-25T01:44:32.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.054+0000 7f1aeca00700 1 Processor -- start 2024-04-25T01:44:32.115 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.054+0000 7f1aeca00700 1 -- start start 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.054+0000 7f1aeca00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.054+0000 7f1aeca00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae810c1e0 con 0x7f1ae8107c30 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.055+0000 7f1ae7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.055+0000 7f1ae7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae819de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55298/0 (socket says 172.21.15.102:55298) 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.055+0000 7f1ae7400700 1 -- 172.21.15.102:0/2566489601 learned_addr learned my addr 172.21.15.102:0/2566489601 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.055+0000 7f1ae7400700 1 -- 172.21.15.102:0/2566489601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae819e340 con 0x7f1ae8107c30 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.055+0000 7f1ae7400700 1 --2- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae819de00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8000c00 tx=0x7f1ad8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.056+0000 7f1adfe00700 1 -- 172.21.15.102:0/2566489601 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ad8003b60 con 0x7f1ae8107c30 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.056+0000 7f1adfe00700 1 -- 172.21.15.102:0/2566489601 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad801a910 con 0x7f1ae8107c30 2024-04-25T01:44:32.116 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.056+0000 7f1adfe00700 1 -- 172.21.15.102:0/2566489601 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ad8024930 con 0x7f1ae8107c30 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.056+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae819e5d0 con 0x7f1ae8107c30 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.056+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae819ea30 con 0x7f1ae8107c30 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.057+0000 7f1adfe00700 1 -- 172.21.15.102:0/2566489601 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1ad8022020 con 0x7f1ae8107c30 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.057+0000 7f1adfe00700 1 --2- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f1acc0390a0 0x7f1acc03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.057+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae8069390 con 0x7f1ae8107c30 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.058+0000 7f1adfe00700 1 -- 172.21.15.102:0/2566489601 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1ad804d260 con 0x7f1ae8107c30 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.058+0000 7f1ae6a00700 1 --2- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f1acc0390a0 0x7f1acc03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.059+0000 7f1ae6a00700 1 --2- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f1acc0390a0 0x7f1acc03b560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4006fd0 tx=0x7f1ad4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.063+0000 7f1adfe00700 1 -- 172.21.15.102:0/2566489601 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1ad802ddb0 con 0x7f1ae8107c30 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.330+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f1ae81a14c0 con 0x7f1ae8107c30 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.335+0000 7f1adfe00700 1 -- 172.21.15.102:0/2566489601 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f1ad800b3e0 con 0x7f1ae8107c30 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.338+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f1acc0390a0 msgr2=0x7f1acc03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:32.117 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.338+0000 7f1aeca00700 1 --2- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f1acc0390a0 0x7f1acc03b560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4006fd0 tx=0x7f1ad4006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:32.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.338+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 msgr2=0x7f1ae819de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:32.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.338+0000 7f1aeca00700 1 --2- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae819de00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8000c00 tx=0x7f1ad8011c10 comp rx=0 tx=0).stop 2024-04-25T01:44:32.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.338+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 shutdown_connections 2024-04-25T01:44:32.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.338+0000 7f1aeca00700 1 --2- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f1acc0390a0 0x7f1acc03b560 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:32.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.338+0000 7f1aeca00700 1 --2- 172.21.15.102:0/2566489601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8107c30 0x7f1ae819de00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:32.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.338+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 >> 172.21.15.102:0/2566489601 conn(0x7f1ae81031d0 msgr2=0x7f1ae8103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:32.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.338+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 shutdown_connections 2024-04-25T01:44:32.118 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:31.339+0000 7f1aeca00700 1 -- 172.21.15.102:0/2566489601 wait complete. 2024-04-25T01:44:32.118 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling the dashboard module... 2024-04-25T01:44:33.883 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:33 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: ignoring --setuser ceph since I am not root 2024-04-25T01:44:33.883 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:33 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: ignoring --setgroup ceph since I am not root 2024-04-25T01:44:33.883 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:33 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:33.891+0000 7fd2a5647200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:44:33.884 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:33 smithi102 ceph-mon[27796]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:44:33.884 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:33 smithi102 ceph-mon[27796]: from='mgr.14120 172.21.15.102:0/2866736534' entity='mgr.x' 2024-04-25T01:44:33.884 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:33 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3741951733' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T01:44:34.164 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:34 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:34.133+0000 7fd2a5647200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:44:34.526 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:34 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:34.262+0000 7fd2a5647200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.736+0000 7feb35c00700 1 Processor -- start 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.736+0000 7feb35c00700 1 -- start start 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.736+0000 7feb35c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3007c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.736+0000 7feb35c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feb3007c7a0 con 0x7feb3007dd60 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.736+0000 7feb26a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3007c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.737+0000 7feb26a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3007c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55306/0 (socket says 172.21.15.102:55306) 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.737+0000 7feb26a00700 1 -- 172.21.15.102:0/3847146306 learned_addr learned my addr 172.21.15.102:0/3847146306 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.737+0000 7feb26a00700 1 -- 172.21.15.102:0/3847146306 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3007c920 con 0x7feb3007dd60 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.737+0000 7feb26a00700 1 --2- 172.21.15.102:0/3847146306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3007c260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feb200098a0 tx=0x7feb20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447c042c41fd0590 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.738+0000 7feb2e000700 1 -- 172.21.15.102:0/3847146306 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb20004160 con 0x7feb3007dd60 2024-04-25T01:44:34.758 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.738+0000 7feb2e000700 1 -- 172.21.15.102:0/3847146306 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb20004300 con 0x7feb3007dd60 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.738+0000 7feb2e000700 1 -- 172.21.15.102:0/3847146306 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb20011b20 con 0x7feb3007dd60 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.738+0000 7feb35c00700 1 -- 172.21.15.102:0/3847146306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 msgr2=0x7feb3007c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.738+0000 7feb35c00700 1 --2- 172.21.15.102:0/3847146306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3007c260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feb200098a0 tx=0x7feb20009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.738+0000 7feb35c00700 1 -- 172.21.15.102:0/3847146306 shutdown_connections 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.738+0000 7feb35c00700 1 --2- 172.21.15.102:0/3847146306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3007c260 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.738+0000 7feb35c00700 1 -- 172.21.15.102:0/3847146306 >> 172.21.15.102:0/3847146306 conn(0x7feb30103420 msgr2=0x7feb30105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.739+0000 7feb35c00700 1 -- 172.21.15.102:0/3847146306 shutdown_connections 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.739+0000 7feb35c00700 1 -- 172.21.15.102:0/3847146306 wait complete. 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.740+0000 7feb35c00700 1 Processor -- start 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.740+0000 7feb35c00700 1 -- start start 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.740+0000 7feb35c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.740+0000 7feb35c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feb3010e3e0 con 0x7feb3007dd60 2024-04-25T01:44:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.741+0000 7feb26a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.741+0000 7feb26a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55308/0 (socket says 172.21.15.102:55308) 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.741+0000 7feb26a00700 1 -- 172.21.15.102:0/3741951733 learned_addr learned my addr 172.21.15.102:0/3741951733 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.741+0000 7feb26a00700 1 -- 172.21.15.102:0/3741951733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3019e540 con 0x7feb3007dd60 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.742+0000 7feb26a00700 1 --2- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3019e000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb200117e0 tx=0x7feb20011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.742+0000 7feb27e00700 1 -- 172.21.15.102:0/3741951733 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb20003ba0 con 0x7feb3007dd60 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.742+0000 7feb27e00700 1 -- 172.21.15.102:0/3741951733 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb2001a910 con 0x7feb3007dd60 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.742+0000 7feb27e00700 1 -- 172.21.15.102:0/3741951733 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb200248f0 con 0x7feb3007dd60 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.742+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb3019e7d0 con 0x7feb3007dd60 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.742+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb3019f2f0 con 0x7feb3007dd60 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.743+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb30063890 con 0x7feb3007dd60 2024-04-25T01:44:34.760 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.744+0000 7feb27e00700 1 -- 172.21.15.102:0/3741951733 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7feb20022020 con 0x7feb3007dd60 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.744+0000 7feb27e00700 1 --2- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7feb140390a0 0x7feb1403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.744+0000 7feb27e00700 1 -- 172.21.15.102:0/3741951733 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7feb2004d450 con 0x7feb3007dd60 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.744+0000 7feb2ea00700 1 --2- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7feb140390a0 0x7feb1403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.745+0000 7feb2ea00700 1 --2- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7feb140390a0 0x7feb1403b560 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb18006fd0 tx=0x7feb18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:32.750+0000 7feb27e00700 1 -- 172.21.15.102:0/3741951733 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb2000b3e0 con 0x7feb3007dd60 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.032+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7feb3019f6e0 con 0x7feb3007dd60 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.589+0000 7feb27e00700 1 -- 172.21.15.102:0/3741951733 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7feb2001ab70 con 0x7feb3007dd60 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.593+0000 7feb27e00700 1 -- 172.21.15.102:0/3741951733 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7feb20024a90 con 0x7feb3007dd60 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7feb140390a0 msgr2=0x7feb1403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 --2- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7feb140390a0 0x7feb1403b560 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb18006fd0 tx=0x7feb18006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 msgr2=0x7feb3019e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 --2- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3019e000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb200117e0 tx=0x7feb20011ba0 comp rx=0 tx=0).stop 2024-04-25T01:44:34.761 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 shutdown_connections 2024-04-25T01:44:34.762 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 --2- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7feb140390a0 0x7feb1403b560 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:34.762 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 --2- 172.21.15.102:0/3741951733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3007dd60 0x7feb3019e000 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:34.762 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 >> 172.21.15.102:0/3741951733 conn(0x7feb30103420 msgr2=0x7feb30103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:34.762 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 shutdown_connections 2024-04-25T01:44:34.762 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:33.595+0000 7feb35c00700 1 -- 172.21.15.102:0/3741951733 wait complete. 2024-04-25T01:44:34.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:34 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3741951733' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T01:44:34.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:34 smithi102 ceph-mon[27796]: mgrmap e9: x(active, since 25s) 2024-04-25T01:44:34.903 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:34 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:34.660+0000 7fd2a5647200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:44:34.903 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:34 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:34.776+0000 7fd2a5647200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:44:36.567 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:36 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2961446287' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T01:44:36.567 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:36 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:36.342+0000 7fd2a5647200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:44:36.567 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:36 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:36.575+0000 7fd2a5647200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:44:36.889 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:36 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:36.793+0000 7fd2a5647200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:44:36.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:36.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T01:44:36.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T01:44:36.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T01:44:36.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T01:44:36.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:36.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.908+0000 7f77da400700 1 Processor -- start 2024-04-25T01:44:36.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.908+0000 7f77da400700 1 -- start start 2024-04-25T01:44:36.904 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.908+0000 7f77da400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d4108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.908+0000 7f77da400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4108850 con 0x7f77d4107e60 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.909+0000 7f77d3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d4108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.909+0000 7f77d3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d4108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39246/0 (socket says 172.21.15.102:39246) 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.909+0000 7f77d3400700 1 -- 172.21.15.102:0/663546725 learned_addr learned my addr 172.21.15.102:0/663546725 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.909+0000 7f77d3400700 1 -- 172.21.15.102:0/663546725 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d4108fc0 con 0x7f77d4107e60 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.910+0000 7f77d3400700 1 --2- 172.21.15.102:0/663546725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d4108280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77c40098a0 tx=0x7f77c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56014ab4859b547e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.910+0000 7f77d2000700 1 -- 172.21.15.102:0/663546725 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c4004160 con 0x7f77d4107e60 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.910+0000 7f77d2000700 1 -- 172.21.15.102:0/663546725 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c4004300 con 0x7f77d4107e60 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.910+0000 7f77d2000700 1 -- 172.21.15.102:0/663546725 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c4011b20 con 0x7f77d4107e60 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.911+0000 7f77da400700 1 -- 172.21.15.102:0/663546725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 msgr2=0x7f77d4108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.911+0000 7f77da400700 1 --2- 172.21.15.102:0/663546725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d4108280 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77c40098a0 tx=0x7f77c4009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.911+0000 7f77da400700 1 -- 172.21.15.102:0/663546725 shutdown_connections 2024-04-25T01:44:36.905 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.911+0000 7f77da400700 1 --2- 172.21.15.102:0/663546725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d4108280 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.911+0000 7f77da400700 1 -- 172.21.15.102:0/663546725 >> 172.21.15.102:0/663546725 conn(0x7f77d4103420 msgr2=0x7f77d4105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.911+0000 7f77da400700 1 -- 172.21.15.102:0/663546725 shutdown_connections 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.911+0000 7f77da400700 1 -- 172.21.15.102:0/663546725 wait complete. 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.912+0000 7f77da400700 1 Processor -- start 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.913+0000 7f77da400700 1 -- start start 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.913+0000 7f77da400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d419e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.913+0000 7f77da400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f77d410c370 con 0x7f77d4107e60 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.913+0000 7f77d3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d419e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.913+0000 7f77d3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d419e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39250/0 (socket says 172.21.15.102:39250) 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.913+0000 7f77d3400700 1 -- 172.21.15.102:0/2961446287 learned_addr learned my addr 172.21.15.102:0/2961446287 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.914+0000 7f77d3400700 1 -- 172.21.15.102:0/2961446287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d419e570 con 0x7f77d4107e60 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.914+0000 7f77d3400700 1 --2- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d419e030 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f77c40117e0 tx=0x7f77c4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.914+0000 7f77cbe00700 1 -- 172.21.15.102:0/2961446287 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c4003bf0 con 0x7f77d4107e60 2024-04-25T01:44:36.906 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.914+0000 7f77cbe00700 1 -- 172.21.15.102:0/2961446287 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c401a910 con 0x7f77d4107e60 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.914+0000 7f77cbe00700 1 -- 172.21.15.102:0/2961446287 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f77c4024970 con 0x7f77d4107e60 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.914+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d419e800 con 0x7f77d4107e60 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.915+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d419f320 con 0x7f77d4107e60 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.916+0000 7f77cbe00700 1 -- 172.21.15.102:0/2961446287 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f77c4022020 con 0x7f77d4107e60 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.916+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77d4063890 con 0x7f77d4107e60 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.916+0000 7f77cbe00700 1 --2- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f77b8039100 0x7f77b803b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.916+0000 7f77cbe00700 1 -- 172.21.15.102:0/2961446287 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f77c404da60 con 0x7f77d4107e60 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.916+0000 7f77d2a00700 1 -- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f77b8039100 msgr2=0x7f77b803b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1124203809 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.916+0000 7f77d2a00700 1 --2- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f77b8039100 0x7f77b803b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:44:36.907 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:35.921+0000 7f77cbe00700 1 -- 172.21.15.102:0/2961446287 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f77c402cdf0 con 0x7f77d4107e60 2024-04-25T01:44:36.908 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.117+0000 7f77d2a00700 1 -- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f77b8039100 msgr2=0x7f77b803b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1124203809 2024-04-25T01:44:36.908 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.117+0000 7f77d2a00700 1 --2- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f77b8039100 0x7f77b803b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:44:36.908 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.202+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f77d419f760 con 0x7f77d4107e60 2024-04-25T01:44:36.908 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.202+0000 7f77cbe00700 1 -- 172.21.15.102:0/2961446287 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f77c400b5a0 con 0x7f77d4107e60 2024-04-25T01:44:36.908 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f77b8039100 msgr2=0x7f77b803b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 --2- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f77b8039100 0x7f77b803b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 msgr2=0x7f77d419e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 --2- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d419e030 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f77c40117e0 tx=0x7f77c4011ba0 comp rx=0 tx=0).stop 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 shutdown_connections 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 --2- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7f77b8039100 0x7f77b803b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 --2- 172.21.15.102:0/2961446287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77d4107e60 0x7f77d419e030 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 >> 172.21.15.102:0/2961446287 conn(0x7f77d4103420 msgr2=0x7f77d4103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 shutdown_connections 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:36.205+0000 7f77da400700 1 -- 172.21.15.102:0/2961446287 wait complete. 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-04-25T01:44:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 9... 2024-04-25T01:44:37.446 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:37 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:37.183+0000 7fd2a5647200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:44:37.447 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:37 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:37.312+0000 7fd2a5647200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:44:37.705 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:37 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:37.572+0000 7fd2a5647200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:44:38.106 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:37 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:37.824+0000 7fd2a5647200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:44:38.907 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:38 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:38.604+0000 7fd2a5647200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:44:39.164 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:39.156+0000 7fd2a5647200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:44:39.541 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:39.282+0000 7fd2a5647200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:44:39.541 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:39.548+0000 7fd2a5647200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:44:40.508 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:40 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:40.216+0000 7fd2a5647200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:44:40.508 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:40 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:40.340+0000 7fd2a5647200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:44:40.830 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:40 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:40.661+0000 7fd2a5647200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:44:40.830 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:40 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:40.837+0000 7fd2a5647200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:44:41.111 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:40 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:40.965+0000 7fd2a5647200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:44:41.111 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:41 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:41.118+0000 7fd2a5647200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:44:41.414 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:41 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:41.246+0000 7fd2a5647200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:44:42.570 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:42.296+0000 7fd2a5647200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:44:42.834 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:44:42.578+0000 7fd2a5647200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:44:42.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: Active manager daemon x restarted 2024-04-25T01:44:42.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: Activating manager daemon x 2024-04-25T01:44:42.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T01:44:42.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: mgrmap e10: x(active, starting, since 0.0090482s) 2024-04-25T01:44:42.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:44:42.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:44:42.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:44:42.835 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:44:42.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:44:42.836 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:42 smithi102 ceph-mon[27796]: Manager daemon x is now available 2024-04-25T01:44:43.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:43.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:44:43.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:44:43.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:43 smithi102 ceph-mon[27796]: mgrmap e11: x(active, since 1.01061s) 2024-04-25T01:44:44.516 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-04-25T01:44:44.517 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.504+0000 7fca6ca00700 1 Processor -- start 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.504+0000 7fca6ca00700 1 -- start start 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.505+0000 7fca6ca00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca68107110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.505+0000 7fca6ca00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fca681076e0 con 0x7fca68106cf0 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.505+0000 7fca65e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca68107110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.505+0000 7fca65e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca68107110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39262/0 (socket says 172.21.15.102:39262) 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.505+0000 7fca65e00700 1 -- 172.21.15.102:0/3279610983 learned_addr learned my addr 172.21.15.102:0/3279610983 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:44.518 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.506+0000 7fca65e00700 1 -- 172.21.15.102:0/3279610983 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca68107ef0 con 0x7fca68106cf0 2024-04-25T01:44:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.506+0000 7fca65e00700 1 --2- 172.21.15.102:0/3279610983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca68107110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca580098a0 tx=0x7fca58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b7d5a0f175d9d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.506+0000 7fca64a00700 1 -- 172.21.15.102:0/3279610983 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca58004160 con 0x7fca68106cf0 2024-04-25T01:44:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.506+0000 7fca64a00700 1 -- 172.21.15.102:0/3279610983 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca58004300 con 0x7fca68106cf0 2024-04-25T01:44:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.506+0000 7fca64a00700 1 -- 172.21.15.102:0/3279610983 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca58011b20 con 0x7fca68106cf0 2024-04-25T01:44:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.507+0000 7fca6ca00700 1 -- 172.21.15.102:0/3279610983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 msgr2=0x7fca68107110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.507+0000 7fca6ca00700 1 --2- 172.21.15.102:0/3279610983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca68107110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca580098a0 tx=0x7fca58009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.507+0000 7fca6ca00700 1 -- 172.21.15.102:0/3279610983 shutdown_connections 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.507+0000 7fca6ca00700 1 --2- 172.21.15.102:0/3279610983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca68107110 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.507+0000 7fca6ca00700 1 -- 172.21.15.102:0/3279610983 >> 172.21.15.102:0/3279610983 conn(0x7fca681022d0 msgr2=0x7fca681046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.507+0000 7fca6ca00700 1 -- 172.21.15.102:0/3279610983 shutdown_connections 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.507+0000 7fca6ca00700 1 -- 172.21.15.102:0/3279610983 wait complete. 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.508+0000 7fca6ca00700 1 Processor -- start 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.508+0000 7fca6ca00700 1 -- start start 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.509+0000 7fca6ca00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca6819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.509+0000 7fca6ca00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fca6807af70 con 0x7fca68106cf0 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.509+0000 7fca65e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca6819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.509+0000 7fca65e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca6819dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39278/0 (socket says 172.21.15.102:39278) 2024-04-25T01:44:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.509+0000 7fca65e00700 1 -- 172.21.15.102:0/4188565299 learned_addr learned my addr 172.21.15.102:0/4188565299 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.509+0000 7fca65e00700 1 -- 172.21.15.102:0/4188565299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca6819e4e0 con 0x7fca68106cf0 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.510+0000 7fca65e00700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca6819dfa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca58000c00 tx=0x7fca58011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.510+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca58003b60 con 0x7fca68106cf0 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.510+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca5801a910 con 0x7fca68106cf0 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.510+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca58024930 con 0x7fca68106cf0 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.510+0000 7fca6ca00700 1 -- 172.21.15.102:0/4188565299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca6819e770 con 0x7fca68106cf0 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.510+0000 7fca6ca00700 1 -- 172.21.15.102:0/4188565299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca6819ebd0 con 0x7fca68106cf0 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.511+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fca58022020 con 0x7fca68106cf0 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.512+0000 7fca5ea00700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 0x7fca4c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:44.526 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.512+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 --> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fca4c03bcc0 con 0x7fca4c0390f0 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.512+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fca5804d1f0 con 0x7fca68106cf0 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.512+0000 7fca65400700 1 -- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 msgr2=0x7fca4c03b5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1124203809 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.512+0000 7fca65400700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 0x7fca4c03b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.713+0000 7fca65400700 1 -- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 msgr2=0x7fca4c03b5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1124203809 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:37.713+0000 7fca65400700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 0x7fca4c03b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:38.114+0000 7fca65400700 1 -- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 msgr2=0x7fca4c03b5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1124203809 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:38.114+0000 7fca65400700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 0x7fca4c03b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:38.915+0000 7fca65400700 1 -- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 msgr2=0x7fca4c03b5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1124203809 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:38.915+0000 7fca65400700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 0x7fca4c03b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:40.516+0000 7fca65400700 1 -- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 msgr2=0x7fca4c03b5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/1124203809 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:40.516+0000 7fca65400700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 0x7fca4c03b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T01:44:44.527 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.587+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7fca5802d3f0 con 0x7fca68106cf0 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.587+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 msgr2=0x7fca4c03b5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:42.587+0000 7fca5ea00700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 0x7fca4c03b5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.588+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7fca5804edd0 con 0x7fca68106cf0 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.588+0000 7fca5ea00700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fca4c03cbb0 0x7fca4c03efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.589+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fca58008f30 con 0x7fca4c03cbb0 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.589+0000 7fca65400700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fca4c03cbb0 0x7fca4c03efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.591+0000 7fca65400700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fca4c03cbb0 0x7fca4c03efa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fca54003a10 tx=0x7fca54009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.592+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fca58008f30 con 0x7fca4c03cbb0 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.600+0000 7fca6ca00700 1 -- 172.21.15.102:0/4188565299 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fca68069390 con 0x7fca4c03cbb0 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.600+0000 7fca5ea00700 1 -- 172.21.15.102:0/4188565299 <== mgr.14152 v2:172.21.15.102:6800/83904525 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fca68069390 con 0x7fca4c03cbb0 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.600+0000 7fca6ca00700 1 -- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fca4c03cbb0 msgr2=0x7fca4c03efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.600+0000 7fca6ca00700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fca4c03cbb0 0x7fca4c03efa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fca54003a10 tx=0x7fca54009270 comp rx=0 tx=0).stop 2024-04-25T01:44:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.600+0000 7fca6ca00700 1 -- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 msgr2=0x7fca6819dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.601+0000 7fca6ca00700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca6819dfa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca58000c00 tx=0x7fca58011c10 comp rx=0 tx=0).stop 2024-04-25T01:44:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.601+0000 7fca6ca00700 1 -- 172.21.15.102:0/4188565299 shutdown_connections 2024-04-25T01:44:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.601+0000 7fca6ca00700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fca4c03cbb0 0x7fca4c03efa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.601+0000 7fca6ca00700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca68106cf0 0x7fca6819dfa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.601+0000 7fca6ca00700 1 --2- 172.21.15.102:0/4188565299 >> [v2:172.21.15.102:6800/1124203809,v1:172.21.15.102:6801/1124203809] conn(0x7fca4c0390f0 0x7fca4c03b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.601+0000 7fca6ca00700 1 -- 172.21.15.102:0/4188565299 >> 172.21.15.102:0/4188565299 conn(0x7fca681022d0 msgr2=0x7fca681945e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.601+0000 7fca6ca00700 1 -- 172.21.15.102:0/4188565299 shutdown_connections 2024-04-25T01:44:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:43.601+0000 7fca6ca00700 1 -- 172.21.15.102:0/4188565299 wait complete. 2024-04-25T01:44:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 9 is available 2024-04-25T01:44:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:Generating a dashboard self-signed certificate... 2024-04-25T01:44:44.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:44 smithi102 ceph-mon[27796]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T01:44:44.778 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:44 smithi102 ceph-mon[27796]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T01:44:45.914 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:45 smithi102 ceph-mon[27796]: [25/Apr/2024:01:44:44] ENGINE Bus STARTING 2024-04-25T01:44:45.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:45 smithi102 ceph-mon[27796]: mgrmap e12: x(active, since 2s) 2024-04-25T01:44:45.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:45 smithi102 ceph-mon[27796]: [25/Apr/2024:01:44:44] ENGINE Serving on https://172.21.15.102:7150 2024-04-25T01:44:45.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:45 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:44:46.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:46 smithi102 ceph-mon[27796]: [25/Apr/2024:01:44:44] ENGINE Serving on http://172.21.15.102:8765 2024-04-25T01:44:46.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:46 smithi102 ceph-mon[27796]: [25/Apr/2024:01:44:44] ENGINE Bus STARTED 2024-04-25T01:44:46.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:46 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:44:46.790 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:46 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:44:46.804 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T01:44:46.804 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.716+0000 7f11e2600700 1 Processor -- start 2024-04-25T01:44:46.804 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.716+0000 7f11e2600700 1 -- start start 2024-04-25T01:44:46.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.717+0000 7f11e2600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc07c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:46.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.717+0000 7f11e2600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc07c850 con 0x7f11dc07de20 2024-04-25T01:44:46.809 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.717+0000 7f11e1200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc07c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.717+0000 7f11e1200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc07c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43464/0 (socket says 172.21.15.102:43464) 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.717+0000 7f11e1200700 1 -- 172.21.15.102:0/1229790780 learned_addr learned my addr 172.21.15.102:0/1229790780 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.718+0000 7f11e1200700 1 -- 172.21.15.102:0/1229790780 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc07c9d0 con 0x7f11dc07de20 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.718+0000 7f11e1200700 1 --2- 172.21.15.102:0/1229790780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc07c280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f11d00098a0 tx=0x7f11d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d23ac9f8dba995 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.718+0000 7f11db400700 1 -- 172.21.15.102:0/1229790780 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11d0004160 con 0x7f11dc07de20 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.718+0000 7f11db400700 1 -- 172.21.15.102:0/1229790780 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d0004300 con 0x7f11dc07de20 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.718+0000 7f11db400700 1 -- 172.21.15.102:0/1229790780 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11d0011b20 con 0x7f11dc07de20 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.719+0000 7f11e2600700 1 -- 172.21.15.102:0/1229790780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 msgr2=0x7f11dc07c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.719+0000 7f11e2600700 1 --2- 172.21.15.102:0/1229790780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc07c280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f11d00098a0 tx=0x7f11d0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.719+0000 7f11e2600700 1 -- 172.21.15.102:0/1229790780 shutdown_connections 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.719+0000 7f11e2600700 1 --2- 172.21.15.102:0/1229790780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc07c280 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.719+0000 7f11e2600700 1 -- 172.21.15.102:0/1229790780 >> 172.21.15.102:0/1229790780 conn(0x7f11dc1031d0 msgr2=0x7f11dc105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:46.810 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.720+0000 7f11e2600700 1 -- 172.21.15.102:0/1229790780 shutdown_connections 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.720+0000 7f11e2600700 1 -- 172.21.15.102:0/1229790780 wait complete. 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.721+0000 7f11e2600700 1 Processor -- start 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.721+0000 7f11e2600700 1 -- start start 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.721+0000 7f11e2600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.721+0000 7f11e2600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc10e2b0 con 0x7f11dc07de20 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.722+0000 7f11e1200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.722+0000 7f11e1200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43478/0 (socket says 172.21.15.102:43478) 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.722+0000 7f11e1200700 1 -- 172.21.15.102:0/2192042413 learned_addr learned my addr 172.21.15.102:0/2192042413 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.722+0000 7f11e1200700 1 -- 172.21.15.102:0/2192042413 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc19e2e0 con 0x7f11dc07de20 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.722+0000 7f11e1200700 1 --2- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc19dda0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f11d0011b60 tx=0x7f11d0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.723+0000 7f11d9600700 1 -- 172.21.15.102:0/2192042413 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11d0003b60 con 0x7f11dc07de20 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.723+0000 7f11d9600700 1 -- 172.21.15.102:0/2192042413 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d001a910 con 0x7f11dc07de20 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.723+0000 7f11d9600700 1 -- 172.21.15.102:0/2192042413 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f11d0024930 con 0x7f11dc07de20 2024-04-25T01:44:46.811 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.723+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11dc19e570 con 0x7f11dc07de20 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.723+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11dc1a1220 con 0x7f11dc07de20 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.724+0000 7f11d9600700 1 -- 172.21.15.102:0/2192042413 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f11d0022020 con 0x7f11dc07de20 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.724+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11dc069390 con 0x7f11dc07de20 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.724+0000 7f11d9600700 1 --2- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f11c4039100 0x7f11c403b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.725+0000 7f11d9600700 1 -- 172.21.15.102:0/2192042413 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f11d004d330 con 0x7f11dc07de20 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.725+0000 7f11dbe00700 1 --2- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f11c4039100 0x7f11c403b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.725+0000 7f11dbe00700 1 --2- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f11c4039100 0x7f11c403b5c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f11c8006fd0 tx=0x7f11c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.735+0000 7f11d9600700 1 -- 172.21.15.102:0/2192042413 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f11d002ddb0 con 0x7f11dc07de20 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:45.965+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f11dc104350 con 0x7f11c4039100 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.108+0000 7f11d9600700 1 -- 172.21.15.102:0/2192042413 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f11dc104350 con 0x7f11c4039100 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.111+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f11c4039100 msgr2=0x7f11c403b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:46.812 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.111+0000 7f11e2600700 1 --2- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f11c4039100 0x7f11c403b5c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f11c8006fd0 tx=0x7f11c8006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:46.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.111+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 msgr2=0x7f11dc19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:46.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.111+0000 7f11e2600700 1 --2- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc19dda0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f11d0011b60 tx=0x7f11d0011c40 comp rx=0 tx=0).stop 2024-04-25T01:44:46.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.111+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 shutdown_connections 2024-04-25T01:44:46.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.111+0000 7f11e2600700 1 --2- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f11c4039100 0x7f11c403b5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:46.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.111+0000 7f11e2600700 1 --2- 172.21.15.102:0/2192042413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11dc07de20 0x7f11dc19dda0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:46.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.111+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 >> 172.21.15.102:0/2192042413 conn(0x7f11dc1031d0 msgr2=0x7f11dc103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:46.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.111+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 shutdown_connections 2024-04-25T01:44:46.813 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:46.112+0000 7f11e2600700 1 -- 172.21.15.102:0/2192042413 wait complete. 2024-04-25T01:44:46.813 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial admin user... 2024-04-25T01:44:47.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:47 smithi102 ceph-mon[27796]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:47.664 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:47 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:44:48.563 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$YsVxY3yzx2cG.6oQGdZOs.d7jwVyE8ngV7CAcjfYboShoS6aRDe1S", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714009487, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T01:44:48.564 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.414+0000 7f1136800700 1 Processor -- start 2024-04-25T01:44:48.564 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.414+0000 7f1136800700 1 -- start start 2024-04-25T01:44:48.564 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.414+0000 7f1136800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f1130108210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:48.564 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.415+0000 7f1136800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f11301087e0 con 0x7f1130107df0 2024-04-25T01:44:48.564 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.415+0000 7f112f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f1130108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:48.564 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.415+0000 7f112f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f1130108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43480/0 (socket says 172.21.15.102:43480) 2024-04-25T01:44:48.564 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.415+0000 7f112f400700 1 -- 172.21.15.102:0/61021749 learned_addr learned my addr 172.21.15.102:0/61021749 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.416+0000 7f112f400700 1 -- 172.21.15.102:0/61021749 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1130108ff0 con 0x7f1130107df0 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.416+0000 7f112f400700 1 --2- 172.21.15.102:0/61021749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f1130108210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f11200098a0 tx=0x7f1120009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aabc58fabbfca43b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.416+0000 7f112e000700 1 -- 172.21.15.102:0/61021749 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1120004160 con 0x7f1130107df0 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.416+0000 7f112e000700 1 -- 172.21.15.102:0/61021749 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1120004300 con 0x7f1130107df0 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.416+0000 7f112e000700 1 -- 172.21.15.102:0/61021749 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1120011b20 con 0x7f1130107df0 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.417+0000 7f1136800700 1 -- 172.21.15.102:0/61021749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 msgr2=0x7f1130108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.417+0000 7f1136800700 1 --2- 172.21.15.102:0/61021749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f1130108210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f11200098a0 tx=0x7f1120009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.417+0000 7f1136800700 1 -- 172.21.15.102:0/61021749 shutdown_connections 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.417+0000 7f1136800700 1 --2- 172.21.15.102:0/61021749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f1130108210 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.417+0000 7f1136800700 1 -- 172.21.15.102:0/61021749 >> 172.21.15.102:0/61021749 conn(0x7f1130103370 msgr2=0x7f11301057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:48.566 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.418+0000 7f1136800700 1 -- 172.21.15.102:0/61021749 shutdown_connections 2024-04-25T01:44:48.567 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.418+0000 7f1136800700 1 -- 172.21.15.102:0/61021749 wait complete. 2024-04-25T01:44:48.567 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.419+0000 7f1136800700 1 Processor -- start 2024-04-25T01:44:48.574 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.419+0000 7f1136800700 1 -- start start 2024-04-25T01:44:48.574 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.419+0000 7f1136800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f113019be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:48.574 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.419+0000 7f1136800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f113010c3a0 con 0x7f1130107df0 2024-04-25T01:44:48.574 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.420+0000 7f112f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f113019be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:48.574 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.420+0000 7f112f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f113019be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43488/0 (socket says 172.21.15.102:43488) 2024-04-25T01:44:48.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.420+0000 7f112f400700 1 -- 172.21.15.102:0/3775268156 learned_addr learned my addr 172.21.15.102:0/3775268156 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:48.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.420+0000 7f112f400700 1 -- 172.21.15.102:0/3775268156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113019c390 con 0x7f1130107df0 2024-04-25T01:44:48.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.420+0000 7f112f400700 1 --2- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f113019be50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f11200117e0 tx=0x7f1120011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:48.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.420+0000 7f1127e00700 1 -- 172.21.15.102:0/3775268156 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1120003bf0 con 0x7f1130107df0 2024-04-25T01:44:48.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.421+0000 7f1127e00700 1 -- 172.21.15.102:0/3775268156 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f112001a910 con 0x7f1130107df0 2024-04-25T01:44:48.575 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.421+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113019c620 con 0x7f1130107df0 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.421+0000 7f1127e00700 1 -- 172.21.15.102:0/3775268156 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1120024970 con 0x7f1130107df0 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.421+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f113019d090 con 0x7f1130107df0 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.422+0000 7f1127e00700 1 -- 172.21.15.102:0/3775268156 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f1120022020 con 0x7f1130107df0 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.422+0000 7f1127e00700 1 --2- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1114039100 0x7f111403b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.422+0000 7f1127e00700 1 -- 172.21.15.102:0/3775268156 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f112004d5b0 con 0x7f1130107df0 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.423+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1130063890 con 0x7f1130107df0 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.423+0000 7f112ea00700 1 --2- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1114039100 0x7f111403b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.424+0000 7f112ea00700 1 --2- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1114039100 0x7f111403b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f111c006fd0 tx=0x7f111c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.429+0000 7f1127e00700 1 -- 172.21.15.102:0/3775268156 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f112002cdf0 con 0x7f1130107df0 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.691+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f1130104540 con 0x7f1114039100 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.933+0000 7f1127e00700 1 -- 172.21.15.102:0/3775268156 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f1130104540 con 0x7f1114039100 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1114039100 msgr2=0x7f111403b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:48.576 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 --2- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1114039100 0x7f111403b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f111c006fd0 tx=0x7f111c006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:48.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 msgr2=0x7f113019be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:48.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 --2- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f113019be50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f11200117e0 tx=0x7f1120011ba0 comp rx=0 tx=0).stop 2024-04-25T01:44:48.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 shutdown_connections 2024-04-25T01:44:48.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 --2- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1114039100 0x7f111403b5c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:48.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 --2- 172.21.15.102:0/3775268156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1130107df0 0x7f113019be50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:48.577 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 >> 172.21.15.102:0/3775268156 conn(0x7f1130103370 msgr2=0x7f1130103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:48.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 shutdown_connections 2024-04-25T01:44:48.578 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:47.936+0000 7f1136800700 1 -- 172.21.15.102:0/3775268156 wait complete. 2024-04-25T01:44:48.578 INFO:teuthology.orchestra.run.smithi102.stdout:Fetching dashboard port number... 2024-04-25T01:44:48.858 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:48 smithi102 ceph-mon[27796]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:44:48.858 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:44:50.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:49 smithi102 ceph-mon[27796]: mgrmap e13: x(active, since 6s) 2024-04-25T01:44:50.127 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:49 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3688244076' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T01:44:50.984 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T01:44:50.984 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.215+0000 7fb83fe00700 1 Processor -- start 2024-04-25T01:44:50.984 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.215+0000 7fb83fe00700 1 -- start start 2024-04-25T01:44:50.984 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.216+0000 7fb83fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb838111060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:50.984 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.216+0000 7fb83fe00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb8381115a0 con 0x7fb83810ec70 2024-04-25T01:44:50.984 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.216+0000 7fb83d400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb838111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.216+0000 7fb83d400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb838111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43490/0 (socket says 172.21.15.102:43490) 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.216+0000 7fb83d400700 1 -- 172.21.15.102:0/1918899602 learned_addr learned my addr 172.21.15.102:0/1918899602 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.216+0000 7fb83d400700 1 -- 172.21.15.102:0/1918899602 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb838111720 con 0x7fb83810ec70 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.217+0000 7fb83d400700 1 --2- 172.21.15.102:0/1918899602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb838111060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0098a0 tx=0x7fb82c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94effe8a5ca2d7da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.217+0000 7fb837e00700 1 -- 172.21.15.102:0/1918899602 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb82c004160 con 0x7fb83810ec70 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.217+0000 7fb837e00700 1 -- 172.21.15.102:0/1918899602 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb82c004300 con 0x7fb83810ec70 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.217+0000 7fb837e00700 1 -- 172.21.15.102:0/1918899602 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb82c011b20 con 0x7fb83810ec70 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.218+0000 7fb83fe00700 1 -- 172.21.15.102:0/1918899602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 msgr2=0x7fb838111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.218+0000 7fb83fe00700 1 --2- 172.21.15.102:0/1918899602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb838111060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0098a0 tx=0x7fb82c009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.218+0000 7fb83fe00700 1 -- 172.21.15.102:0/1918899602 shutdown_connections 2024-04-25T01:44:50.985 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.218+0000 7fb83fe00700 1 --2- 172.21.15.102:0/1918899602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb838111060 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.218+0000 7fb83fe00700 1 -- 172.21.15.102:0/1918899602 >> 172.21.15.102:0/1918899602 conn(0x7fb83807d3c0 msgr2=0x7fb83807f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.218+0000 7fb83fe00700 1 -- 172.21.15.102:0/1918899602 shutdown_connections 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.218+0000 7fb83fe00700 1 -- 172.21.15.102:0/1918899602 wait complete. 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.219+0000 7fb83fe00700 1 Processor -- start 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.219+0000 7fb83fe00700 1 -- start start 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.220+0000 7fb83fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb8381a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.220+0000 7fb83fe00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb838112840 con 0x7fb83810ec70 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.220+0000 7fb83d400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb8381a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.220+0000 7fb83d400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb8381a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43502/0 (socket says 172.21.15.102:43502) 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.220+0000 7fb83d400700 1 -- 172.21.15.102:0/3688244076 learned_addr learned my addr 172.21.15.102:0/3688244076 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.221+0000 7fb83d400700 1 -- 172.21.15.102:0/3688244076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8381a2910 con 0x7fb83810ec70 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.221+0000 7fb83d400700 1 --2- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb8381a23d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0117e0 tx=0x7fb82c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:50.986 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.221+0000 7fb836000700 1 -- 172.21.15.102:0/3688244076 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb82c003ba0 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.221+0000 7fb836000700 1 -- 172.21.15.102:0/3688244076 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb82c01a910 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.221+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8381a2ba0 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.221+0000 7fb836000700 1 -- 172.21.15.102:0/3688244076 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb82c0248f0 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.221+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8381a36c0 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.222+0000 7fb836000700 1 -- 172.21.15.102:0/3688244076 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fb82c022020 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.223+0000 7fb836000700 1 --2- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb8200391a0 0x7fb82003b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.223+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb838063890 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.223+0000 7fb836000700 1 -- 172.21.15.102:0/3688244076 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb82c04e6c0 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.224+0000 7fb83ca00700 1 --2- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb8200391a0 0x7fb82003b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.225+0000 7fb83ca00700 1 --2- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb8200391a0 0x7fb82003b660 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb824006fd0 tx=0x7fb824006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.229+0000 7fb836000700 1 -- 172.21.15.102:0/3688244076 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb82c020c10 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.461+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fb8381a3ad0 con 0x7fb83810ec70 2024-04-25T01:44:50.987 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.462+0000 7fb836000700 1 -- 172.21.15.102:0/3688244076 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fb82c00b5a0 con 0x7fb83810ec70 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb8200391a0 msgr2=0x7fb82003b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 --2- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb8200391a0 0x7fb82003b660 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb824006fd0 tx=0x7fb824006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 msgr2=0x7fb8381a23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 --2- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb8381a23d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb82c0117e0 tx=0x7fb82c011ba0 comp rx=0 tx=0).stop 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 shutdown_connections 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 --2- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb8200391a0 0x7fb82003b660 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 --2- 172.21.15.102:0/3688244076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb83810ec70 0x7fb8381a23d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 >> 172.21.15.102:0/3688244076 conn(0x7fb83807d3c0 msgr2=0x7fb83807dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 shutdown_connections 2024-04-25T01:44:50.988 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:49.466+0000 7fb83fe00700 1 -- 172.21.15.102:0/3688244076 wait complete. 2024-04-25T01:44:51.010 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T01:44:51.010 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-04-25T01:44:51.032 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T01:44:51.032 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-04-25T01:44:51.032 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-04-25T01:44:51.032 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T01:44:51.040 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph Dashboard is now available at: 2024-04-25T01:44:51.040 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:51.040 INFO:teuthology.orchestra.run.smithi102.stdout: URL: https://smithi102.front.sepia.ceph.com:8443/ 2024-04-25T01:44:51.040 INFO:teuthology.orchestra.run.smithi102.stdout: User: admin 2024-04-25T01:44:51.040 INFO:teuthology.orchestra.run.smithi102.stdout: Password: 3giu7b20e2 2024-04-25T01:44:51.040 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:51.041 INFO:teuthology.orchestra.run.smithi102.stdout:Saving cluster configuration to /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config directory 2024-04-25T01:44:51.043 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling autotune for osd_memory_target 2024-04-25T01:44:52.666 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.705+0000 7f12f6e00700 1 Processor -- start 2024-04-25T01:44:52.667 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.705+0000 7f12f6e00700 1 -- start start 2024-04-25T01:44:52.667 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.705+0000 7f12f6e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f010a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:52.667 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.705+0000 7f12f6e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f12f010a9e0 con 0x7f12f0109ff0 2024-04-25T01:44:52.667 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.706+0000 7f12efe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f010a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:52.667 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.706+0000 7f12efe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f010a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43514/0 (socket says 172.21.15.102:43514) 2024-04-25T01:44:52.667 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.706+0000 7f12efe00700 1 -- 172.21.15.102:0/1119126017 learned_addr learned my addr 172.21.15.102:0/1119126017 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:52.667 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.706+0000 7f12efe00700 1 -- 172.21.15.102:0/1119126017 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f010b1f0 con 0x7f12f0109ff0 2024-04-25T01:44:52.667 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.707+0000 7f12efe00700 1 --2- 172.21.15.102:0/1119126017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f010a410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f12e00098a0 tx=0x7f12e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bb9500e77af758 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.707+0000 7f12eea00700 1 -- 172.21.15.102:0/1119126017 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12e0004160 con 0x7f12f0109ff0 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.707+0000 7f12eea00700 1 -- 172.21.15.102:0/1119126017 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e0004300 con 0x7f12f0109ff0 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.707+0000 7f12eea00700 1 -- 172.21.15.102:0/1119126017 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12e0011b20 con 0x7f12f0109ff0 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.708+0000 7f12f6e00700 1 -- 172.21.15.102:0/1119126017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 msgr2=0x7f12f010a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.708+0000 7f12f6e00700 1 --2- 172.21.15.102:0/1119126017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f010a410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f12e00098a0 tx=0x7f12e0009bb0 comp rx=0 tx=0).stop 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.708+0000 7f12f6e00700 1 -- 172.21.15.102:0/1119126017 shutdown_connections 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.708+0000 7f12f6e00700 1 --2- 172.21.15.102:0/1119126017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f010a410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.708+0000 7f12f6e00700 1 -- 172.21.15.102:0/1119126017 >> 172.21.15.102:0/1119126017 conn(0x7f12f007d3c0 msgr2=0x7f12f007d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.708+0000 7f12f6e00700 1 -- 172.21.15.102:0/1119126017 shutdown_connections 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.708+0000 7f12f6e00700 1 -- 172.21.15.102:0/1119126017 wait complete. 2024-04-25T01:44:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.709+0000 7f12f6e00700 1 Processor -- start 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.709+0000 7f12f6e00700 1 -- start start 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.710+0000 7f12f6e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.710+0000 7f12f6e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f12f010e5a0 con 0x7f12f0109ff0 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.710+0000 7f12efe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.710+0000 7f12efe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f019e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43522/0 (socket says 172.21.15.102:43522) 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.710+0000 7f12efe00700 1 -- 172.21.15.102:0/827453953 learned_addr learned my addr 172.21.15.102:0/827453953 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.710+0000 7f12efe00700 1 -- 172.21.15.102:0/827453953 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f019e550 con 0x7f12f0109ff0 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.711+0000 7f12efe00700 1 --2- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f019e010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12e00117e0 tx=0x7f12e0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.711+0000 7f12ecc00700 1 -- 172.21.15.102:0/827453953 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12e0003bf0 con 0x7f12f0109ff0 2024-04-25T01:44:52.669 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.711+0000 7f12ecc00700 1 -- 172.21.15.102:0/827453953 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12e001a910 con 0x7f12f0109ff0 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.711+0000 7f12ecc00700 1 -- 172.21.15.102:0/827453953 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12e0024970 con 0x7f12f0109ff0 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.711+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f019e7e0 con 0x7f12f0109ff0 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.711+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f019f300 con 0x7f12f0109ff0 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.712+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f0063890 con 0x7f12f0109ff0 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.712+0000 7f12ecc00700 1 -- 172.21.15.102:0/827453953 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f12e0022020 con 0x7f12f0109ff0 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.713+0000 7f12ecc00700 1 --2- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f12d40391a0 0x7f12d403b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.713+0000 7f12ecc00700 1 -- 172.21.15.102:0/827453953 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f12e004d680 con 0x7f12f0109ff0 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.713+0000 7f12ef400700 1 --2- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f12d40391a0 0x7f12d403b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.714+0000 7f12ef400700 1 --2- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f12d40391a0 0x7f12d403b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f12dc006fd0 tx=0x7f12dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:52.670 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.718+0000 7f12ecc00700 1 -- 172.21.15.102:0/827453953 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f12e0010380 con 0x7f12f0109ff0 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.940+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f12f019f7b0 con 0x7f12f0109ff0 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.941+0000 7f12ecc00700 1 -- 172.21.15.102:0/827453953 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f12e00212c0 con 0x7f12f0109ff0 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.944+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f12d40391a0 msgr2=0x7f12d403b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.944+0000 7f12f6e00700 1 --2- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f12d40391a0 0x7f12d403b660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f12dc006fd0 tx=0x7f12dc006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.944+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 msgr2=0x7f12f019e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.944+0000 7f12f6e00700 1 --2- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f019e010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12e00117e0 tx=0x7f12e0011ba0 comp rx=0 tx=0).stop 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.944+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 shutdown_connections 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.944+0000 7f12f6e00700 1 --2- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f12d40391a0 0x7f12d403b660 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.944+0000 7f12f6e00700 1 --2- 172.21.15.102:0/827453953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12f0109ff0 0x7f12f019e010 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:52.671 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.944+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 >> 172.21.15.102:0/827453953 conn(0x7f12f007d3c0 msgr2=0x7f12f01073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:52.672 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.945+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 shutdown_connections 2024-04-25T01:44:52.672 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:51.945+0000 7f12f6e00700 1 -- 172.21.15.102:0/827453953 wait complete. 2024-04-25T01:44:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.308+0000 7f59d6600700 1 Processor -- start 2024-04-25T01:44:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.308+0000 7f59d6600700 1 -- start start 2024-04-25T01:44:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.309+0000 7f59d6600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d010ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.309+0000 7f59d6600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f59d007bd90 con 0x7f59d0108930 2024-04-25T01:44:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.309+0000 7f59cf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d010ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:54.368 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.309+0000 7f59cf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d010ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36758/0 (socket says 172.21.15.102:36758) 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.309+0000 7f59cf400700 1 -- 172.21.15.102:0/1578313489 learned_addr learned my addr 172.21.15.102:0/1578313489 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.310+0000 7f59cf400700 1 -- 172.21.15.102:0/1578313489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d010b260 con 0x7f59d0108930 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.310+0000 7f59cf400700 1 --2- 172.21.15.102:0/1578313489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d010ad20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009e80 tx=0x7f59c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f2765caaaa95b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.310+0000 7f59ce000700 1 -- 172.21.15.102:0/1578313489 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c0017070 con 0x7f59d0108930 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.311+0000 7f59ce000700 1 -- 172.21.15.102:0/1578313489 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59c000b8c0 con 0x7f59d0108930 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.311+0000 7f59ce000700 1 -- 172.21.15.102:0/1578313489 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c00045c0 con 0x7f59d0108930 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.311+0000 7f59d6600700 1 -- 172.21.15.102:0/1578313489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 msgr2=0x7f59d010ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.311+0000 7f59d6600700 1 --2- 172.21.15.102:0/1578313489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d010ad20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009e80 tx=0x7f59c000b330 comp rx=0 tx=0).stop 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.312+0000 7f59d6600700 1 -- 172.21.15.102:0/1578313489 shutdown_connections 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.312+0000 7f59d6600700 1 --2- 172.21.15.102:0/1578313489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d010ad20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.312+0000 7f59d6600700 1 -- 172.21.15.102:0/1578313489 >> 172.21.15.102:0/1578313489 conn(0x7f59d0102280 msgr2=0x7f59d01046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:54.369 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.312+0000 7f59d6600700 1 -- 172.21.15.102:0/1578313489 shutdown_connections 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.312+0000 7f59d6600700 1 -- 172.21.15.102:0/1578313489 wait complete. 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.313+0000 7f59d6600700 1 Processor -- start 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.313+0000 7f59d6600700 1 -- start start 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.313+0000 7f59d6600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.313+0000 7f59d6600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f59d010c210 con 0x7f59d0108930 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.313+0000 7f59cf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.313+0000 7f59cf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d019beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36768/0 (socket says 172.21.15.102:36768) 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.313+0000 7f59cf400700 1 -- 172.21.15.102:0/3918210209 learned_addr learned my addr 172.21.15.102:0/3918210209 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.314+0000 7f59cf400700 1 -- 172.21.15.102:0/3918210209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d019c3f0 con 0x7f59d0108930 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.314+0000 7f59cf400700 1 --2- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d019beb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009fd0 tx=0x7f59c0003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.314+0000 7f59c7e00700 1 -- 172.21.15.102:0/3918210209 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c0020020 con 0x7f59d0108930 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.314+0000 7f59c7e00700 1 -- 172.21.15.102:0/3918210209 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59c000bba0 con 0x7f59d0108930 2024-04-25T01:44:54.370 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.314+0000 7f59c7e00700 1 -- 172.21.15.102:0/3918210209 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f59c0017040 con 0x7f59d0108930 2024-04-25T01:44:54.371 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.314+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d019c680 con 0x7f59d0108930 2024-04-25T01:44:54.371 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.315+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d019d2b0 con 0x7f59d0108930 2024-04-25T01:44:54.373 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.316+0000 7f59c7e00700 1 -- 172.21.15.102:0/3918210209 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f59c0023020 con 0x7f59d0108930 2024-04-25T01:44:54.373 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.316+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d0063890 con 0x7f59d0108930 2024-04-25T01:44:54.373 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.316+0000 7f59c7e00700 1 --2- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f59b40391a0 0x7f59b403b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.316+0000 7f59c7e00700 1 -- 172.21.15.102:0/3918210209 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f59c000f050 con 0x7f59d0108930 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.316+0000 7f59cea00700 1 --2- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f59b40391a0 0x7f59b403b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.317+0000 7f59cea00700 1 --2- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f59b40391a0 0x7f59b403b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f59bc006fd0 tx=0x7f59bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.322+0000 7f59c7e00700 1 -- 172.21.15.102:0/3918210209 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f59c002a020 con 0x7f59d0108930 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.660+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f59d019d5f0 con 0x7f59d0108930 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.664+0000 7f59c7e00700 1 -- 172.21.15.102:0/3918210209 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f59c001bbb0 con 0x7f59d0108930 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.666+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f59b40391a0 msgr2=0x7f59b403b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.666+0000 7f59d6600700 1 --2- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f59b40391a0 0x7f59b403b660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f59bc006fd0 tx=0x7f59bc006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.666+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 msgr2=0x7f59d019beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.666+0000 7f59d6600700 1 --2- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d019beb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009fd0 tx=0x7f59c0003d00 comp rx=0 tx=0).stop 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.667+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 shutdown_connections 2024-04-25T01:44:54.374 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.667+0000 7f59d6600700 1 --2- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f59b40391a0 0x7f59b403b660 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.667+0000 7f59d6600700 1 --2- 172.21.15.102:0/3918210209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59d0108930 0x7f59d019beb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.667+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 >> 172.21.15.102:0/3918210209 conn(0x7f59d0102280 msgr2=0x7f59d0102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.667+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 shutdown_connections 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr 2024-04-25T01:44:53.667+0000 7f59d6600700 1 -- 172.21.15.102:0/3918210209 wait complete. 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:54.375 INFO:teuthology.orchestra.run.smithi102.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T01:44:54.376 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:54.376 INFO:teuthology.orchestra.run.smithi102.stdout: ceph telemetry on 2024-04-25T01:44:54.376 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:54.376 INFO:teuthology.orchestra.run.smithi102.stdout:For more information see: 2024-04-25T01:44:54.376 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:54.376 INFO:teuthology.orchestra.run.smithi102.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T01:44:54.376 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:44:54.376 INFO:teuthology.orchestra.run.smithi102.stdout:Bootstrap complete. 2024-04-25T01:44:54.468 INFO:tasks.cephadm:Fetching config... 2024-04-25T01:44:54.468 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:44:54.468 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T01:44:54.518 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T01:44:54.519 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:44:54.519 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T01:44:54.560 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T01:44:54.560 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:44:54.561 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T01:44:54.671 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T01:44:54.671 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:44:54.671 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T01:44:54.767 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T01:44:54.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD17LQrULNXZsl7XlVg8EDFzlzV2tnRj/lWveRrUVLJhTdy1wdceXgIrLl4X8jy88/aFkq074Mqbmkv//iOdxUDO2DnDi3AFLYlklNbrZEDh3fF1lGMsLG9xbhFgPjmhVbzt8+B/6GPznzm8zgdSLoq1n/rMAOlldteoBwMsGj4/enumFpmmu1r8pOlSZFoC7happTPbdAYR3KMWOV9zxxtcoUCm6dbWLpLEQ72NgxBv6ezfKw2UqyuY8clkhRxuXVxCN7rS5Qpy6KurcWdtdNzwOMJRda8hTPlPKghowIhpHy9apGMre8XtRtk+lYmNQvnD1g/bzwbXDLd7XoY9zMi5LIpkdEOEt4Vl8U1V8f4QQHph6e9wkFw8Nj1mwz/gmchYpjE1hRzK8e1ouIV4SZXbSkH4ZIBRlIspk6VMe515GlZIbjYZySvbx53cI/sdO5eSP3M2kw27AZpK63XMe/xVHhrMJ6AwYShS5Ygp9hLnegLitq7dpabe6Sk0dCmIvc= ceph-145ca8d0-02a5-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:44:54.863 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD17LQrULNXZsl7XlVg8EDFzlzV2tnRj/lWveRrUVLJhTdy1wdceXgIrLl4X8jy88/aFkq074Mqbmkv//iOdxUDO2DnDi3AFLYlklNbrZEDh3fF1lGMsLG9xbhFgPjmhVbzt8+B/6GPznzm8zgdSLoq1n/rMAOlldteoBwMsGj4/enumFpmmu1r8pOlSZFoC7happTPbdAYR3KMWOV9zxxtcoUCm6dbWLpLEQ72NgxBv6ezfKw2UqyuY8clkhRxuXVxCN7rS5Qpy6KurcWdtdNzwOMJRda8hTPlPKghowIhpHy9apGMre8XtRtk+lYmNQvnD1g/bzwbXDLd7XoY9zMi5LIpkdEOEt4Vl8U1V8f4QQHph6e9wkFw8Nj1mwz/gmchYpjE1hRzK8e1ouIV4SZXbSkH4ZIBRlIspk6VMe515GlZIbjYZySvbx53cI/sdO5eSP3M2kw27AZpK63XMe/xVHhrMJ6AwYShS5Ygp9hLnegLitq7dpabe6Sk0dCmIvc= ceph-145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:54.912 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:54 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3918210209' entity='client.admin' 2024-04-25T01:44:54.916 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD17LQrULNXZsl7XlVg8EDFzlzV2tnRj/lWveRrUVLJhTdy1wdceXgIrLl4X8jy88/aFkq074Mqbmkv//iOdxUDO2DnDi3AFLYlklNbrZEDh3fF1lGMsLG9xbhFgPjmhVbzt8+B/6GPznzm8zgdSLoq1n/rMAOlldteoBwMsGj4/enumFpmmu1r8pOlSZFoC7happTPbdAYR3KMWOV9zxxtcoUCm6dbWLpLEQ72NgxBv6ezfKw2UqyuY8clkhRxuXVxCN7rS5Qpy6KurcWdtdNzwOMJRda8hTPlPKghowIhpHy9apGMre8XtRtk+lYmNQvnD1g/bzwbXDLd7XoY9zMi5LIpkdEOEt4Vl8U1V8f4QQHph6e9wkFw8Nj1mwz/gmchYpjE1hRzK8e1ouIV4SZXbSkH4ZIBRlIspk6VMe515GlZIbjYZySvbx53cI/sdO5eSP3M2kw27AZpK63XMe/xVHhrMJ6AwYShS5Ygp9hLnegLitq7dpabe6Sk0dCmIvc= ceph-145ca8d0-02a5-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:44:54.955 INFO:teuthology.orchestra.run.smithi110.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD17LQrULNXZsl7XlVg8EDFzlzV2tnRj/lWveRrUVLJhTdy1wdceXgIrLl4X8jy88/aFkq074Mqbmkv//iOdxUDO2DnDi3AFLYlklNbrZEDh3fF1lGMsLG9xbhFgPjmhVbzt8+B/6GPznzm8zgdSLoq1n/rMAOlldteoBwMsGj4/enumFpmmu1r8pOlSZFoC7happTPbdAYR3KMWOV9zxxtcoUCm6dbWLpLEQ72NgxBv6ezfKw2UqyuY8clkhRxuXVxCN7rS5Qpy6KurcWdtdNzwOMJRda8hTPlPKghowIhpHy9apGMre8XtRtk+lYmNQvnD1g/bzwbXDLd7XoY9zMi5LIpkdEOEt4Vl8U1V8f4QQHph6e9wkFw8Nj1mwz/gmchYpjE1hRzK8e1ouIV4SZXbSkH4ZIBRlIspk6VMe515GlZIbjYZySvbx53cI/sdO5eSP3M2kw27AZpK63XMe/xVHhrMJ6AwYShS5Ygp9hLnegLitq7dpabe6Sk0dCmIvc= ceph-145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:54.966 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD17LQrULNXZsl7XlVg8EDFzlzV2tnRj/lWveRrUVLJhTdy1wdceXgIrLl4X8jy88/aFkq074Mqbmkv//iOdxUDO2DnDi3AFLYlklNbrZEDh3fF1lGMsLG9xbhFgPjmhVbzt8+B/6GPznzm8zgdSLoq1n/rMAOlldteoBwMsGj4/enumFpmmu1r8pOlSZFoC7happTPbdAYR3KMWOV9zxxtcoUCm6dbWLpLEQ72NgxBv6ezfKw2UqyuY8clkhRxuXVxCN7rS5Qpy6KurcWdtdNzwOMJRda8hTPlPKghowIhpHy9apGMre8XtRtk+lYmNQvnD1g/bzwbXDLd7XoY9zMi5LIpkdEOEt4Vl8U1V8f4QQHph6e9wkFw8Nj1mwz/gmchYpjE1hRzK8e1ouIV4SZXbSkH4ZIBRlIspk6VMe515GlZIbjYZySvbx53cI/sdO5eSP3M2kw27AZpK63XMe/xVHhrMJ6AwYShS5Ygp9hLnegLitq7dpabe6Sk0dCmIvc= ceph-145ca8d0-02a5-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T01:44:55.004 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD17LQrULNXZsl7XlVg8EDFzlzV2tnRj/lWveRrUVLJhTdy1wdceXgIrLl4X8jy88/aFkq074Mqbmkv//iOdxUDO2DnDi3AFLYlklNbrZEDh3fF1lGMsLG9xbhFgPjmhVbzt8+B/6GPznzm8zgdSLoq1n/rMAOlldteoBwMsGj4/enumFpmmu1r8pOlSZFoC7happTPbdAYR3KMWOV9zxxtcoUCm6dbWLpLEQ72NgxBv6ezfKw2UqyuY8clkhRxuXVxCN7rS5Qpy6KurcWdtdNzwOMJRda8hTPlPKghowIhpHy9apGMre8XtRtk+lYmNQvnD1g/bzwbXDLd7XoY9zMi5LIpkdEOEt4Vl8U1V8f4QQHph6e9wkFw8Nj1mwz/gmchYpjE1hRzK8e1ouIV4SZXbSkH4ZIBRlIspk6VMe515GlZIbjYZySvbx53cI/sdO5eSP3M2kw27AZpK63XMe/xVHhrMJ6AwYShS5Ygp9hLnegLitq7dpabe6Sk0dCmIvc= ceph-145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:44:55.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T01:44:55.449 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.758+0000 7f134b000700 1 -- 172.21.15.102:0/3770144177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 msgr2=0x7f1344108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.758+0000 7f134b000700 1 --2- 172.21.15.102:0/3770144177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 0x7f1344108860 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1334009910 tx=0x7f1334009c20 comp rx=0 tx=0).stop 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.759+0000 7f134b000700 1 -- 172.21.15.102:0/3770144177 shutdown_connections 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.759+0000 7f134b000700 1 --2- 172.21.15.102:0/3770144177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 0x7f1344108860 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.759+0000 7f134b000700 1 -- 172.21.15.102:0/3770144177 >> 172.21.15.102:0/3770144177 conn(0x7f1344077520 msgr2=0x7f1344079980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.759+0000 7f134b000700 1 -- 172.21.15.102:0/3770144177 shutdown_connections 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.759+0000 7f134b000700 1 -- 172.21.15.102:0/3770144177 wait complete. 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.760+0000 7f134b000700 1 Processor -- start 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.760+0000 7f134b000700 1 -- start start 2024-04-25T01:44:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.760+0000 7f134b000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 0x7f134419d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.760+0000 7f134b000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f134410e2b0 con 0x7f1344108440 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.761+0000 7f1343e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 0x7f134419d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.761+0000 7f1343e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 0x7f134419d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36790/0 (socket says 172.21.15.102:36790) 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.761+0000 7f1343e00700 1 -- 172.21.15.102:0/1109281647 learned_addr learned my addr 172.21.15.102:0/1109281647 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.761+0000 7f1343e00700 1 -- 172.21.15.102:0/1109281647 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134419dec0 con 0x7f1344108440 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.761+0000 7f1343e00700 1 --2- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 0x7f134419d980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1334005950 tx=0x7f1334005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.761+0000 7f1340c00700 1 -- 172.21.15.102:0/1109281647 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1334027070 con 0x7f1344108440 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.762+0000 7f1340c00700 1 -- 172.21.15.102:0/1109281647 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1334022470 con 0x7f1344108440 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.762+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134419e150 con 0x7f1344108440 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.762+0000 7f1340c00700 1 -- 172.21.15.102:0/1109281647 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f133400f5d0 con 0x7f1344108440 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.762+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13441a0e40 con 0x7f1344108440 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.763+0000 7f1340c00700 1 -- 172.21.15.102:0/1109281647 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f133402a070 con 0x7f1344108440 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.763+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134410ea30 con 0x7f1344108440 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.763+0000 7f1340c00700 1 --2- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f13280390f0 0x7f132803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:44:57.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.763+0000 7f1340c00700 1 -- 172.21.15.102:0/1109281647 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f133401e030 con 0x7f1344108440 2024-04-25T01:44:57.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.764+0000 7f1343400700 1 --2- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f13280390f0 0x7f132803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:44:57.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.764+0000 7f1343400700 1 --2- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f13280390f0 0x7f132803b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1330006fd0 tx=0x7f1330006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:44:57.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.769+0000 7f1340c00700 1 -- 172.21.15.102:0/1109281647 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1334013070 con 0x7f1344108440 2024-04-25T01:44:57.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:57.997+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f134402dc40 con 0x7f1344108440 2024-04-25T01:44:57.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.001+0000 7f1340c00700 1 -- 172.21.15.102:0/1109281647 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f1334028260 con 0x7f1344108440 2024-04-25T01:44:57.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.004+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f13280390f0 msgr2=0x7f132803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:57.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.004+0000 7f134b000700 1 --2- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f13280390f0 0x7f132803b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1330006fd0 tx=0x7f1330006e40 comp rx=0 tx=0).stop 2024-04-25T01:44:57.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.004+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 msgr2=0x7f134419d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:44:57.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.004+0000 7f134b000700 1 --2- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 0x7f134419d980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1334005950 tx=0x7f1334005030 comp rx=0 tx=0).stop 2024-04-25T01:44:58.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.005+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 shutdown_connections 2024-04-25T01:44:58.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.005+0000 7f134b000700 1 --2- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f13280390f0 0x7f132803b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:58.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.005+0000 7f134b000700 1 --2- 172.21.15.102:0/1109281647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344108440 0x7f134419d980 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:44:58.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.005+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 >> 172.21.15.102:0/1109281647 conn(0x7f1344077520 msgr2=0x7f1344077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:44:58.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.005+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 shutdown_connections 2024-04-25T01:44:58.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:44:58.005+0000 7f134b000700 1 -- 172.21.15.102:0/1109281647 wait complete. 2024-04-25T01:44:59.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:59 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1109281647' entity='client.admin' 2024-04-25T01:44:59.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:44:59.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:44:59.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:44:59.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:44:59.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:44:59.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:44:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:44:59.141 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T01:44:59.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T01:44:59.555 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:02.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.015+0000 7f2d1de00700 1 -- 172.21.15.102:0/4281549141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 msgr2=0x7f2d18074660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:02.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.015+0000 7f2d1de00700 1 --2- 172.21.15.102:0/4281549141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 0x7f2d18074660 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2d08009910 tx=0x7f2d08009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.015+0000 7f2d1de00700 1 -- 172.21.15.102:0/4281549141 shutdown_connections 2024-04-25T01:45:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.015+0000 7f2d1de00700 1 --2- 172.21.15.102:0/4281549141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 0x7f2d18074660 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.015+0000 7f2d1de00700 1 -- 172.21.15.102:0/4281549141 >> 172.21.15.102:0/4281549141 conn(0x7f2d180fd250 msgr2=0x7f2d180ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.015+0000 7f2d1de00700 1 -- 172.21.15.102:0/4281549141 shutdown_connections 2024-04-25T01:45:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.015+0000 7f2d1de00700 1 -- 172.21.15.102:0/4281549141 wait complete. 2024-04-25T01:45:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.016+0000 7f2d1de00700 1 Processor -- start 2024-04-25T01:45:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.016+0000 7f2d1de00700 1 -- start start 2024-04-25T01:45:02.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.017+0000 7f2d1de00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 0x7f2d18199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:02.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.017+0000 7f2d1de00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2d18109f10 con 0x7f2d18076200 2024-04-25T01:45:02.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.017+0000 7f2d16a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 0x7f2d18199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:02.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.017+0000 7f2d16a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 0x7f2d18199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36802/0 (socket says 172.21.15.102:36802) 2024-04-25T01:45:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.017+0000 7f2d16a00700 1 -- 172.21.15.102:0/3358551133 learned_addr learned my addr 172.21.15.102:0/3358551133 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:45:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.017+0000 7f2d16a00700 1 -- 172.21.15.102:0/3358551133 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d18199ac0 con 0x7f2d18076200 2024-04-25T01:45:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.018+0000 7f2d16a00700 1 --2- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 0x7f2d18199580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d08005950 tx=0x7f2d08005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.018+0000 7f2d1ca00700 1 -- 172.21.15.102:0/3358551133 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d08026070 con 0x7f2d18076200 2024-04-25T01:45:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.018+0000 7f2d1ca00700 1 -- 172.21.15.102:0/3358551133 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d0800bc30 con 0x7f2d18076200 2024-04-25T01:45:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.018+0000 7f2d1de00700 1 -- 172.21.15.102:0/3358551133 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d18199d50 con 0x7f2d18076200 2024-04-25T01:45:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.018+0000 7f2d1ca00700 1 -- 172.21.15.102:0/3358551133 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d08003b30 con 0x7f2d18076200 2024-04-25T01:45:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.018+0000 7f2d1de00700 1 -- 172.21.15.102:0/3358551133 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1819ca40 con 0x7f2d18076200 2024-04-25T01:45:02.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.019+0000 7f2d1ca00700 1 -- 172.21.15.102:0/3358551133 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f2d08029070 con 0x7f2d18076200 2024-04-25T01:45:02.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.019+0000 7f2d0d600700 1 -- 172.21.15.102:0/3358551133 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf0005330 con 0x7f2d18076200 2024-04-25T01:45:02.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.020+0000 7f2d1ca00700 1 --2- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2cfc0390f0 0x7f2cfc03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:02.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.020+0000 7f2d1ca00700 1 -- 172.21.15.102:0/3358551133 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2d0801e070 con 0x7f2d18076200 2024-04-25T01:45:02.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.020+0000 7f2d16000700 1 --2- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2cfc0390f0 0x7f2cfc03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:02.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.021+0000 7f2d16000700 1 --2- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2cfc0390f0 0x7f2cfc03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2d04006fd0 tx=0x7f2d04006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:02.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.025+0000 7f2d1ca00700 1 -- 172.21.15.102:0/3358551133 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2d08013070 con 0x7f2d18076200 2024-04-25T01:45:02.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.257+0000 7f2d0d600700 1 -- 172.21.15.102:0/3358551133 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f2cf0000bc0 con 0x7f2cfc0390f0 2024-04-25T01:45:02.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.265+0000 7f2d1ca00700 1 -- 172.21.15.102:0/3358551133 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2cf0000bc0 con 0x7f2cfc0390f0 2024-04-25T01:45:02.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.267+0000 7f2d0d600700 1 -- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2cfc0390f0 msgr2=0x7f2cfc03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.267+0000 7f2d0d600700 1 --2- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2cfc0390f0 0x7f2cfc03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2d04006fd0 tx=0x7f2d04006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.267+0000 7f2d0d600700 1 -- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 msgr2=0x7f2d18199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.267+0000 7f2d0d600700 1 --2- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 0x7f2d18199580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d08005950 tx=0x7f2d08005030 comp rx=0 tx=0).stop 2024-04-25T01:45:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.268+0000 7f2d0d600700 1 -- 172.21.15.102:0/3358551133 shutdown_connections 2024-04-25T01:45:02.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.268+0000 7f2d0d600700 1 --2- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2cfc0390f0 0x7f2cfc03b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:02.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.268+0000 7f2d0d600700 1 --2- 172.21.15.102:0/3358551133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d18076200 0x7f2d18199580 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:02.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.268+0000 7f2d0d600700 1 -- 172.21.15.102:0/3358551133 >> 172.21.15.102:0/3358551133 conn(0x7f2d180fd250 msgr2=0x7f2d180fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:02.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.268+0000 7f2d0d600700 1 -- 172.21.15.102:0/3358551133 shutdown_connections 2024-04-25T01:45:02.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:02.268+0000 7f2d0d600700 1 -- 172.21.15.102:0/3358551133 wait complete. 2024-04-25T01:45:03.071 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi110 2024-04-25T01:45:03.071 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:45:03.072 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:45:03.090 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:45:03.090 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:03.147 INFO:tasks.cephadm:Adding host smithi110 to orchestrator... 2024-04-25T01:45:03.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch host add smithi110 2024-04-25T01:45:03.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:03 smithi102 ceph-mon[27796]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:03.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:03 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:03.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:03 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:03.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:03 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:03.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:03 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:03.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:03 smithi102 ceph-mon[27796]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-25T01:45:03.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:03 smithi102 ceph-mon[27796]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:03.552 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:04.545 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:04 smithi102 ceph-mon[27796]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:04.545 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:04 smithi102 ceph-mon[27796]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:05.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:05 smithi102 ceph-mon[27796]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:45:05.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:05 smithi102 ceph-mon[27796]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:05.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:05.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:05.416 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:05.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.602+0000 7f450e400700 1 -- 172.21.15.102:0/3048547185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 msgr2=0x7f450810e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:05.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.602+0000 7f450e400700 1 --2- 172.21.15.102:0/3048547185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 0x7f450810e700 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f44f8009910 tx=0x7f44f8009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:05.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.602+0000 7f450e400700 1 -- 172.21.15.102:0/3048547185 shutdown_connections 2024-04-25T01:45:05.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.602+0000 7f450e400700 1 --2- 172.21.15.102:0/3048547185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 0x7f450810e700 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:05.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.602+0000 7f450e400700 1 -- 172.21.15.102:0/3048547185 >> 172.21.15.102:0/3048547185 conn(0x7f4508077a90 msgr2=0x7f4508079eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:05.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.602+0000 7f450e400700 1 -- 172.21.15.102:0/3048547185 shutdown_connections 2024-04-25T01:45:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.602+0000 7f450e400700 1 -- 172.21.15.102:0/3048547185 wait complete. 2024-04-25T01:45:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.603+0000 7f450e400700 1 Processor -- start 2024-04-25T01:45:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.603+0000 7f450e400700 1 -- start start 2024-04-25T01:45:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.604+0000 7f450e400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 0x7f45081a3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:05.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.604+0000 7f450e400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4508112870 con 0x7f450810e320 2024-04-25T01:45:05.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.604+0000 7f4507400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 0x7f45081a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:05.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.604+0000 7f4507400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 0x7f45081a3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38836/0 (socket says 172.21.15.102:38836) 2024-04-25T01:45:05.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.604+0000 7f4507400700 1 -- 172.21.15.102:0/4088473846 learned_addr learned my addr 172.21.15.102:0/4088473846 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:45:05.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.604+0000 7f4507400700 1 -- 172.21.15.102:0/4088473846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45081a3e60 con 0x7f450810e320 2024-04-25T01:45:05.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.605+0000 7f4507400700 1 --2- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 0x7f45081a3920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44f8006010 tx=0x7f44f8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:05.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.605+0000 7f44ffe00700 1 -- 172.21.15.102:0/4088473846 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44f8026070 con 0x7f450810e320 2024-04-25T01:45:05.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.605+0000 7f44ffe00700 1 -- 172.21.15.102:0/4088473846 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f800bbd0 con 0x7f450810e320 2024-04-25T01:45:05.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.605+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45081a4150 con 0x7f450810e320 2024-04-25T01:45:05.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.605+0000 7f44ffe00700 1 -- 172.21.15.102:0/4088473846 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44f80039d0 con 0x7f450810e320 2024-04-25T01:45:05.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.605+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f450819dae0 con 0x7f450810e320 2024-04-25T01:45:05.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.606+0000 7f44ffe00700 1 -- 172.21.15.102:0/4088473846 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f44f8029070 con 0x7f450810e320 2024-04-25T01:45:05.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.606+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4508109960 con 0x7f450810e320 2024-04-25T01:45:05.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.607+0000 7f44ffe00700 1 --2- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f44ec039150 0x7f44ec03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:05.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.607+0000 7f44ffe00700 1 -- 172.21.15.102:0/4088473846 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f44f801e070 con 0x7f450810e320 2024-04-25T01:45:05.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.607+0000 7f4506a00700 1 --2- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f44ec039150 0x7f44ec03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:05.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.608+0000 7f4506a00700 1 --2- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f44ec039150 0x7f44ec03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f44f4006fd0 tx=0x7f44f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:05.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.612+0000 7f44ffe00700 1 -- 172.21.15.102:0/4088473846 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f44f8013070 con 0x7f450810e320 2024-04-25T01:45:05.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:05.839+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}) -- 0x7f450819efd0 con 0x7f44ec039150 2024-04-25T01:45:07.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:07 smithi102 ceph-mon[27796]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:07.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:07 smithi102 ceph-mon[27796]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:07.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:07 smithi102 ceph-mon[27796]: Deploying cephadm binary to smithi110 2024-04-25T01:45:09.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:09 smithi102 ceph-mon[27796]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:10.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.591+0000 7f44ffe00700 1 -- 172.21.15.102:0/4088473846 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f450819efd0 con 0x7f44ec039150 2024-04-25T01:45:10.585 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi110' with addr '172.21.15.110' 2024-04-25T01:45:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.595+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f44ec039150 msgr2=0x7f44ec03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.595+0000 7f450e400700 1 --2- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f44ec039150 0x7f44ec03b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f44f4006fd0 tx=0x7f44f4006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.595+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 msgr2=0x7f45081a3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.595+0000 7f450e400700 1 --2- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 0x7f45081a3920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44f8006010 tx=0x7f44f8005030 comp rx=0 tx=0).stop 2024-04-25T01:45:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.595+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 shutdown_connections 2024-04-25T01:45:10.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.595+0000 7f450e400700 1 --2- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f44ec039150 0x7f44ec03b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.595+0000 7f450e400700 1 --2- 172.21.15.102:0/4088473846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f450810e320 0x7f45081a3920 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.595+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 >> 172.21.15.102:0/4088473846 conn(0x7f4508077a90 msgr2=0x7f4508079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.596+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 shutdown_connections 2024-04-25T01:45:10.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:10.596+0000 7f450e400700 1 -- 172.21.15.102:0/4088473846 wait complete. 2024-04-25T01:45:11.312 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:45:11.732 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:11.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:11.876 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:11 smithi102 ceph-mon[27796]: Added host smithi110 2024-04-25T01:45:11.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:11 smithi102 ceph-mon[27796]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:11.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:11.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:12.729 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:12 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:13.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.690+0000 7fa8d4c00700 1 -- 172.21.15.102:0/1016408895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 msgr2=0x7fa8d01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:13.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.690+0000 7fa8d4c00700 1 --2- 172.21.15.102:0/1016408895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 0x7fa8d01031e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009910 tx=0x7fa8c0009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:13.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.690+0000 7fa8d4c00700 1 -- 172.21.15.102:0/1016408895 shutdown_connections 2024-04-25T01:45:13.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.690+0000 7fa8d4c00700 1 --2- 172.21.15.102:0/1016408895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 0x7fa8d01031e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:13.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.690+0000 7fa8d4c00700 1 -- 172.21.15.102:0/1016408895 >> 172.21.15.102:0/1016408895 conn(0x7fa8d00fc670 msgr2=0x7fa8d00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:13.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.690+0000 7fa8d4c00700 1 -- 172.21.15.102:0/1016408895 shutdown_connections 2024-04-25T01:45:13.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.690+0000 7fa8d4c00700 1 -- 172.21.15.102:0/1016408895 wait complete. 2024-04-25T01:45:13.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.691+0000 7fa8d4c00700 1 Processor -- start 2024-04-25T01:45:13.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.692+0000 7fa8d4c00700 1 -- start start 2024-04-25T01:45:13.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.692+0000 7fa8d4c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 0x7fa8d019f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:13.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.692+0000 7fa8d4c00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa8d010c2d0 con 0x7fa8d0102e00 2024-04-25T01:45:13.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.692+0000 7fa8cde00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 0x7fa8d019f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:13.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.692+0000 7fa8cde00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 0x7fa8d019f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51474/0 (socket says 172.21.15.102:51474) 2024-04-25T01:45:13.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.692+0000 7fa8cde00700 1 -- 172.21.15.102:0/3460807883 learned_addr learned my addr 172.21.15.102:0/3460807883 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:45:13.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.693+0000 7fa8cde00700 1 -- 172.21.15.102:0/3460807883 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8d019fb90 con 0x7fa8d0102e00 2024-04-25T01:45:13.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.693+0000 7fa8cde00700 1 --2- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 0x7fa8d019f650 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0006010 tx=0x7fa8c0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:13.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.693+0000 7fa8c6a00700 1 -- 172.21.15.102:0/3460807883 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8c0026070 con 0x7fa8d0102e00 2024-04-25T01:45:13.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.693+0000 7fa8c6a00700 1 -- 172.21.15.102:0/3460807883 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8c000bc30 con 0x7fa8d0102e00 2024-04-25T01:45:13.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.693+0000 7fa8c6a00700 1 -- 172.21.15.102:0/3460807883 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8c00039a0 con 0x7fa8d0102e00 2024-04-25T01:45:13.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.693+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8d019fe20 con 0x7fa8d0102e00 2024-04-25T01:45:13.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.693+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8d01997d0 con 0x7fa8d0102e00 2024-04-25T01:45:13.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.694+0000 7fa8c6a00700 1 -- 172.21.15.102:0/3460807883 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fa8c0029070 con 0x7fa8d0102e00 2024-04-25T01:45:13.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.694+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8d006a930 con 0x7fa8d0102e00 2024-04-25T01:45:13.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.695+0000 7fa8c6a00700 1 --2- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa8b4039150 0x7fa8b403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:13.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.695+0000 7fa8c6a00700 1 -- 172.21.15.102:0/3460807883 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa8c001e070 con 0x7fa8d0102e00 2024-04-25T01:45:13.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.695+0000 7fa8cd400700 1 --2- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa8b4039150 0x7fa8b403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:13.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.696+0000 7fa8cd400700 1 --2- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa8b4039150 0x7fa8b403b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc006fd0 tx=0x7fa8bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:13.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.701+0000 7fa8c6a00700 1 -- 172.21.15.102:0/3460807883 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa8c0013070 con 0x7fa8d0102e00 2024-04-25T01:45:13.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.915+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa8d019ad00 con 0x7fa8b4039150 2024-04-25T01:45:13.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:13 smithi102 ceph-mon[27796]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:13.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:13 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:13.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.917+0000 7fa8c6a00700 1 -- 172.21.15.102:0/3460807883 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7fa8d019ad00 con 0x7fa8b4039150 2024-04-25T01:45:13.910 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:45:13.910 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}] 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa8b4039150 msgr2=0x7fa8b403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 --2- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa8b4039150 0x7fa8b403b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc006fd0 tx=0x7fa8bc006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 msgr2=0x7fa8d019f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 --2- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 0x7fa8d019f650 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0006010 tx=0x7fa8c0005030 comp rx=0 tx=0).stop 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 shutdown_connections 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 --2- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa8b4039150 0x7fa8b403b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 --2- 172.21.15.102:0/3460807883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8d0102e00 0x7fa8d019f650 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 >> 172.21.15.102:0/3460807883 conn(0x7fa8d00fc670 msgr2=0x7fa8d0107fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 shutdown_connections 2024-04-25T01:45:13.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:13.920+0000 7fa8d4c00700 1 -- 172.21.15.102:0/3460807883 wait complete. 2024-04-25T01:45:14.650 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-04-25T01:45:14.650 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:45:14.651 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-04-25T01:45:14.670 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:45:14.670 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:14.728 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-04-25T01:45:14.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch host add smithi193 2024-04-25T01:45:15.163 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:15.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:15 smithi102 ceph-mon[27796]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:45:15.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:15 smithi102 ceph-mon[27796]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:17.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.266+0000 7fa466000700 1 -- 172.21.15.102:0/1818302907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 msgr2=0x7fa460102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:17.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.266+0000 7fa466000700 1 --2- 172.21.15.102:0/1818302907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 0x7fa460102260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa450009910 tx=0x7fa450009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:17.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.266+0000 7fa466000700 1 -- 172.21.15.102:0/1818302907 shutdown_connections 2024-04-25T01:45:17.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.266+0000 7fa466000700 1 --2- 172.21.15.102:0/1818302907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 0x7fa460102260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:17.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.266+0000 7fa466000700 1 -- 172.21.15.102:0/1818302907 >> 172.21.15.102:0/1818302907 conn(0x7fa4600fd750 msgr2=0x7fa4600ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.266+0000 7fa466000700 1 -- 172.21.15.102:0/1818302907 shutdown_connections 2024-04-25T01:45:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.266+0000 7fa466000700 1 -- 172.21.15.102:0/1818302907 wait complete. 2024-04-25T01:45:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.267+0000 7fa466000700 1 Processor -- start 2024-04-25T01:45:17.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.268+0000 7fa466000700 1 -- start start 2024-04-25T01:45:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.268+0000 7fa466000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 0x7fa460074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.268+0000 7fa466000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa46010c250 con 0x7fa460101e80 2024-04-25T01:45:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.268+0000 7fa45f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 0x7fa460074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.268+0000 7fa45f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 0x7fa460074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51496/0 (socket says 172.21.15.102:51496) 2024-04-25T01:45:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.268+0000 7fa45f400700 1 -- 172.21.15.102:0/2042371415 learned_addr learned my addr 172.21.15.102:0/2042371415 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:45:17.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.269+0000 7fa45f400700 1 -- 172.21.15.102:0/2042371415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa460076420 con 0x7fa460101e80 2024-04-25T01:45:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.269+0000 7fa45f400700 1 --2- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 0x7fa460074530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa450006010 tx=0x7fa450005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.269+0000 7fa457e00700 1 -- 172.21.15.102:0/2042371415 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa450026070 con 0x7fa460101e80 2024-04-25T01:45:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.269+0000 7fa457e00700 1 -- 172.21.15.102:0/2042371415 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa45000bc30 con 0x7fa460101e80 2024-04-25T01:45:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.269+0000 7fa457e00700 1 -- 172.21.15.102:0/2042371415 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4500039d0 con 0x7fa460101e80 2024-04-25T01:45:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.269+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa460074ad0 con 0x7fa460101e80 2024-04-25T01:45:17.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.270+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa460074f90 con 0x7fa460101e80 2024-04-25T01:45:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.271+0000 7fa457e00700 1 -- 172.21.15.102:0/2042371415 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fa450029070 con 0x7fa460101e80 2024-04-25T01:45:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.271+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa460103360 con 0x7fa460101e80 2024-04-25T01:45:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.271+0000 7fa457e00700 1 --2- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa444039150 0x7fa44403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.271+0000 7fa457e00700 1 -- 172.21.15.102:0/2042371415 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa45001e070 con 0x7fa460101e80 2024-04-25T01:45:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.271+0000 7fa45ea00700 1 --2- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa444039150 0x7fa44403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.272+0000 7fa45ea00700 1 --2- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa444039150 0x7fa44403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa44c006fd0 tx=0x7fa44c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:17.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.277+0000 7fa457e00700 1 -- 172.21.15.102:0/2042371415 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa450050030 con 0x7fa460101e80 2024-04-25T01:45:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:17.510+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}) -- 0x7fa460075a60 con 0x7fa444039150 2024-04-25T01:45:17.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:17.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:17.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:17.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:17.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:45:17.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:17.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:17.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-25T01:45:17.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:17.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:17 smithi102 ceph-mon[27796]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:19.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:18 smithi102 ceph-mon[27796]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:19.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:18 smithi102 ceph-mon[27796]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:45:19.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:18 smithi102 ceph-mon[27796]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:19.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:19.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:19.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:20.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:19 smithi102 ceph-mon[27796]: Deploying cephadm binary to smithi193 2024-04-25T01:45:20.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:19 smithi102 ceph-mon[27796]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:22.165 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:21 smithi102 ceph-mon[27796]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:22.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.224+0000 7fa457e00700 1 -- 172.21.15.102:0/2042371415 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa460075a60 con 0x7fa444039150 2024-04-25T01:45:22.218 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-04-25T01:45:22.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.229+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa444039150 msgr2=0x7fa44403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:22.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.229+0000 7fa466000700 1 --2- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa444039150 0x7fa44403b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa44c006fd0 tx=0x7fa44c006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:22.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.229+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 msgr2=0x7fa460074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:22.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.229+0000 7fa466000700 1 --2- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 0x7fa460074530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa450006010 tx=0x7fa450005030 comp rx=0 tx=0).stop 2024-04-25T01:45:22.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.229+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 shutdown_connections 2024-04-25T01:45:22.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.229+0000 7fa466000700 1 --2- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa444039150 0x7fa44403b610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:22.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.229+0000 7fa466000700 1 --2- 172.21.15.102:0/2042371415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa460101e80 0x7fa460074530 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:22.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.229+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 >> 172.21.15.102:0/2042371415 conn(0x7fa4600fd750 msgr2=0x7fa460103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:22.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.229+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 shutdown_connections 2024-04-25T01:45:22.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:22.230+0000 7fa466000700 1 -- 172.21.15.102:0/2042371415 wait complete. 2024-04-25T01:45:22.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T01:45:23.443 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:23.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:23.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:23 smithi102 ceph-mon[27796]: Added host smithi193 2024-04-25T01:45:23.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:23.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:23 smithi102 ceph-mon[27796]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:23.589 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:24.561 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:25.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.326+0000 7f58d4a00700 1 -- 172.21.15.102:0/458338026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 msgr2=0x7f58d010e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:25.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.326+0000 7f58d4a00700 1 --2- 172.21.15.102:0/458338026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 0x7f58d010e700 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009910 tx=0x7f58c0009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:25.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.326+0000 7f58d4a00700 1 -- 172.21.15.102:0/458338026 shutdown_connections 2024-04-25T01:45:25.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.326+0000 7f58d4a00700 1 --2- 172.21.15.102:0/458338026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 0x7f58d010e700 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.326+0000 7f58d4a00700 1 -- 172.21.15.102:0/458338026 >> 172.21.15.102:0/458338026 conn(0x7f58d0077a90 msgr2=0x7f58d0079eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:25.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.326+0000 7f58d4a00700 1 -- 172.21.15.102:0/458338026 shutdown_connections 2024-04-25T01:45:25.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.326+0000 7f58d4a00700 1 -- 172.21.15.102:0/458338026 wait complete. 2024-04-25T01:45:25.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.327+0000 7f58d4a00700 1 Processor -- start 2024-04-25T01:45:25.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.327+0000 7f58d4a00700 1 -- start start 2024-04-25T01:45:25.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.327+0000 7f58d4a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 0x7f58d01a3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:25.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.327+0000 7f58d4a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f58d0112870 con 0x7f58d010e320 2024-04-25T01:45:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.328+0000 7f58cde00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 0x7f58d01a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.328+0000 7f58cde00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 0x7f58d01a3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36836/0 (socket says 172.21.15.102:36836) 2024-04-25T01:45:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.328+0000 7f58cde00700 1 -- 172.21.15.102:0/535004329 learned_addr learned my addr 172.21.15.102:0/535004329 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:45:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.328+0000 7f58cde00700 1 -- 172.21.15.102:0/535004329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d01a3e60 con 0x7f58d010e320 2024-04-25T01:45:25.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.328+0000 7f58cde00700 1 --2- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 0x7f58d01a3920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f58c0006010 tx=0x7f58c0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.329+0000 7f58c6a00700 1 -- 172.21.15.102:0/535004329 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c0026070 con 0x7f58d010e320 2024-04-25T01:45:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.329+0000 7f58c6a00700 1 -- 172.21.15.102:0/535004329 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58c000bbd0 con 0x7f58d010e320 2024-04-25T01:45:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.329+0000 7f58c6a00700 1 -- 172.21.15.102:0/535004329 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58c00039d0 con 0x7f58d010e320 2024-04-25T01:45:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.329+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d01a4150 con 0x7f58d010e320 2024-04-25T01:45:25.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.329+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d019dae0 con 0x7f58d010e320 2024-04-25T01:45:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.330+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d0109960 con 0x7f58d010e320 2024-04-25T01:45:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.330+0000 7f58c6a00700 1 -- 172.21.15.102:0/535004329 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f58c0029070 con 0x7f58d010e320 2024-04-25T01:45:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.331+0000 7f58c6a00700 1 --2- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f58b4039150 0x7f58b403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:25.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.331+0000 7f58c6a00700 1 -- 172.21.15.102:0/535004329 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f58c001e070 con 0x7f58d010e320 2024-04-25T01:45:25.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.331+0000 7f58cd400700 1 --2- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f58b4039150 0x7f58b403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:25.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.332+0000 7f58cd400700 1 --2- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f58b4039150 0x7f58b403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f58bc006fd0 tx=0x7f58bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:25.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.337+0000 7f58c6a00700 1 -- 172.21.15.102:0/535004329 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f58c0013070 con 0x7f58d010e320 2024-04-25T01:45:25.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:25 smithi102 ceph-mon[27796]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:25.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:25 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:25.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.573+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f58d019efd0 con 0x7f58b4039150 2024-04-25T01:45:25.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.575+0000 7f58c6a00700 1 -- 172.21.15.102:0/535004329 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f58d019efd0 con 0x7f58b4039150 2024-04-25T01:45:25.568 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:45:25.568 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f58b4039150 msgr2=0x7f58b403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 --2- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f58b4039150 0x7f58b403b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f58bc006fd0 tx=0x7f58bc006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 msgr2=0x7f58d01a3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 --2- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 0x7f58d01a3920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f58c0006010 tx=0x7f58c0005030 comp rx=0 tx=0).stop 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 shutdown_connections 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 --2- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f58b4039150 0x7f58b403b610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 --2- 172.21.15.102:0/535004329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58d010e320 0x7f58d01a3920 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 >> 172.21.15.102:0/535004329 conn(0x7f58d0077a90 msgr2=0x7f58d00792b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 shutdown_connections 2024-04-25T01:45:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:25.579+0000 7f58d4a00700 1 -- 172.21.15.102:0/535004329 wait complete. 2024-04-25T01:45:26.248 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T01:45:26.248 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T01:45:26.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:26 smithi102 ceph-mon[27796]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:45:26.678 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:27.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:27 smithi102 ceph-mon[27796]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.790+0000 7f53a3000700 1 -- 172.21.15.102:0/1250404068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 msgr2=0x7f539c074a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:28.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.790+0000 7f53a3000700 1 --2- 172.21.15.102:0/1250404068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 0x7f539c074a40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f539000b0a0 tx=0x7f539000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:45:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.790+0000 7f53a3000700 1 -- 172.21.15.102:0/1250404068 shutdown_connections 2024-04-25T01:45:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.790+0000 7f53a3000700 1 --2- 172.21.15.102:0/1250404068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 0x7f539c074a40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.790+0000 7f53a3000700 1 -- 172.21.15.102:0/1250404068 >> 172.21.15.102:0/1250404068 conn(0x7f539c0fd580 msgr2=0x7f539c0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.790+0000 7f53a3000700 1 -- 172.21.15.102:0/1250404068 shutdown_connections 2024-04-25T01:45:28.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.790+0000 7f53a3000700 1 -- 172.21.15.102:0/1250404068 wait complete. 2024-04-25T01:45:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.791+0000 7f53a3000700 1 Processor -- start 2024-04-25T01:45:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.791+0000 7f53a3000700 1 -- start start 2024-04-25T01:45:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.792+0000 7f53a3000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 0x7f539c19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.792+0000 7f53a3000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f539c10e300 con 0x7f539c074660 2024-04-25T01:45:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.792+0000 7f53a1c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 0x7f539c19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.792+0000 7f53a1c00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 0x7f539c19f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36856/0 (socket says 172.21.15.102:36856) 2024-04-25T01:45:28.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.792+0000 7f53a1c00700 1 -- 172.21.15.102:0/2355028769 learned_addr learned my addr 172.21.15.102:0/2355028769 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:45:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.793+0000 7f53a1c00700 1 -- 172.21.15.102:0/2355028769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c19f940 con 0x7f539c074660 2024-04-25T01:45:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.793+0000 7f53a1c00700 1 --2- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 0x7f539c19f400 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53900062a0 tx=0x7f539000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.793+0000 7f539a000700 1 -- 172.21.15.102:0/2355028769 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5390017050 con 0x7f539c074660 2024-04-25T01:45:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.793+0000 7f539a000700 1 -- 172.21.15.102:0/2355028769 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5390023070 con 0x7f539c074660 2024-04-25T01:45:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.793+0000 7f539a000700 1 -- 172.21.15.102:0/2355028769 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5390004000 con 0x7f539c074660 2024-04-25T01:45:28.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.793+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539c19fbd0 con 0x7f539c074660 2024-04-25T01:45:28.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.793+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f539c199610 con 0x7f539c074660 2024-04-25T01:45:28.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.794+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f539c075d10 con 0x7f539c074660 2024-04-25T01:45:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.795+0000 7f539a000700 1 -- 172.21.15.102:0/2355028769 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f539001e450 con 0x7f539c074660 2024-04-25T01:45:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.796+0000 7f539a000700 1 --2- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f53840390f0 0x7f538403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:28.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.796+0000 7f539a000700 1 -- 172.21.15.102:0/2355028769 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f539000f050 con 0x7f539c074660 2024-04-25T01:45:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.796+0000 7f53a1200700 1 --2- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f53840390f0 0x7f538403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:28.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.797+0000 7f53a1200700 1 --2- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f53840390f0 0x7f538403b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5388006fd0 tx=0x7f5388006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:28.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:28.801+0000 7f539a000700 1 -- 172.21.15.102:0/2355028769 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5390008cb0 con 0x7f539c074660 2024-04-25T01:45:29.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.029+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f539c063890 con 0x7f539c074660 2024-04-25T01:45:29.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.263+0000 7f539a000700 1 -- 172.21.15.102:0/2355028769 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f539001be20 con 0x7f539c074660 2024-04-25T01:45:29.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.268+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f53840390f0 msgr2=0x7f538403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:29.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.268+0000 7f53a3000700 1 --2- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f53840390f0 0x7f538403b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5388006fd0 tx=0x7f5388006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:29.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.268+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 msgr2=0x7f539c19f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:29.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.268+0000 7f53a3000700 1 --2- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 0x7f539c19f400 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53900062a0 tx=0x7f539000bdd0 comp rx=0 tx=0).stop 2024-04-25T01:45:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.269+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 shutdown_connections 2024-04-25T01:45:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.269+0000 7f53a3000700 1 --2- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f53840390f0 0x7f538403b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.269+0000 7f53a3000700 1 --2- 172.21.15.102:0/2355028769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f539c074660 0x7f539c19f400 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.269+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 >> 172.21.15.102:0/2355028769 conn(0x7f539c0fd580 msgr2=0x7f539c105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.269+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 shutdown_connections 2024-04-25T01:45:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:29.269+0000 7f53a3000700 1 -- 172.21.15.102:0/2355028769 wait complete. 2024-04-25T01:45:29.263 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-04-25T01:45:29.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:29.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:29.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:29.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:29.285 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:45:29.285 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:29.285 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:29.285 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-25T01:45:29.285 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:29.285 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:29.285 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:29 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2355028769' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:45:30.047 INFO:tasks.cephadm:Adding mon.a on smithi102 2024-04-25T01:45:30.047 INFO:tasks.cephadm:Adding mon.b on smithi110 2024-04-25T01:45:30.047 INFO:tasks.cephadm:Adding mon.c on smithi193 2024-04-25T01:45:30.047 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi102:172.21.15.102=a;smithi110:172.21.15.110=b;smithi193:172.21.15.193=c' 2024-04-25T01:45:30.277 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:30.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:30 smithi102 ceph-mon[27796]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:30.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:30 smithi102 ceph-mon[27796]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:45:30.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:30 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2355028769' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:45:30.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:30 smithi102 ceph-mon[27796]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:30.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:30 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:30.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:30 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:30.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:30 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:31.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.625+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2320253022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 msgr2=0x7fcb04102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:31.620 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.625+0000 7fcb0b05a700 1 --2- 172.21.15.193:0/2320253022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 0x7fcb04102260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009910 tx=0x7fcaf4009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.625+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2320253022 shutdown_connections 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.625+0000 7fcb0b05a700 1 --2- 172.21.15.193:0/2320253022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 0x7fcb04102260 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.625+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2320253022 >> 172.21.15.193:0/2320253022 conn(0x7fcb040fd750 msgr2=0x7fcb040ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.625+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2320253022 shutdown_connections 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.625+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2320253022 wait complete. 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.626+0000 7fcb0b05a700 1 Processor -- start 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.626+0000 7fcb0b05a700 1 -- start start 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.626+0000 7fcb0b05a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 0x7fcb04074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.626+0000 7fcb0b05a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0410c250 con 0x7fcb04101e80 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.627+0000 7fcb08df6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 0x7fcb04074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:31.621 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.627+0000 7fcb08df6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 0x7fcb04074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.193:46160/0 (socket says 172.21.15.193:46160) 2024-04-25T01:45:31.622 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.627+0000 7fcb08df6700 1 -- 172.21.15.193:0/2089656784 learned_addr learned my addr 172.21.15.193:0/2089656784 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:45:31.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.627+0000 7fcb08df6700 1 -- 172.21.15.193:0/2089656784 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb04076420 con 0x7fcb04101e80 2024-04-25T01:45:31.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.628+0000 7fcb08df6700 1 --2- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 0x7fcb04074530 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4005850 tx=0x7fcaf4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:31.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.628+0000 7fcb01ffb700 1 -- 172.21.15.193:0/2089656784 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcaf4026070 con 0x7fcb04101e80 2024-04-25T01:45:31.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.628+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb04074ad0 con 0x7fcb04101e80 2024-04-25T01:45:31.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.628+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb04074f90 con 0x7fcb04101e80 2024-04-25T01:45:31.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.628+0000 7fcb01ffb700 1 -- 172.21.15.193:0/2089656784 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf400bbd0 con 0x7fcb04101e80 2024-04-25T01:45:31.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.628+0000 7fcb01ffb700 1 -- 172.21.15.193:0/2089656784 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcaf40039b0 con 0x7fcb04101e80 2024-04-25T01:45:31.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.629+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb04103360 con 0x7fcb04101e80 2024-04-25T01:45:31.623 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.629+0000 7fcb01ffb700 1 -- 172.21.15.193:0/2089656784 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fcaf4029070 con 0x7fcb04101e80 2024-04-25T01:45:31.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.630+0000 7fcb01ffb700 1 --2- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcaec0390f0 0x7fcaec03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:31.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.630+0000 7fcb01ffb700 1 -- 172.21.15.193:0/2089656784 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcaf401e070 con 0x7fcb04101e80 2024-04-25T01:45:31.624 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.630+0000 7fcb03fff700 1 --2- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcaec0390f0 0x7fcaec03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:31.625 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.631+0000 7fcb03fff700 1 --2- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcaec0390f0 0x7fcaec03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8006fd0 tx=0x7fcaf8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:31.637 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.644+0000 7fcb01ffb700 1 -- 172.21.15.193:0/2089656784 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcaf4013070 con 0x7fcb04101e80 2024-04-25T01:45:31.665 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:31 smithi102 ceph-mon[27796]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:31.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.853+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi110:172.21.15.110=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}) -- 0x7fcb04075ad0 con 0x7fcaec0390f0 2024-04-25T01:45:31.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.869+0000 7fcb01ffb700 1 -- 172.21.15.193:0/2089656784 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcb04075ad0 con 0x7fcaec0390f0 2024-04-25T01:45:31.863 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-04-25T01:45:31.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcaec0390f0 msgr2=0x7fcaec03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:31.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 --2- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcaec0390f0 0x7fcaec03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8006fd0 tx=0x7fcaf8006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:31.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 msgr2=0x7fcb04074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:31.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 --2- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 0x7fcb04074530 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4005850 tx=0x7fcaf4005030 comp rx=0 tx=0).stop 2024-04-25T01:45:31.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 shutdown_connections 2024-04-25T01:45:31.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 --2- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcaec0390f0 0x7fcaec03b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:31.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 --2- 172.21.15.193:0/2089656784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb04101e80 0x7fcb04074530 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:31.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 >> 172.21.15.193:0/2089656784 conn(0x7fcb040fd750 msgr2=0x7fcb04104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:31.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 shutdown_connections 2024-04-25T01:45:31.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:31.871+0000 7fcb0b05a700 1 -- 172.21.15.193:0/2089656784 wait complete. 2024-04-25T01:45:32.486 DEBUG:teuthology.orchestra.run.smithi110:mon.b> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T01:45:32.489 DEBUG:teuthology.orchestra.run.smithi193:mon.c> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T01:45:32.491 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:45:32.491 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:45:33.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi110:172.21.15.110=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:33.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi110:172.21.15.110=b;smithi193:172.21.15.193=c;count:3 2024-04-25T01:45:33.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:33.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:33.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:33.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:33.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:33.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:33.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:33.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: Deploying daemon mon.c on smithi193 2024-04-25T01:45:33.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:32 smithi102 ceph-mon[27796]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:33.619 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:35.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.865+0000 7fe85c621700 1 -- 172.21.15.193:0/3720111771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 msgr2=0x7fe854106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.865+0000 7fe85c621700 1 --2- 172.21.15.193:0/3720111771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 0x7fe854106fc0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe844009910 tx=0x7fe844009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.865+0000 7fe85c621700 1 -- 172.21.15.193:0/3720111771 shutdown_connections 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.865+0000 7fe85c621700 1 --2- 172.21.15.193:0/3720111771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 0x7fe854106fc0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.865+0000 7fe85c621700 1 -- 172.21.15.193:0/3720111771 >> 172.21.15.193:0/3720111771 conn(0x7fe854077800 msgr2=0x7fe8540feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.865+0000 7fe85c621700 1 -- 172.21.15.193:0/3720111771 shutdown_connections 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.865+0000 7fe85c621700 1 -- 172.21.15.193:0/3720111771 wait complete. 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.866+0000 7fe85c621700 1 Processor -- start 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.866+0000 7fe85c621700 1 -- start start 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.866+0000 7fe85c621700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 0x7fe8540fcb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.866+0000 7fe85c621700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe85410b1a0 con 0x7fe854106be0 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.867+0000 7fe85a3bd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 0x7fe8540fcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.867+0000 7fe85a3bd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 0x7fe8540fcb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.193:46180/0 (socket says 172.21.15.193:46180) 2024-04-25T01:45:35.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.867+0000 7fe85a3bd700 1 -- 172.21.15.193:0/2803688902 learned_addr learned my addr 172.21.15.193:0/2803688902 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:45:35.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.867+0000 7fe85a3bd700 1 -- 172.21.15.193:0/2803688902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8540fea60 con 0x7fe854106be0 2024-04-25T01:45:35.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.868+0000 7fe85a3bd700 1 --2- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 0x7fe8540fcb70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe844005850 tx=0x7fe844005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:35.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.868+0000 7fe84b7fe700 1 -- 172.21.15.193:0/2803688902 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe844026070 con 0x7fe854106be0 2024-04-25T01:45:35.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.868+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8540fd0b0 con 0x7fe854106be0 2024-04-25T01:45:35.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.868+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8540fd570 con 0x7fe854106be0 2024-04-25T01:45:35.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.868+0000 7fe84b7fe700 1 -- 172.21.15.193:0/2803688902 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe84400b750 con 0x7fe854106be0 2024-04-25T01:45:35.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.868+0000 7fe84b7fe700 1 -- 172.21.15.193:0/2803688902 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe844030060 con 0x7fe854106be0 2024-04-25T01:45:35.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.869+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe838005330 con 0x7fe854106be0 2024-04-25T01:45:35.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.870+0000 7fe84b7fe700 1 -- 172.21.15.193:0/2803688902 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fe844029070 con 0x7fe854106be0 2024-04-25T01:45:35.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.870+0000 7fe84b7fe700 1 --2- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe8400390f0 0x7fe84003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:35.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.870+0000 7fe84b7fe700 1 -- 172.21.15.193:0/2803688902 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe84401e070 con 0x7fe854106be0 2024-04-25T01:45:35.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.871+0000 7fe859bbc700 1 --2- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe8400390f0 0x7fe84003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:35.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.872+0000 7fe859bbc700 1 --2- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe8400390f0 0x7fe84003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe850006fd0 tx=0x7fe850006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:35.880 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:35.886+0000 7fe84b7fe700 1 -- 172.21.15.193:0/2803688902 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe844013070 con 0x7fe854106be0 2024-04-25T01:45:35.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:35 smithi102 ceph-mon[27796]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:36.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.185+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe8380058f0 con 0x7fe854106be0 2024-04-25T01:45:36.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.186+0000 7fe84b7fe700 1 -- 172.21.15.193:0/2803688902 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7fe84400d340 con 0x7fe854106be0 2024-04-25T01:45:36.181 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:45:36.181 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","modified":"2024-04-25T01:43:30.536087Z","created":"2024-04-25T01:43:30.536087Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T01:45:36.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe8400390f0 msgr2=0x7fe84003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:36.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 --2- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe8400390f0 0x7fe84003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe850006fd0 tx=0x7fe850006e40 comp rx=0 tx=0).stop 2024-04-25T01:45:36.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 msgr2=0x7fe8540fcb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:36.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 --2- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 0x7fe8540fcb70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe844005850 tx=0x7fe844005030 comp rx=0 tx=0).stop 2024-04-25T01:45:36.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 shutdown_connections 2024-04-25T01:45:36.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 --2- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe8400390f0 0x7fe84003b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:36.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 --2- 172.21.15.193:0/2803688902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe854106be0 0x7fe8540fcb70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:36.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 >> 172.21.15.193:0/2803688902 conn(0x7fe854077800 msgr2=0x7fe854069660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:36.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 shutdown_connections 2024-04-25T01:45:36.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:36.188+0000 7fe85c621700 1 -- 172.21.15.193:0/2803688902 wait complete. 2024-04-25T01:45:36.183 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-04-25T01:45:36.915 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:36 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/2803688902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:45:37.404 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).mds e1 new map 2024-04-25T01:45:37.404 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-25T01:45:37.404 INFO:journalctl@ceph.mon.c.smithi193.stdout: e1 2024-04-25T01:45:37.404 INFO:journalctl@ceph.mon.c.smithi193.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:45:37.405 INFO:journalctl@ceph.mon.c.smithi193.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:45:37.405 INFO:journalctl@ceph.mon.c.smithi193.stdout: legacy client fscid: -1 2024-04-25T01:45:37.405 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-04-25T01:45:37.405 INFO:journalctl@ceph.mon.c.smithi193.stdout: No filesystems configured 2024-04-25T01:45:37.405 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T01:45:37.405 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T01:45:37.748 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T01:45:37.748 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:37.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Deploying cephadm binary to smithi193 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Added host smithi193 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2355028769' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2355028769' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi110:172.21.15.110=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi110:172.21.15.110=b;smithi193:172.21.15.193=c;count:3 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:37.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: Deploying daemon mon.c on smithi193 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/2803688902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 ceph-mon[32981]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 bash[32962]: 0ded0404a1ee47d12e26f343884593cd2a83266418e9ee2108596e7443c3b6dd 2024-04-25T01:45:37.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:37 smithi193 systemd[1]: Started Ceph mon.c for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:45:37.914 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:45:37.915 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:45:38.154 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:45:42.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.669+0000 7f7b0b81b700 1 -- 172.21.15.193:0/1948243450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aec0035c0 msgr2=0x7f7aec005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:42.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.669+0000 7f7b0b81b700 1 --2- 172.21.15.193:0/1948243450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aec0035c0 0x7f7aec005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b04101ca0 tx=0x7f7afc009e80 comp rx=0 tx=0).stop 2024-04-25T01:45:42.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.669+0000 7f7b0b81b700 1 -- 172.21.15.193:0/1948243450 shutdown_connections 2024-04-25T01:45:42.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.669+0000 7f7b0b81b700 1 --2- 172.21.15.193:0/1948243450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aec0035c0 0x7f7aec005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:42.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.669+0000 7f7b0b81b700 1 -- 172.21.15.193:0/1948243450 >> 172.21.15.193:0/1948243450 conn(0x7f7b040fd880 msgr2=0x7f7b040ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:42.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.669+0000 7f7b0b81b700 1 -- 172.21.15.193:0/1948243450 shutdown_connections 2024-04-25T01:45:42.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.669+0000 7f7b0b81b700 1 -- 172.21.15.193:0/1948243450 wait complete. 2024-04-25T01:45:42.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.670+0000 7f7b0b81b700 1 Processor -- start 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.670+0000 7f7b0b81b700 1 -- start start 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.670+0000 7f7b0b81b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aec0035c0 0x7f7b0419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.670+0000 7f7b0b81b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 0x7f7b04199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.670+0000 7f7b0b81b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b04109e00 con 0x7f7aec0035c0 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.670+0000 7f7b0b81b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b04109c80 con 0x7f7b0419fce0 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.671+0000 7f7b08db6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 0x7f7b04199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.671+0000 7f7b08db6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 0x7f7b04199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:58820/0 (socket says 172.21.15.193:58820) 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.671+0000 7f7b08db6700 1 -- 172.21.15.193:0/896645886 learned_addr learned my addr 172.21.15.193:0/896645886 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.671+0000 7f7b08db6700 1 -- 172.21.15.193:0/896645886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 msgr2=0x7f7b04199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.671+0000 7f7b08db6700 1 -- 172.21.15.193:0/896645886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 msgr2=0x7f7b04199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.671+0000 7f7b08db6700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 0x7f7b04199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:45:42.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.671+0000 7f7b08db6700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 0x7f7b04199960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:45:42.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.671+0000 7f7b095b7700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aec0035c0 0x7f7b0419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:42.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.672+0000 7f7b095b7700 1 -- 172.21.15.193:0/896645886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 msgr2=0x7f7b04199960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:45:42.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.672+0000 7f7b095b7700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 0x7f7b04199960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.672+0000 7f7b095b7700 1 -- 172.21.15.193:0/896645886 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b041a0320 con 0x7f7aec0035c0 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.672+0000 7f7b095b7700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aec0035c0 0x7f7b0419f7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b04101ca0 tx=0x7f7afc018fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.672+0000 7f7b027fc700 1 -- 172.21.15.193:0/896645886 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7afc005a30 con 0x7f7aec0035c0 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.672+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b04199ea0 con 0x7f7aec0035c0 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.672+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0419a430 con 0x7f7aec0035c0 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.672+0000 7f7b027fc700 1 -- 172.21.15.193:0/896645886 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7afc007dc0 con 0x7f7aec0035c0 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.672+0000 7f7b027fc700 1 -- 172.21.15.193:0/896645886 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7afc020930 con 0x7f7aec0035c0 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.674+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aec002100 con 0x7f7aec0035c0 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.674+0000 7f7b027fc700 1 -- 172.21.15.193:0/896645886 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f7afc01e070 con 0x7f7aec0035c0 2024-04-25T01:45:42.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.674+0000 7f7b027fc700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7af0038e90 0x7f7af003b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:42.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.674+0000 7f7b027fc700 1 -- 172.21.15.193:0/896645886 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7afc01c070 con 0x7f7aec0035c0 2024-04-25T01:45:42.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.675+0000 7f7b08db6700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7af0038e90 0x7f7af003b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:42.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.676+0000 7f7b08db6700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7af0038e90 0x7f7af003b350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009180 tx=0x7f7af8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:42.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.690+0000 7f7b027fc700 1 -- 172.21.15.193:0/896645886 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7afc0283b0 con 0x7f7aec0035c0 2024-04-25T01:45:42.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:45:42.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: mon.a calling monitor election 2024-04-25T01:45:42.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:42.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: mon.c calling monitor election 2024-04-25T01:45:42.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: monmap epoch 2 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: last_changed 2024-04-25T01:45:37.418234+0000 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: min_mon_release 19 (squid) 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: election_strategy: 1 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: fsmap 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:42.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: mgrmap e13: x(active, since 59s) 2024-04-25T01:45:42.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: overall HEALTH_OK 2024-04-25T01:45:42.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:42.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:42.958 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.964+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7aec002bf0 con 0x7f7aec0035c0 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: mon.a calling monitor election 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: mon.c calling monitor election 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:42.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: monmap epoch 2 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: last_changed 2024-04-25T01:45:37.418234+0000 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: min_mon_release 19 (squid) 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: election_strategy: 1 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: fsmap 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: mgrmap e13: x(active, since 59s) 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: overall HEALTH_OK 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:42.961 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.965+0000 7f7b027fc700 1 -- 172.21.15.193:0/896645886 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f7afc0504d0 con 0x7f7aec0035c0 2024-04-25T01:45:42.961 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:45:42.961 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","modified":"2024-04-25T01:45:37.418234Z","created":"2024-04-25T01:43:30.536087Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T01:45:42.962 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.967+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7af0038e90 msgr2=0x7f7af003b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:42.962 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.967+0000 7f7b0b81b700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7af0038e90 0x7f7af003b350 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009180 tx=0x7f7af8008040 comp rx=0 tx=0).stop 2024-04-25T01:45:42.962 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.967+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aec0035c0 msgr2=0x7f7b0419f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:42.962 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.967+0000 7f7b0b81b700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aec0035c0 0x7f7b0419f7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b04101ca0 tx=0x7f7afc018fb0 comp rx=0 tx=0).stop 2024-04-25T01:45:42.962 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.967+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 shutdown_connections 2024-04-25T01:45:42.962 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.967+0000 7f7b0b81b700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7af0038e90 0x7f7af003b350 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:42.963 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.967+0000 7f7b0b81b700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7aec0035c0 0x7f7b0419f7a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:42.963 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.967+0000 7f7b0b81b700 1 --2- 172.21.15.193:0/896645886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b0419fce0 0x7f7b04199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:42.963 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.967+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 >> 172.21.15.193:0/896645886 conn(0x7f7b040fd880 msgr2=0x7f7b040feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:42.963 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.968+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 shutdown_connections 2024-04-25T01:45:42.963 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:42.968+0000 7f7b0b81b700 1 -- 172.21.15.193:0/896645886 wait complete. 2024-04-25T01:45:42.963 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-04-25T01:45:43.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:43.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:43.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:43.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:43 smithi102 ceph-mon[27796]: Deploying daemon mon.b on smithi110 2024-04-25T01:45:43.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:43 smithi102 ceph-mon[27796]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:43.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:43 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/896645886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:45:43.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:43.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:43.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:43.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:43.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:43 smithi193 ceph-mon[32981]: Deploying daemon mon.b on smithi110 2024-04-25T01:45:43.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:43 smithi193 ceph-mon[32981]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:43.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:43 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/896645886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:45:43.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:44.555 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T01:45:44.555 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T01:45:44.637 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:45:44 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:45:44.418+0000 7fd261c00700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T01:45:44.797 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:45:45.831 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:45 smithi193 ceph-mon[32981]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:45.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:45 smithi102 ceph-mon[27796]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.272 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(???) e0 preinit fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:46.272 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T01:45:46.272 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T01:45:46.272 INFO:journalctl@ceph.mon.b.smithi110.stdout: e1 2024-04-25T01:45:46.273 INFO:journalctl@ceph.mon.b.smithi110.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:45:46.273 INFO:journalctl@ceph.mon.b.smithi110.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:45:46.273 INFO:journalctl@ceph.mon.b.smithi110.stdout: legacy client fscid: -1 2024-04-25T01:45:46.273 INFO:journalctl@ceph.mon.b.smithi110.stdout: 2024-04-25T01:45:46.273 INFO:journalctl@ceph.mon.b.smithi110.stdout: No filesystems configured 2024-04-25T01:45:46.273 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T01:45:46.730 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:45:46.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:46.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Deploying cephadm binary to smithi193 2024-04-25T01:45:46.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Added host smithi193 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2355028769' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2355028769' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T01:45:46.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi110:172.21.15.110=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi110:172.21.15.110=b;smithi193:172.21.15.193=c;count:3 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:46.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Deploying daemon mon.c on smithi193 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/2803688902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.a calling monitor election 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.c calling monitor election 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: monmap epoch 2 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: last_changed 2024-04-25T01:45:37.418234+0000 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: min_mon_release 19 (squid) 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: election_strategy: 1 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: fsmap 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:46.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mgrmap e13: x(active, since 59s) 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: overall HEALTH_OK 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: Deploying daemon mon.b on smithi110 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/896645886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 ceph-mon[32848]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 bash[32829]: b5efaa94d32a25aecb916be99cb717878be4328379788bba6117289998654c66 2024-04-25T01:45:46.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:46 smithi110 systemd[1]: Started Ceph mon.b for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:45:51.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: mon.c calling monitor election 2024-04-25T01:45:51.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:45:51.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: mon.a calling monitor election 2024-04-25T01:45:51.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:51.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:51.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:51.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: monmap epoch 3 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: last_changed 2024-04-25T01:45:46.285407+0000 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: min_mon_release 19 (squid) 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: election_strategy: 1 2024-04-25T01:45:51.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:45:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-25T01:45:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-25T01:45:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: fsmap 2024-04-25T01:45:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: mgrmap e13: x(active, since 68s) 2024-04-25T01:45:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: overall HEALTH_OK 2024-04-25T01:45:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:51.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:51.748 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: mon.c calling monitor election 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: mon.a calling monitor election 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:45:51.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: monmap epoch 3 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: last_changed 2024-04-25T01:45:46.285407+0000 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: min_mon_release 19 (squid) 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: election_strategy: 1 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: fsmap 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: mgrmap e13: x(active, since 68s) 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: overall HEALTH_OK 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:51.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:52.346 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.351+0000 7efe3824f700 1 -- 172.21.15.193:0/2929375627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1c0084b0 msgr2=0x7efe1c00a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:52.346 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.351+0000 7efe3824f700 1 --2- 172.21.15.193:0/2929375627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1c0084b0 0x7efe1c00a920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efe30101ce0 tx=0x7efe2000fe90 comp rx=0 tx=0).stop 2024-04-25T01:45:52.346 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.351+0000 7efe3824f700 1 -- 172.21.15.193:0/2929375627 shutdown_connections 2024-04-25T01:45:52.346 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.351+0000 7efe3824f700 1 --2- 172.21.15.193:0/2929375627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1c0084b0 0x7efe1c00a920 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:52.346 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.351+0000 7efe3824f700 1 -- 172.21.15.193:0/2929375627 >> 172.21.15.193:0/2929375627 conn(0x7efe300fc710 msgr2=0x7efe300feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:52.346 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.352+0000 7efe3824f700 1 -- 172.21.15.193:0/2929375627 shutdown_connections 2024-04-25T01:45:52.346 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.352+0000 7efe3824f700 1 -- 172.21.15.193:0/2929375627 wait complete. 2024-04-25T01:45:52.349 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.352+0000 7efe3824f700 1 Processor -- start 2024-04-25T01:45:52.349 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe3824f700 1 -- start start 2024-04-25T01:45:52.349 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe3824f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1c0084b0 0x7efe3019b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:52.349 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe3824f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 0x7efe30195760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:52.349 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe3824f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe30195ca0 0x7efe30196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:52.349 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe3824f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efe30107c80 con 0x7efe1c0084b0 2024-04-25T01:45:52.349 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe3824f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efe30107b00 con 0x7efe30195ca0 2024-04-25T01:45:52.349 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe3824f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efe30107e00 con 0x7efe3019bbd0 2024-04-25T01:45:52.349 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe357ea700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 0x7efe30195760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe357ea700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 0x7efe30195760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45804/0 (socket says 172.21.15.193:45804) 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe357ea700 1 -- 172.21.15.193:0/2534836490 learned_addr learned my addr 172.21.15.193:0/2534836490 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe357ea700 1 -- 172.21.15.193:0/2534836490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 msgr2=0x7efe30195760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe357ea700 1 -- 172.21.15.193:0/2534836490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 msgr2=0x7efe30195760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe357ea700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 0x7efe30195760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe357ea700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 0x7efe30195760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe35feb700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1c0084b0 0x7efe3019b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe367ec700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe30195ca0 0x7efe30196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe367ec700 1 -- 172.21.15.193:0/2534836490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe30195ca0 msgr2=0x7efe30196150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe367ec700 1 -- 172.21.15.193:0/2534836490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe30195ca0 msgr2=0x7efe30196150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe367ec700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe30195ca0 0x7efe30196150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.353+0000 7efe367ec700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe30195ca0 0x7efe30196150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:45:52.350 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe35feb700 1 -- 172.21.15.193:0/2534836490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 msgr2=0x7efe30195760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe35feb700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 0x7efe30195760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe35feb700 1 -- 172.21.15.193:0/2534836490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe30195ca0 msgr2=0x7efe30196150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe35feb700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe30195ca0 0x7efe30196150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe35feb700 1 -- 172.21.15.193:0/2534836490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe301a2440 con 0x7efe1c0084b0 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe35feb700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1c0084b0 0x7efe3019b690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe20010010 tx=0x7efe20012cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe26ffd700 1 -- 172.21.15.193:0/2534836490 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20011d20 con 0x7efe1c0084b0 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe26ffd700 1 -- 172.21.15.193:0/2534836490 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe2002a070 con 0x7efe1c0084b0 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe301a2670 con 0x7efe1c0084b0 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe26ffd700 1 -- 172.21.15.193:0/2534836490 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2001db50 con 0x7efe1c0084b0 2024-04-25T01:45:52.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.354+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe301a2ba0 con 0x7efe1c0084b0 2024-04-25T01:45:52.352 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.355+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe30102360 con 0x7efe1c0084b0 2024-04-25T01:45:52.352 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.356+0000 7efe26ffd700 1 -- 172.21.15.193:0/2534836490 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7efe2002c020 con 0x7efe1c0084b0 2024-04-25T01:45:52.352 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.356+0000 7efe26ffd700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7efe14039580 0x7efe1403ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:52.352 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.356+0000 7efe26ffd700 1 -- 172.21.15.193:0/2534836490 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efe2004ebc0 con 0x7efe1c0084b0 2024-04-25T01:45:52.352 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.356+0000 7efe357ea700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7efe14039580 0x7efe1403ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:52.352 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.357+0000 7efe357ea700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7efe14039580 0x7efe1403ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009180 tx=0x7efe2c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:52.356 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.363+0000 7efe26ffd700 1 -- 172.21.15.193:0/2534836490 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efe2000c2d0 con 0x7efe1c0084b0 2024-04-25T01:45:52.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.647+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7efe3002db30 con 0x7efe1c0084b0 2024-04-25T01:45:52.642 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:52.642 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:52.642 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:52.642 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:52.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.648+0000 7efe26ffd700 1 -- 172.21.15.193:0/2534836490 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7efe2000de30 con 0x7efe1c0084b0 2024-04-25T01:45:52.643 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-04-25T01:45:52.643 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","modified":"2024-04-25T01:45:46.285407Z","created":"2024-04-25T01:43:30.536087Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T01:45:52.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7efe14039580 msgr2=0x7efe1403ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:52.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7efe14039580 0x7efe1403ba40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efe2c009180 tx=0x7efe2c008040 comp rx=0 tx=0).stop 2024-04-25T01:45:52.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1c0084b0 msgr2=0x7efe3019b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:52.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1c0084b0 0x7efe3019b690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efe20010010 tx=0x7efe20012cb0 comp rx=0 tx=0).stop 2024-04-25T01:45:52.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 shutdown_connections 2024-04-25T01:45:52.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7efe14039580 0x7efe1403ba40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:52.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe30195ca0 0x7efe30196150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:52.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe1c0084b0 0x7efe3019b690 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:52.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 --2- 172.21.15.193:0/2534836490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe3019bbd0 0x7efe30195760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:52.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 >> 172.21.15.193:0/2534836490 conn(0x7efe300fc710 msgr2=0x7efe300fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:52.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 shutdown_connections 2024-04-25T01:45:52.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:45:52.650+0000 7efe3824f700 1 -- 172.21.15.193:0/2534836490 wait complete. 2024-04-25T01:45:52.646 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-04-25T01:45:52.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:52.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:52.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:52.666 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:53.583 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T01:45:53.584 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T01:45:53.730 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mon.c calling monitor election 2024-04-25T01:45:53.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:45:53.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mon.a calling monitor election 2024-04-25T01:45:53.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:53.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:45:53.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:53.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:53.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:53.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:53.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:53.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:53.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:53.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:53.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T01:45:53.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: monmap epoch 3 2024-04-25T01:45:53.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:53.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: last_changed 2024-04-25T01:45:46.285407+0000 2024-04-25T01:45:53.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: min_mon_release 19 (squid) 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: election_strategy: 1 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: fsmap 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mgrmap e13: x(active, since 68s) 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: overall HEALTH_OK 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:53.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:45:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:45:54.207 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: mon.b calling monitor election 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: mon.b calling monitor election 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: mon.c calling monitor election 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: mon.a calling monitor election 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: monmap epoch 3 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:54.208 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: last_changed 2024-04-25T01:45:46.285407+0000 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: min_mon_release 19 (squid) 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: election_strategy: 1 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: fsmap 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: mgrmap e13: x(active, since 70s) 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: overall HEALTH_OK 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.209 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:53 smithi102 ceph-mon[27796]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:54.230 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mon.b calling monitor election 2024-04-25T01:45:54.230 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-25T01:45:54.230 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mon.b calling monitor election 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mon.c calling monitor election 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mon.a calling monitor election 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: monmap epoch 3 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: last_changed 2024-04-25T01:45:46.285407+0000 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: min_mon_release 19 (squid) 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: election_strategy: 1 2024-04-25T01:45:54.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: fsmap 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: mgrmap e13: x(active, since 70s) 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: overall HEALTH_OK 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:53 smithi110 ceph-mon[32848]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:54.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: mon.b calling monitor election 2024-04-25T01:45:54.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-25T01:45:54.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: mon.b calling monitor election 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: mon.c calling monitor election 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: mon.a calling monitor election 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: monmap epoch 3 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: last_changed 2024-04-25T01:45:46.285407+0000 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: created 2024-04-25T01:43:30.536087+0000 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: min_mon_release 19 (squid) 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: election_strategy: 1 2024-04-25T01:45:54.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: 0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: 2: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: fsmap 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: mgrmap e13: x(active, since 70s) 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: overall HEALTH_OK 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:54.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:53 smithi193 ceph-mon[32981]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:45:55.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:55.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.214 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:55.215 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: Reconfiguring daemon mon.a on smithi102 2024-04-25T01:45:55.216 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:54 smithi102 ceph-mon[27796]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:55.230 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:55.230 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.230 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: Reconfiguring daemon mon.a on smithi102 2024-04-25T01:45:55.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:54 smithi110 ceph-mon[32848]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:55.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:45:55.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: Reconfiguring daemon mon.a on smithi102 2024-04-25T01:45:55.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:54 smithi193 ceph-mon[32981]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.329+0000 7f4e0dc00700 1 -- 172.21.15.102:0/3140461810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e08101e70 msgr2=0x7f4e08102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.329+0000 7f4e0dc00700 1 --2- 172.21.15.102:0/3140461810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e08101e70 0x7f4e08102250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4df8009910 tx=0x7f4df8009c20 comp rx=0 tx=0).stop 2024-04-25T01:45:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.329+0000 7f4e0dc00700 1 -- 172.21.15.102:0/3140461810 shutdown_connections 2024-04-25T01:45:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.329+0000 7f4e0dc00700 1 --2- 172.21.15.102:0/3140461810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e08101e70 0x7f4e08102250 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.329+0000 7f4e0dc00700 1 -- 172.21.15.102:0/3140461810 >> 172.21.15.102:0/3140461810 conn(0x7f4e080fd720 msgr2=0x7f4e080ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.329+0000 7f4e0dc00700 1 -- 172.21.15.102:0/3140461810 shutdown_connections 2024-04-25T01:45:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.329+0000 7f4e0dc00700 1 -- 172.21.15.102:0/3140461810 wait complete. 2024-04-25T01:45:57.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.330+0000 7f4e0dc00700 1 Processor -- start 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.330+0000 7f4e0dc00700 1 -- start start 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e0dc00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e08101e70 0x7f4e0819f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e0dc00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e0819fd40 0x7f4e081999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e0dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e08199f00 0x7f4e0819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e0dc00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0810c240 con 0x7f4e0819fd40 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e0dc00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0810c0c0 con 0x7f4e08101e70 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e0dc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0810c3c0 con 0x7f4e08199f00 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e06a00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e08101e70 0x7f4e0819f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e06000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e0819fd40 0x7f4e081999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e06a00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e08101e70 0x7f4e0819f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:39940/0 (socket says 172.21.15.102:39940) 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e06a00700 1 -- 172.21.15.102:0/3787787359 learned_addr learned my addr 172.21.15.102:0/3787787359 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:45:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.331+0000 7f4e07400700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e08199f00 0x7f4e0819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:57.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4e06a00700 1 -- 172.21.15.102:0/3787787359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e08199f00 msgr2=0x7f4e0819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:57.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4e06a00700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e08199f00 0x7f4e0819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:57.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4e06a00700 1 -- 172.21.15.102:0/3787787359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e0819fd40 msgr2=0x7f4e081999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:57.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4e06a00700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e0819fd40 0x7f4e081999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4e06a00700 1 -- 172.21.15.102:0/3787787359 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e081a66d0 con 0x7f4e08101e70 2024-04-25T01:45:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4e07400700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e08199f00 0x7f4e0819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:45:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4e06a00700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e08101e70 0x7f4e0819f800 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4df8005850 tx=0x7f4df8004f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4dff400700 1 -- 172.21.15.102:0/3787787359 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df8026070 con 0x7f4e08101e70 2024-04-25T01:45:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4dff400700 1 -- 172.21.15.102:0/3787787359 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4df800b6f0 con 0x7f4e08101e70 2024-04-25T01:45:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4dff400700 1 -- 172.21.15.102:0/3787787359 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df80038e0 con 0x7f4e08101e70 2024-04-25T01:45:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.332+0000 7f4e0dc00700 1 -- 172.21.15.102:0/3787787359 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e081a6900 con 0x7f4e08101e70 2024-04-25T01:45:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.333+0000 7f4e0dc00700 1 -- 172.21.15.102:0/3787787359 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e081a6e90 con 0x7f4e08101e70 2024-04-25T01:45:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.334+0000 7f4dff400700 1 -- 172.21.15.102:0/3787787359 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f4df8029070 con 0x7f4e08101e70 2024-04-25T01:45:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.334+0000 7f4dfcc00700 1 -- 172.21.15.102:0/3787787359 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de0005330 con 0x7f4e08101e70 2024-04-25T01:45:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.334+0000 7f4dff400700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dec039580 0x7f4dec03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:45:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.334+0000 7f4dff400700 1 -- 172.21.15.102:0/3787787359 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4df804ceb0 con 0x7f4e08101e70 2024-04-25T01:45:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.334+0000 7f4e06000700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dec039580 0x7f4dec03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:45:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.335+0000 7f4e06000700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dec039580 0x7f4dec03ba40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009ad0 tx=0x7f4df4009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:45:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.339+0000 7f4dff400700 1 -- 172.21.15.102:0/3787787359 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4df8013070 con 0x7f4e08101e70 2024-04-25T01:45:57.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.564+0000 7f4dfcc00700 1 -- 172.21.15.102:0/3787787359 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f4de0005620 con 0x7f4e08101e70 2024-04-25T01:45:57.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.565+0000 7f4dff400700 1 -- 172.21.15.102:0/3787787359 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f4df800bc80 con 0x7f4e08101e70 2024-04-25T01:45:57.558 INFO:teuthology.orchestra.run.smithi102.stdout:# minimal ceph.conf for 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:57.558 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-04-25T01:45:57.558 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:45:57.558 INFO:teuthology.orchestra.run.smithi102.stdout: mon_host = [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-04-25T01:45:57.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 -- 172.21.15.102:0/3787787359 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dec039580 msgr2=0x7f4dec03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:57.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dec039580 0x7f4dec03ba40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009ad0 tx=0x7f4df4009210 comp rx=0 tx=0).stop 2024-04-25T01:45:57.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 -- 172.21.15.102:0/3787787359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e08101e70 msgr2=0x7f4e0819f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:45:57.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e08101e70 0x7f4e0819f800 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4df8005850 tx=0x7f4df8004f80 comp rx=0 tx=0).stop 2024-04-25T01:45:57.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 -- 172.21.15.102:0/3787787359 shutdown_connections 2024-04-25T01:45:57.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dec039580 0x7f4dec03ba40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:57.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e08101e70 0x7f4e0819f800 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:57.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e08199f00 0x7f4e0819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:57.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 --2- 172.21.15.102:0/3787787359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e0819fd40 0x7f4e081999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:45:57.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 -- 172.21.15.102:0/3787787359 >> 172.21.15.102:0/3787787359 conn(0x7f4e080fd720 msgr2=0x7f4e08074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:45:57.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 -- 172.21.15.102:0/3787787359 shutdown_connections 2024-04-25T01:45:57.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:45:57.569+0000 7f4dfcc00700 1 -- 172.21.15.102:0/3787787359 wait complete. 2024-04-25T01:45:57.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:57 smithi102 ceph-mon[27796]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:57.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:57.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:57.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:57.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:45:57.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:57.872 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:57 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3787787359' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:57.963 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:57 smithi110 ceph-mon[32848]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:57.963 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:57.964 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:57.964 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:57.964 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:45:57.964 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:57.964 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:57 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3787787359' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:57 smithi193 ceph-mon[32981]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:45:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:57 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3787787359' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:58.189 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T01:45:58.189 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:45:58.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:45:58.275 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:45:58.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:58.396 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:45:58.397 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:45:58.423 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:45:58.423 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:58.487 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:45:58.487 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T01:45:58.519 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:45:58.520 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:45:58.586 INFO:tasks.cephadm:Adding mgr.x on smithi102 2024-04-25T01:45:58.586 INFO:tasks.cephadm:Adding mgr.y on smithi110 2024-04-25T01:45:58.586 INFO:tasks.cephadm:Adding mgr.z on smithi193 2024-04-25T01:45:58.587 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi102=x;smithi110=y;smithi193=z' 2024-04-25T01:45:58.706 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:58 smithi110 ceph-mon[32848]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:45:58.706 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:58 smithi110 ceph-mon[32848]: Reconfiguring daemon mon.b on smithi110 2024-04-25T01:45:58.706 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:58.706 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:58.706 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:58.706 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:45:58.707 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:58.862 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:45:58.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:58 smithi102 ceph-mon[27796]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:45:58.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:58 smithi102 ceph-mon[27796]: Reconfiguring daemon mon.b on smithi110 2024-04-25T01:45:58.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:58.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:58.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:58.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:45:58.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:58.983 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:58 smithi193 ceph-mon[32981]: Reconfiguring mon.b (monmap changed)... 2024-04-25T01:45:58.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:58 smithi193 ceph-mon[32981]: Reconfiguring daemon mon.b on smithi110 2024-04-25T01:45:58.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:58.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:45:58.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T01:45:58.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T01:45:58.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:45:59.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:59 smithi102 ceph-mon[27796]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:45:59.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:59 smithi102 ceph-mon[27796]: Reconfiguring daemon mon.c on smithi193 2024-04-25T01:45:59.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:45:59 smithi102 ceph-mon[27796]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:59.934 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:59 smithi193 ceph-mon[32981]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:45:59.934 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:59 smithi193 ceph-mon[32981]: Reconfiguring daemon mon.c on smithi193 2024-04-25T01:45:59.934 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:45:59 smithi193 ceph-mon[32981]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:45:59.980 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:59 smithi110 ceph-mon[32848]: Reconfiguring mon.c (monmap changed)... 2024-04-25T01:45:59.980 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:59 smithi110 ceph-mon[32848]: Reconfiguring daemon mon.c on smithi193 2024-04-25T01:45:59.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:45:59 smithi110 ceph-mon[32848]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:00.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.487+0000 7f5be37fe700 1 -- 172.21.15.193:0/1228065119 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80056d0 con 0x7f5be4100cb0 2024-04-25T01:46:00.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.488+0000 7f5beaf38700 1 -- 172.21.15.193:0/1228065119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5be4100cb0 msgr2=0x7f5be41010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:00.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.488+0000 7f5beaf38700 1 --2- 172.21.15.193:0/1228065119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5be4100cb0 0x7f5be41010d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009910 tx=0x7f5bd8009c20 comp rx=0 tx=0).stop 2024-04-25T01:46:00.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.488+0000 7f5beaf38700 1 -- 172.21.15.193:0/1228065119 shutdown_connections 2024-04-25T01:46:00.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.488+0000 7f5beaf38700 1 --2- 172.21.15.193:0/1228065119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5be4100cb0 0x7f5be41010d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:00.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.488+0000 7f5beaf38700 1 -- 172.21.15.193:0/1228065119 >> 172.21.15.193:0/1228065119 conn(0x7f5be40fc270 msgr2=0x7f5be40fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:00.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.488+0000 7f5beaf38700 1 -- 172.21.15.193:0/1228065119 shutdown_connections 2024-04-25T01:46:00.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.488+0000 7f5beaf38700 1 -- 172.21.15.193:0/1228065119 wait complete. 2024-04-25T01:46:00.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5beaf38700 1 Processor -- start 2024-04-25T01:46:00.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5beaf38700 1 -- start start 2024-04-25T01:46:00.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5beaf38700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5be4100cb0 0x7f5be41953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:00.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5beaf38700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5be41958e0 0x7f5be419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:00.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5beaf38700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5be4195d60 0x7f5be419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:00.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5beaf38700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5be4105be0 con 0x7f5be4100cb0 2024-04-25T01:46:00.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5beaf38700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5be4105a60 con 0x7f5be4195d60 2024-04-25T01:46:00.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5beaf38700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5be4105d60 con 0x7f5be41958e0 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5be3fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5be41958e0 0x7f5be419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5be3fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5be41958e0 0x7f5be419a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:52164/0 (socket says 172.21.15.193:52164) 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5be3fff700 1 -- 172.21.15.193:0/1303132336 learned_addr learned my addr 172.21.15.193:0/1303132336 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5be94d5700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5be4195d60 0x7f5be419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5be3fff700 1 -- 172.21.15.193:0/1303132336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5be4195d60 msgr2=0x7f5be419cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5be3fff700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5be4195d60 0x7f5be419cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5be3fff700 1 -- 172.21.15.193:0/1303132336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5be4100cb0 msgr2=0x7f5be41953a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5be3fff700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5be4100cb0 0x7f5be41953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.489+0000 7f5be3fff700 1 -- 172.21.15.193:0/1303132336 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be419d7b0 con 0x7f5be41958e0 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.490+0000 7f5be3fff700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5be41958e0 0x7f5be419a950 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000b780 tx=0x7f5bd000bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.490+0000 7f5be1ffb700 1 -- 172.21.15.193:0/1303132336 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0004230 con 0x7f5be41958e0 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.490+0000 7f5be1ffb700 1 -- 172.21.15.193:0/1303132336 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd00043d0 con 0x7f5be41958e0 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.490+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be419daa0 con 0x7f5be41958e0 2024-04-25T01:46:00.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.490+0000 7f5be1ffb700 1 -- 172.21.15.193:0/1303132336 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0011c60 con 0x7f5be41958e0 2024-04-25T01:46:00.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.490+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be419e0b0 con 0x7f5be41958e0 2024-04-25T01:46:00.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.491+0000 7f5be1ffb700 1 -- 172.21.15.193:0/1303132336 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f5bd001a400 con 0x7f5be41958e0 2024-04-25T01:46:00.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.491+0000 7f5be1ffb700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5bd403b790 0x7f5bd403dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:00.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.491+0000 7f5be1ffb700 1 -- 172.21.15.193:0/1303132336 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5bd004be30 con 0x7f5be41958e0 2024-04-25T01:46:00.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.491+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5be4063890 con 0x7f5be41958e0 2024-04-25T01:46:00.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.491+0000 7f5be8cd4700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5bd403b790 0x7f5bd403dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:00.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.492+0000 7f5be8cd4700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5bd403b790 0x7f5bd403dc50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8005330 tx=0x7f5bd8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:00.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.496+0000 7f5be1ffb700 1 -- 172.21.15.193:0/1303132336 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5bd0003c80 con 0x7f5be41958e0 2024-04-25T01:46:00.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.708+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi102=x;smithi110=y;smithi193=z", "target": ["mon-mgr", ""]}) -- 0x7f5be40627c0 con 0x7f5bd403b790 2024-04-25T01:46:00.714 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.721+0000 7f5be1ffb700 1 -- 172.21.15.193:0/1303132336 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5be40627c0 con 0x7f5bd403b790 2024-04-25T01:46:00.715 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5bd403b790 msgr2=0x7f5bd403dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5bd403b790 0x7f5bd403dc50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8005330 tx=0x7f5bd8005fb0 comp rx=0 tx=0).stop 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5be41958e0 msgr2=0x7f5be419a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5be41958e0 0x7f5be419a950 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000b780 tx=0x7f5bd000bb40 comp rx=0 tx=0).stop 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 shutdown_connections 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5bd403b790 0x7f5bd403dc50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5be4195d60 0x7f5be419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5be4100cb0 0x7f5be41953a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 --2- 172.21.15.193:0/1303132336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5be41958e0 0x7f5be419a950 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:00.717 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 >> 172.21.15.193:0/1303132336 conn(0x7f5be40fc270 msgr2=0x7f5be40fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:00.718 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 shutdown_connections 2024-04-25T01:46:00.718 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:46:00.723+0000 7f5beaf38700 1 -- 172.21.15.193:0/1303132336 wait complete. 2024-04-25T01:46:01.947 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:01.947 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi102=x;smithi110=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: Saving service mgr spec with placement smithi102=x;smithi110=y;smithi193=z;count:3 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:01.948 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:01.976 DEBUG:teuthology.orchestra.run.smithi110:mgr.y> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T01:46:01.979 DEBUG:teuthology.orchestra.run.smithi193:mgr.z> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T01:46:01.981 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T01:46:01.981 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:46:01.981 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:46:01.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:01.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi102=x;smithi110=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:01.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: Saving service mgr spec with placement smithi102=x;smithi110=y;smithi193=z;count:3 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:01.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:02.042 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:46:02.042 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-04-25T01:46:02.095 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:46:02.095 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 634 Links: 1 2024-04-25T01:46:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:02.095 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:02.096 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-25 01:44:59.796715259 +0000 2024-04-25T01:46:02.096 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-25 01:40:02.622187518 +0000 2024-04-25T01:46:02.096 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-25 01:40:02.622187518 +0000 2024-04-25T01:46:02.096 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-25 01:40:02.622187518 +0000 2024-04-25T01:46:02.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:46:02.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:02.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi102=x;smithi110=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:02.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: Saving service mgr spec with placement smithi102=x;smithi110=y;smithi193=z;count:3 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:02.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:02.176 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-25T01:46:02.176 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-25T01:46:02.176 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000343084 s, 1.5 MB/s 2024-04-25T01:46:02.181 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:46:02.233 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-04-25T01:46:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:46:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:02.280 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-25T01:46:02.280 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:02.280 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:02.280 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-25 01:44:59.895713207 +0000 2024-04-25T01:46:02.280 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-25 01:40:03.200175415 +0000 2024-04-25T01:46:02.280 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-25 01:40:03.200175415 +0000 2024-04-25T01:46:02.280 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-25 01:40:03.200175415 +0000 2024-04-25T01:46:02.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:46:02.356 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-25T01:46:02.356 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-25T01:46:02.356 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000338828 s, 1.5 MB/s 2024-04-25T01:46:02.360 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:46:02.415 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-04-25T01:46:02.466 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:46:02.466 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:02.466 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-04-25T01:46:02.466 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:02.466 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:02.466 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-25 01:44:59.965711756 +0000 2024-04-25T01:46:02.467 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-25 01:40:03.865161491 +0000 2024-04-25T01:46:02.467 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-25 01:40:03.865161491 +0000 2024-04-25T01:46:02.467 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-25 01:40:03.865161491 +0000 2024-04-25T01:46:02.467 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:46:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-25T01:46:02.547 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-25T01:46:02.547 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000345762 s, 1.5 MB/s 2024-04-25T01:46:02.550 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:46:02.601 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-04-25T01:46:02.653 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:46:02.653 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:02.653 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-25T01:46:02.653 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:02.653 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:02.653 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-04-25 01:45:00.041710180 +0000 2024-04-25T01:46:02.653 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-04-25 01:40:04.471148802 +0000 2024-04-25T01:46:02.653 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-04-25 01:40:04.471148802 +0000 2024-04-25T01:46:02.653 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-04-25 01:40:04.471148802 +0000 2024-04-25T01:46:02.654 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:46:02.731 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-04-25T01:46:02.731 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-04-25T01:46:02.731 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000405583 s, 1.3 MB/s 2024-04-25T01:46:02.735 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:46:02.759 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:02 smithi110 ceph-mon[32848]: Deploying daemon mgr.z on smithi193 2024-04-25T01:46:02.793 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-04-25T01:46:02.793 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:46:02.810 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:46:02.810 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-04-25T01:46:02.866 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:46:02.866 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:02.866 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T01:46:02.866 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:02.866 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:02.866 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-25 01:45:45.738759781 +0000 2024-04-25T01:46:02.866 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-25 01:39:32.731943825 +0000 2024-04-25T01:46:02.866 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-25 01:39:32.731943825 +0000 2024-04-25T01:46:02.866 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-04-25 01:39:32.731943825 +0000 2024-04-25T01:46:02.867 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:46:02.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:02 smithi102 ceph-mon[27796]: Deploying daemon mgr.z on smithi193 2024-04-25T01:46:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-25T01:46:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-25T01:46:02.933 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.0506e-05 s, 5.7 MB/s 2024-04-25T01:46:02.935 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:46:02.991 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-04-25T01:46:02.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:02 smithi193 ceph-mon[32981]: Deploying daemon mgr.z on smithi193 2024-04-25T01:46:03.046 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:46:03.046 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:03.046 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T01:46:03.046 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:03.046 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:03.046 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-25 01:45:45.739759763 +0000 2024-04-25T01:46:03.046 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-25 01:39:33.046938301 +0000 2024-04-25T01:46:03.047 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-25 01:39:33.046938301 +0000 2024-04-25T01:46:03.047 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-04-25 01:39:33.046938301 +0000 2024-04-25T01:46:03.047 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:46:03.109 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-25T01:46:03.109 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-25T01:46:03.109 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.8912e-05 s, 5.8 MB/s 2024-04-25T01:46:03.110 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:46:03.166 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-04-25T01:46:03.222 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:46:03.222 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:03.222 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T01:46:03.222 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:03.222 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:03.223 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-25 01:45:45.739759763 +0000 2024-04-25T01:46:03.223 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-25 01:39:33.369932637 +0000 2024-04-25T01:46:03.223 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-25 01:39:33.369932637 +0000 2024-04-25T01:46:03.223 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-04-25 01:39:33.369932637 +0000 2024-04-25T01:46:03.223 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:46:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-25T01:46:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-25T01:46:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.9031e-05 s, 5.8 MB/s 2024-04-25T01:46:03.286 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:46:03.341 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-04-25T01:46:03.396 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:46:03.396 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:03.396 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-25T01:46:03.396 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:03.397 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:03.397 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-04-25 01:45:45.739759763 +0000 2024-04-25T01:46:03.397 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-04-25 01:39:33.695926921 +0000 2024-04-25T01:46:03.397 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-04-25 01:39:33.695926921 +0000 2024-04-25T01:46:03.397 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-04-25 01:39:33.695926921 +0000 2024-04-25T01:46:03.397 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:46:03.457 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-04-25T01:46:03.458 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-04-25T01:46:03.458 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.6882e-05 s, 5.9 MB/s 2024-04-25T01:46:03.459 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:46:03.515 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-04-25T01:46:03.515 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T01:46:03.531 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T01:46:03.531 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-04-25T01:46:03.586 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T01:46:03.586 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:03.586 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T01:46:03.586 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:03.586 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:03.586 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-04-25 01:46:03.543419687 +0000 2024-04-25T01:46:03.586 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-04-25 01:39:00.419960471 +0000 2024-04-25T01:46:03.586 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-04-25 01:39:00.419960471 +0000 2024-04-25T01:46:03.586 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-04-25 01:39:00.419960471 +0000 2024-04-25T01:46:03.587 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T01:46:03.652 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-04-25T01:46:03.652 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-04-25T01:46:03.652 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000200191 s, 2.6 MB/s 2024-04-25T01:46:03.653 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T01:46:03.711 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-04-25T01:46:03.766 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T01:46:03.766 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:03.766 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T01:46:03.767 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:03.767 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:03.767 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-04-25 01:46:03.544419672 +0000 2024-04-25T01:46:03.767 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-04-25 01:39:00.758955061 +0000 2024-04-25T01:46:03.767 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-04-25 01:39:00.758955061 +0000 2024-04-25T01:46:03.767 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-04-25 01:39:00.758955061 +0000 2024-04-25T01:46:03.767 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T01:46:03.828 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-04-25T01:46:03.828 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-04-25T01:46:03.828 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000208609 s, 2.5 MB/s 2024-04-25T01:46:03.829 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T01:46:03.886 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-04-25T01:46:03.941 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T01:46:03.942 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:03.942 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T01:46:03.942 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:03.942 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:03.942 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-04-25 01:46:03.544419672 +0000 2024-04-25T01:46:03.942 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-04-25 01:39:01.089949780 +0000 2024-04-25T01:46:03.942 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-04-25 01:39:01.089949780 +0000 2024-04-25T01:46:03.942 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-04-25 01:39:01.089949780 +0000 2024-04-25T01:46:03.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T01:46:03.980 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:03 smithi110 ceph-mon[32848]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:03.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:03 smithi193 ceph-mon[32981]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:04.005 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-04-25T01:46:04.005 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-04-25T01:46:04.005 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000193669 s, 2.6 MB/s 2024-04-25T01:46:04.006 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T01:46:04.062 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-04-25T01:46:04.117 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T01:46:04.117 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T01:46:04.117 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T01:46:04.117 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T01:46:04.117 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T01:46:04.117 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-04-25 01:46:03.544419672 +0000 2024-04-25T01:46:04.118 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-04-25 01:39:01.425944419 +0000 2024-04-25T01:46:04.118 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-04-25 01:39:01.425944419 +0000 2024-04-25T01:46:04.118 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-04-25 01:39:01.425944419 +0000 2024-04-25T01:46:04.118 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T01:46:04.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:03 smithi102 ceph-mon[27796]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:04.181 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-04-25T01:46:04.181 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-04-25T01:46:04.181 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000197738 s, 2.6 MB/s 2024-04-25T01:46:04.183 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T01:46:04.238 INFO:tasks.cephadm:Deploying osd.0 on smithi102 with /dev/vg_nvme/lv_4... 2024-04-25T01:46:04.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:46:04.395 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:04.214+0000 7fb64db57200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:46:04.396 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:04.402+0000 7fb64db57200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:46:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:46:04.748 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:04.505+0000 7fb64db57200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:46:05.233 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:04.814+0000 7fb64db57200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:46:05.233 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:04.899+0000 7fb64db57200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:46:05.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:05.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:05.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:05.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:05.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:05.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:05.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:05.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:05 smithi102 ceph-mon[27796]: Deploying daemon mgr.y on smithi110 2024-04-25T01:46:05.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:05 smithi102 ceph-mon[27796]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:05.480 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:05.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:05.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:05.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:05.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:05.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:05.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:05.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:05 smithi110 ceph-mon[32848]: Deploying daemon mgr.y on smithi110 2024-04-25T01:46:05.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:05 smithi110 ceph-mon[32848]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:05.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:05.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:05.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:05.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:05.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T01:46:05.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:05.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:05.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:05 smithi193 ceph-mon[32981]: Deploying daemon mgr.y on smithi110 2024-04-25T01:46:05.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:05 smithi193 ceph-mon[32981]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:06.426 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 systemd[1]: Starting Ceph mgr.y for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:46:06.426 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 podman[34223]: 2024-04-25 01:46:06.334577125 +0000 UTC m=+0.017589374 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:46:06.436 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:06.103+0000 7fb64db57200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:46:06.436 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:06.270+0000 7fb64db57200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:46:06.436 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:06.442+0000 7fb64db57200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:46:06.730 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 podman[34223]: 2024-04-25 01:46:06.449362184 +0000 UTC m=+0.132374452 container create 9c2040968d4de02d200a1af591d19e8a03b23f7529c8c058822387b8d7968c89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9) 2024-04-25T01:46:06.730 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 podman[34223]: 2024-04-25 01:46:06.651787323 +0000 UTC m=+0.334799554 container init 9c2040968d4de02d200a1af591d19e8a03b23f7529c8c058822387b8d7968c89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T01:46:06.730 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 podman[34223]: 2024-04-25 01:46:06.654245747 +0000 UTC m=+0.337257981 container start 9c2040968d4de02d200a1af591d19e8a03b23f7529c8c058822387b8d7968c89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T01:46:06.730 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 ceph-mgr[34243]: -- 172.21.15.110:0/4062470440 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566b54a05a0 con 0x5566b5475000 2024-04-25T01:46:06.981 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 bash[34223]: 9c2040968d4de02d200a1af591d19e8a03b23f7529c8c058822387b8d7968c89 2024-04-25T01:46:06.981 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 systemd[1]: Started Ceph mgr.y for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:46:06.981 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:06.812+0000 7f5d596d9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:46:06.998 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:06.728+0000 7fb64db57200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:46:06.998 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:06.820+0000 7fb64db57200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:46:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:46:07.351 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:46:07.351 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-04-25T01:46:07.352 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-04-25T01:46:07.352 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0371176 s, 283 MB/s 2024-04-25T01:46:07.352 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-04-25T01:46:07.414 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:06.996+0000 7f5d596d9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:46:07.414 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:07 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:07.102+0000 7f5d596d9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:46:07.414 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:07 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:07.420+0000 7f5d596d9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:46:07.498 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:07.025+0000 7fb64db57200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:46:07.498 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:07.221+0000 7fb64db57200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:46:07.730 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:07 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:07.506+0000 7f5d596d9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:46:07.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:07 smithi110 ceph-mon[32848]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:07.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:07.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:07 smithi102 ceph-mon[27796]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:07.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:07.998 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:07.858+0000 7fb64db57200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:46:07.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:07 smithi193 ceph-mon[32981]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:07.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:07.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:08.207 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_4 2024-04-25T01:46:08.387 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:08.293+0000 7fb64db57200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:46:08.629 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:46:08.674 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:08.393+0000 7fb64db57200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:46:08.674 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:08.600+0000 7fb64db57200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:46:08.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:08.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:08.937 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.980 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:08.751+0000 7f5d596d9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:46:08.981 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:08.919+0000 7f5d596d9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:46:08.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:08.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:08.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:08.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:08.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:08.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:09.363 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:09.086+0000 7f5d596d9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:46:09.363 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:09.369+0000 7f5d596d9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:46:09.452 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:09.126+0000 7fb64db57200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:46:09.452 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:09.217+0000 7fb64db57200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:46:09.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:09.460+0000 7f5d596d9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:46:09.653 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:09.659+0000 7f5d596d9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:46:09.748 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:09.458+0000 7fb64db57200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:46:09.748 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:09.589+0000 7fb64db57200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:46:09.748 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:09.684+0000 7fb64db57200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:46:09.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-mon[32981]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:09.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:09.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:09.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:09.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:09 smithi102 ceph-mon[27796]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:09.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:09 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:09.782 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:09 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:09.783 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:09 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:09.980 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:09 smithi110 ceph-mon[32848]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:09.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:09 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T01:46:09.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:09 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T01:46:09.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:09 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:09.981 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:09.850+0000 7f5d596d9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:46:10.248 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:09.799+0000 7fb64db57200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:46:10.248 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:09.889+0000 7fb64db57200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:46:10.684 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:10.438+0000 7f5d596d9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:46:10.843 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:10 smithi102 ceph-mon[27796]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:46:10.843 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:10 smithi102 ceph-mon[27796]: Reconfiguring daemon mgr.x on smithi102 2024-04-25T01:46:10.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:10 smithi110 ceph-mon[32848]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:46:10.942 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:10 smithi110 ceph-mon[32848]: Reconfiguring daemon mgr.x on smithi102 2024-04-25T01:46:10.942 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:10.853+0000 7f5d596d9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:46:10.942 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:10.948+0000 7f5d596d9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:46:10.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:10 smithi193 ceph-mon[32981]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T01:46:10.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:10 smithi193 ceph-mon[32981]: Reconfiguring daemon mgr.x on smithi102 2024-04-25T01:46:10.999 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:10.726+0000 7fb64db57200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:46:10.999 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:46:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:46:10.936+0000 7fb64db57200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:46:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.132+0000 7f762fc00700 1 -- 172.21.15.102:0/3960343359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628106080 msgr2=0x7f762810c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.132+0000 7f762fc00700 1 --2- 172.21.15.102:0/3960343359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628106080 0x7f762810c930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f762000b0d0 tx=0x7f762000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:46:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.132+0000 7f762fc00700 1 -- 172.21.15.102:0/3960343359 shutdown_connections 2024-04-25T01:46:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.132+0000 7f762fc00700 1 --2- 172.21.15.102:0/3960343359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628106080 0x7f762810c930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.132+0000 7f762fc00700 1 --2- 172.21.15.102:0/3960343359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76281044e0 0x7f7628104900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.132+0000 7f762fc00700 1 --2- 172.21.15.102:0/3960343359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76281056e0 0x7f7628105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.132+0000 7f762fc00700 1 -- 172.21.15.102:0/3960343359 >> 172.21.15.102:0/3960343359 conn(0x7f76280ffa80 msgr2=0x7f7628101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.132+0000 7f762fc00700 1 -- 172.21.15.102:0/3960343359 shutdown_connections 2024-04-25T01:46:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.132+0000 7f762fc00700 1 -- 172.21.15.102:0/3960343359 wait complete. 2024-04-25T01:46:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.134+0000 7f762fc00700 1 Processor -- start 2024-04-25T01:46:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.134+0000 7f762fc00700 1 -- start start 2024-04-25T01:46:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.134+0000 7f762fc00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76281044e0 0x7f762819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.134+0000 7f762fc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76281056e0 0x7f762819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.134+0000 7f762fc00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628106080 0x7f76281a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.134+0000 7f762fc00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7628110100 con 0x7f76281044e0 2024-04-25T01:46:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.134+0000 7f762fc00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f762810ff80 con 0x7f7628106080 2024-04-25T01:46:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.134+0000 7f762fc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7628110280 con 0x7f76281056e0 2024-04-25T01:46:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f762d200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76281044e0 0x7f762819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f762d200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76281044e0 0x7f762819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43212/0 (socket says 172.21.15.102:43212) 2024-04-25T01:46:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f762d200700 1 -- 172.21.15.102:0/1579071715 learned_addr learned my addr 172.21.15.102:0/1579071715 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:46:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f7627e00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76281056e0 0x7f762819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f762dc00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628106080 0x7f76281a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f762d200700 1 -- 172.21.15.102:0/1579071715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76281056e0 msgr2=0x7f762819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f762d200700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76281056e0 0x7f762819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:11.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f762d200700 1 -- 172.21.15.102:0/1579071715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628106080 msgr2=0x7f76281a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f762d200700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628106080 0x7f76281a1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.135+0000 7f762d200700 1 -- 172.21.15.102:0/1579071715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76281a22b0 con 0x7f76281044e0 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.136+0000 7f762d200700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76281044e0 0x7f762819a650 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f761c00ea80 tx=0x7f761c00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.136+0000 7f7625600700 1 -- 172.21.15.102:0/1579071715 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c00cbd0 con 0x7f76281044e0 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.136+0000 7f7625600700 1 -- 172.21.15.102:0/1579071715 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f761c00cd70 con 0x7f76281044e0 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.136+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76281a25a0 con 0x7f76281044e0 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.136+0000 7f7625600700 1 -- 172.21.15.102:0/1579071715 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c018780 con 0x7f76281044e0 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.136+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76281a2ae0 con 0x7f76281044e0 2024-04-25T01:46:11.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.137+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7628068f20 con 0x7f76281044e0 2024-04-25T01:46:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.138+0000 7f7625600700 1 -- 172.21.15.102:0/1579071715 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f761c01f020 con 0x7f76281044e0 2024-04-25T01:46:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.138+0000 7f7625600700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7610039330 0x7f761003b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.138+0000 7f7625600700 1 -- 172.21.15.102:0/1579071715 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f761c04c370 con 0x7f76281044e0 2024-04-25T01:46:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.139+0000 7f7627e00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7610039330 0x7f761003b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:11.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.139+0000 7f7627e00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7610039330 0x7f761003b7f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7614009c60 tx=0x7f7614009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:11.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.144+0000 7f7625600700 1 -- 172.21.15.102:0/1579071715 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f761c018ca0 con 0x7f76281044e0 2024-04-25T01:46:11.230 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:11.147+0000 7f5d596d9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:46:11.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.378+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7628062670 con 0x7f7610039330 2024-04-25T01:46:11.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:11.715+0000 7f7625600700 1 -- 172.21.15.102:0/1579071715 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 14) ==== 92061+0+0 (secure 0 0 0) 0x7f761c01f310 con 0x7f76281044e0 2024-04-25T01:46:11.980 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:11.686+0000 7f5d596d9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:46:11.981 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:11.780+0000 7f5d596d9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:46:11.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:11.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:46:11.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: Standby manager daemon z started 2024-04-25T01:46:11.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:11.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:46:11.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:46:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:46:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:11.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:11.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:46:11.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: Standby manager daemon z started 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: Standby manager daemon z started 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.193:0/1489943732' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:12.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:12.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:12.353 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:12.020+0000 7f5d596d9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:46:12.353 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:12.151+0000 7f5d596d9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:46:12.353 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:12.245+0000 7f5d596d9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:46:12.353 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:12.359+0000 7f5d596d9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:46:12.701 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:12.449+0000 7f5d596d9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:46:12.759 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:12 smithi102 ceph-mon[27796]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:12.760 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:12 smithi102 ceph-mon[27796]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-25T01:46:12.760 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:12 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:46:12.980 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:12 smithi110 ceph-mon[32848]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:12.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:12 smithi110 ceph-mon[32848]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-25T01:46:12.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:12 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:46:12.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:12 smithi193 ceph-mon[32981]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:12.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:12 smithi193 ceph-mon[32981]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-25T01:46:12.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:12 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:46:13.465 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:13 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:13.259+0000 7f5d596d9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:46:13.465 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:46:13 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:46:13.470+0000 7f5d596d9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:46:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:13.736+0000 7f7625600700 1 -- 172.21.15.102:0/1579071715 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f761c00f3c0 con 0x7f76281044e0 2024-04-25T01:46:13.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:13 smithi102 ceph-mon[27796]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:13.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:46:13.916 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:13.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:13 smithi102 ceph-mon[27796]: Standby manager daemon y started 2024-04-25T01:46:13.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:46:13.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:13.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:13 smithi193 ceph-mon[32981]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:13.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:46:13.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:13.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:13 smithi193 ceph-mon[32981]: Standby manager daemon y started 2024-04-25T01:46:13.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:46:13.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:14.230 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:13 smithi110 ceph-mon[32848]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:14.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:46:14.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:46:14.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:13 smithi110 ceph-mon[32848]: Standby manager daemon y started 2024-04-25T01:46:14.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:46:14.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.110:0/3129275461' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:46:14.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:14 smithi193 ceph-mon[32981]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-25T01:46:14.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:46:15.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:14 smithi102 ceph-mon[27796]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-25T01:46:15.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:46:15.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:14 smithi110 ceph-mon[32848]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-25T01:46:15.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:46:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:15 smithi102 ceph-mon[27796]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:15 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2626983177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "caf505e1-b2d5-43da-848f-89b616616084"}]: dispatch 2024-04-25T01:46:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:15 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2626983177' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "caf505e1-b2d5-43da-848f-89b616616084"}]': finished 2024-04-25T01:46:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:15 smithi102 ceph-mon[27796]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:46:15.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:15.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:15 smithi193 ceph-mon[32981]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:15.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:15 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2626983177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "caf505e1-b2d5-43da-848f-89b616616084"}]: dispatch 2024-04-25T01:46:15.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:15 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2626983177' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "caf505e1-b2d5-43da-848f-89b616616084"}]': finished 2024-04-25T01:46:15.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:15 smithi193 ceph-mon[32981]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:46:15.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:16.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:15 smithi110 ceph-mon[32848]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:16.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:15 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2626983177' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "caf505e1-b2d5-43da-848f-89b616616084"}]: dispatch 2024-04-25T01:46:16.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:15 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2626983177' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "caf505e1-b2d5-43da-848f-89b616616084"}]': finished 2024-04-25T01:46:16.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:15 smithi110 ceph-mon[32848]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T01:46:16.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:17.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:16 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1476435051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:46:17.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:16 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1476435051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:46:17.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:16 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1476435051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:46:18.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:17 smithi102 ceph-mon[27796]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:17 smithi110 ceph-mon[32848]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:18.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:17 smithi193 ceph-mon[32981]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:19.166 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:18 smithi102 ceph-mon[27796]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:19.231 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:18 smithi110 ceph-mon[32848]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:19.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:18 smithi193 ceph-mon[32981]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:21.883 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:21 smithi102 ceph-mon[27796]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:21.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:21 smithi110 ceph-mon[32848]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:21.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:21 smithi193 ceph-mon[32981]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:23.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:23 smithi102 ceph-mon[27796]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:23.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:23 smithi110 ceph-mon[32848]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:23.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:23 smithi193 ceph-mon[32981]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:24.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:46:24.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:24.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:46:24.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:24.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T01:46:24.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:25.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:25 smithi102 ceph-mon[27796]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:25.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:25 smithi102 ceph-mon[27796]: Deploying daemon osd.0 on smithi102 2024-04-25T01:46:25.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:25 smithi110 ceph-mon[32848]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:26.047 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:25 smithi110 ceph-mon[32848]: Deploying daemon osd.0 on smithi102 2024-04-25T01:46:26.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:25 smithi193 ceph-mon[32981]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:26.048 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:25 smithi193 ceph-mon[32981]: Deploying daemon osd.0 on smithi102 2024-04-25T01:46:27.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:27 smithi110 ceph-mon[32848]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:27.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:27 smithi193 ceph-mon[32981]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:28.043 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:27 smithi102 ceph-mon[27796]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:29.917 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:29 smithi102 ceph-mon[27796]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:29.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:29 smithi110 ceph-mon[32848]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:29.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:29 smithi193 ceph-mon[32981]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:31.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:31 smithi102 ceph-mon[27796]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:31.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:31 smithi110 ceph-mon[32848]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:31.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:31 smithi193 ceph-mon[32981]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:33.954 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:33 smithi102 ceph-mon[27796]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:33.955 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:33 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:33.955 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:33 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:33.955 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:33 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:33.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:33 smithi110 ceph-mon[32848]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:33.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:33 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:33.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:33 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:33.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:33 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:33.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:33 smithi193 ceph-mon[32981]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:33.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:33 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:33.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:33 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:33.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:33 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:35.873 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:35 smithi102 ceph-mon[27796]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:35.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:35 smithi110 ceph-mon[32848]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:35.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:35 smithi193 ceph-mon[32981]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:37.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:37.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:37.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:37.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:37.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:37.360 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:37 smithi102 ceph-mon[27796]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:37.360 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:37 smithi102 ceph-mon[27796]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:46:37.470 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 0 on host 'smithi102' 2024-04-25T01:46:37.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.473+0000 7f7625600700 1 -- 172.21.15.102:0/1579071715 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7628062670 con 0x7f7610039330 2024-04-25T01:46:37.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7610039330 msgr2=0x7f761003b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:37.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7610039330 0x7f761003b7f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7614009c60 tx=0x7f7614009260 comp rx=0 tx=0).stop 2024-04-25T01:46:37.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76281044e0 msgr2=0x7f762819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:37.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76281044e0 0x7f762819a650 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f761c00ea80 tx=0x7f761c00ed90 comp rx=0 tx=0).stop 2024-04-25T01:46:37.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 shutdown_connections 2024-04-25T01:46:37.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7610039330 0x7f761003b7f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628106080 0x7f76281a1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76281044e0 0x7f762819a650 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 --2- 172.21.15.102:0/1579071715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76281056e0 0x7f762819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:37.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 >> 172.21.15.102:0/1579071715 conn(0x7f76280ffa80 msgr2=0x7f7628108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:37.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 shutdown_connections 2024-04-25T01:46:37.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:46:37.478+0000 7f762fc00700 1 -- 172.21.15.102:0/1579071715 wait complete. 2024-04-25T01:46:37.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:37.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:37.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:37.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:37.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:37.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:37 smithi110 ceph-mon[32848]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:37.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:37 smithi110 ceph-mon[32848]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:46:37.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:37.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:37.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:37.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:37.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:37.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:37 smithi193 ceph-mon[32981]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:37.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:37 smithi193 ceph-mon[32981]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T01:46:38.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:38 smithi102 ceph-mon[27796]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:46:38.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:38 smithi102 ceph-mon[27796]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:46:38.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:38 smithi102 ceph-mon[27796]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:46:38.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:38 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:38.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:38 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:38.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:38 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:38.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:38 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:38.207 DEBUG:teuthology.orchestra.run.smithi102:osd.0> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T01:46:38.211 INFO:tasks.cephadm:Deploying osd.1 on smithi110 with /dev/vg_nvme/lv_4... 2024-04-25T01:46:38.212 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:46:38.454 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:46:38.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:38 smithi110 ceph-mon[32848]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:46:38.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:38 smithi110 ceph-mon[32848]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:46:38.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:38 smithi110 ceph-mon[32848]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:46:38.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:38 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:38.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:38 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:38.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:38 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:38.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:38 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:38.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:38 smithi193 ceph-mon[32981]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T01:46:38.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:38 smithi193 ceph-mon[32981]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T01:46:38.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:38 smithi193 ceph-mon[32981]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:46:38.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:38 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:38.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:38 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:38.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:38 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:38.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:38 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:39.414 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:39 smithi102 ceph-mon[27796]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:46:39.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:39 smithi102 ceph-mon[27796]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:46:39.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:39.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:39.415 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:39 smithi102 ceph-mon[27796]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:39.415 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:46:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0[42920]: 2024-04-25T01:46:39.129+0000 7f2301e00700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T01:46:39.415 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:46:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0[42920]: 2024-04-25T01:46:39.147+0000 7f22f8600700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:46:39.436 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:39 smithi110 ceph-mon[32848]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:46:39.436 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:39 smithi110 ceph-mon[32848]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:46:39.436 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:39.436 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:39.436 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:39 smithi110 ceph-mon[32848]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:39.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:39 smithi193 ceph-mon[32981]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:46:39.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:39 smithi193 ceph-mon[32981]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T01:46:39.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:39.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:39.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:39 smithi193 ceph-mon[32981]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:40.227 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:46:40.227 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:46:40.227 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-04-25T01:46:40.228 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-04-25T01:46:40.228 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00853215 s, 1.2 GB/s 2024-04-25T01:46:40.228 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-04-25T01:46:40.251 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:40 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:46:40.251 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:40 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:46:40.251 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:40 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:40.251 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:40 smithi110 ceph-mon[32848]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' 2024-04-25T01:46:40.251 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:40 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:40.407 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:40 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:46:40.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:40 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:46:40.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:40 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:40.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:40 smithi102 ceph-mon[27796]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' 2024-04-25T01:46:40.408 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:40 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:40.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:40 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:46:40.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:40 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:46:40.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:40 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:40.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:40 smithi193 ceph-mon[32981]: from='osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417]' entity='osd.0' 2024-04-25T01:46:40.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:40 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:40.787 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_4 2024-04-25T01:46:41.018 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:46:41.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:41 smithi102 ceph-mon[27796]: osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] boot 2024-04-25T01:46:41.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:41 smithi102 ceph-mon[27796]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:46:41.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:41 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:41.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:41 smithi102 ceph-mon[27796]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:41.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:41 smithi110 ceph-mon[32848]: osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] boot 2024-04-25T01:46:41.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:41 smithi110 ceph-mon[32848]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:46:41.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:41 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:41.445 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:41 smithi110 ceph-mon[32848]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:41.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:41 smithi193 ceph-mon[32981]: osd.0 [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] boot 2024-04-25T01:46:41.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:41 smithi193 ceph-mon[32981]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T01:46:41.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:41 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:46:41.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:41 smithi193 ceph-mon[32981]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T01:46:42.335 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:42 smithi110 ceph-mon[32848]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:46:42.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:42 smithi102 ceph-mon[27796]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:46:42.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:42 smithi193 ceph-mon[32981]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T01:46:42.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.535+0000 7f91779a5700 1 -- 172.21.15.110:0/3953050039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 msgr2=0x7f91701026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:42.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.535+0000 7f91779a5700 1 --2- 172.21.15.110:0/3953050039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 0x7f91701026b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f916c0099c0 tx=0x7f916c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:46:42.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.535+0000 7f91779a5700 1 -- 172.21.15.110:0/3953050039 shutdown_connections 2024-04-25T01:46:42.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.535+0000 7f91779a5700 1 --2- 172.21.15.110:0/3953050039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 0x7f91701026b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.535+0000 7f91779a5700 1 --2- 172.21.15.110:0/3953050039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9170103e30 0x7f917010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.535+0000 7f91779a5700 1 --2- 172.21.15.110:0/3953050039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9170103490 0x7f91701038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.535+0000 7f91779a5700 1 -- 172.21.15.110:0/3953050039 >> 172.21.15.110:0/3953050039 conn(0x7f91700fd830 msgr2=0x7f91700ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 -- 172.21.15.110:0/3953050039 shutdown_connections 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 -- 172.21.15.110:0/3953050039 wait complete. 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 Processor -- start 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 -- start start 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 0x7f9170198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9170103490 0x7f9170198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9170103e30 0x7f917019fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f917010df40 con 0x7f9170103e30 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f917010ddc0 con 0x7f9170102290 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f91779a5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f917010e0c0 con 0x7f9170103490 2024-04-25T01:46:42.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.536+0000 7f9175741700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 0x7f9170198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9175741700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 0x7f9170198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:45274/0 (socket says 172.21.15.110:45274) 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9175741700 1 -- 172.21.15.110:0/701885894 learned_addr learned my addr 172.21.15.110:0/701885894 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9174f40700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9170103490 0x7f9170198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9175741700 1 -- 172.21.15.110:0/701885894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9170103490 msgr2=0x7f9170198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9175741700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9170103490 0x7f9170198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9175741700 1 -- 172.21.15.110:0/701885894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9170103e30 msgr2=0x7f917019fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9175741700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9170103e30 0x7f917019fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9175741700 1 -- 172.21.15.110:0/701885894 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91701a01f0 con 0x7f9170102290 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9174f40700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9170103490 0x7f9170198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f9175741700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 0x7f9170198590 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f916c0098e0 tx=0x7f916c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f91667fc700 1 -- 172.21.15.110:0/701885894 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c027070 con 0x7f9170102290 2024-04-25T01:46:42.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f91667fc700 1 -- 172.21.15.110:0/701885894 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f916c022470 con 0x7f9170102290 2024-04-25T01:46:42.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91701a0480 con 0x7f9170102290 2024-04-25T01:46:42.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f91667fc700 1 -- 172.21.15.110:0/701885894 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c00f6a0 con 0x7f9170102290 2024-04-25T01:46:42.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.537+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91701a0950 con 0x7f9170102290 2024-04-25T01:46:42.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.538+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9170068f20 con 0x7f9170102290 2024-04-25T01:46:42.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.539+0000 7f91667fc700 1 -- 172.21.15.110:0/701885894 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f916c02a070 con 0x7f9170102290 2024-04-25T01:46:42.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.540+0000 7f91667fc700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f915c0a2b90 0x7f915c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:46:42.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.540+0000 7f91667fc700 1 -- 172.21.15.110:0/701885894 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f916c0ce440 con 0x7f9170102290 2024-04-25T01:46:42.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.540+0000 7f9174f40700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f915c0a2b90 0x7f915c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:46:42.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.541+0000 7f9174f40700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f915c0a2b90 0x7f915c0a5050 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9160005950 tx=0x7f9160005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:46:42.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.544+0000 7f91667fc700 1 -- 172.21.15.110:0/701885894 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f916c013070 con 0x7f9170102290 2024-04-25T01:46:42.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:46:42.746+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9170062670 con 0x7f915c0a2b90 2024-04-25T01:46:43.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:43.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:43.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:43.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:46:43.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:46:43.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:43.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:46:43.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:43.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:43.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:43.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:46:43.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:46:43.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:46:43.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:43.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:46:43.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:43.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:43.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:46:43.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:44.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:44 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi102 2024-04-25T01:46:44.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:44 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi102 to 17089M 2024-04-25T01:46:44.420 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:44 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi102 2024-04-25T01:46:44.421 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:44 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi102 to 17089M 2024-04-25T01:46:44.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:44 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi102 2024-04-25T01:46:44.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:44 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi102 to 17089M 2024-04-25T01:46:45.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:45 smithi102 ceph-mon[27796]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:45.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:45 smithi110 ceph-mon[32848]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:45.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:45 smithi193 ceph-mon[32981]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:46.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:46 smithi110 ceph-mon[32848]: from='client.? 172.21.15.110:0/3372705467' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "437ccdd6-3c94-47c1-9350-b252f8aec22f"}]: dispatch 2024-04-25T01:46:46.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:46 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "437ccdd6-3c94-47c1-9350-b252f8aec22f"}]: dispatch 2024-04-25T01:46:46.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:46 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "437ccdd6-3c94-47c1-9350-b252f8aec22f"}]': finished 2024-04-25T01:46:46.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:46 smithi110 ceph-mon[32848]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:46:46.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:46:46.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:46 smithi110 ceph-mon[32848]: from='client.? 172.21.15.110:0/77877872' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:46:46.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:46 smithi193 ceph-mon[32981]: from='client.? 172.21.15.110:0/3372705467' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "437ccdd6-3c94-47c1-9350-b252f8aec22f"}]: dispatch 2024-04-25T01:46:46.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:46 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "437ccdd6-3c94-47c1-9350-b252f8aec22f"}]: dispatch 2024-04-25T01:46:46.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:46 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "437ccdd6-3c94-47c1-9350-b252f8aec22f"}]': finished 2024-04-25T01:46:46.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:46 smithi193 ceph-mon[32981]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:46:46.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:46 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:46:46.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:46 smithi193 ceph-mon[32981]: from='client.? 172.21.15.110:0/77877872' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:46:46.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:46 smithi102 ceph-mon[27796]: from='client.? 172.21.15.110:0/3372705467' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "437ccdd6-3c94-47c1-9350-b252f8aec22f"}]: dispatch 2024-04-25T01:46:46.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:46 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "437ccdd6-3c94-47c1-9350-b252f8aec22f"}]: dispatch 2024-04-25T01:46:46.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:46 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "437ccdd6-3c94-47c1-9350-b252f8aec22f"}]': finished 2024-04-25T01:46:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:46 smithi102 ceph-mon[27796]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T01:46:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:46 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:46:46.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:46 smithi102 ceph-mon[27796]: from='client.? 172.21.15.110:0/77877872' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:46:47.481 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:47 smithi110 ceph-mon[32848]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:47.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:47 smithi193 ceph-mon[32981]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:47.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:47 smithi102 ceph-mon[27796]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:49.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:49 smithi110 ceph-mon[32848]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:49.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:49 smithi193 ceph-mon[32981]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:50.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:49 smithi102 ceph-mon[27796]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:51.981 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:51 smithi110 ceph-mon[32848]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:51.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:51 smithi193 ceph-mon[32981]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:52.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:51 smithi102 ceph-mon[27796]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:52.936 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:46:52.936 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:52.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:46:52.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:53.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T01:46:53.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:46:53.937 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:53 smithi110 ceph-mon[32848]: Deploying daemon osd.1 on smithi110 2024-04-25T01:46:53.937 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:53 smithi110 ceph-mon[32848]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:53.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:53 smithi193 ceph-mon[32981]: Deploying daemon osd.1 on smithi110 2024-04-25T01:46:53.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:53 smithi193 ceph-mon[32981]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:54.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:53 smithi102 ceph-mon[27796]: Deploying daemon osd.1 on smithi110 2024-04-25T01:46:54.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:53 smithi102 ceph-mon[27796]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:55.957 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:55 smithi110 ceph-mon[32848]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:55.989 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:55 smithi102 ceph-mon[27796]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:55.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:55 smithi193 ceph-mon[32981]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:57.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:57 smithi110 ceph-mon[32848]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:57.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:57.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:57.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:57 smithi193 ceph-mon[32981]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:57.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:57 smithi102 ceph-mon[27796]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:46:58.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:46:59.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:46:59 smithi110 ceph-mon[32848]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:46:59.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:46:59 smithi193 ceph-mon[32981]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:00.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:46:59 smithi102 ceph-mon[27796]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:01.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.285+0000 7f91667fc700 1 -- 172.21.15.110:0/701885894 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9170062670 con 0x7f915c0a2b90 2024-04-25T01:47:01.280 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 1 on host 'smithi110' 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f915c0a2b90 msgr2=0x7f915c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f915c0a2b90 0x7f915c0a5050 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9160005950 tx=0x7f9160005f30 comp rx=0 tx=0).stop 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 msgr2=0x7f9170198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 0x7f9170198590 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f916c0098e0 tx=0x7f916c0047b0 comp rx=0 tx=0).stop 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 shutdown_connections 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f915c0a2b90 0x7f915c0a5050 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9170102290 0x7f9170198590 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9170103e30 0x7f917019fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 --2- 172.21.15.110:0/701885894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9170103490 0x7f9170198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 >> 172.21.15.110:0/701885894 conn(0x7f91700fd830 msgr2=0x7f9170106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 shutdown_connections 2024-04-25T01:47:01.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:47:01.288+0000 7f91779a5700 1 -- 172.21.15.110:0/701885894 wait complete. 2024-04-25T01:47:01.731 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='osd.1 [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:01.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:01 smithi110 ceph-mon[32848]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:01.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:01.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:01.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:01.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:01.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='osd.1 [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:01.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:01 smithi193 ceph-mon[32981]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:01.848 DEBUG:teuthology.orchestra.run.smithi110:osd.1> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T01:47:01.850 INFO:tasks.cephadm:Deploying osd.2 on smithi193 with /dev/vg_nvme/lv_4... 2024-04-25T01:47:01.850 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T01:47:01.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:01.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:01.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='osd.1 [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:01.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:01 smithi102 ceph-mon[27796]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T01:47:02.092 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:47:03.369 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:03 smithi110 ceph-mon[32848]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:47:03.369 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:03 smithi110 ceph-mon[32848]: from='osd.1 [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:47:03.369 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:03 smithi110 ceph-mon[32848]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:47:03.369 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:03 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:03.369 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:03 smithi110 ceph-mon[32848]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:47:03.369 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:03 smithi110 ceph-mon[32848]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:03.370 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1[37452]: 2024-04-25T01:47:03.281+0000 7fb84ab08700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T01:47:03.370 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1[37452]: 2024-04-25T01:47:03.285+0000 7fb8470fc700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:47:03.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:03 smithi193 ceph-mon[32981]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:47:03.384 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:03 smithi193 ceph-mon[32981]: from='osd.1 [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:47:03.384 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:03 smithi193 ceph-mon[32981]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:47:03.384 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:03 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:03.384 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:03 smithi193 ceph-mon[32981]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:47:03.384 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:03 smithi193 ceph-mon[32981]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:03 smithi102 ceph-mon[27796]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T01:47:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:03 smithi102 ceph-mon[27796]: from='osd.1 [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:47:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:03 smithi102 ceph-mon[27796]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T01:47:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:03 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:03 smithi102 ceph-mon[27796]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:47:03.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:03 smithi102 ceph-mon[27796]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:03.975 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T01:47:03.975 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T01:47:03.975 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-04-25T01:47:03.976 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-04-25T01:47:03.976 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012432 s, 843 MB/s 2024-04-25T01:47:03.976 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-04-25T01:47:04.563 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:04 smithi193 ceph-mon[32981]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:47:04.563 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:04 smithi193 ceph-mon[32981]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:47:04.563 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:04.563 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:04.590 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-04-25T01:47:04.667 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:04 smithi102 ceph-mon[27796]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:47:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:04 smithi102 ceph-mon[27796]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:47:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:04.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:04.714 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:04 smithi110 ceph-mon[32848]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:47:04.715 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:04 smithi110 ceph-mon[32848]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T01:47:04.715 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:04.715 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:04.838 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:47:05.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:47:05.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:47:05.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: osd.1 [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] boot 2024-04-25T01:47:05.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:47:05.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:05.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:05.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.482 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:05.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:05.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:05.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:05 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: osd.1 [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] boot 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:05 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:47:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:47:05.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: osd.1 [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] boot 2024-04-25T01:47:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T01:47:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:47:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-25T01:47:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:05.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:05.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:05.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:05.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:05 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:06.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.289+0000 7f93a1f82700 1 -- 172.21.15.193:0/1407466311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c101190 msgr2=0x7f939c1015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:06.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.289+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1407466311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c101190 0x7f939c1015b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f938c009990 tx=0x7f938c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:47:06.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.289+0000 7f93a1f82700 1 -- 172.21.15.193:0/1407466311 shutdown_connections 2024-04-25T01:47:06.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.289+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1407466311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f939c102d30 0x7f939c1095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:06.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.289+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1407466311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c101190 0x7f939c1015b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:06.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.289+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1407466311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f939c102390 0x7f939c1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:06.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.289+0000 7f93a1f82700 1 -- 172.21.15.193:0/1407466311 >> 172.21.15.193:0/1407466311 conn(0x7f939c0fc730 msgr2=0x7f939c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:06.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.289+0000 7f93a1f82700 1 -- 172.21.15.193:0/1407466311 shutdown_connections 2024-04-25T01:47:06.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.289+0000 7f93a1f82700 1 -- 172.21.15.193:0/1407466311 wait complete. 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f93a1f82700 1 Processor -- start 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f93a1f82700 1 -- start start 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f93a1f82700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f939c101190 0x7f939c1974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f93a1f82700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f939c102390 0x7f939c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f93a1f82700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c102d30 0x7f939c19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f93a1f82700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f939c10ce20 con 0x7f939c102390 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f93a1f82700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f939c10cca0 con 0x7f939c101190 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f93a1f82700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f939c10cfa0 con 0x7f939c102d30 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f939bfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c102d30 0x7f939c19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f939bfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c102d30 0x7f939c19eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:42404/0 (socket says 172.21.15.193:42404) 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f939bfff700 1 -- 172.21.15.193:0/1816248511 learned_addr learned my addr 172.21.15.193:0/1816248511 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f939b7fe700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f939c101190 0x7f939c1974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:06.287 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f939bfff700 1 -- 172.21.15.193:0/1816248511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f939c101190 msgr2=0x7f939c1974f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.290+0000 7f939bfff700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f939c101190 0x7f939c1974f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.291+0000 7f939bfff700 1 -- 172.21.15.193:0/1816248511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f939c102390 msgr2=0x7f939c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.291+0000 7f939bfff700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f939c102390 0x7f939c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.291+0000 7f939bfff700 1 -- 172.21.15.193:0/1816248511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939c19f150 con 0x7f939c102d30 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.291+0000 7f939bfff700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c102d30 0x7f939c19eab0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9394007f10 tx=0x7f939400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.291+0000 7f93997fa700 1 -- 172.21.15.193:0/1816248511 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9394004bb0 con 0x7f939c102d30 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.291+0000 7f93997fa700 1 -- 172.21.15.193:0/1816248511 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9394004d50 con 0x7f939c102d30 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.291+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939c19f440 con 0x7f939c102d30 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.291+0000 7f93997fa700 1 -- 172.21.15.193:0/1816248511 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9394005a60 con 0x7f939c102d30 2024-04-25T01:47:06.288 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.291+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939c19fa50 con 0x7f939c102d30 2024-04-25T01:47:06.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.292+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f939c068f20 con 0x7f939c102d30 2024-04-25T01:47:06.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.293+0000 7f93997fa700 1 -- 172.21.15.193:0/1816248511 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f939401a020 con 0x7f939c102d30 2024-04-25T01:47:06.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.294+0000 7f93997fa700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f93880a2c90 0x7f93880a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:06.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.294+0000 7f93997fa700 1 -- 172.21.15.193:0/1816248511 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f93940cd5f0 con 0x7f939c102d30 2024-04-25T01:47:06.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.294+0000 7f939b7fe700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f93880a2c90 0x7f93880a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:06.290 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.295+0000 7f939b7fe700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f93880a2c90 0x7f93880a5150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f938c00b5c0 tx=0x7f938c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:06.292 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.298+0000 7f93997fa700 1 -- 172.21.15.193:0/1816248511 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9394010070 con 0x7f939c102d30 2024-04-25T01:47:06.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:06.501+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f939c062670 con 0x7f93880a2c90 2024-04-25T01:47:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:06 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi110 2024-04-25T01:47:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:06 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi110 to 17088M 2024-04-25T01:47:06.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:06 smithi102 ceph-mon[27796]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:47:06.701 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:06 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi110 2024-04-25T01:47:06.701 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:06 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi110 to 17088M 2024-04-25T01:47:06.701 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:06 smithi193 ceph-mon[32981]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:47:06.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:06 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi110 2024-04-25T01:47:06.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:06 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi110 to 17088M 2024-04-25T01:47:06.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:06 smithi110 ceph-mon[32848]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T01:47:07.434 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:07 smithi193 ceph-mon[32981]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:07.435 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:07.435 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:07.435 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:07.435 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:07 smithi193 ceph-mon[32981]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:07 smithi102 ceph-mon[27796]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:07.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:07 smithi102 ceph-mon[27796]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:07.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:07 smithi110 ceph-mon[32848]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:07.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:07.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:07.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:07.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:07 smithi110 ceph-mon[32848]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:09.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:09 smithi193 ceph-mon[32981]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:09.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:09 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/2832789777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c730312e-3b4f-4839-bcc4-587ad4c5c576"}]: dispatch 2024-04-25T01:47:09.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:09 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c730312e-3b4f-4839-bcc4-587ad4c5c576"}]: dispatch 2024-04-25T01:47:09.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:09 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c730312e-3b4f-4839-bcc4-587ad4c5c576"}]': finished 2024-04-25T01:47:09.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:09 smithi193 ceph-mon[32981]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:47:09.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:09 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:09.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:09 smithi110 ceph-mon[32848]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:09.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:09 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/2832789777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c730312e-3b4f-4839-bcc4-587ad4c5c576"}]: dispatch 2024-04-25T01:47:09.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:09 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c730312e-3b4f-4839-bcc4-587ad4c5c576"}]: dispatch 2024-04-25T01:47:09.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:09 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c730312e-3b4f-4839-bcc4-587ad4c5c576"}]': finished 2024-04-25T01:47:09.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:09 smithi110 ceph-mon[32848]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:47:09.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:09 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:09 smithi102 ceph-mon[27796]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:09 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/2832789777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c730312e-3b4f-4839-bcc4-587ad4c5c576"}]: dispatch 2024-04-25T01:47:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:09 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c730312e-3b4f-4839-bcc4-587ad4c5c576"}]: dispatch 2024-04-25T01:47:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:09 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c730312e-3b4f-4839-bcc4-587ad4c5c576"}]': finished 2024-04-25T01:47:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:09 smithi102 ceph-mon[27796]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T01:47:10.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:09 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:10.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:10 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/1856006663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:10.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:10 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/1856006663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:11.167 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:10 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/1856006663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:11.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:11 smithi110 ceph-mon[32848]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:11.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:11 smithi193 ceph-mon[32981]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:12.069 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:11 smithi102 ceph-mon[27796]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:13.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:13 smithi110 ceph-mon[32848]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:13.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:13 smithi193 ceph-mon[32981]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:14.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:13 smithi102 ceph-mon[27796]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:15.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:15 smithi193 ceph-mon[32981]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:15.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:15 smithi110 ceph-mon[32848]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:16.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:15 smithi102 ceph-mon[27796]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:16.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:16 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:47:16.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:16 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:16.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:16 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:47:16.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:16 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:16 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T01:47:17.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:16 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:17.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:17 smithi193 ceph-mon[32981]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:17.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:17 smithi193 ceph-mon[32981]: Deploying daemon osd.2 on smithi193 2024-04-25T01:47:17.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:17 smithi110 ceph-mon[32848]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:17.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:17 smithi110 ceph-mon[32848]: Deploying daemon osd.2 on smithi193 2024-04-25T01:47:18.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:17 smithi102 ceph-mon[27796]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:18.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:17 smithi102 ceph-mon[27796]: Deploying daemon osd.2 on smithi193 2024-04-25T01:47:19.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:19 smithi110 ceph-mon[32848]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:19.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:19 smithi193 ceph-mon[32981]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:20.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:19 smithi102 ceph-mon[27796]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:21.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:21 smithi110 ceph-mon[32848]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:21.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:21 smithi193 ceph-mon[32981]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:22.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:21 smithi102 ceph-mon[27796]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:22 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:22 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:22 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:23.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:22 smithi102 ceph-mon[27796]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:23.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:22 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:23.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:22 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:23.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:22 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:23.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:22 smithi110 ceph-mon[32848]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:23.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:22 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:23.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:22 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:23.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:22 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:23.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:22 smithi193 ceph-mon[32981]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:25.279 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 2 on host 'smithi193' 2024-04-25T01:47:25.279 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.283+0000 7f93997fa700 1 -- 172.21.15.193:0/1816248511 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f939c062670 con 0x7f93880a2c90 2024-04-25T01:47:25.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.285+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f93880a2c90 msgr2=0x7f93880a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:25.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.285+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f93880a2c90 0x7f93880a5150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f938c00b5c0 tx=0x7f938c005fd0 comp rx=0 tx=0).stop 2024-04-25T01:47:25.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c102d30 msgr2=0x7f939c19eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:25.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c102d30 0x7f939c19eab0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9394007f10 tx=0x7f939400d3b0 comp rx=0 tx=0).stop 2024-04-25T01:47:25.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 shutdown_connections 2024-04-25T01:47:25.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f93880a2c90 0x7f93880a5150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:25.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f939c101190 0x7f939c1974f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:25.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f939c102d30 0x7f939c19eab0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:25.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 --2- 172.21.15.193:0/1816248511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f939c102390 0x7f939c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:25.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 >> 172.21.15.193:0/1816248511 conn(0x7f939c0fc730 msgr2=0x7f939c105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:25.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 shutdown_connections 2024-04-25T01:47:25.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:47:25.286+0000 7f93a1f82700 1 -- 172.21.15.193:0/1816248511 wait complete. 2024-04-25T01:47:25.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:25 smithi193 ceph-mon[32981]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:25.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:25 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:25 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:25 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:25 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.916 DEBUG:teuthology.orchestra.run.smithi193:osd.2> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T01:47:25.918 INFO:tasks.cephadm:Deploying osd.3 on smithi102 with /dev/vg_nvme/lv_3... 2024-04-25T01:47:25.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:47:25.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:25 smithi110 ceph-mon[32848]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:25.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:25 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:25 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:25 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:25 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:25 smithi102 ceph-mon[27796]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:25.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:25 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:25 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:25 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:25.997 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:25 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:26.352 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:47:26.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:26 smithi193 ceph-mon[32981]: from='osd.2 [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:26.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:26 smithi193 ceph-mon[32981]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:26.767 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:26 smithi102 ceph-mon[27796]: from='osd.2 [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:26.767 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:26 smithi102 ceph-mon[27796]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:26.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:26 smithi110 ceph-mon[32848]: from='osd.2 [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:26.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:26 smithi110 ceph-mon[32848]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T01:47:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:27 smithi102 ceph-mon[27796]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:27 smithi102 ceph-mon[27796]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:47:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:27 smithi102 ceph-mon[27796]: from='osd.2 [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:47:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:27 smithi102 ceph-mon[27796]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:47:27.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:27.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:27 smithi102 ceph-mon[27796]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:47:27.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:27 smithi110 ceph-mon[32848]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:27.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:27 smithi110 ceph-mon[32848]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:47:27.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:27 smithi110 ceph-mon[32848]: from='osd.2 [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:47:27.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:27 smithi110 ceph-mon[32848]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:47:27.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:27.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:27 smithi110 ceph-mon[32848]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:47:28.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:27 smithi193 ceph-mon[32981]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:28.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:27 smithi193 ceph-mon[32981]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T01:47:28.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:27 smithi193 ceph-mon[32981]: from='osd.2 [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:47:28.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:27 smithi193 ceph-mon[32981]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T01:47:28.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:28.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:27 smithi193 ceph-mon[32981]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:47:28.703 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2[38151]: 2024-04-25T01:47:28.527+0000 7f17cd938700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T01:47:28.703 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2[38151]: 2024-04-25T01:47:28.538+0000 7f17c4721700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:47:28.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:28 smithi102 ceph-mon[27796]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:47:28.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:28 smithi102 ceph-mon[27796]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:47:28.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:28.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:28.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:28 smithi102 ceph-mon[27796]: from='osd.2 ' entity='osd.2' 2024-04-25T01:47:28.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:28 smithi110 ceph-mon[32848]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:47:28.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:28 smithi110 ceph-mon[32848]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:47:28.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:28.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:28.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:28 smithi110 ceph-mon[32848]: from='osd.2 ' entity='osd.2' 2024-04-25T01:47:28.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:28 smithi193 ceph-mon[32981]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:47:28.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:28 smithi193 ceph-mon[32981]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T01:47:28.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:28.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:28.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:28 smithi193 ceph-mon[32981]: from='osd.2 ' entity='osd.2' 2024-04-25T01:47:29.209 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:47:29.209 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:47:29.209 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-04-25T01:47:29.210 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-04-25T01:47:29.210 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0312581 s, 335 MB/s 2024-04-25T01:47:29.210 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-04-25T01:47:29.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:47:29.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:47:29.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:29.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:29.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:29.878 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: osd.2 [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] boot 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:47:29.879 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:29.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:47:29.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:47:29.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:29.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:29.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:29.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: osd.2 [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] boot 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:47:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:30.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:47:30.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:47:30.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-25T01:47:30.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:30.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:30.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:30.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:47:30.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: osd.2 [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] boot 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T01:47:30.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:47:30.034 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_3 2024-04-25T01:47:30.464 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:47:30.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:30 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi193 2024-04-25T01:47:30.870 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:30 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi193 to 17088M 2024-04-25T01:47:30.870 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:30 smithi102 ceph-mon[27796]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:47:30.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:30 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi193 2024-04-25T01:47:30.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:30 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi193 to 17088M 2024-04-25T01:47:30.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:30 smithi110 ceph-mon[32848]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:47:31.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:30 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi193 2024-04-25T01:47:31.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:30 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi193 to 17088M 2024-04-25T01:47:31.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:30 smithi193 ceph-mon[32981]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T01:47:31.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:31 smithi110 ceph-mon[32848]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:31.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:31 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:32.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:31 smithi193 ceph-mon[32981]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:32.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:31 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:32.079 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:31 smithi102 ceph-mon[27796]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:32.079 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:31 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:32.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.584+0000 7f6db1a00700 1 -- 172.21.15.102:0/2710745917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dac109ad0 msgr2=0x7f6dac109f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:32.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.584+0000 7f6db1a00700 1 --2- 172.21.15.102:0/2710745917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dac109ad0 0x7f6dac109f30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400b0d0 tx=0x7f6d9400b3e0 comp rx=0 tx=0).stop 2024-04-25T01:47:32.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.584+0000 7f6db1a00700 1 -- 172.21.15.102:0/2710745917 shutdown_connections 2024-04-25T01:47:32.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.584+0000 7f6db1a00700 1 --2- 172.21.15.102:0/2710745917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dac1088d0 0x7f6dac108cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:32.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.584+0000 7f6db1a00700 1 --2- 172.21.15.102:0/2710745917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dac10a470 0x7f6dac110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:32.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.584+0000 7f6db1a00700 1 --2- 172.21.15.102:0/2710745917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dac109ad0 0x7f6dac109f30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:32.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.584+0000 7f6db1a00700 1 -- 172.21.15.102:0/2710745917 >> 172.21.15.102:0/2710745917 conn(0x7f6dac0778a0 msgr2=0x7f6dac079ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:32.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.584+0000 7f6db1a00700 1 -- 172.21.15.102:0/2710745917 shutdown_connections 2024-04-25T01:47:32.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.584+0000 7f6db1a00700 1 -- 172.21.15.102:0/2710745917 wait complete. 2024-04-25T01:47:32.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.585+0000 7f6db1a00700 1 Processor -- start 2024-04-25T01:47:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.585+0000 7f6db1a00700 1 -- start start 2024-04-25T01:47:32.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6db1a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dac1088d0 0x7f6dac19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6db1a00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dac109ad0 0x7f6dac19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6db1a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dac10a470 0x7f6dac1a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6db1a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac114490 con 0x7f6dac10a470 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6db1a00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac114310 con 0x7f6dac109ad0 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6db1a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac114610 con 0x7f6dac1088d0 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6dab400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dac10a470 0x7f6dac1a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6da2000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dac109ad0 0x7f6dac19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6daaa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dac1088d0 0x7f6dac19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6daaa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dac1088d0 0x7f6dac19eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44102/0 (socket says 172.21.15.102:44102) 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6da2000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dac109ad0 0x7f6dac19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54206/0 (socket says 172.21.15.102:54206) 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.586+0000 7f6daaa00700 1 -- 172.21.15.102:0/3729835148 learned_addr learned my addr 172.21.15.102:0/3729835148 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:47:32.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6da2000700 1 -- 172.21.15.102:0/3729835148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dac1088d0 msgr2=0x7f6dac19eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6da2000700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dac1088d0 0x7f6dac19eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6da2000700 1 -- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dac10a470 msgr2=0x7f6dac1a6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6da2000700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dac10a470 0x7f6dac1a6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6da2000700 1 -- 172.21.15.102:0/3729835148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac1a6700 con 0x7f6dac109ad0 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6daaa00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dac1088d0 0x7f6dac19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6dab400700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dac10a470 0x7f6dac1a6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6da2000700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dac109ad0 0x7f6dac19efe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400d010 tx=0x7f6d94007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6da3e00700 1 -- 172.21.15.102:0/3729835148 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94021020 con 0x7f6dac109ad0 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6da3e00700 1 -- 172.21.15.102:0/3729835148 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d9401d950 con 0x7f6dac109ad0 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dac1a6990 con 0x7f6dac109ad0 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6da3e00700 1 -- 172.21.15.102:0/3729835148 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94005340 con 0x7f6dac109ad0 2024-04-25T01:47:32.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.587+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dac1a6e40 con 0x7f6dac109ad0 2024-04-25T01:47:32.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.588+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dac068f20 con 0x7f6dac109ad0 2024-04-25T01:47:32.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.590+0000 7f6da3e00700 1 -- 172.21.15.102:0/3729835148 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f6d94024020 con 0x7f6dac109ad0 2024-04-25T01:47:32.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.592+0000 7f6da3e00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d8c0a2b90 0x7f6d8c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:47:32.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.592+0000 7f6da3e00700 1 -- 172.21.15.102:0/3729835148 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7f6d9400f020 con 0x7f6dac109ad0 2024-04-25T01:47:32.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.592+0000 7f6daaa00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d8c0a2b90 0x7f6d8c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:47:32.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.593+0000 7f6daaa00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d8c0a2b90 0x7f6d8c0a5050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c005950 tx=0x7f6d9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:47:32.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.594+0000 7f6da3e00700 1 -- 172.21.15.102:0/3729835148 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d940cee20 con 0x7f6dac109ad0 2024-04-25T01:47:32.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:32.846+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6dac062670 con 0x7f6d8c0a2b90 2024-04-25T01:47:32.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:32 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:32.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:32 smithi110 ceph-mon[32848]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:47:32.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:32 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:32.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:32 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:33.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:32 smithi193 ceph-mon[32981]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:47:33.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:32 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:32 smithi102 ceph-mon[27796]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T01:47:33.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T01:47:33.732 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:47:33 smithi102 sudo[47516]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:33.732 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:47:33 smithi102 sudo[47516]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:33.732 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:47:33 smithi102 sudo[47516]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:33.732 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:47:33 smithi102 sudo[47516]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:33.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:33 smithi110 ceph-mon[32848]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:33.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:33 smithi110 ceph-mon[32848]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:47:33.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:33 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:33.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:33 smithi110 ceph-mon[32848]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:47:33.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:33 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:33.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:33 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:33.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:33 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:34.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:33 smithi193 ceph-mon[32981]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:34.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:33 smithi193 ceph-mon[32981]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:47:34.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:33 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:34.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:33 smithi193 ceph-mon[32981]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:47:34.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:33 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:34.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:33 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:34.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:33 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:34.063 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:33 smithi102 ceph-mon[27796]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:34.063 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:33 smithi102 ceph-mon[27796]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:47:34.063 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:33 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T01:47:34.063 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:33 smithi102 ceph-mon[27796]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T01:47:34.063 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:33 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:47:34.063 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:33 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:47:34.063 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:33 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:34.418 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:47:34 smithi102 sudo[47575]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:34.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:47:34 smithi102 sudo[47575]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:34.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:47:34 smithi102 sudo[47575]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:34.419 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 01:47:34 smithi102 sudo[47575]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:34.482 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:34 smithi110 sudo[41608]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:34.482 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:34 smithi110 sudo[41608]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:34.482 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:34 smithi110 sudo[41608]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:34.482 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:34 smithi110 sudo[41608]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:34.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:34 smithi102 ceph-mon[27796]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:34.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:34 smithi102 ceph-mon[27796]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T01:47:34.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:34 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:34.908 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:34 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:34.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:34 smithi110 ceph-mon[32848]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:34.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:34 smithi110 ceph-mon[32848]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T01:47:34.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:34 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:34.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:34 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:34.983 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:34 smithi110 sudo[41613]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:34.983 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:34 smithi110 sudo[41613]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:34.983 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:34 smithi110 sudo[41613]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:34.983 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 01:47:34 smithi110 sudo[41613]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:35.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:34 smithi193 ceph-mon[32981]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:47:35.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:34 smithi193 ceph-mon[32981]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T01:47:35.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:34 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:35.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:34 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:35.000 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:34 smithi193 sudo[41787]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:35.000 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:34 smithi193 sudo[41787]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:35.000 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:34 smithi193 sudo[41787]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:35.000 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:34 smithi193 sudo[41787]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:35.500 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:35 smithi193 sudo[41790]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:35.500 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:35 smithi193 sudo[41790]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:35.500 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:35 smithi193 sudo[41790]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:35.500 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 01:47:35 smithi193 sudo[41790]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:36.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:35 smithi102 sudo[47625]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:36.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:35 smithi102 sudo[47625]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:36.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:35 smithi102 sudo[47625]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:36.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:35 smithi102 sudo[47625]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:36.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:36.394+0000 7f6da3e00700 1 -- 172.21.15.102:0/3729835148 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6d9401d470 con 0x7f6dac109ad0 2024-04-25T01:47:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:36 smithi102 sudo[47648]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:36 smithi102 sudo[47648]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:36 smithi102 sudo[47648]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:36.668 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:36 smithi102 sudo[47648]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:36.871 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 sudo[41793]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:36.872 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 sudo[41793]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:36.872 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 sudo[41793]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:36.872 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 sudo[41793]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:37.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:36 smithi102 ceph-mon[27796]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:37.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:36 smithi102 ceph-mon[27796]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:47:37.168 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:36 smithi102 ceph-mon[27796]: Cluster is now healthy 2024-04-25T01:47:37.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:36 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:37.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:36 smithi102 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:37.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 sudo[41796]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:37.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 sudo[41796]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:37.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 sudo[41796]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:37.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 sudo[41796]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:37.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 ceph-mon[32981]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:37.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 ceph-mon[32981]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:47:37.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 ceph-mon[32981]: Cluster is now healthy 2024-04-25T01:47:37.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:37.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:36 smithi193 ceph-mon[32981]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:37.369 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 sudo[41617]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T01:47:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 sudo[41617]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 sudo[41617]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 sudo[41617]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 sudo[41620]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T01:47:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 sudo[41620]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T01:47:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 sudo[41620]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T01:47:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 sudo[41620]: pam_unix(sudo:session): session closed for user root 2024-04-25T01:47:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:37.370 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:47:37.371 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: Cluster is now healthy 2024-04-25T01:47:37.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:37.732 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:47:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/174474256' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "240b29df-2fe9-4354-83c2-6a319aecefac"}]: dispatch 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/174474256' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "240b29df-2fe9-4354-83c2-6a319aecefac"}]': finished 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:47:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:47:38.232 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/174474256' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "240b29df-2fe9-4354-83c2-6a319aecefac"}]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:38.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:38.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:38.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:38.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/174474256' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "240b29df-2fe9-4354-83c2-6a319aecefac"}]': finished 2024-04-25T01:47:38.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:47:38.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:47:38.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:38.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:38.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:38.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/174474256' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "240b29df-2fe9-4354-83c2-6a319aecefac"}]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:47:38.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T01:47:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/174474256' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "240b29df-2fe9-4354-83c2-6a319aecefac"}]': finished 2024-04-25T01:47:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T01:47:38.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:47:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:38 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3635516396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:39.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:38 smithi102 ceph-mon[27796]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:39.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:38 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3635516396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:39.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:38 smithi110 ceph-mon[32848]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:39.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:38 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3635516396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:47:39.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:38 smithi193 ceph-mon[32981]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:41.951 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:41 smithi102 ceph-mon[27796]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-04-25T01:47:41.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:41 smithi110 ceph-mon[32848]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-04-25T01:47:42.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:41 smithi193 ceph-mon[32981]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-04-25T01:47:43.945 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:43 smithi102 ceph-mon[27796]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-04-25T01:47:43.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:43 smithi110 ceph-mon[32848]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-04-25T01:47:44.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:43 smithi193 ceph-mon[32981]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-04-25T01:47:45.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:45 smithi102 ceph-mon[27796]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-04-25T01:47:45.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:45 smithi110 ceph-mon[32848]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-04-25T01:47:46.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:45 smithi193 ceph-mon[32981]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-04-25T01:47:47.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:47 smithi110 ceph-mon[32848]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:47:47.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:47 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:47:47.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:47 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:47.988 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:47 smithi102 ceph-mon[27796]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:47:47.988 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:47 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:47:47.989 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:47 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:48.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:47 smithi193 ceph-mon[32981]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-04-25T01:47:48.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:47 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T01:47:48.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:47 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:48.918 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:48 smithi102 ceph-mon[27796]: Deploying daemon osd.3 on smithi102 2024-04-25T01:47:48.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:48 smithi110 ceph-mon[32848]: Deploying daemon osd.3 on smithi102 2024-04-25T01:47:49.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:48 smithi193 ceph-mon[32981]: Deploying daemon osd.3 on smithi102 2024-04-25T01:47:49.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:49 smithi110 ceph-mon[32848]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 910 B/s rd, 23 KiB/s wr, 2 op/s 2024-04-25T01:47:50.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:49 smithi193 ceph-mon[32981]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 910 B/s rd, 23 KiB/s wr, 2 op/s 2024-04-25T01:47:50.010 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:49 smithi102 ceph-mon[27796]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 910 B/s rd, 23 KiB/s wr, 2 op/s 2024-04-25T01:47:51.775 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:51 smithi102 ceph-mon[27796]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:47:51.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:51 smithi110 ceph-mon[32848]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:47:52.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:51 smithi193 ceph-mon[32981]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-04-25T01:47:53.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:53 smithi110 ceph-mon[32848]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:54.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:53 smithi193 ceph-mon[32981]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:54.108 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:53 smithi102 ceph-mon[27796]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:55.969 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:55 smithi102 ceph-mon[27796]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:55.970 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:55 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:55.970 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:55 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:55.970 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:55 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:55.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:55 smithi110 ceph-mon[32848]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:55.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:55 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:55.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:55 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:55.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:55 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:56.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:55 smithi193 ceph-mon[32981]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:56.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:55 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:56.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:55 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:56.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:55 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:47:57.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:57 smithi110 ceph-mon[32848]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:58.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:57 smithi193 ceph-mon[32981]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:58.030 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:57 smithi102 ceph-mon[27796]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:59.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:59.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:59.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:59.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:59.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:59 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:59.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:59 smithi102 ceph-mon[27796]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:59.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:59 smithi102 ceph-mon[27796]: from='osd.3 [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:47:59.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:47:59 smithi102 ceph-mon[27796]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:47:59.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:59 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:59.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:59 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:59.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:59 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:59.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:59 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:59.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:59 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:59.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:59 smithi110 ceph-mon[32848]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:59.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:59 smithi110 ceph-mon[32848]: from='osd.3 [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:47:59.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:47:59 smithi110 ceph-mon[32848]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:47:59.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:59 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:59.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:59 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:59.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:59 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:47:59.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:59 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:47:59.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:59 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:47:59.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:59 smithi193 ceph-mon[32981]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:47:59.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:59 smithi193 ceph-mon[32981]: from='osd.3 [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:47:59.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:47:59 smithi193 ceph-mon[32981]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T01:47:59.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.887+0000 7f6da3e00700 1 -- 172.21.15.102:0/3729835148 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6dac062670 con 0x7f6d8c0a2b90 2024-04-25T01:47:59.885 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 3 on host 'smithi102' 2024-04-25T01:47:59.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d8c0a2b90 msgr2=0x7f6d8c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:59.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d8c0a2b90 0x7f6d8c0a5050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c005950 tx=0x7f6d9c009260 comp rx=0 tx=0).stop 2024-04-25T01:47:59.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dac109ad0 msgr2=0x7f6dac19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:47:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dac109ad0 0x7f6dac19efe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400d010 tx=0x7f6d94007a20 comp rx=0 tx=0).stop 2024-04-25T01:47:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 shutdown_connections 2024-04-25T01:47:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d8c0a2b90 0x7f6d8c0a5050 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dac10a470 0x7f6dac1a6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6dac1088d0 0x7f6dac19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 --2- 172.21.15.102:0/3729835148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6dac109ad0 0x7f6dac19efe0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:47:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 >> 172.21.15.102:0/3729835148 conn(0x7f6dac0778a0 msgr2=0x7f6dac10cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:47:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.890+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 shutdown_connections 2024-04-25T01:47:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:47:59.891+0000 7f6db1a00700 1 -- 172.21.15.102:0/3729835148 wait complete. 2024-04-25T01:48:00.297 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:00 smithi102 ceph-mon[27796]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:48:00.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:00 smithi102 ceph-mon[27796]: from='osd.3 [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:48:00.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:00 smithi102 ceph-mon[27796]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:48:00.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:00 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:00.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:00 smithi102 ceph-mon[27796]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:48:00.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:00 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:00.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:00 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:00.298 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:00 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:00.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:00 smithi110 ceph-mon[32848]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:48:00.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:00 smithi110 ceph-mon[32848]: from='osd.3 [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:48:00.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:00 smithi110 ceph-mon[32848]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:48:00.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:00 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:00.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:00 smithi110 ceph-mon[32848]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:48:00.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:00 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:00.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:00 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:00.483 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:00 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:00.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:00 smithi193 ceph-mon[32981]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T01:48:00.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:00 smithi193 ceph-mon[32981]: from='osd.3 [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:48:00.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:00 smithi193 ceph-mon[32981]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T01:48:00.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:00 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:00.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:00 smithi193 ceph-mon[32981]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:48:00.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:00 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:00.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:00 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:00.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:00 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:00.640 DEBUG:teuthology.orchestra.run.smithi102:osd.3> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T01:48:00.644 INFO:tasks.cephadm:Deploying osd.4 on smithi110 with /dev/vg_nvme/lv_3... 2024-04-25T01:48:00.645 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:48:00.885 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:48:01.300 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:01 smithi110 ceph-mon[32848]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:48:01.300 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:01 smithi110 ceph-mon[32848]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T01:48:01.300 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:01.300 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:01.301 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:01 smithi110 ceph-mon[32848]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:01.301 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:01 smithi110 ceph-mon[32848]: osdmap e26: 4 total, 3 up, 4 in 2024-04-25T01:48:01.301 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:01.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:01 smithi102 ceph-mon[27796]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:48:01.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:01 smithi102 ceph-mon[27796]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T01:48:01.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:01.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:01.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:01 smithi102 ceph-mon[27796]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:01.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:01 smithi102 ceph-mon[27796]: osdmap e26: 4 total, 3 up, 4 in 2024-04-25T01:48:01.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:01.420 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 01:48:01 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3[49716]: 2024-04-25T01:48:01.064+0000 7f3fb7600700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T01:48:01.421 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 01:48:01 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3[49716]: 2024-04-25T01:48:01.088+0000 7f3fac000700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:48:01.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:01 smithi193 ceph-mon[32981]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:48:01.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:01 smithi193 ceph-mon[32981]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T01:48:01.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:01.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:01.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:01 smithi193 ceph-mon[32981]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T01:48:01.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:01 smithi193 ceph-mon[32981]: osdmap e26: 4 total, 3 up, 4 in 2024-04-25T01:48:01.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:02.266 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:02 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:48:02.266 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:02 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:48:02.266 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:02 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:02.266 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:02 smithi110 ceph-mon[32848]: from='osd.3 ' entity='osd.3' 2024-04-25T01:48:02.266 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:02 smithi110 ceph-mon[32848]: osd.3 [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] boot 2024-04-25T01:48:02.266 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:02 smithi110 ceph-mon[32848]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:48:02.266 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:02 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:02.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:02 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:48:02.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:02 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:48:02.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:02 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:02.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:02 smithi102 ceph-mon[27796]: from='osd.3 ' entity='osd.3' 2024-04-25T01:48:02.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:02 smithi102 ceph-mon[27796]: osd.3 [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] boot 2024-04-25T01:48:02.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:02 smithi102 ceph-mon[27796]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:48:02.379 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:02 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:02.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:02 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:48:02.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:02 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:48:02.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:02 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:02.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:02 smithi193 ceph-mon[32981]: from='osd.3 ' entity='osd.3' 2024-04-25T01:48:02.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:02 smithi193 ceph-mon[32981]: osd.3 [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] boot 2024-04-25T01:48:02.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:02 smithi193 ceph-mon[32981]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T01:48:02.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:02 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:48:02.810 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:48:02.811 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:48:02.811 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-04-25T01:48:02.811 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-04-25T01:48:02.811 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00854528 s, 1.2 GB/s 2024-04-25T01:48:02.811 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-04-25T01:48:03.329 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:03 smithi110 ceph-mon[32848]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:03.329 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:03 smithi110 ceph-mon[32848]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:48:03.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:03 smithi102 ceph-mon[27796]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:03.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:03 smithi102 ceph-mon[27796]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:48:03.451 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_3 2024-04-25T01:48:03.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:03 smithi193 ceph-mon[32981]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:03.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:03 smithi193 ceph-mon[32981]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T01:48:03.701 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:48:05.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.203+0000 7f0736d6a700 1 -- 172.21.15.110:0/3140305869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 msgr2=0x7f07301095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:05.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.203+0000 7f0736d6a700 1 --2- 172.21.15.110:0/3140305869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 0x7f07301095e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f072800b0d0 tx=0x7f072800b3e0 comp rx=0 tx=0).stop 2024-04-25T01:48:05.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.203+0000 7f0736d6a700 1 -- 172.21.15.110:0/3140305869 shutdown_connections 2024-04-25T01:48:05.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.203+0000 7f0736d6a700 1 --2- 172.21.15.110:0/3140305869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 0x7f07301095e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:05.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.203+0000 7f0736d6a700 1 --2- 172.21.15.110:0/3140305869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0730101190 0x7f07301015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:05.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.203+0000 7f0736d6a700 1 --2- 172.21.15.110:0/3140305869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0730102390 0x7f07301027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:05.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.203+0000 7f0736d6a700 1 -- 172.21.15.110:0/3140305869 >> 172.21.15.110:0/3140305869 conn(0x7f07300fc750 msgr2=0x7f07300feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:48:05.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.203+0000 7f0736d6a700 1 -- 172.21.15.110:0/3140305869 shutdown_connections 2024-04-25T01:48:05.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.203+0000 7f0736d6a700 1 -- 172.21.15.110:0/3140305869 wait complete. 2024-04-25T01:48:05.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0736d6a700 1 Processor -- start 2024-04-25T01:48:05.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0736d6a700 1 -- start start 2024-04-25T01:48:05.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0736d6a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0730101190 0x7f0730198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:05.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0736d6a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0730102390 0x7f07301989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:05.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0736d6a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 0x7f073019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:05.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0736d6a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f073010def0 con 0x7f0730102390 2024-04-25T01:48:05.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0736d6a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f073010dd70 con 0x7f0730102d30 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0736d6a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f073010e070 con 0x7f0730101190 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0735307700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 0x7f073019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0735307700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 0x7f073019fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34170/0 (socket says 172.21.15.110:34170) 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0735307700 1 -- 172.21.15.110:0/1294114399 learned_addr learned my addr 172.21.15.110:0/1294114399 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f072ffff700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0730102390 0x7f07301989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0734b06700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0730101190 0x7f0730198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0735307700 1 -- 172.21.15.110:0/1294114399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0730101190 msgr2=0x7f0730198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0735307700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0730101190 0x7f0730198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0735307700 1 -- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0730102390 msgr2=0x7f07301989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:05.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0735307700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0730102390 0x7f07301989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0735307700 1 -- 172.21.15.110:0/1294114399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07301a00e0 con 0x7f0730102d30 2024-04-25T01:48:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0734b06700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0730101190 0x7f0730198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:48:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.204+0000 7f0735307700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 0x7f073019fa40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f07280062a0 tx=0x7f0728007e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.205+0000 7f072dffb700 1 -- 172.21.15.110:0/1294114399 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0728003c00 con 0x7f0730102d30 2024-04-25T01:48:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.205+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07301a0370 con 0x7f0730102d30 2024-04-25T01:48:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.205+0000 7f072dffb700 1 -- 172.21.15.110:0/1294114399 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0728003da0 con 0x7f0730102d30 2024-04-25T01:48:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.205+0000 7f072dffb700 1 -- 172.21.15.110:0/1294114399 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0728005560 con 0x7f0730102d30 2024-04-25T01:48:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.205+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07301a08d0 con 0x7f0730102d30 2024-04-25T01:48:05.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.206+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0730068f20 con 0x7f0730102d30 2024-04-25T01:48:05.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.207+0000 7f072dffb700 1 -- 172.21.15.110:0/1294114399 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0728005700 con 0x7f0730102d30 2024-04-25T01:48:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.207+0000 7f072dffb700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f07200a2c20 0x7f07200a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.207+0000 7f072dffb700 1 -- 172.21.15.110:0/1294114399 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 3669+0+0 (secure 0 0 0) 0x7f072800f020 con 0x7f0730102d30 2024-04-25T01:48:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.208+0000 7f0734b06700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f07200a2c20 0x7f07200a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:05.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.208+0000 7f0734b06700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f07200a2c20 0x7f07200a50e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f07301021f0 tx=0x7f0724008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:05.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.212+0000 7f072dffb700 1 -- 172.21.15.110:0/1294114399 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f072802b020 con 0x7f0730102d30 2024-04-25T01:48:05.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:05 smithi102 ceph-mon[27796]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:48:05.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:05 smithi102 ceph-mon[27796]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:05.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:05.416+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0730062670 con 0x7f07200a2c20 2024-04-25T01:48:05.413 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:05 smithi110 ceph-mon[32848]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:48:05.413 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:05 smithi110 ceph-mon[32848]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:05.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:05 smithi193 ceph-mon[32981]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T01:48:05.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:05 smithi193 ceph-mon[32981]: pgmap v120: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:06.340 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:06 smithi110 ceph-mon[32848]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T01:48:06.341 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:06 smithi110 ceph-mon[32848]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:06.341 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:06 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:06.341 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:06 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:06.341 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:06 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:06.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:06 smithi102 ceph-mon[27796]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T01:48:06.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:06 smithi102 ceph-mon[27796]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:06.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:06 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:06.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:06 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:06.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:06 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:06.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:06 smithi193 ceph-mon[32981]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T01:48:06.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:06 smithi193 ceph-mon[32981]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:06.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:06 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:06.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:06 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:06.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:06 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:07.322 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: osdmap e31: 4 total, 4 up, 4 in 2024-04-25T01:48:07.322 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi102 2024-04-25T01:48:07.322 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.323 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.323 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:07.323 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:07.323 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi102 to 8544M 2024-04-25T01:48:07.323 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.323 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:07.323 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:07.323 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.323 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:07 smithi110 ceph-mon[32848]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:48:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: osdmap e31: 4 total, 4 up, 4 in 2024-04-25T01:48:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi102 2024-04-25T01:48:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:07.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi102 to 8544M 2024-04-25T01:48:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:07 smithi102 ceph-mon[27796]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:48:07.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: osdmap e31: 4 total, 4 up, 4 in 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi102 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi102 to 8544M 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:07.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:07 smithi193 ceph-mon[32981]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:48:08.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:08 smithi102 ceph-mon[27796]: from='client.? 172.21.15.110:0/76603829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63705793-8fc3-4a74-80d0-41e70f84c5bf"}]: dispatch 2024-04-25T01:48:08.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:08 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63705793-8fc3-4a74-80d0-41e70f84c5bf"}]: dispatch 2024-04-25T01:48:08.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:08 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63705793-8fc3-4a74-80d0-41e70f84c5bf"}]': finished 2024-04-25T01:48:08.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:08 smithi102 ceph-mon[27796]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:48:08.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:08.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:08 smithi110 ceph-mon[32848]: from='client.? 172.21.15.110:0/76603829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63705793-8fc3-4a74-80d0-41e70f84c5bf"}]: dispatch 2024-04-25T01:48:08.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:08 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63705793-8fc3-4a74-80d0-41e70f84c5bf"}]: dispatch 2024-04-25T01:48:08.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:08 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63705793-8fc3-4a74-80d0-41e70f84c5bf"}]': finished 2024-04-25T01:48:08.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:08 smithi110 ceph-mon[32848]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:48:08.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:08.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:08 smithi193 ceph-mon[32981]: from='client.? 172.21.15.110:0/76603829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63705793-8fc3-4a74-80d0-41e70f84c5bf"}]: dispatch 2024-04-25T01:48:08.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:08 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63705793-8fc3-4a74-80d0-41e70f84c5bf"}]: dispatch 2024-04-25T01:48:08.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:08 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63705793-8fc3-4a74-80d0-41e70f84c5bf"}]': finished 2024-04-25T01:48:08.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:08 smithi193 ceph-mon[32981]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T01:48:08.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:09 smithi102 ceph-mon[27796]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T01:48:09.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:09 smithi102 ceph-mon[27796]: from='client.? 172.21.15.110:0/1086854732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:09.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:09 smithi110 ceph-mon[32848]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T01:48:09.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:09 smithi110 ceph-mon[32848]: from='client.? 172.21.15.110:0/1086854732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:09.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:09 smithi193 ceph-mon[32981]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T01:48:09.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:09 smithi193 ceph-mon[32981]: from='client.? 172.21.15.110:0/1086854732' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:11.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:11 smithi110 ceph-mon[32848]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:48:12.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:11 smithi193 ceph-mon[32981]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:48:12.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:11 smithi102 ceph-mon[27796]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:48:13.809 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:13 smithi110 ceph-mon[32848]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T01:48:13.971 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:13 smithi102 ceph-mon[27796]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T01:48:14.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:13 smithi193 ceph-mon[32981]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T01:48:15.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:15 smithi102 ceph-mon[27796]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:48:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:15.969 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:15 smithi110 ceph-mon[32848]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:15.969 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:48:15.969 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:16.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:15 smithi193 ceph-mon[32981]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:16.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T01:48:16.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:16.982 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:16 smithi110 ceph-mon[32848]: Deploying daemon osd.4 on smithi110 2024-04-25T01:48:17.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:16 smithi193 ceph-mon[32981]: Deploying daemon osd.4 on smithi110 2024-04-25T01:48:17.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:16 smithi102 ceph-mon[27796]: Deploying daemon osd.4 on smithi110 2024-04-25T01:48:17.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:17 smithi110 ceph-mon[32848]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:18.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:17 smithi193 ceph-mon[32981]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:18.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:17 smithi102 ceph-mon[27796]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:19.897 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:19 smithi110 ceph-mon[32848]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:20.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:19 smithi193 ceph-mon[32981]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:20.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:19 smithi102 ceph-mon[27796]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:21.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:21.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:21.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:21.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:21 smithi110 ceph-mon[32848]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:21.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:21.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:21.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:21.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:21 smithi193 ceph-mon[32981]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:21.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:21.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:21.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:21.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:21 smithi102 ceph-mon[27796]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:23.875 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:23 smithi110 ceph-mon[32848]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:23.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.946+0000 7f072dffb700 1 -- 172.21.15.110:0/1294114399 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0730062670 con 0x7f07200a2c20 2024-04-25T01:48:23.943 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 4 on host 'smithi110' 2024-04-25T01:48:23.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f07200a2c20 msgr2=0x7f07200a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:23.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f07200a2c20 0x7f07200a50e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f07301021f0 tx=0x7f0724008040 comp rx=0 tx=0).stop 2024-04-25T01:48:23.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 msgr2=0x7f073019fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:23.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 0x7f073019fa40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f07280062a0 tx=0x7f0728007e40 comp rx=0 tx=0).stop 2024-04-25T01:48:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 shutdown_connections 2024-04-25T01:48:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f07200a2c20 0x7f07200a50e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0730102d30 0x7f073019fa40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0730101190 0x7f0730198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 --2- 172.21.15.110:0/1294114399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0730102390 0x7f07301989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 >> 172.21.15.110:0/1294114399 conn(0x7f07300fc750 msgr2=0x7f0730105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:48:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 shutdown_connections 2024-04-25T01:48:23.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:48:23.949+0000 7f0736d6a700 1 -- 172.21.15.110:0/1294114399 wait complete. 2024-04-25T01:48:24.000 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:23 smithi193 ceph-mon[32981]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:24.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:23 smithi102 ceph-mon[27796]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:24.603 DEBUG:teuthology.orchestra.run.smithi110:osd.4> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T01:48:24.605 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-04-25T01:48:24.605 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T01:48:24.851 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:48:25.167 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.167 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:24 smithi193 ceph-mon[32981]: from='osd.4 [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:25.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:24 smithi193 ceph-mon[32981]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:25.168 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:24 smithi193 ceph-mon[32981]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:24 smithi102 ceph-mon[27796]: from='osd.4 [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:24 smithi102 ceph-mon[27796]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:25.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:24 smithi102 ceph-mon[27796]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:25.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:25.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:24 smithi110 ceph-mon[32848]: from='osd.4 [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:25.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:24 smithi110 ceph-mon[32848]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T01:48:25.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:24 smithi110 ceph-mon[32848]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:26.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:26 smithi110 ceph-mon[32848]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:48:26.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:26 smithi110 ceph-mon[32848]: from='osd.4 [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:48:26.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:26 smithi110 ceph-mon[32848]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T01:48:26.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:26 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:26.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:26 smithi110 ceph-mon[32848]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:48:26.234 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 01:48:26 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T01:48:26.030+0000 7f74ddae9700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T01:48:26.234 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 01:48:26 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T01:48:26.036+0000 7f74da0dd700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:48:26.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:26 smithi193 ceph-mon[32981]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:48:26.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:26 smithi193 ceph-mon[32981]: from='osd.4 [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:48:26.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:26 smithi193 ceph-mon[32981]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T01:48:26.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:26 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:26.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:26 smithi193 ceph-mon[32981]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:48:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:26 smithi102 ceph-mon[27796]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T01:48:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:26 smithi102 ceph-mon[27796]: from='osd.4 [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:48:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:26 smithi102 ceph-mon[27796]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T01:48:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:26 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:26.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:26 smithi102 ceph-mon[27796]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:48:26.913 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T01:48:26.913 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T01:48:26.914 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-04-25T01:48:26.914 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-04-25T01:48:26.914 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012262 s, 855 MB/s 2024-04-25T01:48:26.914 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 2024-04-25T01:48:26.914 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-04-25T01:48:27.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:27 smithi110 ceph-mon[32848]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:48:27.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:27 smithi110 ceph-mon[32848]: osdmap e34: 5 total, 4 up, 5 in 2024-04-25T01:48:27.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:27.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:27 smithi110 ceph-mon[32848]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:27.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:27.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:27 smithi102 ceph-mon[27796]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:48:27.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:27 smithi102 ceph-mon[27796]: osdmap e34: 5 total, 4 up, 5 in 2024-04-25T01:48:27.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:27.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:27 smithi102 ceph-mon[27796]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:27.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:27.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:27 smithi193 ceph-mon[32981]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:48:27.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:27 smithi193 ceph-mon[32981]: osdmap e34: 5 total, 4 up, 5 in 2024-04-25T01:48:27.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:27.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:27 smithi193 ceph-mon[32981]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T01:48:27.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:27.577 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-04-25T01:48:27.816 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: osd.4 [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] boot 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi110 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:28.097 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:28.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi110 to 8544M 2024-04-25T01:48:28.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:28.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:28.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:28.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:28.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:28.098 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:28 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: osd.4 [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] boot 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi110 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:28.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi110 to 8544M 2024-04-25T01:48:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:28.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:28 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: osd.4 [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] boot 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi110 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi110 to 8544M 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:28.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:28.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:28.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:28.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:28.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:28.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:28 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:29.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.471+0000 7f36bb63f700 1 -- 172.21.15.193:0/3382818476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 msgr2=0x7f36b41027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:29.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.471+0000 7f36bb63f700 1 --2- 172.21.15.193:0/3382818476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 0x7f36b41027f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f36a4009990 tx=0x7f36a4009ca0 comp rx=0 tx=0).stop 2024-04-25T01:48:29.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.471+0000 7f36bb63f700 1 -- 172.21.15.193:0/3382818476 shutdown_connections 2024-04-25T01:48:29.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.471+0000 7f36bb63f700 1 --2- 172.21.15.193:0/3382818476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4102d30 0x7f36b41095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:29.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.471+0000 7f36bb63f700 1 --2- 172.21.15.193:0/3382818476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36b4101190 0x7f36b41015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:29.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.471+0000 7f36bb63f700 1 --2- 172.21.15.193:0/3382818476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 0x7f36b41027f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.471+0000 7f36bb63f700 1 -- 172.21.15.193:0/3382818476 >> 172.21.15.193:0/3382818476 conn(0x7f36b40fc730 msgr2=0x7f36b40feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.471+0000 7f36bb63f700 1 -- 172.21.15.193:0/3382818476 shutdown_connections 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.471+0000 7f36bb63f700 1 -- 172.21.15.193:0/3382818476 wait complete. 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36bb63f700 1 Processor -- start 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36bb63f700 1 -- start start 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36bb63f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36b4101190 0x7f36b406aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36bb63f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 0x7f36b419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36bb63f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4102d30 0x7f36b419fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36bb63f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f36b410cec0 con 0x7f36b4101190 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36bb63f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f36b410cd40 con 0x7f36b4102d30 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36bb63f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f36b410d040 con 0x7f36b4102390 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36b8bda700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 0x7f36b419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36b8bda700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 0x7f36b419d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:58554/0 (socket says 172.21.15.193:58554) 2024-04-25T01:48:29.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36b8bda700 1 -- 172.21.15.193:0/850390234 learned_addr learned my addr 172.21.15.193:0/850390234 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36b9bdc700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4102d30 0x7f36b419fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36b8bda700 1 -- 172.21.15.193:0/850390234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4102d30 msgr2=0x7f36b419fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36b8bda700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4102d30 0x7f36b419fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36b8bda700 1 -- 172.21.15.193:0/850390234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36b4101190 msgr2=0x7f36b406aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36b8bda700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36b4101190 0x7f36b406aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.472+0000 7f36b8bda700 1 -- 172.21.15.193:0/850390234 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b41a0310 con 0x7f36b4102390 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.473+0000 7f36b8bda700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 0x7f36b419d400 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f36a4005910 tx=0x7f36a4004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.473+0000 7f36aa7fc700 1 -- 172.21.15.193:0/850390234 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4027070 con 0x7f36b4102390 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.473+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b41a05a0 con 0x7f36b4102390 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.473+0000 7f36aa7fc700 1 -- 172.21.15.193:0/850390234 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36a40041d0 con 0x7f36b4102390 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.473+0000 7f36aa7fc700 1 -- 172.21.15.193:0/850390234 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a400f6c0 con 0x7f36b4102390 2024-04-25T01:48:29.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.473+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b41a0a70 con 0x7f36b4102390 2024-04-25T01:48:29.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.474+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b4068f20 con 0x7f36b4102390 2024-04-25T01:48:29.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.476+0000 7f36aa7fc700 1 -- 172.21.15.193:0/850390234 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f36a402a070 con 0x7f36b4102390 2024-04-25T01:48:29.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.476+0000 7f36aa7fc700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f36a00a2c20 0x7f36a00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:29.472 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.476+0000 7f36aa7fc700 1 -- 172.21.15.193:0/850390234 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4101+0+0 (secure 0 0 0) 0x7f36a40cf6e0 con 0x7f36b4102390 2024-04-25T01:48:29.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.477+0000 7f36b93db700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f36a00a2c20 0x7f36a00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:29.473 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.478+0000 7f36b93db700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f36a00a2c20 0x7f36a00a50e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f36b41021f0 tx=0x7f36b0006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:29.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.480+0000 7f36aa7fc700 1 -- 172.21.15.193:0/850390234 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f36a4031070 con 0x7f36b4102390 2024-04-25T01:48:29.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:29 smithi193 ceph-mon[32981]: osdmap e36: 5 total, 5 up, 5 in 2024-04-25T01:48:29.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:29 smithi193 ceph-mon[32981]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:29 smithi102 ceph-mon[27796]: osdmap e36: 5 total, 5 up, 5 in 2024-04-25T01:48:29.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:29 smithi102 ceph-mon[27796]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:29.678 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:29.682+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f36b4062670 con 0x7f36a00a2c20 2024-04-25T01:48:29.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:29 smithi110 ceph-mon[32848]: osdmap e36: 5 total, 5 up, 5 in 2024-04-25T01:48:29.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:29 smithi110 ceph-mon[32848]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:30.652 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:30 smithi193 ceph-mon[32981]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:30.673 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:30 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:30.673 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:30 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:30.673 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:30 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:30.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:30 smithi102 ceph-mon[27796]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:30.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:30 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:30.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:30 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:30.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:30 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:30.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:30 smithi110 ceph-mon[32848]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:30.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:30 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:30.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:30 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:30.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:30 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:31.501 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:31 smithi193 ceph-mon[32981]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:31.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:31 smithi102 ceph-mon[27796]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:31.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:31 smithi110 ceph-mon[32848]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:32.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:32 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/1851612642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8109f99d-f1f9-424a-8d2e-d1ddb0626f0c"}]: dispatch 2024-04-25T01:48:32.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:32 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8109f99d-f1f9-424a-8d2e-d1ddb0626f0c"}]: dispatch 2024-04-25T01:48:32.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:32 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8109f99d-f1f9-424a-8d2e-d1ddb0626f0c"}]': finished 2024-04-25T01:48:32.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:32 smithi110 ceph-mon[32848]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:48:32.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:32 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:32.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:32 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/1851612642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8109f99d-f1f9-424a-8d2e-d1ddb0626f0c"}]: dispatch 2024-04-25T01:48:32.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:32 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8109f99d-f1f9-424a-8d2e-d1ddb0626f0c"}]: dispatch 2024-04-25T01:48:32.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:32 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8109f99d-f1f9-424a-8d2e-d1ddb0626f0c"}]': finished 2024-04-25T01:48:32.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:32 smithi193 ceph-mon[32981]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:48:32.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:32 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:32.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:32 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/1851612642' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8109f99d-f1f9-424a-8d2e-d1ddb0626f0c"}]: dispatch 2024-04-25T01:48:32.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:32 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8109f99d-f1f9-424a-8d2e-d1ddb0626f0c"}]: dispatch 2024-04-25T01:48:32.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:32 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8109f99d-f1f9-424a-8d2e-d1ddb0626f0c"}]': finished 2024-04-25T01:48:32.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:32 smithi102 ceph-mon[27796]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T01:48:32.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:33.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:33 smithi110 ceph-mon[32848]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:33.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:33 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/3202850224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:33.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:33 smithi193 ceph-mon[32981]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:33.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:33 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/3202850224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:33.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:33 smithi102 ceph-mon[27796]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:33.919 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:33 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/3202850224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:48:35.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:35 smithi110 ceph-mon[32848]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:36.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:35 smithi193 ceph-mon[32981]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:36.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:35 smithi102 ceph-mon[27796]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:37.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:37 smithi110 ceph-mon[32848]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:38.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:37 smithi193 ceph-mon[32981]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:38.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:37 smithi102 ceph-mon[27796]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:39.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:39 smithi110 ceph-mon[32848]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:40.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:39 smithi193 ceph-mon[32981]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:40.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:39 smithi102 ceph-mon[27796]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:40.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:40 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:48:40.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:40 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:41.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:40 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:48:41.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:40 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:40 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T01:48:41.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:40 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:41.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:41 smithi110 ceph-mon[32848]: Deploying daemon osd.5 on smithi193 2024-04-25T01:48:41.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:41 smithi110 ceph-mon[32848]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:41.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:41 smithi193 ceph-mon[32981]: Deploying daemon osd.5 on smithi193 2024-04-25T01:48:41.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:41 smithi193 ceph-mon[32981]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:41 smithi102 ceph-mon[27796]: Deploying daemon osd.5 on smithi193 2024-04-25T01:48:42.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:41 smithi102 ceph-mon[27796]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:43.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:43 smithi193 ceph-mon[32981]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:43.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:43 smithi110 ceph-mon[32848]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:44.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:43 smithi102 ceph-mon[27796]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:45.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:45 smithi110 ceph-mon[32848]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:45.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:45 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:45.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:45 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:45.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:45 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:45.985 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:45 smithi193 ceph-mon[32981]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:45.985 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:45 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:45.985 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:45 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:45.985 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:45 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:45 smithi102 ceph-mon[27796]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:45 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:46.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:45 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:46.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:45 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:48.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:47 smithi193 ceph-mon[32981]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:48.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:47 smithi102 ceph-mon[27796]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:48.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:47 smithi110 ceph-mon[32848]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:49.214 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.217+0000 7f36aa7fc700 1 -- 172.21.15.193:0/850390234 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f36b4062670 con 0x7f36a00a2c20 2024-04-25T01:48:49.215 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-04-25T01:48:49.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f36a00a2c20 msgr2=0x7f36a00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:49.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f36a00a2c20 0x7f36a00a50e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f36b41021f0 tx=0x7f36b0006d20 comp rx=0 tx=0).stop 2024-04-25T01:48:49.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 msgr2=0x7f36b419d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:49.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 0x7f36b419d400 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f36a4005910 tx=0x7f36a4004580 comp rx=0 tx=0).stop 2024-04-25T01:48:49.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 shutdown_connections 2024-04-25T01:48:49.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f36a00a2c20 0x7f36a00a50e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:49.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4102d30 0x7f36b419fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:49.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36b4101190 0x7f36b406aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:49.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 --2- 172.21.15.193:0/850390234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36b4102390 0x7f36b419d400 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:49.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 >> 172.21.15.193:0/850390234 conn(0x7f36b40fc730 msgr2=0x7f36b4105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:48:49.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.219+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 shutdown_connections 2024-04-25T01:48:49.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:48:49.220+0000 7f36bb63f700 1 -- 172.21.15.193:0/850390234 wait complete. 2024-04-25T01:48:49.871 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T01:48:49.874 INFO:tasks.cephadm:Deploying osd.6 on smithi102 with /dev/vg_nvme/lv_2... 2024-04-25T01:48:49.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:48:50.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:49 smithi193 ceph-mon[32981]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:50.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:49 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:49 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:49 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:49 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:49 smithi193 ceph-mon[32981]: from='osd.5 [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:48:50.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:49 smithi193 ceph-mon[32981]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:48:50.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:49 smithi102 ceph-mon[27796]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:50.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:49 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:49 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:49 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:49 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:49 smithi102 ceph-mon[27796]: from='osd.5 [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:48:50.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:49 smithi102 ceph-mon[27796]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:48:50.233 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:49 smithi110 ceph-mon[32848]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:50.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:49 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:49 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:49 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:49 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:50.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:49 smithi110 ceph-mon[32848]: from='osd.5 [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:48:50.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:49 smithi110 ceph-mon[32848]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T01:48:50.317 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:48:51.363 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:51 smithi193 ceph-mon[32981]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:48:51.363 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:51 smithi193 ceph-mon[32981]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T01:48:51.363 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:51.363 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:51 smithi193 ceph-mon[32981]: from='osd.5 [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:48:51.363 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:51 smithi193 ceph-mon[32981]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:48:51.363 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:51 smithi193 ceph-mon[32981]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:51.363 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 01:48:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T01:48:51.261+0000 7fccd924b700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T01:48:51.363 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 01:48:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T01:48:51.268+0000 7fccd583f700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:48:51.637 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:51 smithi102 ceph-mon[27796]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:48:51.637 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:51 smithi102 ceph-mon[27796]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T01:48:51.637 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:51.637 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:51 smithi102 ceph-mon[27796]: from='osd.5 [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:48:51.638 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:51 smithi102 ceph-mon[27796]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:48:51.638 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:51 smithi102 ceph-mon[27796]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:51.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:51 smithi110 ceph-mon[32848]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T01:48:51.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:51 smithi110 ceph-mon[32848]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T01:48:51.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:51 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:51.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:51 smithi110 ceph-mon[32848]: from='osd.5 [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:48:51.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:51 smithi110 ceph-mon[32848]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:48:51.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:51 smithi110 ceph-mon[32848]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T01:48:52.365 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:52 smithi193 ceph-mon[32981]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:48:52.365 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:52 smithi193 ceph-mon[32981]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T01:48:52.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:52.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:52.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:52 smithi193 ceph-mon[32981]: osd.5 [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] boot 2024-04-25T01:48:52.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:52 smithi193 ceph-mon[32981]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:48:52.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:52.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:52 smithi102 ceph-mon[27796]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:48:52.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:52 smithi102 ceph-mon[27796]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T01:48:52.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:52.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:52.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:52 smithi102 ceph-mon[27796]: osd.5 [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] boot 2024-04-25T01:48:52.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:52 smithi102 ceph-mon[27796]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:48:52.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:52.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:52 smithi110 ceph-mon[32848]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:48:52.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:52 smithi110 ceph-mon[32848]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T01:48:52.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:52.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:52.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:52 smithi110 ceph-mon[32848]: osd.5 [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] boot 2024-04-25T01:48:52.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:52 smithi110 ceph-mon[32848]: osdmap e40: 6 total, 6 up, 6 in 2024-04-25T01:48:52.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:48:53.147 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:48:53.147 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:48:53.147 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-04-25T01:48:53.147 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-04-25T01:48:53.147 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0590489 s, 178 MB/s 2024-04-25T01:48:53.147 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-04-25T01:48:53.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:48:53.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:48:53.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:48:53.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi193 2024-04-25T01:48:53.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:53.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:53 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi193 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:53.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:53.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:48:53.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:53.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:53.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:53.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:53 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:48:53.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:48:53.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:48:53.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi193 2024-04-25T01:48:53.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:48:53.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:53 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:48:54.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_2 2024-04-25T01:48:54.430 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:48:54.579 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:54 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi193 to 8544M 2024-04-25T01:48:54.579 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:54 smithi102 ceph-mon[27796]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:48:54.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:54 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi193 to 8544M 2024-04-25T01:48:54.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:54 smithi110 ceph-mon[32848]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:48:54.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:54 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi193 to 8544M 2024-04-25T01:48:54.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:54 smithi193 ceph-mon[32981]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T01:48:55.606 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:55 smithi102 ceph-mon[27796]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:48:55.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:55 smithi193 ceph-mon[32981]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:48:55.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:55 smithi110 ceph-mon[32848]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:48:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.464+0000 7fa3ea600700 1 -- 172.21.15.102:0/3572542116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e41056c0 msgr2=0x7fa3e4105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.464+0000 7fa3ea600700 1 --2- 172.21.15.102:0/3572542116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e41056c0 0x7fa3e4105b20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009990 tx=0x7fa3d0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:48:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.464+0000 7fa3ea600700 1 -- 172.21.15.102:0/3572542116 shutdown_connections 2024-04-25T01:48:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.464+0000 7fa3ea600700 1 --2- 172.21.15.102:0/3572542116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3e41044c0 0x7fa3e41048e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.464+0000 7fa3ea600700 1 --2- 172.21.15.102:0/3572542116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3e4106060 0x7fa3e410c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.464+0000 7fa3ea600700 1 --2- 172.21.15.102:0/3572542116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e41056c0 0x7fa3e4105b20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.464+0000 7fa3ea600700 1 -- 172.21.15.102:0/3572542116 >> 172.21.15.102:0/3572542116 conn(0x7fa3e40ffa80 msgr2=0x7fa3e4101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:48:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.464+0000 7fa3ea600700 1 -- 172.21.15.102:0/3572542116 shutdown_connections 2024-04-25T01:48:56.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.464+0000 7fa3ea600700 1 -- 172.21.15.102:0/3572542116 wait complete. 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.466+0000 7fa3ea600700 1 Processor -- start 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.466+0000 7fa3ea600700 1 -- start start 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.466+0000 7fa3ea600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3e41044c0 0x7fa3e419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.466+0000 7fa3ea600700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3e41056c0 0x7fa3e419ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.466+0000 7fa3ea600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e4106060 0x7fa3e41a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.466+0000 7fa3ea600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e4110100 con 0x7fa3e41044c0 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3ea600700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e410ff80 con 0x7fa3e41056c0 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e4106060 0x7fa3e41a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3ea600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e4110280 con 0x7fa3e4106060 2024-04-25T01:48:56.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e4106060 0x7fa3e41a1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56580/0 (socket says 172.21.15.102:56580) 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3e00700 1 -- 172.21.15.102:0/130644310 learned_addr learned my addr 172.21.15.102:0/130644310 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3400700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3e41044c0 0x7fa3e419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e2a00700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3e41056c0 0x7fa3e419ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3e00700 1 -- 172.21.15.102:0/130644310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3e41056c0 msgr2=0x7fa3e419ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3e00700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3e41056c0 0x7fa3e419ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3e00700 1 -- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3e41044c0 msgr2=0x7fa3e419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3e00700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3e41044c0 0x7fa3e419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3e00700 1 -- 172.21.15.102:0/130644310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e41a2340 con 0x7fa3e4106060 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3400700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3e41044c0 0x7fa3e419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:48:56.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.467+0000 7fa3e3e00700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e4106060 0x7fa3e41a1ca0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc00ef60 tx=0x7fa3dc00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.468+0000 7fa3dbe00700 1 -- 172.21.15.102:0/130644310 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc0092e0 con 0x7fa3e4106060 2024-04-25T01:48:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.468+0000 7fa3dbe00700 1 -- 172.21.15.102:0/130644310 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3dc004bb0 con 0x7fa3e4106060 2024-04-25T01:48:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.468+0000 7fa3dbe00700 1 -- 172.21.15.102:0/130644310 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc004280 con 0x7fa3e4106060 2024-04-25T01:48:56.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.468+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e41a25d0 con 0x7fa3e4106060 2024-04-25T01:48:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.468+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3e41a2b60 con 0x7fa3e4106060 2024-04-25T01:48:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.469+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3e4068f20 con 0x7fa3e4106060 2024-04-25T01:48:56.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.471+0000 7fa3dbe00700 1 -- 172.21.15.102:0/130644310 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa3dc01b030 con 0x7fa3e4106060 2024-04-25T01:48:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.472+0000 7fa3dbe00700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa3c80a2ca0 0x7fa3c80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:48:56.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.472+0000 7fa3dbe00700 1 -- 172.21.15.102:0/130644310 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4533+0+0 (secure 0 0 0) 0x7fa3dc0cdc70 con 0x7fa3e4106060 2024-04-25T01:48:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.472+0000 7fa3e3400700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa3c80a2ca0 0x7fa3c80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:48:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.473+0000 7fa3e3400700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa3c80a2ca0 0x7fa3c80a5160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4105520 tx=0x7fa3d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:48:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.476+0000 7fa3dbe00700 1 -- 172.21.15.102:0/130644310 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa3dc022020 con 0x7fa3e4106060 2024-04-25T01:48:56.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:56 smithi102 ceph-mon[27796]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T01:48:56.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:48:56.710+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa3e4062670 con 0x7fa3c80a2ca0 2024-04-25T01:48:56.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:56 smithi110 ceph-mon[32848]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T01:48:56.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:56 smithi193 ceph-mon[32981]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T01:48:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:57 smithi102 ceph-mon[27796]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T01:48:57.669 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:57 smithi102 ceph-mon[27796]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:48:57.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:57 smithi102 ceph-mon[27796]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:57.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:57.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:57.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:57.733 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:57 smithi110 ceph-mon[32848]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T01:48:57.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:57 smithi110 ceph-mon[32848]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:48:57.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:57 smithi110 ceph-mon[32848]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:57.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:57.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:57.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:57 smithi193 ceph-mon[32981]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T01:48:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:57 smithi193 ceph-mon[32981]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T01:48:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:57 smithi193 ceph-mon[32981]: from='client.24265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:48:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:48:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:48:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:48:59.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:48:59 smithi110 ceph-mon[32848]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-25T01:49:00.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:48:59 smithi193 ceph-mon[32981]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-25T01:49:00.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:48:59 smithi102 ceph-mon[27796]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-25T01:49:01.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:01 smithi102 ceph-mon[27796]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-25T01:49:01.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:01 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/4291054190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92fb32af-539d-474a-87ff-2305e626f89a"}]: dispatch 2024-04-25T01:49:01.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:01 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/4291054190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92fb32af-539d-474a-87ff-2305e626f89a"}]': finished 2024-04-25T01:49:01.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:01 smithi102 ceph-mon[27796]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:49:01.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:01.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:01 smithi110 ceph-mon[32848]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-25T01:49:01.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:01 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/4291054190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92fb32af-539d-474a-87ff-2305e626f89a"}]: dispatch 2024-04-25T01:49:01.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:01 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/4291054190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92fb32af-539d-474a-87ff-2305e626f89a"}]': finished 2024-04-25T01:49:01.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:01 smithi110 ceph-mon[32848]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:49:01.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:02.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:01 smithi193 ceph-mon[32981]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-25T01:49:02.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:01 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/4291054190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92fb32af-539d-474a-87ff-2305e626f89a"}]: dispatch 2024-04-25T01:49:02.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:01 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/4291054190' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92fb32af-539d-474a-87ff-2305e626f89a"}]': finished 2024-04-25T01:49:02.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:01 smithi193 ceph-mon[32981]: osdmap e45: 7 total, 6 up, 7 in 2024-04-25T01:49:02.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:02.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:02 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1639031610' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:03.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:02 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1639031610' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:03.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:02 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1639031610' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:03.983 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:03 smithi110 ceph-mon[32848]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-25T01:49:04.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:03 smithi193 ceph-mon[32981]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-25T01:49:04.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:03 smithi102 ceph-mon[27796]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-25T01:49:05.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:05 smithi110 ceph-mon[32848]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:06.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:05 smithi193 ceph-mon[32981]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:06.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:05 smithi102 ceph-mon[27796]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:07.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:07 smithi102 ceph-mon[27796]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:08.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:07 smithi193 ceph-mon[32981]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:08.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:07 smithi110 ceph-mon[32848]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:09.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:09 smithi102 ceph-mon[27796]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:10.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:09 smithi193 ceph-mon[32981]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:10.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:09 smithi110 ceph-mon[32848]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:11.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:10 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:49:11.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:10 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:11.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:10 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:49:11.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:10 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:11.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:10 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T01:49:11.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:10 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:12.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:11 smithi102 ceph-mon[27796]: Deploying daemon osd.6 on smithi102 2024-04-25T01:49:12.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:11 smithi102 ceph-mon[27796]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:12.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:11 smithi110 ceph-mon[32848]: Deploying daemon osd.6 on smithi102 2024-04-25T01:49:12.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:11 smithi110 ceph-mon[32848]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:12.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:11 smithi193 ceph-mon[32981]: Deploying daemon osd.6 on smithi102 2024-04-25T01:49:12.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:11 smithi193 ceph-mon[32981]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:14.156 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:13 smithi102 ceph-mon[27796]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:14.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:13 smithi110 ceph-mon[32848]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:14.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:13 smithi193 ceph-mon[32981]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:15.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:15 smithi102 ceph-mon[27796]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:16.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:15 smithi110 ceph-mon[32848]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:16.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:15 smithi193 ceph-mon[32981]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:18.169 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:17 smithi102 ceph-mon[27796]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:18.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:17 smithi110 ceph-mon[32848]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:18.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:17 smithi193 ceph-mon[32981]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:20.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:19 smithi102 ceph-mon[27796]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:20.103 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:19 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:20.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:19 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:20.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:19 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:20.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:19 smithi110 ceph-mon[32848]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:20.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:19 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:20.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:19 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:20.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:19 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:20.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:19 smithi193 ceph-mon[32981]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:20.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:19 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:20.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:19 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:20.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:19 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:22.106 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:21 smithi102 ceph-mon[27796]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:22.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:21 smithi110 ceph-mon[32848]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:22.251 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:21 smithi193 ceph-mon[32981]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:23 smithi102 ceph-mon[27796]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:23.670 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:23 smithi102 ceph-mon[27796]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:49:23.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.828+0000 7fa3dbe00700 1 -- 172.21.15.102:0/130644310 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa3e4062670 con 0x7fa3c80a2ca0 2024-04-25T01:49:23.826 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 6 on host 'smithi102' 2024-04-25T01:49:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa3c80a2ca0 msgr2=0x7fa3c80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa3c80a2ca0 0x7fa3c80a5160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4105520 tx=0x7fa3d4009260 comp rx=0 tx=0).stop 2024-04-25T01:49:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e4106060 msgr2=0x7fa3e41a1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e4106060 0x7fa3e41a1ca0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc00ef60 tx=0x7fa3dc00ef90 comp rx=0 tx=0).stop 2024-04-25T01:49:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 shutdown_connections 2024-04-25T01:49:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa3c80a2ca0 0x7fa3c80a5160 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3e4106060 0x7fa3e41a1ca0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa3e41044c0 0x7fa3e419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:23.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 --2- 172.21.15.102:0/130644310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3e41056c0 0x7fa3e419ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:23.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.831+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 >> 172.21.15.102:0/130644310 conn(0x7fa3e40ffa80 msgr2=0x7fa3e4108960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:23.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.832+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 shutdown_connections 2024-04-25T01:49:23.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:49:23.832+0000 7fa3ea600700 1 -- 172.21.15.102:0/130644310 wait complete. 2024-04-25T01:49:23.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:23 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:23.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:23 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:23.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:23 smithi110 ceph-mon[32848]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:23.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:23 smithi110 ceph-mon[32848]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:49:24.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:23 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:24.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:23 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:24.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:23 smithi193 ceph-mon[32981]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:24.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:23 smithi193 ceph-mon[32981]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T01:49:24.582 DEBUG:teuthology.orchestra.run.smithi102:osd.6> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T01:49:24.586 INFO:tasks.cephadm:Deploying osd.7 on smithi110 with /dev/vg_nvme/lv_2... 2024-04-25T01:49:24.587 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:49:24.834 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:49:24.848 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:49:24.848 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T01:49:24.848 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:49:24.848 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:24.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:24.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:24.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:24.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:24.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:24.849 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:24.953 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:25.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:25.662 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:25 smithi102 ceph-mon[27796]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:49:25.662 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:25 smithi102 ceph-mon[27796]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T01:49:25.662 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:25 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:25.662 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:25 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:25.663 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:25 smithi102 ceph-mon[27796]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:25.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:25 smithi110 ceph-mon[32848]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:49:25.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:25 smithi110 ceph-mon[32848]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T01:49:25.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:25 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:25.734 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:25 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:25.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:25 smithi110 ceph-mon[32848]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:25.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:25 smithi193 ceph-mon[32981]: from='osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:49:25.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:25 smithi193 ceph-mon[32981]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T01:49:25.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:25 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:25.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:25 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:25.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:25 smithi193 ceph-mon[32981]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-04-25T01:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:26 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:26 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:26 smithi102 ceph-mon[27796]: osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] boot 2024-04-25T01:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:26 smithi102 ceph-mon[27796]: osdmap e48: 7 total, 7 up, 7 in 2024-04-25T01:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:26 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:26.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:26 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:49:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:49:26.715 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-04-25T01:49:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-04-25T01:49:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00868612 s, 1.2 GB/s 2024-04-25T01:49:26.715 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-04-25T01:49:26.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:26 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:49:26.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:26 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:49:26.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:26 smithi110 ceph-mon[32848]: osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] boot 2024-04-25T01:49:26.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:26 smithi110 ceph-mon[32848]: osdmap e48: 7 total, 7 up, 7 in 2024-04-25T01:49:26.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:26 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:26.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:26 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:27.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:26 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:49:27.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:26 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:49:27.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:26 smithi193 ceph-mon[32981]: osd.6 [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] boot 2024-04-25T01:49:27.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:26 smithi193 ceph-mon[32981]: osdmap e48: 7 total, 7 up, 7 in 2024-04-25T01:49:27.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:26 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:27.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:26 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:49:27.305 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_2 2024-04-25T01:49:27.554 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:49:27.914 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:27 smithi110 ceph-mon[32848]: osdmap e49: 7 total, 7 up, 7 in 2024-04-25T01:49:27.914 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:27 smithi110 ceph-mon[32848]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:27.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:27 smithi102 ceph-mon[27796]: osdmap e49: 7 total, 7 up, 7 in 2024-04-25T01:49:27.920 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:27 smithi102 ceph-mon[27796]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:28.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:27 smithi193 ceph-mon[32981]: osdmap e49: 7 total, 7 up, 7 in 2024-04-25T01:49:28.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:27 smithi193 ceph-mon[32981]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.131+0000 7f962f28b700 1 -- 172.21.15.110:0/2808453215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628102280 msgr2=0x7f96281026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.131+0000 7f962f28b700 1 --2- 172.21.15.110:0/2808453215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628102280 0x7f96281026a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f96180099c0 tx=0x7f9618009cd0 comp rx=0 tx=0).stop 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.131+0000 7f962f28b700 1 -- 172.21.15.110:0/2808453215 shutdown_connections 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.131+0000 7f962f28b700 1 --2- 172.21.15.110:0/2808453215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628102280 0x7f96281026a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.131+0000 7f962f28b700 1 --2- 172.21.15.110:0/2808453215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9628103e20 0x7f962810a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.131+0000 7f962f28b700 1 --2- 172.21.15.110:0/2808453215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9628103480 0x7f96281038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.131+0000 7f962f28b700 1 -- 172.21.15.110:0/2808453215 >> 172.21.15.110:0/2808453215 conn(0x7f96280fd820 msgr2=0x7f96280ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.131+0000 7f962f28b700 1 -- 172.21.15.110:0/2808453215 shutdown_connections 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.131+0000 7f962f28b700 1 -- 172.21.15.110:0/2808453215 wait complete. 2024-04-25T01:49:29.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962f28b700 1 Processor -- start 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962f28b700 1 -- start start 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962f28b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9628102280 0x7f96281985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962f28b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628103480 0x7f9628198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962f28b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9628103e20 0x7f962819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962f28b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f962810df40 con 0x7f9628103e20 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962f28b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f962810ddc0 con 0x7f9628103480 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962f28b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f962810e0c0 con 0x7f9628102280 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962c826700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628103480 0x7f9628198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962c826700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628103480 0x7f9628198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39722/0 (socket says 172.21.15.110:39722) 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.132+0000 7f962c826700 1 -- 172.21.15.110:0/2267038567 learned_addr learned my addr 172.21.15.110:0/2267038567 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f962d027700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9628102280 0x7f96281985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f962c826700 1 -- 172.21.15.110:0/2267038567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9628102280 msgr2=0x7f96281985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:29.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f962c826700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9628102280 0x7f96281985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:29.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f962c826700 1 -- 172.21.15.110:0/2267038567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9628103e20 msgr2=0x7f962819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:29.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f962c826700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9628103e20 0x7f962819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:29.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f962c826700 1 -- 172.21.15.110:0/2267038567 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96281a0240 con 0x7f9628103480 2024-04-25T01:49:29.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f962c826700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628103480 0x7f9628198b20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f962400b530 tx=0x7f962400b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:29.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f961e7fc700 1 -- 172.21.15.110:0/2267038567 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9624015020 con 0x7f9628103480 2024-04-25T01:49:29.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96281a0530 con 0x7f9628103480 2024-04-25T01:49:29.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f961e7fc700 1 -- 172.21.15.110:0/2267038567 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9624011d10 con 0x7f9628103480 2024-04-25T01:49:29.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f961e7fc700 1 -- 172.21.15.110:0/2267038567 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962400f2e0 con 0x7f9628103480 2024-04-25T01:49:29.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.133+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96281a0af0 con 0x7f9628103480 2024-04-25T01:49:29.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.134+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9628068f20 con 0x7f9628103480 2024-04-25T01:49:29.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.135+0000 7f961e7fc700 1 -- 172.21.15.110:0/2267038567 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9624018020 con 0x7f9628103480 2024-04-25T01:49:29.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.136+0000 7f961e7fc700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f96140a2c20 0x7f96140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:29.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.136+0000 7f961e7fc700 1 -- 172.21.15.110:0/2267038567 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4965+0+0 (secure 0 0 0) 0x7f96240ce0c0 con 0x7f9628103480 2024-04-25T01:49:29.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.136+0000 7f962d027700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f96140a2c20 0x7f96140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:29.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.137+0000 7f962d027700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f96140a2c20 0x7f96140a50e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f961800b580 tx=0x7f9618005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:29.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.139+0000 7f961e7fc700 1 -- 172.21.15.110:0/2267038567 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f962401f020 con 0x7f9628103480 2024-04-25T01:49:29.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:29.343+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9628062670 con 0x7f96140a2c20 2024-04-25T01:49:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:29.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:29.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:30.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:30.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:30.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:30.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:30.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:30.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:30.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:30 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi102 2024-04-25T01:49:30.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:30 smithi110 ceph-mon[32848]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:30.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:30 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi102 to 5696M 2024-04-25T01:49:31.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:30 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi102 2024-04-25T01:49:31.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:30 smithi193 ceph-mon[32981]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:31.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:30 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi102 to 5696M 2024-04-25T01:49:31.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:30 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi102 2024-04-25T01:49:31.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:30 smithi102 ceph-mon[27796]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:31.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:30 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi102 to 5696M 2024-04-25T01:49:31.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:31 smithi110 ceph-mon[32848]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:32.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:31 smithi193 ceph-mon[32981]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:32.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:31 smithi102 ceph-mon[27796]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:32.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:32 smithi110 ceph-mon[32848]: from='client.? 172.21.15.110:0/499225202' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99d65788-2491-4fc8-b15c-1861814a488a"}]: dispatch 2024-04-25T01:49:32.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:32 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99d65788-2491-4fc8-b15c-1861814a488a"}]: dispatch 2024-04-25T01:49:32.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:32 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99d65788-2491-4fc8-b15c-1861814a488a"}]': finished 2024-04-25T01:49:32.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:32 smithi110 ceph-mon[32848]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:49:32.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:32 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:32.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:32 smithi110 ceph-mon[32848]: from='client.? 172.21.15.110:0/3422891597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:33.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:32 smithi193 ceph-mon[32981]: from='client.? 172.21.15.110:0/499225202' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99d65788-2491-4fc8-b15c-1861814a488a"}]: dispatch 2024-04-25T01:49:33.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:32 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99d65788-2491-4fc8-b15c-1861814a488a"}]: dispatch 2024-04-25T01:49:33.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:32 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99d65788-2491-4fc8-b15c-1861814a488a"}]': finished 2024-04-25T01:49:33.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:32 smithi193 ceph-mon[32981]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:49:33.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:32 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:33.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:32 smithi193 ceph-mon[32981]: from='client.? 172.21.15.110:0/3422891597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:33.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:32 smithi102 ceph-mon[27796]: from='client.? 172.21.15.110:0/499225202' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99d65788-2491-4fc8-b15c-1861814a488a"}]: dispatch 2024-04-25T01:49:33.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:32 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99d65788-2491-4fc8-b15c-1861814a488a"}]: dispatch 2024-04-25T01:49:33.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:32 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99d65788-2491-4fc8-b15c-1861814a488a"}]': finished 2024-04-25T01:49:33.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:32 smithi102 ceph-mon[27796]: osdmap e50: 8 total, 7 up, 8 in 2024-04-25T01:49:33.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:32 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:33.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:32 smithi102 ceph-mon[27796]: from='client.? 172.21.15.110:0/3422891597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:33.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:33 smithi110 ceph-mon[32848]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:34.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:33 smithi193 ceph-mon[32981]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:34.062 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:33 smithi102 ceph-mon[27796]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:35.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:35 smithi110 ceph-mon[32848]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:36.001 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:35 smithi193 ceph-mon[32981]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:36.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:35 smithi102 ceph-mon[27796]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:37.865 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:37 smithi110 ceph-mon[32848]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:38.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:37 smithi193 ceph-mon[32981]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:38.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:37 smithi102 ceph-mon[27796]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:39.997 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:39 smithi110 ceph-mon[32848]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:39.997 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:49:39.998 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:40.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:39 smithi193 ceph-mon[32981]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:40.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:49:40.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:40.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:39 smithi102 ceph-mon[27796]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:40.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T01:49:40.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:40.853 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:40 smithi110 ceph-mon[32848]: Deploying daemon osd.7 on smithi110 2024-04-25T01:49:41.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:40 smithi102 ceph-mon[27796]: Deploying daemon osd.7 on smithi110 2024-04-25T01:49:41.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:40 smithi193 ceph-mon[32981]: Deploying daemon osd.7 on smithi110 2024-04-25T01:49:41.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:41 smithi110 ceph-mon[32848]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:42.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:41 smithi102 ceph-mon[27796]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:41 smithi193 ceph-mon[32981]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:43.926 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:43 smithi110 ceph-mon[32848]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:44.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:43 smithi102 ceph-mon[27796]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:44.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:43 smithi193 ceph-mon[32981]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:46.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:45 smithi102 ceph-mon[27796]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:46.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:45 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:46.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:45 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:46.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:45 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:46.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:45 smithi110 ceph-mon[32848]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:46.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:45 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:46.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:45 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:46.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:45 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:45 smithi193 ceph-mon[32981]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:45 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:45 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:46.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:45 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:47.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:47 smithi110 ceph-mon[32848]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:48.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:47 smithi102 ceph-mon[27796]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:48.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:47 smithi193 ceph-mon[32981]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:48.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.782+0000 7f961e7fc700 1 -- 172.21.15.110:0/2267038567 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9628062670 con 0x7f96140a2c20 2024-04-25T01:49:48.780 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 7 on host 'smithi110' 2024-04-25T01:49:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f96140a2c20 msgr2=0x7f96140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f96140a2c20 0x7f96140a50e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f961800b580 tx=0x7f9618005fb0 comp rx=0 tx=0).stop 2024-04-25T01:49:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628103480 msgr2=0x7f9628198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628103480 0x7f9628198b20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f962400b530 tx=0x7f962400b840 comp rx=0 tx=0).stop 2024-04-25T01:49:48.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 shutdown_connections 2024-04-25T01:49:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f96140a2c20 0x7f96140a50e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9628103e20 0x7f962819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9628102280 0x7f96281985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 --2- 172.21.15.110:0/2267038567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9628103480 0x7f9628198b20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.784+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 >> 172.21.15.110:0/2267038567 conn(0x7f96280fd820 msgr2=0x7f9628106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.785+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 shutdown_connections 2024-04-25T01:49:48.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:49:48.785+0000 7f962f28b700 1 -- 172.21.15.110:0/2267038567 wait complete. 2024-04-25T01:49:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:48.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:48.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:48.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='osd.7 [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:48.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:48.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:48.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:48.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:49.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:49.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:49.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='osd.7 [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:49.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='osd.7 [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T01:49:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:49:49.398 DEBUG:teuthology.orchestra.run.smithi110:osd.7> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T01:49:49.401 INFO:tasks.cephadm:Deploying osd.8 on smithi193 with /dev/vg_nvme/lv_2... 2024-04-25T01:49:49.401 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T01:49:49.646 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:49:49.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:49 smithi110 ceph-mon[32848]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:49.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:49 smithi110 ceph-mon[32848]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:49:49.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:49 smithi110 ceph-mon[32848]: from='osd.7 [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:49:49.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:49 smithi110 ceph-mon[32848]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T01:49:49.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:49 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:49.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:49 smithi110 ceph-mon[32848]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:49:49.985 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 01:49:49 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T01:49:49.840+0000 7f1e62320700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T01:49:49.985 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 01:49:49 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T01:49:49.849+0000 7f1e5e914700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:49:50.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:49 smithi193 ceph-mon[32981]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:50.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:49 smithi193 ceph-mon[32981]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:49:50.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:49 smithi193 ceph-mon[32981]: from='osd.7 [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:49:50.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:49 smithi193 ceph-mon[32981]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T01:49:50.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:49 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:50.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:49 smithi193 ceph-mon[32981]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:49:50.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:49 smithi102 ceph-mon[27796]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:50.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:49 smithi102 ceph-mon[27796]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T01:49:50.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:49 smithi102 ceph-mon[27796]: from='osd.7 [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:49:50.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:49 smithi102 ceph-mon[27796]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T01:49:50.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:49 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:50.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:49 smithi102 ceph-mon[27796]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:49:51.125 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:50 smithi193 ceph-mon[32981]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:49:51.126 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:50 smithi193 ceph-mon[32981]: osdmap e52: 8 total, 7 up, 8 in 2024-04-25T01:49:51.126 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:50 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:51.126 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:50 smithi193 ceph-mon[32981]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:51.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:50 smithi102 ceph-mon[27796]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:49:51.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:50 smithi102 ceph-mon[27796]: osdmap e52: 8 total, 7 up, 8 in 2024-04-25T01:49:51.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:50 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:51.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:50 smithi102 ceph-mon[27796]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:51.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:50 smithi110 ceph-mon[32848]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:49:51.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:50 smithi110 ceph-mon[32848]: osdmap e52: 8 total, 7 up, 8 in 2024-04-25T01:49:51.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:50 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:51.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:50 smithi110 ceph-mon[32848]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-04-25T01:49:51.674 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T01:49:51.674 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T01:49:51.674 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-04-25T01:49:51.675 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-04-25T01:49:51.675 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127338 s, 823 MB/s 2024-04-25T01:49:51.675 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-04-25T01:49:52.129 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:51 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:49:52.129 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:51 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:49:52.129 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:51 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:52.129 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:51 smithi110 ceph-mon[32848]: osd.7 [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] boot 2024-04-25T01:49:52.129 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:51 smithi110 ceph-mon[32848]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:49:52.129 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:51 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:52.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:51 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:49:52.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:51 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:49:52.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:52.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:51 smithi102 ceph-mon[27796]: osd.7 [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] boot 2024-04-25T01:49:52.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:51 smithi102 ceph-mon[27796]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:49:52.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:51 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:51 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:49:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:51 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:49:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:51 smithi193 ceph-mon[32981]: osd.7 [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] boot 2024-04-25T01:49:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:51 smithi193 ceph-mon[32981]: osdmap e53: 8 total, 8 up, 8 in 2024-04-25T01:49:52.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:51 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:49:52.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-04-25T01:49:52.549 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:49:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:49:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi110 2024-04-25T01:49:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi110 to 5696M 2024-04-25T01:49:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:53.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:53.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:52 smithi102 ceph-mon[27796]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:53.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:49:53.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi110 2024-04-25T01:49:53.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:53.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:53.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:53.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi110 to 5696M 2024-04-25T01:49:53.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:53.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:53.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:52 smithi110 ceph-mon[32848]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T01:49:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi110 2024-04-25T01:49:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:49:53.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi110 to 5696M 2024-04-25T01:49:53.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:53.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:49:53.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:49:53.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:52 smithi193 ceph-mon[32981]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:53.917 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:53 smithi193 ceph-mon[32981]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:49:53.917 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:53 smithi193 ceph-mon[32981]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-25T01:49:54.135 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.131+0000 7fddd30e4700 1 -- 172.21.15.193:0/1613633916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc107fe0 msgr2=0x7fddcc10a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:54.135 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.131+0000 7fddd30e4700 1 --2- 172.21.15.193:0/1613633916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc107fe0 0x7fddcc10a3f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fddbc009990 tx=0x7fddbc009ca0 comp rx=0 tx=0).stop 2024-04-25T01:49:54.135 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.131+0000 7fddd30e4700 1 -- 172.21.15.193:0/1613633916 shutdown_connections 2024-04-25T01:49:54.135 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.131+0000 7fddd30e4700 1 --2- 172.21.15.193:0/1613633916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddcc102d80 0x7fddcc105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:54.135 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.131+0000 7fddd30e4700 1 --2- 172.21.15.193:0/1613633916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc107fe0 0x7fddcc10a3f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.131+0000 7fddd30e4700 1 --2- 172.21.15.193:0/1613633916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddcc1056b0 0x7fddcc107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.131+0000 7fddd30e4700 1 -- 172.21.15.193:0/1613633916 >> 172.21.15.193:0/1613633916 conn(0x7fddcc0fc730 msgr2=0x7fddcc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.131+0000 7fddd30e4700 1 -- 172.21.15.193:0/1613633916 shutdown_connections 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.131+0000 7fddd30e4700 1 -- 172.21.15.193:0/1613633916 wait complete. 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd30e4700 1 Processor -- start 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd30e4700 1 -- start start 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd30e4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc102d80 0x7fddcc198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd30e4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddcc1056b0 0x7fddcc198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd30e4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddcc107fe0 0x7fddcc19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd30e4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc10df90 con 0x7fddcc1056b0 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd30e4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc10de10 con 0x7fddcc107fe0 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd30e4700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc10e110 con 0x7fddcc102d80 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd0e80700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc102d80 0x7fddcc198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:54.136 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd0e80700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc102d80 0x7fddcc198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:44260/0 (socket says 172.21.15.193:44260) 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd0e80700 1 -- 172.21.15.193:0/2584636777 learned_addr learned my addr 172.21.15.193:0/2584636777 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd1681700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddcc107fe0 0x7fddcc19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd0e80700 1 -- 172.21.15.193:0/2584636777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddcc107fe0 msgr2=0x7fddcc19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd0e80700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddcc107fe0 0x7fddcc19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd0e80700 1 -- 172.21.15.193:0/2584636777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddcc1056b0 msgr2=0x7fddcc198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd0e80700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddcc1056b0 0x7fddcc198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd0e80700 1 -- 172.21.15.193:0/2584636777 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddcc1a02a0 con 0x7fddcc102d80 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.132+0000 7fddd1681700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddcc107fe0 0x7fddcc19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.133+0000 7fddd0e80700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc102d80 0x7fddcc198640 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fddc400ba80 tx=0x7fddc400be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.133+0000 7fddc9ffb700 1 -- 172.21.15.193:0/2584636777 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4019070 con 0x7fddcc102d80 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.133+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddcc1a0590 con 0x7fddcc102d80 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.133+0000 7fddc9ffb700 1 -- 172.21.15.193:0/2584636777 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddc4014440 con 0x7fddcc102d80 2024-04-25T01:49:54.137 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.133+0000 7fddc9ffb700 1 -- 172.21.15.193:0/2584636777 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc40075d0 con 0x7fddcc102d80 2024-04-25T01:49:54.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.133+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddcc1a0b50 con 0x7fddcc102d80 2024-04-25T01:49:54.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.134+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddcc068f20 con 0x7fddcc102d80 2024-04-25T01:49:54.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.135+0000 7fddc9ffb700 1 -- 172.21.15.193:0/2584636777 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fddc401c070 con 0x7fddcc102d80 2024-04-25T01:49:54.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.135+0000 7fddc9ffb700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fddb80a2c20 0x7fddb80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:49:54.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.136+0000 7fddc9ffb700 1 -- 172.21.15.193:0/2584636777 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5397+0+0 (secure 0 0 0) 0x7fddc40cd9e0 con 0x7fddcc102d80 2024-04-25T01:49:54.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.136+0000 7fddcbfff700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fddb80a2c20 0x7fddb80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:49:54.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.137+0000 7fddcbfff700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fddb80a2c20 0x7fddb80a50e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fddb40097b0 tx=0x7fddb4006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:49:54.138 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.140+0000 7fddc9ffb700 1 -- 172.21.15.193:0/2584636777 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fddc409b4a0 con 0x7fddcc102d80 2024-04-25T01:49:54.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:53 smithi102 ceph-mon[27796]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:49:54.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:53 smithi102 ceph-mon[27796]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-25T01:49:54.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:53 smithi110 ceph-mon[32848]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T01:49:54.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:53 smithi110 ceph-mon[32848]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-25T01:49:54.343 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:49:54.345+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fddcc062670 con 0x7fddb80a2c20 2024-04-25T01:49:55.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:54 smithi193 ceph-mon[32981]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T01:49:55.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:54 smithi193 ceph-mon[32981]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:55.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:55.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:55.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:55.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:54 smithi193 ceph-mon[32981]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:55.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:54 smithi102 ceph-mon[27796]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T01:49:55.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:54 smithi102 ceph-mon[27796]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:55.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:55.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:55.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:55.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:54 smithi102 ceph-mon[27796]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:55.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:54 smithi110 ceph-mon[32848]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T01:49:55.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:54 smithi110 ceph-mon[32848]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:49:55.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:49:55.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:49:55.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:49:55.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:54 smithi110 ceph-mon[32848]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:56.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:55 smithi102 ceph-mon[27796]: osdmap e57: 8 total, 8 up, 8 in 2024-04-25T01:49:56.177 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:55 smithi193 ceph-mon[32981]: osdmap e57: 8 total, 8 up, 8 in 2024-04-25T01:49:56.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:55 smithi110 ceph-mon[32848]: osdmap e57: 8 total, 8 up, 8 in 2024-04-25T01:49:57.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:56 smithi102 ceph-mon[27796]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:57.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:56 smithi110 ceph-mon[32848]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:57.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:56 smithi193 ceph-mon[32981]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:58.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:57 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/4119898814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7180db-d4e2-468d-a845-3846caa7a043"}]: dispatch 2024-04-25T01:49:58.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:57 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7180db-d4e2-468d-a845-3846caa7a043"}]: dispatch 2024-04-25T01:49:58.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:57 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc7180db-d4e2-468d-a845-3846caa7a043"}]': finished 2024-04-25T01:49:58.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:57 smithi102 ceph-mon[27796]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:49:58.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:49:58.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:57 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/330801589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:58.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:57 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/4119898814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7180db-d4e2-468d-a845-3846caa7a043"}]: dispatch 2024-04-25T01:49:58.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:57 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7180db-d4e2-468d-a845-3846caa7a043"}]: dispatch 2024-04-25T01:49:58.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:57 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc7180db-d4e2-468d-a845-3846caa7a043"}]': finished 2024-04-25T01:49:58.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:57 smithi110 ceph-mon[32848]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:49:58.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:49:58.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:57 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/330801589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:57 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/4119898814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7180db-d4e2-468d-a845-3846caa7a043"}]: dispatch 2024-04-25T01:49:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:57 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc7180db-d4e2-468d-a845-3846caa7a043"}]: dispatch 2024-04-25T01:49:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:57 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc7180db-d4e2-468d-a845-3846caa7a043"}]': finished 2024-04-25T01:49:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:57 smithi193 ceph-mon[32981]: osdmap e58: 9 total, 8 up, 9 in 2024-04-25T01:49:58.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:49:58.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:57 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/330801589' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:49:59.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:58 smithi102 ceph-mon[27796]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:59.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:58 smithi110 ceph-mon[32848]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:49:59.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:58 smithi193 ceph-mon[32981]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 230 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:00.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:59 smithi102 ceph-mon[27796]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-25T01:50:00.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:49:59 smithi102 ceph-mon[27796]: Cluster is now healthy 2024-04-25T01:50:00.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:59 smithi110 ceph-mon[32848]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-25T01:50:00.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:49:59 smithi110 ceph-mon[32848]: Cluster is now healthy 2024-04-25T01:50:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:59 smithi193 ceph-mon[32981]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-25T01:50:00.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:49:59 smithi193 ceph-mon[32981]: Cluster is now healthy 2024-04-25T01:50:01.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:00 smithi110 ceph-mon[32848]: overall HEALTH_OK 2024-04-25T01:50:01.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:00 smithi110 ceph-mon[32848]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:01.240 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:00 smithi193 ceph-mon[32981]: overall HEALTH_OK 2024-04-25T01:50:01.240 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:00 smithi193 ceph-mon[32981]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:01.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:00 smithi102 ceph-mon[27796]: overall HEALTH_OK 2024-04-25T01:50:01.420 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:00 smithi102 ceph-mon[27796]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:03.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:03 smithi110 ceph-mon[32848]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:04.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:03 smithi193 ceph-mon[32981]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:04.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:03 smithi102 ceph-mon[27796]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:04.996 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:50:04.997 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:05.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:50:05.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:05.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T01:50:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:06.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:05 smithi193 ceph-mon[32981]: Deploying daemon osd.8 on smithi193 2024-04-25T01:50:06.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:05 smithi193 ceph-mon[32981]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:06.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:05 smithi102 ceph-mon[27796]: Deploying daemon osd.8 on smithi193 2024-04-25T01:50:06.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:05 smithi102 ceph-mon[27796]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:06.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:05 smithi110 ceph-mon[32848]: Deploying daemon osd.8 on smithi193 2024-04-25T01:50:06.174 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:05 smithi110 ceph-mon[32848]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:07.987 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:07 smithi193 ceph-mon[32981]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:08.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:07 smithi102 ceph-mon[27796]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:08.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:07 smithi110 ceph-mon[32848]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:09.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:09 smithi193 ceph-mon[32981]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:10.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:09 smithi102 ceph-mon[27796]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:10.234 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:09 smithi110 ceph-mon[32848]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:11.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:11.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:11.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:11.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:11 smithi102 ceph-mon[27796]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:11.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:11.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:11.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:11.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:11 smithi110 ceph-mon[32848]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:11.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:11.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:11.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:11 smithi193 ceph-mon[32981]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:13.778 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 8 on host 'smithi193' 2024-04-25T01:50:13.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.779+0000 7fddc9ffb700 1 -- 172.21.15.193:0/2584636777 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fddcc062670 con 0x7fddb80a2c20 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fddb80a2c20 msgr2=0x7fddb80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fddb80a2c20 0x7fddb80a50e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fddb40097b0 tx=0x7fddb4006d20 comp rx=0 tx=0).stop 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc102d80 msgr2=0x7fddcc198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc102d80 0x7fddcc198640 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fddc400ba80 tx=0x7fddc400be40 comp rx=0 tx=0).stop 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 shutdown_connections 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fddb80a2c20 0x7fddb80a50e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddcc107fe0 0x7fddcc19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc102d80 0x7fddcc198640 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 --2- 172.21.15.193:0/2584636777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddcc1056b0 0x7fddcc198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:13.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 >> 172.21.15.193:0/2584636777 conn(0x7fddcc0fc730 msgr2=0x7fddcc103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:50:13.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 shutdown_connections 2024-04-25T01:50:13.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:50:13.782+0000 7fddd30e4700 1 -- 172.21.15.193:0/2584636777 wait complete. 2024-04-25T01:50:13.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:13 smithi110 ceph-mon[32848]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:14.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:13 smithi193 ceph-mon[32981]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:14.125 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:13 smithi102 ceph-mon[27796]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:14.400 DEBUG:teuthology.orchestra.run.smithi193:osd.8> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T01:50:14.403 INFO:tasks.cephadm:Deploying osd.9 on smithi102 with /dev/vg_nvme/lv_1... 2024-04-25T01:50:14.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:50:14.850 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:50:14.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:14.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:14.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:14 smithi110 ceph-mon[32848]: from='osd.8 [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:14.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:14 smithi110 ceph-mon[32848]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:14.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:14.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:14.994 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:14.995 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:14.995 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:14 smithi102 ceph-mon[27796]: from='osd.8 [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:14.995 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:14 smithi102 ceph-mon[27796]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:14.995 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:14.995 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:15.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:15.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:15.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:14 smithi193 ceph-mon[32981]: from='osd.8 [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:15.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:14 smithi193 ceph-mon[32981]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T01:50:15.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:15.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:15.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:15 smithi110 ceph-mon[32848]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:15.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:15 smithi110 ceph-mon[32848]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:50:15.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:15 smithi110 ceph-mon[32848]: from='osd.8 [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:50:15.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:15 smithi110 ceph-mon[32848]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T01:50:15.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:15.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:15 smithi110 ceph-mon[32848]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:50:16.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:15 smithi102 ceph-mon[27796]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:16.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:15 smithi102 ceph-mon[27796]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:50:16.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:15 smithi102 ceph-mon[27796]: from='osd.8 [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:50:16.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:15 smithi102 ceph-mon[27796]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T01:50:16.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:16.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:15 smithi102 ceph-mon[27796]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:50:16.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:15 smithi193 ceph-mon[32981]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:16.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:15 smithi193 ceph-mon[32981]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T01:50:16.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:15 smithi193 ceph-mon[32981]: from='osd.8 [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:50:16.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:15 smithi193 ceph-mon[32981]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T01:50:16.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:16.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:15 smithi193 ceph-mon[32981]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:50:16.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 01:50:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T01:50:16.561+0000 7fc34e67f700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T01:50:16.753 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 01:50:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T01:50:16.574+0000 7fc345468700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:50:16.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:16 smithi102 ceph-mon[27796]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:50:16.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:16 smithi102 ceph-mon[27796]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T01:50:16.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:16 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:16.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:16 smithi102 ceph-mon[27796]: from='osd.8 ' entity='osd.8' 2024-04-25T01:50:16.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:16 smithi102 ceph-mon[27796]: osd.8 [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] boot 2024-04-25T01:50:16.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:16 smithi102 ceph-mon[27796]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:50:16.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:16 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:17.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:16 smithi193 ceph-mon[32981]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:50:17.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:16 smithi193 ceph-mon[32981]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T01:50:17.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:16 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:17.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:16 smithi193 ceph-mon[32981]: from='osd.8 ' entity='osd.8' 2024-04-25T01:50:17.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:16 smithi193 ceph-mon[32981]: osd.8 [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] boot 2024-04-25T01:50:17.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:16 smithi193 ceph-mon[32981]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:50:17.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:16 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:17.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:16 smithi110 ceph-mon[32848]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:50:17.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:16 smithi110 ceph-mon[32848]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T01:50:17.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:16 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:17.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:16 smithi110 ceph-mon[32848]: from='osd.8 ' entity='osd.8' 2024-04-25T01:50:17.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:16 smithi110 ceph-mon[32848]: osd.8 [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] boot 2024-04-25T01:50:17.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:16 smithi110 ceph-mon[32848]: osdmap e61: 9 total, 9 up, 9 in 2024-04-25T01:50:17.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:16 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:50:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:50:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:50:17.773 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-04-25T01:50:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-04-25T01:50:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0358835 s, 292 MB/s 2024-04-25T01:50:17.773 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-04-25T01:50:18.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:17 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:50:18.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:17 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:50:18.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:17 smithi102 ceph-mon[27796]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:18.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:17 smithi102 ceph-mon[27796]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:50:18.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:17 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:50:18.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:17 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:50:18.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:17 smithi110 ceph-mon[32848]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:18.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:17 smithi110 ceph-mon[32848]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:50:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:17 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:50:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:17 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:50:18.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:17 smithi193 ceph-mon[32981]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 233 MiB used, 715 GiB / 715 GiB avail 2024-04-25T01:50:18.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:17 smithi193 ceph-mon[32981]: osdmap e62: 9 total, 9 up, 9 in 2024-04-25T01:50:18.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi102:vg_nvme/lv_1 2024-04-25T01:50:19.041 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:50:19.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi193 2024-04-25T01:50:19.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi193 to 5696M 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:18 smithi102 ceph-mon[27796]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 596 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:19.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi193 2024-04-25T01:50:19.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:19.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:19.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:19.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi193 to 5696M 2024-04-25T01:50:19.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:50:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:18 smithi110 ceph-mon[32848]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 596 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:19.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi193 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi193 to 5696M 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:19.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:19.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:19.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:18 smithi193 ceph-mon[32981]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 596 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:20.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:19 smithi193 ceph-mon[32981]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T01:50:20.418 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:19 smithi102 ceph-mon[27796]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T01:50:20.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:19 smithi110 ceph-mon[32848]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T01:50:21.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.226+0000 7fbc01a00700 1 -- 172.21.15.102:0/910000322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc102d20 msgr2=0x7fbbfc105110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:21.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.226+0000 7fbc01a00700 1 --2- 172.21.15.102:0/910000322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc102d20 0x7fbbfc105110 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009f90 tx=0x7fbbec00b330 comp rx=0 tx=0).stop 2024-04-25T01:50:21.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.226+0000 7fbc01a00700 1 -- 172.21.15.102:0/910000322 shutdown_connections 2024-04-25T01:50:21.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.226+0000 7fbc01a00700 1 --2- 172.21.15.102:0/910000322 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbfc107f80 0x7fbbfc10a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.226+0000 7fbc01a00700 1 --2- 172.21.15.102:0/910000322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc102d20 0x7fbbfc105110 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.226+0000 7fbc01a00700 1 --2- 172.21.15.102:0/910000322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbfc105650 0x7fbbfc107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.226+0000 7fbc01a00700 1 -- 172.21.15.102:0/910000322 >> 172.21.15.102:0/910000322 conn(0x7fbbfc0fc750 msgr2=0x7fbbfc0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:50:21.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.226+0000 7fbc01a00700 1 -- 172.21.15.102:0/910000322 shutdown_connections 2024-04-25T01:50:21.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.226+0000 7fbc01a00700 1 -- 172.21.15.102:0/910000322 wait complete. 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.227+0000 7fbc01a00700 1 Processor -- start 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.227+0000 7fbc01a00700 1 -- start start 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbc01a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbfc102d20 0x7fbbfc19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbc01a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc105650 0x7fbbfc19ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbc01a00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbfc107f80 0x7fbbfc1a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbc01a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc10deb0 con 0x7fbbfc102d20 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbc01a00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc10dd30 con 0x7fbbfc107f80 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbc01a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbbfc10e030 con 0x7fbbfc105650 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbbfaa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbfc102d20 0x7fbbfc19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbbf2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc105650 0x7fbbfc19ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:21.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbbfaa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbfc102d20 0x7fbbfc19a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57804/0 (socket says 172.21.15.102:57804) 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbbfaa00700 1 -- 172.21.15.102:0/2128550022 learned_addr learned my addr 172.21.15.102:0/2128550022 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbbf2000700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc105650 0x7fbbfc19ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:47568/0 (socket says 172.21.15.102:47568) 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.228+0000 7fbbfb400700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbfc107f80 0x7fbbfc1a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbfaa00700 1 -- 172.21.15.102:0/2128550022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc105650 msgr2=0x7fbbfc19ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbfaa00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc105650 0x7fbbfc19ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbfaa00700 1 -- 172.21.15.102:0/2128550022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbfc107f80 msgr2=0x7fbbfc1a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbfaa00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbfc107f80 0x7fbbfc1a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbfaa00700 1 -- 172.21.15.102:0/2128550022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbfc1a2350 con 0x7fbbfc102d20 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbf2000700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc105650 0x7fbbfc19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbfb400700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbfc107f80 0x7fbbfc1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbfaa00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbfc102d20 0x7fbbfc19a6f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbbec007d30 tx=0x7fbbec007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:50:21.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbf3e00700 1 -- 172.21.15.102:0/2128550022 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec01f070 con 0x7fbbfc102d20 2024-04-25T01:50:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbfc1a25e0 con 0x7fbbfc102d20 2024-04-25T01:50:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbf3e00700 1 -- 172.21.15.102:0/2128550022 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbec004e40 con 0x7fbbfc102d20 2024-04-25T01:50:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbbf3e00700 1 -- 172.21.15.102:0/2128550022 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec01b750 con 0x7fbbfc102d20 2024-04-25T01:50:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.229+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbfc1a2ae0 con 0x7fbbfc102d20 2024-04-25T01:50:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.231+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbfc068f20 con 0x7fbbfc102d20 2024-04-25T01:50:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.232+0000 7fbbf3e00700 1 -- 172.21.15.102:0/2128550022 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbbec022070 con 0x7fbbfc102d20 2024-04-25T01:50:21.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.234+0000 7fbbf3e00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbe00a2c20 0x7fbbe00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:21.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.234+0000 7fbbf3e00700 1 -- 172.21.15.102:0/2128550022 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5829+0+0 (secure 0 0 0) 0x7fbbec00f020 con 0x7fbbfc102d20 2024-04-25T01:50:21.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.234+0000 7fbbf2000700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbe00a2c20 0x7fbbe00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:21.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.235+0000 7fbbf2000700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbe00a2c20 0x7fbbe00a50e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009750 tx=0x7fbbe4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:50:21.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.242+0000 7fbbf3e00700 1 -- 172.21.15.102:0/2128550022 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbbec029020 con 0x7fbbfc102d20 2024-04-25T01:50:21.243 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:21 smithi102 ceph-mon[27796]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T01:50:21.243 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:21 smithi102 ceph-mon[27796]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 596 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:21.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:21.477+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbbfc062670 con 0x7fbbe00a2c20 2024-04-25T01:50:21.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:21 smithi110 ceph-mon[32848]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T01:50:21.484 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:21 smithi110 ceph-mon[32848]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 596 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:21.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:21 smithi193 ceph-mon[32981]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T01:50:21.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:21 smithi193 ceph-mon[32981]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 596 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:22.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:22 smithi102 ceph-mon[27796]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:22.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:22 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:22.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:22 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:22.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:22 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:22.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:22 smithi110 ceph-mon[32848]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:22.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:22 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:22.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:22 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:22.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:22 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:22.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:22 smithi193 ceph-mon[32981]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:22.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:22 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:22.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:22 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:22.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:22 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:23.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:23 smithi102 ceph-mon[27796]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-25T01:50:23.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:23 smithi110 ceph-mon[32848]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-25T01:50:23.502 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:23 smithi193 ceph-mon[32981]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-25T01:50:25.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:25 smithi102 ceph-mon[27796]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-25T01:50:25.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:25 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/692191476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15d150e-08b4-45c2-8f9d-cd64dba163fb"}]: dispatch 2024-04-25T01:50:25.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:25 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15d150e-08b4-45c2-8f9d-cd64dba163fb"}]: dispatch 2024-04-25T01:50:25.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:25 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d15d150e-08b4-45c2-8f9d-cd64dba163fb"}]': finished 2024-04-25T01:50:25.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:25 smithi102 ceph-mon[27796]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:50:25.887 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:25 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:25 smithi110 ceph-mon[32848]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:25 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/692191476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15d150e-08b4-45c2-8f9d-cd64dba163fb"}]: dispatch 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:25 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15d150e-08b4-45c2-8f9d-cd64dba163fb"}]: dispatch 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:25 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d15d150e-08b4-45c2-8f9d-cd64dba163fb"}]': finished 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:25 smithi110 ceph-mon[32848]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:50:25.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:25 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:26.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:25 smithi193 ceph-mon[32981]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-25T01:50:26.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:25 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/692191476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15d150e-08b4-45c2-8f9d-cd64dba163fb"}]: dispatch 2024-04-25T01:50:26.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:25 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15d150e-08b4-45c2-8f9d-cd64dba163fb"}]: dispatch 2024-04-25T01:50:26.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:25 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d15d150e-08b4-45c2-8f9d-cd64dba163fb"}]': finished 2024-04-25T01:50:26.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:25 smithi193 ceph-mon[32981]: osdmap e66: 10 total, 9 up, 10 in 2024-04-25T01:50:26.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:25 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:26.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:26 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2065541829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:27.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:26 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2065541829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:27.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:26 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2065541829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:28.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:27 smithi193 ceph-mon[32981]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-25T01:50:28.170 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:27 smithi102 ceph-mon[27796]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-25T01:50:28.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:27 smithi110 ceph-mon[32848]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-04-25T01:50:30.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:29 smithi193 ceph-mon[32981]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-25T01:50:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:29 smithi102 ceph-mon[27796]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-25T01:50:30.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:29 smithi110 ceph-mon[32848]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-25T01:50:32.008 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:31 smithi102 ceph-mon[27796]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:50:32.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:31 smithi110 ceph-mon[32848]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:50:32.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:31 smithi193 ceph-mon[32981]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:50:33.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:33 smithi102 ceph-mon[27796]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:34.037 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:33 smithi110 ceph-mon[32848]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:34.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:33 smithi193 ceph-mon[32981]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:36.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:35 smithi102 ceph-mon[27796]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:36.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:50:36.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:36.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:35 smithi110 ceph-mon[32848]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:36.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:50:36.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:36.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:35 smithi193 ceph-mon[32981]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T01:50:36.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:36.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:36 smithi102 ceph-mon[27796]: Deploying daemon osd.9 on smithi102 2024-04-25T01:50:37.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:36 smithi193 ceph-mon[32981]: Deploying daemon osd.9 on smithi102 2024-04-25T01:50:37.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:36 smithi110 ceph-mon[32848]: Deploying daemon osd.9 on smithi102 2024-04-25T01:50:38.020 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:37 smithi102 ceph-mon[27796]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:38.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:37 smithi110 ceph-mon[32848]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:38.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:37 smithi193 ceph-mon[32981]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:39.865 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:39 smithi102 ceph-mon[27796]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:40.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:39 smithi110 ceph-mon[32848]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:40.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:39 smithi193 ceph-mon[32981]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:42.098 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:41 smithi102 ceph-mon[27796]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:42.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:41 smithi110 ceph-mon[32848]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:42.252 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:41 smithi193 ceph-mon[32981]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:44.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:43 smithi102 ceph-mon[27796]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:44.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:44.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:44.130 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:44.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:43 smithi110 ceph-mon[32848]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:44.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:44.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:44.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:44.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:43 smithi193 ceph-mon[32981]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:44.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:44.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:44.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:46.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:45 smithi102 ceph-mon[27796]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:46.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:45 smithi110 ceph-mon[32848]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:46.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:45 smithi193 ceph-mon[32981]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:47.817 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 9 on host 'smithi102' 2024-04-25T01:50:47.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.818+0000 7fbbf3e00700 1 -- 172.21.15.102:0/2128550022 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbbfc062670 con 0x7fbbe00a2c20 2024-04-25T01:50:47.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.821+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbe00a2c20 msgr2=0x7fbbe00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:47.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.821+0000 7fbc01a00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbe00a2c20 0x7fbbe00a50e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009750 tx=0x7fbbe4009310 comp rx=0 tx=0).stop 2024-04-25T01:50:47.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.821+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbfc102d20 msgr2=0x7fbbfc19a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:47.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.821+0000 7fbc01a00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbfc102d20 0x7fbbfc19a6f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbbec007d30 tx=0x7fbbec007d60 comp rx=0 tx=0).stop 2024-04-25T01:50:47.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.822+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 shutdown_connections 2024-04-25T01:50:47.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.822+0000 7fbc01a00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbe00a2c20 0x7fbbe00a50e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:47.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.822+0000 7fbc01a00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbfc107f80 0x7fbbfc1a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:47.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.822+0000 7fbc01a00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbfc102d20 0x7fbbfc19a6f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:47.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.822+0000 7fbc01a00700 1 --2- 172.21.15.102:0/2128550022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbfc105650 0x7fbbfc19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:47.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.822+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 >> 172.21.15.102:0/2128550022 conn(0x7fbbfc0fc750 msgr2=0x7fbbfc102520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:50:47.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.822+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 shutdown_connections 2024-04-25T01:50:47.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:50:47.822+0000 7fbc01a00700 1 -- 172.21.15.102:0/2128550022 wait complete. 2024-04-25T01:50:48.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:47 smithi102 ceph-mon[27796]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:48.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:47 smithi102 ceph-mon[27796]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:50:48.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:47 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:47 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:47 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:47 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:47 smithi110 ceph-mon[32848]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:48.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:47 smithi110 ceph-mon[32848]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:50:48.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:47 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:47 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:47 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:47 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:47 smithi193 ceph-mon[32981]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:47 smithi193 ceph-mon[32981]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T01:50:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:47 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:47 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:47 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:47 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:48.611 DEBUG:teuthology.orchestra.run.smithi102:osd.9> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T01:50:48.615 INFO:tasks.cephadm:Deploying osd.10 on smithi110 with /dev/vg_nvme/lv_1... 2024-04-25T01:50:48.615 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:50:48.862 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:50:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:48 smithi110 ceph-mon[32848]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:50:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:48 smithi110 ceph-mon[32848]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T01:50:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:48 smithi110 ceph-mon[32848]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:50:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:48 smithi110 ceph-mon[32848]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:48 smithi110 ceph-mon[32848]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:50:48.984 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:48 smithi110 ceph-mon[32848]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T01:50:48.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:48 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:49.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-mon[27796]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:50:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-mon[27796]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T01:50:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-mon[27796]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:50:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-mon[27796]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-mon[27796]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:50:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-mon[27796]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T01:50:49.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:49.172 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T01:50:48.820+0000 7f5d40a00700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T01:50:49.172 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 01:50:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T01:50:48.841+0000 7f5d3c200700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:50:49.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:48 smithi193 ceph-mon[32981]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T01:50:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:48 smithi193 ceph-mon[32981]: osdmap e67: 10 total, 9 up, 10 in 2024-04-25T01:50:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:48 smithi193 ceph-mon[32981]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-04-25T01:50:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:48 smithi193 ceph-mon[32981]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:48 smithi193 ceph-mon[32981]: from='osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-04-25T01:50:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:48 smithi193 ceph-mon[32981]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T01:50:49.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:48 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:50.143 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:49 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:50.143 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:49 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:50.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:49 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:50.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:49 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:50.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:49 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:50.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:49 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:50.849 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:50:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:50:50.850 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-04-25T01:50:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-04-25T01:50:50.850 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123547 s, 849 MB/s 2024-04-25T01:50:50.850 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-04-25T01:50:51.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:50 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:50:51.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:50 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:50:51.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:50 smithi110 ceph-mon[32848]: osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] boot 2024-04-25T01:50:51.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:50 smithi110 ceph-mon[32848]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T01:50:51.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:50 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:51.157 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:50 smithi110 ceph-mon[32848]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:51.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:50 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:50:51.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:50 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:50:51.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:50 smithi102 ceph-mon[27796]: osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] boot 2024-04-25T01:50:51.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:50 smithi102 ceph-mon[27796]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T01:50:51.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:50 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:51.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:50 smithi102 ceph-mon[27796]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:51.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:50 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:50:51.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:50 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:50:51.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:50 smithi193 ceph-mon[32981]: osd.9 [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] boot 2024-04-25T01:50:51.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:50 smithi193 ceph-mon[32981]: osdmap e69: 10 total, 10 up, 10 in 2024-04-25T01:50:51.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:50 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:50:51.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:50 smithi193 ceph-mon[32981]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 265 MiB used, 804 GiB / 805 GiB avail 2024-04-25T01:50:51.510 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi110:vg_nvme/lv_1 2024-04-25T01:50:51.752 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T01:50:52.111 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:51 smithi110 ceph-mon[32848]: osdmap e70: 10 total, 10 up, 10 in 2024-04-25T01:50:52.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:51 smithi102 ceph-mon[27796]: osdmap e70: 10 total, 10 up, 10 in 2024-04-25T01:50:52.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:51 smithi193 ceph-mon[32981]: osdmap e70: 10 total, 10 up, 10 in 2024-04-25T01:50:53.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:52 smithi110 ceph-mon[32848]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:53.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:52 smithi102 ceph-mon[27796]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:53.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:52 smithi193 ceph-mon[32981]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 -- 172.21.15.110:0/2205905200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 msgr2=0x7ff6a410a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/2205905200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 0x7ff6a410a7d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00b0a0 tx=0x7ff69c00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:50:53.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 -- 172.21.15.110:0/2205905200 shutdown_connections 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/2205905200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 0x7ff6a410a7d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/2205905200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6a41005a0 0x7ff6a41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/2205905200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a4103580 0x7ff6a41039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 -- 172.21.15.110:0/2205905200 >> 172.21.15.110:0/2205905200 conn(0x7ff6a40fbf00 msgr2=0x7ff6a40fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 -- 172.21.15.110:0/2205905200 shutdown_connections 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 -- 172.21.15.110:0/2205905200 wait complete. 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 Processor -- start 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 -- start start 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6a41005a0 0x7ff6a4198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a4103580 0x7ff6a4198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 0x7ff6a419f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.405+0000 7ff6ab26e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a410df10 con 0x7ff6a4103580 2024-04-25T01:50:53.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6ab26e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a410dd90 con 0x7ff6a4103f20 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6ab26e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a410e090 con 0x7ff6a41005a0 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a980b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 0x7ff6a419f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a980b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 0x7ff6a419f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42852/0 (socket says 172.21.15.110:42852) 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a980b700 1 -- 172.21.15.110:0/4162676514 learned_addr learned my addr 172.21.15.110:0/4162676514 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a900a700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6a41005a0 0x7ff6a4198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a980b700 1 -- 172.21.15.110:0/4162676514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6a41005a0 msgr2=0x7ff6a4198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a980b700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6a41005a0 0x7ff6a4198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a980b700 1 -- 172.21.15.110:0/4162676514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a4103580 msgr2=0x7ff6a4198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a980b700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a4103580 0x7ff6a4198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a980b700 1 -- 172.21.15.110:0/4162676514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a41a0060 con 0x7ff6a4103f20 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6a980b700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 0x7ff6a419f9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff69c007f00 tx=0x7ff69c00be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff69a7fc700 1 -- 172.21.15.110:0/4162676514 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c021030 con 0x7ff6a4103f20 2024-04-25T01:50:53.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff69a7fc700 1 -- 172.21.15.110:0/4162676514 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff69c007b00 con 0x7ff6a4103f20 2024-04-25T01:50:53.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a41a02f0 con 0x7ff6a4103f20 2024-04-25T01:50:53.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff69a7fc700 1 -- 172.21.15.110:0/4162676514 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c01d470 con 0x7ff6a4103f20 2024-04-25T01:50:53.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.406+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a41a07a0 con 0x7ff6a4103f20 2024-04-25T01:50:53.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.407+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a4068f20 con 0x7ff6a4103f20 2024-04-25T01:50:53.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.408+0000 7ff69a7fc700 1 -- 172.21.15.110:0/4162676514 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff69c024070 con 0x7ff6a4103f20 2024-04-25T01:50:53.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.409+0000 7ff69a7fc700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6900a2c20 0x7ff6900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:50:53.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.409+0000 7ff69a7fc700 1 -- 172.21.15.110:0/4162676514 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7ff69c00f020 con 0x7ff6a4103f20 2024-04-25T01:50:53.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.409+0000 7ff6a900a700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6900a2c20 0x7ff6900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:50:53.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.410+0000 7ff6a900a700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6900a2c20 0x7ff6900a50e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6a406a780 tx=0x7ff6a000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:50:53.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.414+0000 7ff69a7fc700 1 -- 172.21.15.110:0/4162676514 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff69c007cb0 con 0x7ff6a4103f20 2024-04-25T01:50:53.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:50:53.623+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff6a4062670 con 0x7ff6900a2c20 2024-04-25T01:50:54.553 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi102 2024-04-25T01:50:54.553 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi102 to 4272M 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.554 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:54.555 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:54.555 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:54.555 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:54.555 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:54.555 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:54.555 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:54.555 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:54 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi102 2024-04-25T01:50:54.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi102 to 4272M 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:54.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:54.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:54.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:54.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:54.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:54 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi102 2024-04-25T01:50:54.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:50:54.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi102 to 4272M 2024-04-25T01:50:54.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:50:54.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:54 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:50:55.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:55 smithi110 ceph-mon[32848]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:55.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:55 smithi193 ceph-mon[32981]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:55.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:55 smithi102 ceph-mon[27796]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:56.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:56 smithi110 ceph-mon[32848]: from='client.? 172.21.15.110:0/1863223669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb0f9f59-1254-4fbb-9f87-566e43a05d06"}]: dispatch 2024-04-25T01:50:56.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:56 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb0f9f59-1254-4fbb-9f87-566e43a05d06"}]: dispatch 2024-04-25T01:50:56.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:56 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb0f9f59-1254-4fbb-9f87-566e43a05d06"}]': finished 2024-04-25T01:50:56.735 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:56 smithi110 ceph-mon[32848]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:50:56.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:56 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:50:56.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:56 smithi193 ceph-mon[32981]: from='client.? 172.21.15.110:0/1863223669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb0f9f59-1254-4fbb-9f87-566e43a05d06"}]: dispatch 2024-04-25T01:50:56.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:56 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb0f9f59-1254-4fbb-9f87-566e43a05d06"}]: dispatch 2024-04-25T01:50:56.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:56 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb0f9f59-1254-4fbb-9f87-566e43a05d06"}]': finished 2024-04-25T01:50:56.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:56 smithi193 ceph-mon[32981]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:50:56.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:56 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:50:56.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:56 smithi102 ceph-mon[27796]: from='client.? 172.21.15.110:0/1863223669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb0f9f59-1254-4fbb-9f87-566e43a05d06"}]: dispatch 2024-04-25T01:50:56.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:56 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb0f9f59-1254-4fbb-9f87-566e43a05d06"}]: dispatch 2024-04-25T01:50:56.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:56 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb0f9f59-1254-4fbb-9f87-566e43a05d06"}]': finished 2024-04-25T01:50:56.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:56 smithi102 ceph-mon[27796]: osdmap e71: 11 total, 10 up, 11 in 2024-04-25T01:50:56.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:56 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:50:57.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:57 smithi193 ceph-mon[32981]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:57.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:57 smithi193 ceph-mon[32981]: from='client.? 172.21.15.110:0/288545782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:57.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:57 smithi102 ceph-mon[27796]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:57.921 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:57 smithi102 ceph-mon[27796]: from='client.? 172.21.15.110:0/288545782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:50:57.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:57 smithi110 ceph-mon[32848]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:50:57.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:57 smithi110 ceph-mon[32848]: from='client.? 172.21.15.110:0/288545782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:51:00.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:50:59 smithi193 ceph-mon[32981]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:00.045 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:50:59 smithi110 ceph-mon[32848]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:00.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:50:59 smithi102 ceph-mon[27796]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:02.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:01 smithi193 ceph-mon[32981]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:02.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:01 smithi102 ceph-mon[27796]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:02.223 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:01 smithi110 ceph-mon[32848]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:04.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:03 smithi102 ceph-mon[27796]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:04.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:03 smithi110 ceph-mon[32848]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:04.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:03 smithi193 ceph-mon[32981]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:04.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:51:04.985 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:51:05.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:05.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T01:51:05.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:05.936 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:05 smithi110 ceph-mon[32848]: Deploying daemon osd.10 on smithi110 2024-04-25T01:51:05.937 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:05 smithi110 ceph-mon[32848]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:06.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:05 smithi102 ceph-mon[27796]: Deploying daemon osd.10 on smithi110 2024-04-25T01:51:06.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:05 smithi102 ceph-mon[27796]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:05 smithi193 ceph-mon[32981]: Deploying daemon osd.10 on smithi110 2024-04-25T01:51:06.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:05 smithi193 ceph-mon[32981]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:07.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:07 smithi110 ceph-mon[32848]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:08.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:07 smithi102 ceph-mon[27796]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:08.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:07 smithi193 ceph-mon[32981]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:10.036 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:09 smithi110 ceph-mon[32848]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:10.037 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:09 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:10.037 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:09 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:10.037 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:09 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:09 smithi102 ceph-mon[27796]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:09 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:10.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:09 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:10.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:09 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:09 smithi193 ceph-mon[32981]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:09 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:09 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:10.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:09 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:12.156 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:11 smithi110 ceph-mon[32848]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:12.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:11 smithi102 ceph-mon[27796]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:12.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:11 smithi193 ceph-mon[32981]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:13.260 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:13 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:13.260 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:13 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:13.260 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:13 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:13.260 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:13 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:13.261 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:13 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:13.261 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:13 smithi110 ceph-mon[32848]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:13.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.408+0000 7ff69a7fc700 1 -- 172.21.15.110:0/4162676514 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff6a4062670 con 0x7ff6900a2c20 2024-04-25T01:51:13.407 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 10 on host 'smithi110' 2024-04-25T01:51:13.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6900a2c20 msgr2=0x7ff6900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:13.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6900a2c20 0x7ff6900a50e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6a406a780 tx=0x7ff6a000a3b0 comp rx=0 tx=0).stop 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 msgr2=0x7ff6a419f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 0x7ff6a419f9c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff69c007f00 tx=0x7ff69c00be60 comp rx=0 tx=0).stop 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 shutdown_connections 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6900a2c20 0x7ff6900a50e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6a4103f20 0x7ff6a419f9c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6a41005a0 0x7ff6a4198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 --2- 172.21.15.110:0/4162676514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a4103580 0x7ff6a4198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 >> 172.21.15.110:0/4162676514 conn(0x7ff6a40fbf00 msgr2=0x7ff6a41067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 shutdown_connections 2024-04-25T01:51:13.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-04-25T01:51:13.410+0000 7ff6ab26e700 1 -- 172.21.15.110:0/4162676514 wait complete. 2024-04-25T01:51:13.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:13 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:13.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:13 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:13.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:13 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:13.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:13 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:13.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:13 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:13.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:13 smithi102 ceph-mon[27796]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:13.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:13 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:13.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:13 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:13.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:13 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:13.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:13 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:13.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:13 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:13.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:13 smithi193 ceph-mon[32981]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:14.060 DEBUG:teuthology.orchestra.run.smithi110:osd.10> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T01:51:14.063 INFO:tasks.cephadm:Deploying osd.11 on smithi193 with /dev/vg_nvme/lv_1... 2024-04-25T01:51:14.063 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T01:51:14.314 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:51:14.351 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:14 smithi110 ceph-mon[32848]: from='osd.10 [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:14.352 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:14 smithi110 ceph-mon[32848]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:14.352 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:14.352 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:14.352 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:14.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:14 smithi102 ceph-mon[27796]: from='osd.10 [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:14.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:14 smithi102 ceph-mon[27796]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:14.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:14.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:14.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:14.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:14 smithi193 ceph-mon[32981]: from='osd.10 [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:14.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:14 smithi193 ceph-mon[32981]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T01:51:14.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:14.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:14.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:15.217 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:15 smithi110 ceph-mon[32848]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:51:15.217 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:15 smithi110 ceph-mon[32848]: from='osd.10 [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:51:15.217 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:15 smithi110 ceph-mon[32848]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T01:51:15.217 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:15.218 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:15 smithi110 ceph-mon[32848]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:51:15.218 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:15 smithi110 ceph-mon[32848]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:15.414 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:15 smithi193 ceph-mon[32981]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:51:15.415 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:15 smithi193 ceph-mon[32981]: from='osd.10 [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:51:15.415 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:15 smithi193 ceph-mon[32981]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T01:51:15.415 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:15.415 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:15 smithi193 ceph-mon[32981]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:51:15.415 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:15 smithi193 ceph-mon[32981]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:15.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:15 smithi102 ceph-mon[27796]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T01:51:15.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:15 smithi102 ceph-mon[27796]: from='osd.10 [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:51:15.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:15 smithi102 ceph-mon[27796]: osdmap e72: 11 total, 10 up, 11 in 2024-04-25T01:51:15.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:15.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:15 smithi102 ceph-mon[27796]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-04-25T01:51:15.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:15 smithi102 ceph-mon[27796]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 293 MiB used, 894 GiB / 894 GiB avail 2024-04-25T01:51:16.141 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 01:51:15 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T01:51:15.845+0000 7fdfe0524700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T01:51:16.141 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 01:51:15 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T01:51:15.859+0000 7fdfd8b10700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:51:16.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:16 smithi102 ceph-mon[27796]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:51:16.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:16 smithi102 ceph-mon[27796]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T01:51:16.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:16 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:16.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:16 smithi102 ceph-mon[27796]: from='osd.10 ' entity='osd.10' 2024-04-25T01:51:16.279 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:16 smithi193 ceph-mon[32981]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:51:16.279 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:16 smithi193 ceph-mon[32981]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T01:51:16.279 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:16 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:16.279 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:16 smithi193 ceph-mon[32981]: from='osd.10 ' entity='osd.10' 2024-04-25T01:51:16.280 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T01:51:16.280 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T01:51:16.280 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-04-25T01:51:16.280 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-04-25T01:51:16.280 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129783 s, 808 MB/s 2024-04-25T01:51:16.280 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-04-25T01:51:16.399 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:16 smithi110 ceph-mon[32848]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-04-25T01:51:16.399 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:16 smithi110 ceph-mon[32848]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T01:51:16.399 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:16 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:16.399 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:16 smithi110 ceph-mon[32848]: from='osd.10 ' entity='osd.10' 2024-04-25T01:51:16.906 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: osd.10 [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] boot 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 615 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.151 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.151 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.151 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.151 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.151 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:17.151 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:17.151 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.153 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: osd.10 [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] boot 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 615 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:17.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:17.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:51:17.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: osd.10 [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] boot 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: osdmap e74: 11 total, 11 up, 11 in 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 615 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: osdmap e75: 11 total, 11 up, 11 in 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:17.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:17.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:18.278 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:18 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi110 2024-04-25T01:51:18.278 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:18 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi110 to 4272M 2024-04-25T01:51:18.278 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:18 smithi193 ceph-mon[32981]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:51:18.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:18 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi110 2024-04-25T01:51:18.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:18 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi110 to 4272M 2024-04-25T01:51:18.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:18 smithi102 ceph-mon[27796]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:51:18.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:18 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi110 2024-04-25T01:51:18.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:18 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi110 to 4272M 2024-04-25T01:51:18.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:18 smithi110 ceph-mon[32848]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T01:51:18.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.810+0000 7fd160805700 1 -- 172.21.15.193:0/2547701951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102380 msgr2=0x7fd15c1027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:18.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.810+0000 7fd160805700 1 --2- 172.21.15.193:0/2547701951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102380 0x7fd15c1027e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009990 tx=0x7fd14c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:51:18.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.810+0000 7fd160805700 1 -- 172.21.15.193:0/2547701951 shutdown_connections 2024-04-25T01:51:18.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.810+0000 7fd160805700 1 --2- 172.21.15.193:0/2547701951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd15c102d20 0x7fd15c1095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:18.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.810+0000 7fd160805700 1 --2- 172.21.15.193:0/2547701951 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15c101180 0x7fd15c1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:18.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.810+0000 7fd160805700 1 --2- 172.21.15.193:0/2547701951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102380 0x7fd15c1027e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:18.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.810+0000 7fd160805700 1 -- 172.21.15.193:0/2547701951 >> 172.21.15.193:0/2547701951 conn(0x7fd15c0fc720 msgr2=0x7fd15c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:18.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.810+0000 7fd160805700 1 -- 172.21.15.193:0/2547701951 shutdown_connections 2024-04-25T01:51:18.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.811+0000 7fd160805700 1 -- 172.21.15.193:0/2547701951 wait complete. 2024-04-25T01:51:18.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.811+0000 7fd160805700 1 Processor -- start 2024-04-25T01:51:18.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.811+0000 7fd160805700 1 -- start start 2024-04-25T01:51:18.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.811+0000 7fd160805700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd15c101180 0x7fd15c198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:18.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.811+0000 7fd160805700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15c102380 0x7fd15c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:18.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.811+0000 7fd160805700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102d20 0x7fd15c19fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:18.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.811+0000 7fd160805700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c10ce00 con 0x7fd15c102380 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.811+0000 7fd160805700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c10cc80 con 0x7fd15c101180 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.811+0000 7fd160805700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c10cf80 con 0x7fd15c102d20 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15ad9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102d20 0x7fd15c19fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15ad9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102d20 0x7fd15c19fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:39614/0 (socket says 172.21.15.193:39614) 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15ad9d700 1 -- 172.21.15.193:0/413079557 learned_addr learned my addr 172.21.15.193:0/413079557 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15a59c700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd15c101180 0x7fd15c198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15ad9d700 1 -- 172.21.15.193:0/413079557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd15c101180 msgr2=0x7fd15c198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15ad9d700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd15c101180 0x7fd15c198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15ad9d700 1 -- 172.21.15.193:0/413079557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15c102380 msgr2=0x7fd15c198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15ad9d700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15c102380 0x7fd15c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15ad9d700 1 -- 172.21.15.193:0/413079557 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15c1a01a0 con 0x7fd15c102d20 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd15ad9d700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102d20 0x7fd15c19fb00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd15400add0 tx=0x7fd15400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:18.813 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd14b7fe700 1 -- 172.21.15.193:0/413079557 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd154017020 con 0x7fd15c102d20 2024-04-25T01:51:18.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15c1a0490 con 0x7fd15c102d20 2024-04-25T01:51:18.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd14b7fe700 1 -- 172.21.15.193:0/413079557 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd15400ce20 con 0x7fd15c102d20 2024-04-25T01:51:18.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd14b7fe700 1 -- 172.21.15.193:0/413079557 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd154005520 con 0x7fd15c102d20 2024-04-25T01:51:18.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.812+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd15c1a0a50 con 0x7fd15c102d20 2024-04-25T01:51:18.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.813+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd15c068f20 con 0x7fd15c102d20 2024-04-25T01:51:18.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.814+0000 7fd14b7fe700 1 -- 172.21.15.193:0/413079557 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd154004ad0 con 0x7fd15c102d20 2024-04-25T01:51:18.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.815+0000 7fd14b7fe700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1440a2ca0 0x7fd1440a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:18.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.815+0000 7fd14b7fe700 1 -- 172.21.15.193:0/413079557 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6723+0+0 (secure 0 0 0) 0x7fd1540ce1b0 con 0x7fd15c102d20 2024-04-25T01:51:18.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.815+0000 7fd15a59c700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1440a2ca0 0x7fd1440a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:18.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.816+0000 7fd15a59c700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1440a2ca0 0x7fd1440a5160 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd150006fd0 tx=0x7fd150008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:18.818 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:18.819+0000 7fd14b7fe700 1 -- 172.21.15.193:0/413079557 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd154021020 con 0x7fd15c102d20 2024-04-25T01:51:19.022 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:19.023+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd15c06b0d0 con 0x7fd1440a2ca0 2024-04-25T01:51:19.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:19 smithi102 ceph-mon[27796]: pgmap v264: 1 pgs: 1 remapped+peering; 577 KiB data, 616 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:19.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:19 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:51:19.421 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:19 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:51:19.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:19 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:19.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:19 smithi110 ceph-mon[32848]: pgmap v264: 1 pgs: 1 remapped+peering; 577 KiB data, 616 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:19.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:19 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:51:19.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:19 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:51:19.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:19 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:19.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:19 smithi193 ceph-mon[32981]: pgmap v264: 1 pgs: 1 remapped+peering; 577 KiB data, 616 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:19.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:19 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T01:51:19.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:19 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T01:51:19.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:19 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:20.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:20 smithi193 ceph-mon[32981]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:20.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:20 smithi193 ceph-mon[32981]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T01:51:20.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:20 smithi110 ceph-mon[32848]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:20.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:20 smithi110 ceph-mon[32848]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T01:51:20.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:20 smithi102 ceph-mon[27796]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:20.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:20 smithi102 ceph-mon[27796]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T01:51:21.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:21 smithi110 ceph-mon[32848]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T01:51:21.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:21 smithi110 ceph-mon[32848]: pgmap v267: 1 pgs: 1 remapped+peering; 577 KiB data, 616 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:21.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:21 smithi193 ceph-mon[32981]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T01:51:21.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:21 smithi193 ceph-mon[32981]: pgmap v267: 1 pgs: 1 remapped+peering; 577 KiB data, 616 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:21.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:21 smithi102 ceph-mon[27796]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T01:51:21.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:21 smithi102 ceph-mon[27796]: pgmap v267: 1 pgs: 1 remapped+peering; 577 KiB data, 616 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:22.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:22 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/3568055369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e8f1ff1-80f8-459e-8fea-733a1b7e54ac"}]: dispatch 2024-04-25T01:51:22.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:22 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e8f1ff1-80f8-459e-8fea-733a1b7e54ac"}]: dispatch 2024-04-25T01:51:22.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:22 smithi110 ceph-mon[32848]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e8f1ff1-80f8-459e-8fea-733a1b7e54ac"}]': finished 2024-04-25T01:51:22.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:22 smithi110 ceph-mon[32848]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:51:22.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:22 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:22.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:22 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/3568055369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e8f1ff1-80f8-459e-8fea-733a1b7e54ac"}]: dispatch 2024-04-25T01:51:22.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:22 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e8f1ff1-80f8-459e-8fea-733a1b7e54ac"}]: dispatch 2024-04-25T01:51:22.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:22 smithi193 ceph-mon[32981]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e8f1ff1-80f8-459e-8fea-733a1b7e54ac"}]': finished 2024-04-25T01:51:22.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:22 smithi193 ceph-mon[32981]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:51:22.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:22 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:22.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:22 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/3568055369' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e8f1ff1-80f8-459e-8fea-733a1b7e54ac"}]: dispatch 2024-04-25T01:51:22.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:22 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e8f1ff1-80f8-459e-8fea-733a1b7e54ac"}]: dispatch 2024-04-25T01:51:22.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:22 smithi102 ceph-mon[27796]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e8f1ff1-80f8-459e-8fea-733a1b7e54ac"}]': finished 2024-04-25T01:51:22.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:22 smithi102 ceph-mon[27796]: osdmap e79: 12 total, 11 up, 12 in 2024-04-25T01:51:22.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:22 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:23.485 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:23 smithi110 ceph-mon[32848]: from='client.? 172.21.15.193:0/3007464949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:51:23.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:23 smithi110 ceph-mon[32848]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-25T01:51:23.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:23 smithi193 ceph-mon[32981]: from='client.? 172.21.15.193:0/3007464949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:51:23.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:23 smithi193 ceph-mon[32981]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-25T01:51:23.671 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:23 smithi102 ceph-mon[27796]: from='client.? 172.21.15.193:0/3007464949' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T01:51:23.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:23 smithi102 ceph-mon[27796]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-25T01:51:25.937 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:25 smithi193 ceph-mon[32981]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:51:26.003 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:25 smithi102 ceph-mon[27796]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:51:26.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:25 smithi110 ceph-mon[32848]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T01:51:28.056 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:27 smithi193 ceph-mon[32981]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T01:51:28.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:27 smithi102 ceph-mon[27796]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T01:51:28.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:27 smithi110 ceph-mon[32848]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T01:51:30.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:29 smithi193 ceph-mon[32981]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T01:51:30.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:51:30.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:29 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:30.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:29 smithi102 ceph-mon[27796]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T01:51:30.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:51:30.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:29 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:30.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:29 smithi110 ceph-mon[32848]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T01:51:30.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T01:51:30.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:29 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:30.977 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:30 smithi193 ceph-mon[32981]: Deploying daemon osd.11 on smithi193 2024-04-25T01:51:31.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:30 smithi102 ceph-mon[27796]: Deploying daemon osd.11 on smithi193 2024-04-25T01:51:31.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:30 smithi110 ceph-mon[32848]: Deploying daemon osd.11 on smithi193 2024-04-25T01:51:31.957 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:31 smithi193 ceph-mon[32981]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:51:32.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:31 smithi102 ceph-mon[27796]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:51:32.235 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:31 smithi110 ceph-mon[32848]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T01:51:34.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:33 smithi193 ceph-mon[32981]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-25T01:51:34.038 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:33 smithi110 ceph-mon[32848]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-25T01:51:34.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:33 smithi102 ceph-mon[27796]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-25T01:51:36.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:35 smithi193 ceph-mon[32981]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:36.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:36.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:36.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:36.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:35 smithi102 ceph-mon[27796]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:36.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:36.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:36.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:36.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:35 smithi110 ceph-mon[32848]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:36.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:36.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:36.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:38.100 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:37 smithi193 ceph-mon[32981]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:38.100 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:38.100 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:38.100 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:38.100 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:38.100 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:38.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:37 smithi102 ceph-mon[27796]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:38.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:38.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:38.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:38.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:38.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:38.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:37 smithi110 ceph-mon[32848]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:38.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:38.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:38.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:38.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:38.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:38.460 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.461+0000 7fd14b7fe700 1 -- 172.21.15.193:0/413079557 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fd15c06b0d0 con 0x7fd1440a2ca0 2024-04-25T01:51:38.460 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 11 on host 'smithi193' 2024-04-25T01:51:38.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1440a2ca0 msgr2=0x7fd1440a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:38.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1440a2ca0 0x7fd1440a5160 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd150006fd0 tx=0x7fd150008040 comp rx=0 tx=0).stop 2024-04-25T01:51:38.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102d20 msgr2=0x7fd15c19fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:38.462 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102d20 0x7fd15c19fb00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd15400add0 tx=0x7fd15400c330 comp rx=0 tx=0).stop 2024-04-25T01:51:38.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 shutdown_connections 2024-04-25T01:51:38.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1440a2ca0 0x7fd1440a5160 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd15c101180 0x7fd15c198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15c102d20 0x7fd15c19fb00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 --2- 172.21.15.193:0/413079557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd15c102380 0x7fd15c198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:38.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 >> 172.21.15.193:0/413079557 conn(0x7fd15c0fc720 msgr2=0x7fd15c06a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:38.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 shutdown_connections 2024-04-25T01:51:38.463 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:38.463+0000 7fd160805700 1 -- 172.21.15.193:0/413079557 wait complete. 2024-04-25T01:51:39.173 DEBUG:teuthology.orchestra.run.smithi193:osd.11> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T01:51:39.175 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T01:51:39.176 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T01:51:39.623 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:51:39.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:39.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:39.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:39.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:39 smithi102 ceph-mon[27796]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:39.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:39 smithi102 ceph-mon[27796]: from='osd.11 [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:39.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:39 smithi102 ceph-mon[27796]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:39.699 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:39.699 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:39.699 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:39.699 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:39 smithi193 ceph-mon[32981]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:39.699 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:39 smithi193 ceph-mon[32981]: from='osd.11 [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:39.699 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:39 smithi193 ceph-mon[32981]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:39.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:39.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:39.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:39.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:39 smithi110 ceph-mon[32848]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:39.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:39 smithi110 ceph-mon[32848]: from='osd.11 [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:39.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:39 smithi110 ceph-mon[32848]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T01:51:40.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:40 smithi110 ceph-mon[32848]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:51:40.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:40 smithi110 ceph-mon[32848]: from='osd.11 [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:51:40.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:40 smithi110 ceph-mon[32848]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T01:51:40.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:40 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:40.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:40 smithi110 ceph-mon[32848]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:51:40.747 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:40 smithi193 ceph-mon[32981]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:51:40.747 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:40 smithi193 ceph-mon[32981]: from='osd.11 [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:51:40.747 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:40 smithi193 ceph-mon[32981]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T01:51:40.747 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:40 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:40.747 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:40 smithi193 ceph-mon[32981]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:51:40.785 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:40 smithi102 ceph-mon[27796]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T01:51:40.786 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:40 smithi102 ceph-mon[27796]: from='osd.11 [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:51:40.786 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:40 smithi102 ceph-mon[27796]: osdmap e80: 12 total, 11 up, 12 in 2024-04-25T01:51:40.786 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:40 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:40.786 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:40 smithi102 ceph-mon[27796]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-04-25T01:51:41.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:41 smithi193 ceph-mon[32981]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:51:41.503 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:41 smithi193 ceph-mon[32981]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T01:51:41.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:41 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:41.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:41 smithi193 ceph-mon[32981]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:41.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:41 smithi193 ceph-mon[32981]: from='osd.11 ' entity='osd.11' 2024-04-25T01:51:41.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:41 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:41.504 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 01:51:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T01:51:41.251+0000 7f3c06125700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T01:51:41.504 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 01:51:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T01:51:41.275+0000 7f3bfcf0e700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T01:51:41.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:41 smithi102 ceph-mon[27796]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:51:41.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:41 smithi102 ceph-mon[27796]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T01:51:41.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:41 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:41.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:41 smithi102 ceph-mon[27796]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:41.639 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:41 smithi102 ceph-mon[27796]: from='osd.11 ' entity='osd.11' 2024-04-25T01:51:41.640 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:41 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:41.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:41 smithi110 ceph-mon[32848]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-04-25T01:51:41.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:41 smithi110 ceph-mon[32848]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T01:51:41.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:41 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:41.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:41 smithi110 ceph-mon[32848]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-25T01:51:41.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:41 smithi110 ceph-mon[32848]: from='osd.11 ' entity='osd.11' 2024-04-25T01:51:41.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:41 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.818+0000 7f17e1a00700 1 -- 172.21.15.102:0/4116479507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1044d0 msgr2=0x7f17dc1048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.818+0000 7f17e1a00700 1 --2- 172.21.15.102:0/4116479507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1044d0 0x7f17dc1048f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009920 tx=0x7f17cc009c30 comp rx=0 tx=0).stop 2024-04-25T01:51:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.818+0000 7f17e1a00700 1 -- 172.21.15.102:0/4116479507 shutdown_connections 2024-04-25T01:51:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.818+0000 7f17e1a00700 1 --2- 172.21.15.102:0/4116479507 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1044d0 0x7f17dc1048f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.818+0000 7f17e1a00700 1 --2- 172.21.15.102:0/4116479507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17dc106070 0x7f17dc10c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.818+0000 7f17e1a00700 1 --2- 172.21.15.102:0/4116479507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17dc1056d0 0x7f17dc105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.818+0000 7f17e1a00700 1 -- 172.21.15.102:0/4116479507 >> 172.21.15.102:0/4116479507 conn(0x7f17dc0ffa90 msgr2=0x7f17dc101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.818+0000 7f17e1a00700 1 -- 172.21.15.102:0/4116479507 shutdown_connections 2024-04-25T01:51:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.818+0000 7f17e1a00700 1 -- 172.21.15.102:0/4116479507 wait complete. 2024-04-25T01:51:41.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.819+0000 7f17e1a00700 1 Processor -- start 2024-04-25T01:51:41.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.819+0000 7f17e1a00700 1 -- start start 2024-04-25T01:51:41.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17e1a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17dc1044d0 0x7f17dc19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:41.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17e1a00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1056d0 0x7f17dc19acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17e1a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17dc106070 0x7f17dc1a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17e1a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc110100 con 0x7f17dc1044d0 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17e1a00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10ff80 con 0x7f17dc1056d0 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17e1a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc110280 con 0x7f17dc106070 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17da000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1056d0 0x7f17dc19acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17daa00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17dc1044d0 0x7f17dc19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17da000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1056d0 0x7f17dc19acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:49622/0 (socket says 172.21.15.102:49622) 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17da000700 1 -- 172.21.15.102:0/324579182 learned_addr learned my addr 172.21.15.102:0/324579182 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.820+0000 7f17db400700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17dc106070 0x7f17dc1a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17daa00700 1 -- 172.21.15.102:0/324579182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17dc106070 msgr2=0x7f17dc1a1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17daa00700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17dc106070 0x7f17dc1a1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17daa00700 1 -- 172.21.15.102:0/324579182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1056d0 msgr2=0x7f17dc19acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:41.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17daa00700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1056d0 0x7f17dc19acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:41.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17daa00700 1 -- 172.21.15.102:0/324579182 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17dc1a23e0 con 0x7f17dc1044d0 2024-04-25T01:51:41.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17db400700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17dc106070 0x7f17dc1a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:51:41.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17da000700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1056d0 0x7f17dc19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:51:41.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17daa00700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17dc1044d0 0x7f17dc19a780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009920 tx=0x7f17cc005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:41.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17d3400700 1 -- 172.21.15.102:0/324579182 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc027070 con 0x7f17dc1044d0 2024-04-25T01:51:41.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17dc1a2670 con 0x7f17dc1044d0 2024-04-25T01:51:41.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17d3400700 1 -- 172.21.15.102:0/324579182 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17cc00f930 con 0x7f17dc1044d0 2024-04-25T01:51:41.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17d3400700 1 -- 172.21.15.102:0/324579182 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc004a30 con 0x7f17dc1044d0 2024-04-25T01:51:41.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.821+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17dc1a2ac0 con 0x7f17dc1044d0 2024-04-25T01:51:41.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.823+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17dc068f20 con 0x7f17dc1044d0 2024-04-25T01:51:41.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.825+0000 7f17d3400700 1 -- 172.21.15.102:0/324579182 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f17cc02a070 con 0x7f17dc1044d0 2024-04-25T01:51:41.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.826+0000 7f17d3400700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17c00a2ca0 0x7f17c00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:41.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.826+0000 7f17d3400700 1 -- 172.21.15.102:0/324579182 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 6847+0+0 (secure 0 0 0) 0x7f17cc0cf890 con 0x7f17dc1044d0 2024-04-25T01:51:41.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.827+0000 7f17da000700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17c00a2ca0 0x7f17c00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:41.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.827+0000 7f17da000700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17c00a2ca0 0x7f17c00a5160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f17c8011550 tx=0x7f17c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:41.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:41.829+0000 7f17d3400700 1 -- 172.21.15.102:0/324579182 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f17cc031020 con 0x7f17dc1044d0 2024-04-25T01:51:42.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.055+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f17dc062e40 con 0x7f17dc1044d0 2024-04-25T01:51:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.056+0000 7f17d3400700 1 -- 172.21.15.102:0/324579182 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f17cc013030 con 0x7f17dc1044d0 2024-04-25T01:51:42.055 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:51:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17c00a2ca0 msgr2=0x7f17c00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17c00a2ca0 0x7f17c00a5160 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f17c8011550 tx=0x7f17c8009260 comp rx=0 tx=0).stop 2024-04-25T01:51:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17dc1044d0 msgr2=0x7f17dc19a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17dc1044d0 0x7f17dc19a780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009920 tx=0x7f17cc005250 comp rx=0 tx=0).stop 2024-04-25T01:51:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 shutdown_connections 2024-04-25T01:51:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17c00a2ca0 0x7f17c00a5160 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17dc106070 0x7f17dc1a1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17dc1044d0 0x7f17dc19a780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 --2- 172.21.15.102:0/324579182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17dc1056d0 0x7f17dc19acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 >> 172.21.15.102:0/324579182 conn(0x7f17dc0ffa90 msgr2=0x7f17dc108a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 shutdown_connections 2024-04-25T01:51:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:42.059+0000 7f17e1a00700 1 -- 172.21.15.102:0/324579182 wait complete. 2024-04-25T01:51:42.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: purged_snaps scrub starts 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: purged_snaps scrub ok 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/324579182' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:42.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:42.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: osd.11 [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] boot 2024-04-25T01:51:42.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:51:42.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: purged_snaps scrub starts 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: purged_snaps scrub ok 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/324579182' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:42.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:42.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: osd.11 [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] boot 2024-04-25T01:51:42.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:51:42.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:42.896 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"num_osds":12,"num_up_osds":11,"osd_up_since":1714009876,"num_in_osds":12,"osd_in_since":1714009881,"num_remapped_pgs":0} 2024-04-25T01:51:42.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: purged_snaps scrub starts 2024-04-25T01:51:42.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: purged_snaps scrub ok 2024-04-25T01:51:42.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/324579182' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:51:42.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: osd.11 [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] boot 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T01:51:42.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:51:43.757 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:43 smithi102 ceph-mon[27796]: Detected new or changed devices on smithi193 2024-04-25T01:51:43.757 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:43 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi193 to 4272M 2024-04-25T01:51:43.757 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:43 smithi102 ceph-mon[27796]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:43.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T01:51:43.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:43 smithi110 ceph-mon[32848]: Detected new or changed devices on smithi193 2024-04-25T01:51:43.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:43 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi193 to 4272M 2024-04-25T01:51:43.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:43 smithi110 ceph-mon[32848]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:44.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:43 smithi193 ceph-mon[32981]: Detected new or changed devices on smithi193 2024-04-25T01:51:44.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:43 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi193 to 4272M 2024-04-25T01:51:44.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:43 smithi193 ceph-mon[32981]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:44.339 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:51:44.843 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:44 smithi102 ceph-mon[27796]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:51:44.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:44 smithi110 ceph-mon[32848]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:51:45.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:44 smithi193 ceph-mon[32981]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T01:51:45.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:45 smithi110 ceph-mon[32848]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:45.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:45 smithi102 ceph-mon[27796]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:46.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:45 smithi193 ceph-mon[32981]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:46.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.747+0000 7f0d79800700 1 -- 172.21.15.102:0/2885908349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d74100b70 msgr2=0x7f0d74100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.747+0000 7f0d79800700 1 --2- 172.21.15.102:0/2885908349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d74100b70 0x7f0d74100f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009990 tx=0x7f0d64009ca0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.748+0000 7f0d79800700 1 -- 172.21.15.102:0/2885908349 shutdown_connections 2024-04-25T01:51:46.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.748+0000 7f0d79800700 1 --2- 172.21.15.102:0/2885908349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d74100b70 0x7f0d74100f90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.748+0000 7f0d79800700 1 --2- 172.21.15.102:0/2885908349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d74101e90 0x7f0d74102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.748+0000 7f0d79800700 1 --2- 172.21.15.102:0/2885908349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d741014d0 0x7f0d74101950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.748+0000 7f0d79800700 1 -- 172.21.15.102:0/2885908349 >> 172.21.15.102:0/2885908349 conn(0x7f0d740fc730 msgr2=0x7f0d740feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:46.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.748+0000 7f0d79800700 1 -- 172.21.15.102:0/2885908349 shutdown_connections 2024-04-25T01:51:46.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.748+0000 7f0d79800700 1 -- 172.21.15.102:0/2885908349 wait complete. 2024-04-25T01:51:46.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.750+0000 7f0d79800700 1 Processor -- start 2024-04-25T01:51:46.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.750+0000 7f0d79800700 1 -- start start 2024-04-25T01:51:46.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.750+0000 7f0d79800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d74100b70 0x7f0d7419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:46.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.750+0000 7f0d79800700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d741014d0 0x7f0d7419ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:46.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.750+0000 7f0d79800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d74101e90 0x7f0d741a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:46.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d79800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d74110040 con 0x7f0d74100b70 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d79800700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7410fec0 con 0x7f0d741014d0 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d79800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d741101c0 con 0x7f0d74101e90 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d74100b70 0x7f0d7419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d741014d0 0x7f0d7419ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d74100b70 0x7f0d7419a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35030/0 (socket says 172.21.15.102:35030) 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72a00700 1 -- 172.21.15.102:0/677376849 learned_addr learned my addr 172.21.15.102:0/677376849 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d73400700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d74101e90 0x7f0d741a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72000700 1 -- 172.21.15.102:0/677376849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d74101e90 msgr2=0x7f0d741a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72000700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d74101e90 0x7f0d741a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72000700 1 -- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d74100b70 msgr2=0x7f0d7419a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72000700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d74100b70 0x7f0d7419a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72000700 1 -- 172.21.15.102:0/677376849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d741a2360 con 0x7f0d741014d0 2024-04-25T01:51:46.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.751+0000 7f0d72a00700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d74100b70 0x7f0d7419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:51:46.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.752+0000 7f0d72000700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d741014d0 0x7f0d7419ac40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d6000e9f0 tx=0x7f0d6000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:46.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.752+0000 7f0d6b400700 1 -- 172.21.15.102:0/677376849 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6000c9c0 con 0x7f0d741014d0 2024-04-25T01:51:46.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.752+0000 7f0d6b400700 1 -- 172.21.15.102:0/677376849 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d6000cb60 con 0x7f0d741014d0 2024-04-25T01:51:46.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.752+0000 7f0d6b400700 1 -- 172.21.15.102:0/677376849 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d60018680 con 0x7f0d741014d0 2024-04-25T01:51:46.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.752+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d741a2650 con 0x7f0d741014d0 2024-04-25T01:51:46.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.752+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d741a2be0 con 0x7f0d741014d0 2024-04-25T01:51:46.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.753+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d74068f20 con 0x7f0d741014d0 2024-04-25T01:51:46.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.756+0000 7f0d6b400700 1 -- 172.21.15.102:0/677376849 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0d60010430 con 0x7f0d741014d0 2024-04-25T01:51:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.757+0000 7f0d6b400700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d580a2ca0 0x7f0d580a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.757+0000 7f0d6b400700 1 -- 172.21.15.102:0/677376849 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f0d600cf120 con 0x7f0d741014d0 2024-04-25T01:51:46.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.758+0000 7f0d72a00700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d580a2ca0 0x7f0d580a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:46.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.758+0000 7f0d72a00700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d580a2ca0 0x7f0d580a5160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400b7b0 tx=0x7f0d6400b8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:46.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.759+0000 7f0d6b400700 1 -- 172.21.15.102:0/677376849 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0d6009c480 con 0x7f0d741014d0 2024-04-25T01:51:46.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.981+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f0d7402dbd0 con 0x7f0d741014d0 2024-04-25T01:51:46.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.982+0000 7f0d6b400700 1 -- 172.21.15.102:0/677376849 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) ==== 74+0+133 (secure 0 0 0) 0x7f0d7402dbd0 con 0x7f0d741014d0 2024-04-25T01:51:46.982 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.984+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d580a2ca0 msgr2=0x7f0d580a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.984+0000 7f0d79800700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d580a2ca0 0x7f0d580a5160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400b7b0 tx=0x7f0d6400b8a0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.984+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d741014d0 msgr2=0x7f0d7419ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.984+0000 7f0d79800700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d741014d0 0x7f0d7419ac40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0d6000e9f0 tx=0x7f0d6000ed00 comp rx=0 tx=0).stop 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.985+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 shutdown_connections 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.985+0000 7f0d79800700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d580a2ca0 0x7f0d580a5160 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.985+0000 7f0d79800700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d74101e90 0x7f0d741a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.985+0000 7f0d79800700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d74100b70 0x7f0d7419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.985+0000 7f0d79800700 1 --2- 172.21.15.102:0/677376849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d741014d0 0x7f0d7419ac40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.985+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 >> 172.21.15.102:0/677376849 conn(0x7f0d740fc730 msgr2=0x7f0d740fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.985+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 shutdown_connections 2024-04-25T01:51:46.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:46.985+0000 7f0d79800700 1 -- 172.21.15.102:0/677376849 wait complete. 2024-04-25T01:51:47.759 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1714009902,"num_in_osds":12,"osd_in_since":1714009881,"num_remapped_pgs":0} 2024-04-25T01:51:47.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:51:48.171 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:47 smithi102 ceph-mon[27796]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:48.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:47 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/677376849' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:51:48.215 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:51:48.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:47 smithi110 ceph-mon[32848]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:48.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:47 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/677376849' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:51:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:47 smithi193 ceph-mon[32981]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:48.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:47 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/677376849' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T01:51:50.025 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:49 smithi102 ceph-mon[27796]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:50.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:49 smithi110 ceph-mon[32848]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:50.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:49 smithi193 ceph-mon[32981]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:50.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.279+0000 7ff747200700 1 -- 172.21.15.102:0/2369895759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff748113170 msgr2=0x7ff748115560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.279+0000 7ff747200700 1 --2- 172.21.15.102:0/2369895759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff748113170 0x7ff748115560 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff74000b0a0 tx=0x7ff74000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.280+0000 7ff747200700 1 -- 172.21.15.102:0/2369895759 shutdown_connections 2024-04-25T01:51:50.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.280+0000 7ff747200700 1 --2- 172.21.15.102:0/2369895759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7480745d0 0x7ff7480749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.280+0000 7ff747200700 1 --2- 172.21.15.102:0/2369895759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff748113170 0x7ff748115560 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.280+0000 7ff747200700 1 --2- 172.21.15.102:0/2369895759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff748074ef0 0x7ff748112c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.280+0000 7ff747200700 1 -- 172.21.15.102:0/2369895759 >> 172.21.15.102:0/2369895759 conn(0x7ff7480fdcb0 msgr2=0x7ff7481000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:50.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.280+0000 7ff747200700 1 -- 172.21.15.102:0/2369895759 shutdown_connections 2024-04-25T01:51:50.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.280+0000 7ff747200700 1 -- 172.21.15.102:0/2369895759 wait complete. 2024-04-25T01:51:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.281+0000 7ff747200700 1 Processor -- start 2024-04-25T01:51:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.281+0000 7ff747200700 1 -- start start 2024-04-25T01:51:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.281+0000 7ff747200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7480745d0 0x7ff74819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.281+0000 7ff747200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff748074ef0 0x7ff74819f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.281+0000 7ff747200700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff748113170 0x7ff7481a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.281+0000 7ff747200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff748118ae0 con 0x7ff7480745d0 2024-04-25T01:51:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.281+0000 7ff747200700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff748118960 con 0x7ff748113170 2024-04-25T01:51:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.281+0000 7ff747200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff748118c60 con 0x7ff748074ef0 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff745e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7480745d0 0x7ff74819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff73fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff748074ef0 0x7ff74819f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff745e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7480745d0 0x7ff74819ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35042/0 (socket says 172.21.15.102:35042) 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff745e00700 1 -- 172.21.15.102:0/507191125 learned_addr learned my addr 172.21.15.102:0/507191125 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff746800700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff748113170 0x7ff7481a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff73fe00700 1 -- 172.21.15.102:0/507191125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff748113170 msgr2=0x7ff7481a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff73fe00700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff748113170 0x7ff7481a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff73fe00700 1 -- 172.21.15.102:0/507191125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7480745d0 msgr2=0x7ff74819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff73fe00700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7480745d0 0x7ff74819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff73fe00700 1 -- 172.21.15.102:0/507191125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7481a3f20 con 0x7ff748074ef0 2024-04-25T01:51:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.282+0000 7ff73fe00700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff748074ef0 0x7ff74819f450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff73000e9f0 tx=0x7ff73000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:50.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.283+0000 7ff73ea00700 1 -- 172.21.15.102:0/507191125 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730004220 con 0x7ff748074ef0 2024-04-25T01:51:50.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.283+0000 7ff73ea00700 1 -- 172.21.15.102:0/507191125 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff730004d10 con 0x7ff748074ef0 2024-04-25T01:51:50.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.283+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7481a4210 con 0x7ff748074ef0 2024-04-25T01:51:50.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.283+0000 7ff73ea00700 1 -- 172.21.15.102:0/507191125 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff730005020 con 0x7ff748074ef0 2024-04-25T01:51:50.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.283+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7481aba90 con 0x7ff748074ef0 2024-04-25T01:51:50.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.284+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff748068f20 con 0x7ff748074ef0 2024-04-25T01:51:50.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.287+0000 7ff73ea00700 1 -- 172.21.15.102:0/507191125 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff730004750 con 0x7ff748074ef0 2024-04-25T01:51:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.288+0000 7ff73ea00700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff72c0a2c20 0x7ff72c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.288+0000 7ff73ea00700 1 -- 172.21.15.102:0/507191125 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff7300cea80 con 0x7ff748074ef0 2024-04-25T01:51:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.288+0000 7ff745e00700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff72c0a2c20 0x7ff72c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:50.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.289+0000 7ff745e00700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff72c0a2c20 0x7ff72c0a50e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff738009d20 tx=0x7ff738009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:50.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.293+0000 7ff73ea00700 1 -- 172.21.15.102:0/507191125 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff7300d0e20 con 0x7ff748074ef0 2024-04-25T01:51:50.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.517+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7481a0270 con 0x7ff748074ef0 2024-04-25T01:51:50.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.518+0000 7ff73ea00700 1 -- 172.21.15.102:0/507191125 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18490 (secure 0 0 0) 0x7ff73009bde0 con 0x7ff748074ef0 2024-04-25T01:51:50.518 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:51:50.518 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":83,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:51:43.497262+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.521+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff72c0a2c20 msgr2=0x7ff72c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.521+0000 7ff747200700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff72c0a2c20 0x7ff72c0a50e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff738009d20 tx=0x7ff738009260 comp rx=0 tx=0).stop 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.521+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff748074ef0 msgr2=0x7ff74819f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.521+0000 7ff747200700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff748074ef0 0x7ff74819f450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff73000e9f0 tx=0x7ff73000ed00 comp rx=0 tx=0).stop 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.522+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 shutdown_connections 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.522+0000 7ff747200700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff72c0a2c20 0x7ff72c0a50e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.522+0000 7ff747200700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff748113170 0x7ff7481a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.522+0000 7ff747200700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7480745d0 0x7ff74819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.522+0000 7ff747200700 1 --2- 172.21.15.102:0/507191125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff748074ef0 0x7ff74819f450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.522+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 >> 172.21.15.102:0/507191125 conn(0x7ff7480fdcb0 msgr2=0x7ff7480fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.522+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 shutdown_connections 2024-04-25T01:51:50.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:50.522+0000 7ff747200700 1 -- 172.21.15.102:0/507191125 wait complete. 2024-04-25T01:51:51.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:50 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/507191125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:51.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:50 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/507191125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:51.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:50 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/507191125' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:51.334 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:51:51.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T01:51:51.768 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:51:52.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:51 smithi102 ceph-mon[27796]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:52.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:51 smithi110 ceph-mon[32848]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:52.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:51 smithi193 ceph-mon[32981]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.846+0000 7ff703800700 1 -- 172.21.15.102:0/138474815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6fc104450 msgr2=0x7ff6fc1048d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.846+0000 7ff703800700 1 --2- 172.21.15.102:0/138474815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6fc104450 0x7ff6fc1048d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009990 tx=0x7ff6e8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.846+0000 7ff703800700 1 -- 172.21.15.102:0/138474815 shutdown_connections 2024-04-25T01:51:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.846+0000 7ff703800700 1 --2- 172.21.15.102:0/138474815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc104e10 0x7ff6fc111320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.846+0000 7ff703800700 1 --2- 172.21.15.102:0/138474815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc10a4e0 0x7ff6fc10a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.846+0000 7ff703800700 1 --2- 172.21.15.102:0/138474815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6fc104450 0x7ff6fc1048d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.846+0000 7ff703800700 1 -- 172.21.15.102:0/138474815 >> 172.21.15.102:0/138474815 conn(0x7ff6fc0fff40 msgr2=0x7ff6fc102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.846+0000 7ff703800700 1 -- 172.21.15.102:0/138474815 shutdown_connections 2024-04-25T01:51:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.846+0000 7ff703800700 1 -- 172.21.15.102:0/138474815 wait complete. 2024-04-25T01:51:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.847+0000 7ff703800700 1 Processor -- start 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.848+0000 7ff703800700 1 -- start start 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.848+0000 7ff703800700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6fc104450 0x7ff6fc1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.848+0000 7ff703800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc104e10 0x7ff6fc1a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.848+0000 7ff703800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc10a4e0 0x7ff6fc19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.848+0000 7ff703800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc114790 con 0x7ff6fc10a4e0 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.848+0000 7ff703800700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc114610 con 0x7ff6fc104450 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.848+0000 7ff703800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc114910 con 0x7ff6fc104e10 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.848+0000 7ff701800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc10a4e0 0x7ff6fc19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.848+0000 7ff6fbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc104e10 0x7ff6fc1a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff6fbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc104e10 0x7ff6fc1a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46166/0 (socket says 172.21.15.102:46166) 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff6fbe00700 1 -- 172.21.15.102:0/3587029317 learned_addr learned my addr 172.21.15.102:0/3587029317 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:51:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff701800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc10a4e0 0x7ff6fc19aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44224/0 (socket says 172.21.15.102:44224) 2024-04-25T01:51:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff700e00700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6fc104450 0x7ff6fc1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff6fbe00700 1 -- 172.21.15.102:0/3587029317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6fc104450 msgr2=0x7ff6fc1a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff6fbe00700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6fc104450 0x7ff6fc1a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff6fbe00700 1 -- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc10a4e0 msgr2=0x7ff6fc19aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff6fbe00700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc10a4e0 0x7ff6fc19aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff6fbe00700 1 -- 172.21.15.102:0/3587029317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6fc19b1f0 con 0x7ff6fc104e10 2024-04-25T01:51:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff701800700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc10a4e0 0x7ff6fc19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:51:53.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.849+0000 7ff6fbe00700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc104e10 0x7ff6fc1a0dc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8000c00 tx=0x7ff6e800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.850+0000 7ff6f9600700 1 -- 172.21.15.102:0/3587029317 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8027070 con 0x7ff6fc104e10 2024-04-25T01:51:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.850+0000 7ff6f9600700 1 -- 172.21.15.102:0/3587029317 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6e80056d0 con 0x7ff6fc104e10 2024-04-25T01:51:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.850+0000 7ff6f9600700 1 -- 172.21.15.102:0/3587029317 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8022e10 con 0x7ff6fc104e10 2024-04-25T01:51:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.850+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6fc19b480 con 0x7ff6fc104e10 2024-04-25T01:51:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.850+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6fc1a7750 con 0x7ff6fc104e10 2024-04-25T01:51:53.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.851+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6fc068f20 con 0x7ff6fc104e10 2024-04-25T01:51:53.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.853+0000 7ff6f9600700 1 -- 172.21.15.102:0/3587029317 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff6e802a070 con 0x7ff6fc104e10 2024-04-25T01:51:53.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.854+0000 7ff6f9600700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6e40a2ca0 0x7ff6e40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:53.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.854+0000 7ff6f9600700 1 -- 172.21.15.102:0/3587029317 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff6e80cff80 con 0x7ff6fc104e10 2024-04-25T01:51:53.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.855+0000 7ff700e00700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6e40a2ca0 0x7ff6e40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:53.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.855+0000 7ff700e00700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6e40a2ca0 0x7ff6e40a5160 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc102830 tx=0x7ff6f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:53.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:53.857+0000 7ff6f9600700 1 -- 172.21.15.102:0/3587029317 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff6e8013070 con 0x7ff6fc104e10 2024-04-25T01:51:54.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.078+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff6fc02db30 con 0x7ff6fc104e10 2024-04-25T01:51:54.078 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:53 smithi102 ceph-mon[27796]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:54.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.079+0000 7ff6f9600700 1 -- 172.21.15.102:0/3587029317 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7ff6e80a3070 con 0x7ff6fc104e10 2024-04-25T01:51:54.078 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:51:54.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6e40a2ca0 msgr2=0x7ff6e40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:54.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6e40a2ca0 0x7ff6e40a5160 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc102830 tx=0x7ff6f0009260 comp rx=0 tx=0).stop 2024-04-25T01:51:54.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc104e10 msgr2=0x7ff6fc1a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc104e10 0x7ff6fc1a0dc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8000c00 tx=0x7ff6e800bab0 comp rx=0 tx=0).stop 2024-04-25T01:51:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 shutdown_connections 2024-04-25T01:51:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff6e40a2ca0 0x7ff6e40a5160 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6fc104450 0x7ff6fc1a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6fc10a4e0 0x7ff6fc19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 --2- 172.21.15.102:0/3587029317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc104e10 0x7ff6fc1a0dc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 >> 172.21.15.102:0/3587029317 conn(0x7ff6fc0fff40 msgr2=0x7ff6fc100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:54.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 shutdown_connections 2024-04-25T01:51:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:54.082+0000 7ff703800700 1 -- 172.21.15.102:0/3587029317 wait complete. 2024-04-25T01:51:54.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:53 smithi110 ceph-mon[32848]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:54.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:53 smithi193 ceph-mon[32981]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:54.936 INFO:tasks.cephadm:Adding mds.a on smithi102 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.d on smithi102 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.g on smithi102 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.j on smithi102 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.b on smithi110 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.e on smithi110 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.h on smithi110 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.k on smithi110 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.c on smithi193 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.f on smithi193 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.i on smithi193 2024-04-25T01:51:54.937 INFO:tasks.cephadm:Adding mds.l on smithi193 2024-04-25T01:51:54.938 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l' 2024-04-25T01:51:55.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:54 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3587029317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:51:55.187 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T01:51:55.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:54 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3587029317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:51:55.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:54 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3587029317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:51:56.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:55 smithi193 ceph-mon[32981]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:56.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:55 smithi102 ceph-mon[27796]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:56.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:55 smithi110 ceph-mon[32848]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:56.682 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 -- 172.21.15.193:0/1651017733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103490 msgr2=0x7f7d481038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:56.682 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 --2- 172.21.15.193:0/1651017733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103490 0x7f7d481038f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7d38009990 tx=0x7f7d38009ca0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.682 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 -- 172.21.15.193:0/1651017733 shutdown_connections 2024-04-25T01:51:56.682 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 --2- 172.21.15.193:0/1651017733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d48103e30 0x7f7d4810a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.682 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 --2- 172.21.15.193:0/1651017733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48102290 0x7f7d481026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.682 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 --2- 172.21.15.193:0/1651017733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103490 0x7f7d481038f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.682 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 -- 172.21.15.193:0/1651017733 >> 172.21.15.193:0/1651017733 conn(0x7f7d480fd830 msgr2=0x7f7d480ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:56.682 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 -- 172.21.15.193:0/1651017733 shutdown_connections 2024-04-25T01:51:56.682 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 -- 172.21.15.193:0/1651017733 wait complete. 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.680+0000 7f7d507f2700 1 Processor -- start 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d507f2700 1 -- start start 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d507f2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d48102290 0x7f7d48198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d507f2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48103490 0x7f7d48198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d507f2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103e30 0x7f7d4819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d507f2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4810df10 con 0x7f7d48103490 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d507f2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4810dd90 con 0x7f7d48102290 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d507f2700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4810e090 con 0x7f7d48103e30 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4ed8f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103e30 0x7f7d4819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4ed8f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103e30 0x7f7d4819fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:34430/0 (socket says 172.21.15.193:34430) 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4ed8f700 1 -- 172.21.15.193:0/3873934000 learned_addr learned my addr 172.21.15.193:0/3873934000 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4e58e700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d48102290 0x7f7d48198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:56.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4ed8f700 1 -- 172.21.15.193:0/3873934000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d48102290 msgr2=0x7f7d48198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4ed8f700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d48102290 0x7f7d48198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4ed8f700 1 -- 172.21.15.193:0/3873934000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48103490 msgr2=0x7f7d48198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4ed8f700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48103490 0x7f7d48198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4ed8f700 1 -- 172.21.15.193:0/3873934000 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d481a0180 con 0x7f7d48103e30 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4e58e700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d48102290 0x7f7d48198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d4ed8f700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103e30 0x7f7d4819fae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000add0 tx=0x7f7d4000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d3f7fe700 1 -- 172.21.15.193:0/3873934000 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40017020 con 0x7f7d48103e30 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d481a0470 con 0x7f7d48103e30 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d3f7fe700 1 -- 172.21.15.193:0/3873934000 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d4000ce20 con 0x7f7d48103e30 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.681+0000 7f7d3f7fe700 1 -- 172.21.15.193:0/3873934000 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40005520 con 0x7f7d48103e30 2024-04-25T01:51:56.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.682+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d481a0a30 con 0x7f7d48103e30 2024-04-25T01:51:56.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.682+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d48068f20 con 0x7f7d48103e30 2024-04-25T01:51:56.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.683+0000 7f7d3f7fe700 1 -- 172.21.15.193:0/3873934000 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7d40004ad0 con 0x7f7d48103e30 2024-04-25T01:51:56.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.684+0000 7f7d3f7fe700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d340a2c20 0x7f7d340a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:56.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.684+0000 7f7d3f7fe700 1 -- 172.21.15.193:0/3873934000 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f7d400ce3c0 con 0x7f7d48103e30 2024-04-25T01:51:56.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.685+0000 7f7d4e58e700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d340a2c20 0x7f7d340a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:56.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.685+0000 7f7d4e58e700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d340a2c20 0x7f7d340a50e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7d481032f0 tx=0x7f7d44008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:56.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.688+0000 7f7d3f7fe700 1 -- 172.21.15.193:0/3873934000 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7d40021020 con 0x7f7d48103e30 2024-04-25T01:51:56.892 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.892+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}) -- 0x7f7d48062930 con 0x7f7d340a2c20 2024-04-25T01:51:56.908 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.908+0000 7f7d3f7fe700 1 -- 172.21.15.193:0/3873934000 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f7d48062930 con 0x7f7d340a2c20 2024-04-25T01:51:56.909 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mds.all update... 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d340a2c20 msgr2=0x7f7d340a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d340a2c20 0x7f7d340a50e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7d481032f0 tx=0x7f7d44008040 comp rx=0 tx=0).stop 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103e30 msgr2=0x7f7d4819fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103e30 0x7f7d4819fae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000add0 tx=0x7f7d4000c330 comp rx=0 tx=0).stop 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 shutdown_connections 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d340a2c20 0x7f7d340a50e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d48102290 0x7f7d48198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d48103e30 0x7f7d4819fae0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 --2- 172.21.15.193:0/3873934000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48103490 0x7f7d48198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:56.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 >> 172.21.15.193:0/3873934000 conn(0x7f7d480fd830 msgr2=0x7f7d48106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:56.912 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 shutdown_connections 2024-04-25T01:51:56.912 INFO:teuthology.orchestra.run.smithi193.stderr:2024-04-25T01:51:56.911+0000 7f7d507f2700 1 -- 172.21.15.193:0/3873934000 wait complete. 2024-04-25T01:51:57.548 DEBUG:teuthology.orchestra.run.smithi102:mds.a> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T01:51:57.554 DEBUG:teuthology.orchestra.run.smithi102:mds.d> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T01:51:57.599 DEBUG:teuthology.orchestra.run.smithi102:mds.g> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T01:51:57.644 DEBUG:teuthology.orchestra.run.smithi102:mds.j> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T01:51:57.691 DEBUG:teuthology.orchestra.run.smithi110:mds.b> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T01:51:57.694 DEBUG:teuthology.orchestra.run.smithi110:mds.e> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T01:51:57.737 DEBUG:teuthology.orchestra.run.smithi110:mds.h> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T01:51:57.780 DEBUG:teuthology.orchestra.run.smithi110:mds.k> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T01:51:57.823 DEBUG:teuthology.orchestra.run.smithi193:mds.c> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T01:51:57.825 DEBUG:teuthology.orchestra.run.smithi193:mds.f> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T01:51:57.868 DEBUG:teuthology.orchestra.run.smithi193:mds.i> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T01:51:57.912 DEBUG:teuthology.orchestra.run.smithi193:mds.l> sudo journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T01:51:57.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:57 smithi102 ceph-mon[27796]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:57.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:57.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:57 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:57.955 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T01:51:57.956 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:51:57.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:57 smithi110 ceph-mon[32848]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:57.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:57.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:57 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.181+0000 7fa29aa00640 1 -- 172.21.15.102:0/1240142336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa294077ce0 msgr2=0x7fa294078580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.181+0000 7fa29aa00640 1 --2- 172.21.15.102:0/1240142336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa294077ce0 0x7fa294078580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa28c009900 tx=0x7fa28c02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.181+0000 7fa29aa00640 1 -- 172.21.15.102:0/1240142336 shutdown_connections 2024-04-25T01:51:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.181+0000 7fa29aa00640 1 --2- 172.21.15.102:0/1240142336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa294077ce0 0x7fa294078580 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.181+0000 7fa29aa00640 1 --2- 172.21.15.102:0/1240142336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa294077340 0x7fa2940777a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.181+0000 7fa29aa00640 1 --2- 172.21.15.102:0/1240142336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa294076100 0x7fa294076520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.181+0000 7fa29aa00640 1 -- 172.21.15.102:0/1240142336 >> 172.21.15.102:0/1240142336 conn(0x7fa2940fef30 msgr2=0x7fa294101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.182+0000 7fa29aa00640 1 -- 172.21.15.102:0/1240142336 shutdown_connections 2024-04-25T01:51:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.182+0000 7fa29aa00640 1 -- 172.21.15.102:0/1240142336 wait complete. 2024-04-25T01:51:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa29aa00640 1 Processor -- start 2024-04-25T01:51:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa29aa00640 1 -- start start 2024-04-25T01:51:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa29aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa294076100 0x7fa29419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa29aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa294077340 0x7fa29419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa29aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa294077ce0 0x7fa2941a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa29aa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa294113a50 con 0x7fa294077340 2024-04-25T01:51:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa29aa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa2941138d0 con 0x7fa294076100 2024-04-25T01:51:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa29aa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa294113bd0 con 0x7fa294077ce0 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa28b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa294077340 0x7fa29419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa28b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa294077340 0x7fa29419e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44248/0 (socket says 172.21.15.102:44248) 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa28b400640 1 -- 172.21.15.102:0/3876053222 learned_addr learned my addr 172.21.15.102:0/3876053222 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa298a00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa294077ce0 0x7fa2941a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.183+0000 7fa293e00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa294076100 0x7fa29419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa28b400640 1 -- 172.21.15.102:0/3876053222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa294077ce0 msgr2=0x7fa2941a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa28b400640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa294077ce0 0x7fa2941a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa28b400640 1 -- 172.21.15.102:0/3876053222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa294076100 msgr2=0x7fa29419e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa28b400640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa294076100 0x7fa29419e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa28b400640 1 -- 172.21.15.102:0/3876053222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2941a5c40 con 0x7fa294077340 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa28b400640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa294077340 0x7fa29419e560 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00e9f0 tx=0x7fa27c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa291600640 1 -- 172.21.15.102:0/3876053222 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c00cd80 con 0x7fa294077340 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa291600640 1 -- 172.21.15.102:0/3876053222 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa27c010910 con 0x7fa294077340 2024-04-25T01:51:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2941a5f30 con 0x7fa294077340 2024-04-25T01:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.184+0000 7fa291600640 1 -- 172.21.15.102:0/3876053222 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c018990 con 0x7fa294077340 2024-04-25T01:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.185+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2941a63f0 con 0x7fa294077340 2024-04-25T01:51:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.186+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa294110470 con 0x7fa294077340 2024-04-25T01:51:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.188+0000 7fa291600640 1 -- 172.21.15.102:0/3876053222 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa27c010ac0 con 0x7fa294077340 2024-04-25T01:51:58.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.191+0000 7fa291600640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa2780a2b90 0x7fa2780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:58.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.191+0000 7fa291600640 1 -- 172.21.15.102:0/3876053222 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fa27c0cf330 con 0x7fa294077340 2024-04-25T01:51:58.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.191+0000 7fa293e00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa2780a2b90 0x7fa2780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:58.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.192+0000 7fa293e00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa2780a2b90 0x7fa2780a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa284005fd0 tx=0x7fa284009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:58.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.194+0000 7fa291600640 1 -- 172.21.15.102:0/3876053222 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa27c09c040 con 0x7fa294077340 2024-04-25T01:51:58.253 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:57 smithi193 ceph-mon[32981]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:51:58.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:58.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:57 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:51:58.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.393+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa29402d3c0 con 0x7fa294077340 2024-04-25T01:51:58.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.395+0000 7fa291600640 1 -- 172.21.15.102:0/3876053222 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18490 (secure 0 0 0) 0x7fa27c0a0340 con 0x7fa294077340 2024-04-25T01:51:58.394 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:51:58.395 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":83,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:51:43.497262+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:51:58.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa2780a2b90 msgr2=0x7fa2780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa2780a2b90 0x7fa2780a5050 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa284005fd0 tx=0x7fa284009500 comp rx=0 tx=0).stop 2024-04-25T01:51:58.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa294077340 msgr2=0x7fa29419e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa294077340 0x7fa29419e560 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00e9f0 tx=0x7fa27c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 shutdown_connections 2024-04-25T01:51:58.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa2780a2b90 0x7fa2780a5050 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa294077ce0 0x7fa2941a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa294077340 0x7fa29419e560 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 --2- 172.21.15.102:0/3876053222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa294076100 0x7fa29419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 >> 172.21.15.102:0/3876053222 conn(0x7fa2940fef30 msgr2=0x7fa2941005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:58.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 shutdown_connections 2024-04-25T01:51:58.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.398+0000 7fa29aa00640 1 -- 172.21.15.102:0/3876053222 wait complete. 2024-04-25T01:51:58.416 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:51:58.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:51:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.614+0000 7f6203e00640 1 -- 172.21.15.102:0/2761188959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6204109830 msgr2=0x7f62041100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.614+0000 7f6203e00640 1 --2- 172.21.15.102:0/2761188959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6204109830 0x7f62041100c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009ca0 tx=0x7f61fc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.614+0000 7f6203e00640 1 -- 172.21.15.102:0/2761188959 shutdown_connections 2024-04-25T01:51:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.614+0000 7f6203e00640 1 --2- 172.21.15.102:0/2761188959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6204109830 0x7f62041100c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.614+0000 7f6203e00640 1 --2- 172.21.15.102:0/2761188959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6204108e70 0x7f62041092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.614+0000 7f6203e00640 1 --2- 172.21.15.102:0/2761188959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204107c80 0x7f62041080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.614+0000 7f6203e00640 1 -- 172.21.15.102:0/2761188959 >> 172.21.15.102:0/2761188959 conn(0x7f6204076c10 msgr2=0x7f6204079050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.614+0000 7f6203e00640 1 -- 172.21.15.102:0/2761188959 shutdown_connections 2024-04-25T01:51:58.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.614+0000 7f6203e00640 1 -- 172.21.15.102:0/2761188959 wait complete. 2024-04-25T01:51:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.615+0000 7f6203e00640 1 Processor -- start 2024-04-25T01:51:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6203e00640 1 -- start start 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6203e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204107c80 0x7f620419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6203e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6204108e70 0x7f620419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6203e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6204109830 0x7f62041a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6203e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6204113950 con 0x7f6204107c80 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6203e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f62041137d0 con 0x7f6204108e70 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6203e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6204113ad0 con 0x7f6204109830 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6202a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204107c80 0x7f620419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6202a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204107c80 0x7f620419dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44270/0 (socket says 172.21.15.102:44270) 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6203400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6204109830 0x7f62041a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6202a00640 1 -- 172.21.15.102:0/3639780457 learned_addr learned my addr 172.21.15.102:0/3639780457 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.616+0000 7f6202000640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6204108e70 0x7f620419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f6202000640 1 -- 172.21.15.102:0/3639780457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6204109830 msgr2=0x7f62041a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f6202000640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6204109830 0x7f62041a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f6202000640 1 -- 172.21.15.102:0/3639780457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204107c80 msgr2=0x7f620419dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f6202000640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204107c80 0x7f620419dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f6202000640 1 -- 172.21.15.102:0/3639780457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62041a5b00 con 0x7f6204108e70 2024-04-25T01:51:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f6202000640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6204108e70 0x7f620419e420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f61f000e9f0 tx=0x7f61f000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f61fb400640 1 -- 172.21.15.102:0/3639780457 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f000ccc0 con 0x7f6204108e70 2024-04-25T01:51:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f61fb400640 1 -- 172.21.15.102:0/3639780457 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f0005020 con 0x7f6204108e70 2024-04-25T01:51:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62041a5df0 con 0x7f6204108e70 2024-04-25T01:51:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f61fb400640 1 -- 172.21.15.102:0/3639780457 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f00049e0 con 0x7f6204108e70 2024-04-25T01:51:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.617+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62041a6260 con 0x7f6204108e70 2024-04-25T01:51:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.618+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f620410ef40 con 0x7f6204108e70 2024-04-25T01:51:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.621+0000 7f61fb400640 1 -- 172.21.15.102:0/3639780457 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f61f0005560 con 0x7f6204108e70 2024-04-25T01:51:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.622+0000 7f61fb400640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61e80a2b90 0x7f61e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.622+0000 7f61fb400640 1 -- 172.21.15.102:0/3639780457 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f61f00ceb70 con 0x7f6204108e70 2024-04-25T01:51:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.623+0000 7f6202a00640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61e80a2b90 0x7f61e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.623+0000 7f6202a00640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61e80a2b90 0x7f61e80a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f61f40058c0 tx=0x7f61f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:58.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.626+0000 7f61fb400640 1 -- 172.21.15.102:0/3639780457 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f61f00051d0 con 0x7f6204108e70 2024-04-25T01:51:58.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.826+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6204062340 con 0x7f6204108e70 2024-04-25T01:51:58.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.827+0000 7f61fb400640 1 -- 172.21.15.102:0/3639780457 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f61f009bed0 con 0x7f6204108e70 2024-04-25T01:51:58.826 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:51:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61e80a2b90 msgr2=0x7f61e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61e80a2b90 0x7f61e80a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f61f40058c0 tx=0x7f61f4005f90 comp rx=0 tx=0).stop 2024-04-25T01:51:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6204108e70 msgr2=0x7f620419e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6204108e70 0x7f620419e420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f61f000e9f0 tx=0x7f61f000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 shutdown_connections 2024-04-25T01:51:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61e80a2b90 0x7f61e80a5050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6204109830 0x7f62041a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6204108e70 0x7f620419e420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 --2- 172.21.15.102:0/3639780457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204107c80 0x7f620419dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:58.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 >> 172.21.15.102:0/3639780457 conn(0x7f6204076c10 msgr2=0x7f620410c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:58.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 shutdown_connections 2024-04-25T01:51:58.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:58.830+0000 7f6203e00640 1 -- 172.21.15.102:0/3639780457 wait complete. 2024-04-25T01:51:58.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:51:59.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:58 smithi193 ceph-mon[32981]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:59.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:58 smithi193 ceph-mon[32981]: Saving service mds.all spec with placement smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-04-25T01:51:59.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:59.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:59.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:59.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:59.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:59.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:58 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:59.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:58 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3876053222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.049+0000 7f8d61000640 1 -- 172.21.15.102:0/1053866778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c103160 msgr2=0x7f8d5c1099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.049+0000 7f8d61000640 1 --2- 172.21.15.102:0/1053866778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c103160 0x7f8d5c1099f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8d54009ca0 tx=0x7f8d5402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.049+0000 7f8d61000640 1 -- 172.21.15.102:0/1053866778 shutdown_connections 2024-04-25T01:51:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.049+0000 7f8d61000640 1 --2- 172.21.15.102:0/1053866778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c103160 0x7f8d5c1099f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.049+0000 7f8d61000640 1 --2- 172.21.15.102:0/1053866778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d5c1027a0 0x7f8d5c102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.049+0000 7f8d61000640 1 --2- 172.21.15.102:0/1053866778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d5c1015b0 0x7f8d5c1019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.049+0000 7f8d61000640 1 -- 172.21.15.102:0/1053866778 >> 172.21.15.102:0/1053866778 conn(0x7f8d5c0fcb20 msgr2=0x7f8d5c0fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:59.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.049+0000 7f8d61000640 1 -- 172.21.15.102:0/1053866778 shutdown_connections 2024-04-25T01:51:59.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.049+0000 7f8d61000640 1 -- 172.21.15.102:0/1053866778 wait complete. 2024-04-25T01:51:59.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d61000640 1 Processor -- start 2024-04-25T01:51:59.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d61000640 1 -- start start 2024-04-25T01:51:59.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d61000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c1015b0 0x7f8d5c197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:59.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d61000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d5c1027a0 0x7f8d5c197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d61000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d5c103160 0x7f8d5c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d61000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c10d280 con 0x7f8d5c103160 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d61000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c10d100 con 0x7f8d5c1027a0 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d61000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5c10d400 con 0x7f8d5c1015b0 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d5b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c1015b0 0x7f8d5c197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d5b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c1015b0 0x7f8d5c197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46222/0 (socket says 172.21.15.102:46222) 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d5b400640 1 -- 172.21.15.102:0/3882005387 learned_addr learned my addr 172.21.15.102:0/3882005387 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.050+0000 7f8d5be00640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d5c103160 0x7f8d5c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d5aa00640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d5c1027a0 0x7f8d5c197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d5b400640 1 -- 172.21.15.102:0/3882005387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d5c1027a0 msgr2=0x7f8d5c197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d5b400640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d5c1027a0 0x7f8d5c197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d5b400640 1 -- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d5c103160 msgr2=0x7f8d5c19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d5b400640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d5c103160 0x7f8d5c19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d5b400640 1 -- 172.21.15.102:0/3882005387 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d5c19f490 con 0x7f8d5c1015b0 2024-04-25T01:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d5b400640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c1015b0 0x7f8d5c197870 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00e9c0 tx=0x7f8d4c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d53e00640 1 -- 172.21.15.102:0/3882005387 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c00cd80 con 0x7f8d5c1015b0 2024-04-25T01:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d53e00640 1 -- 172.21.15.102:0/3882005387 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d4c0035f0 con 0x7f8d5c1015b0 2024-04-25T01:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d5c19f780 con 0x7f8d5c1015b0 2024-04-25T01:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d53e00640 1 -- 172.21.15.102:0/3882005387 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c010690 con 0x7f8d5c1015b0 2024-04-25T01:51:59.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.051+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d5c19fcd0 con 0x7f8d5c1015b0 2024-04-25T01:51:59.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.052+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d5c108870 con 0x7f8d5c1015b0 2024-04-25T01:51:59.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.054+0000 7f8d53e00640 1 -- 172.21.15.102:0/3882005387 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8d4c0037a0 con 0x7f8d5c1015b0 2024-04-25T01:51:59.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.056+0000 7f8d53e00640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8d400a2b90 0x7f8d400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:51:59.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.056+0000 7f8d53e00640 1 -- 172.21.15.102:0/3882005387 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f8d4c0cefe0 con 0x7f8d5c1015b0 2024-04-25T01:51:59.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.056+0000 7f8d5aa00640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8d400a2b90 0x7f8d400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:51:59.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.057+0000 7f8d5aa00640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8d400a2b90 0x7f8d400a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c198d80 tx=0x7f8d44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:51:59.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.058+0000 7f8d53e00640 1 -- 172.21.15.102:0/3882005387 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d4c09c340 con 0x7f8d5c1015b0 2024-04-25T01:51:59.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:58 smithi102 ceph-mon[27796]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:59.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:58 smithi102 ceph-mon[27796]: Saving service mds.all spec with placement smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-04-25T01:51:59.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:59.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:59.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:59.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:59.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:59.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:58 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:59.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:58 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3876053222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:59.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:58 smithi110 ceph-mon[32848]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:51:59.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:58 smithi110 ceph-mon[32848]: Saving service mds.all spec with placement smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-04-25T01:51:59.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:59.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:51:59.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:51:59.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:51:59.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:51:59.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:58 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:51:59.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:58 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3876053222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:51:59.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.298+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d5c02d170 con 0x7f8d5c1015b0 2024-04-25T01:51:59.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.299+0000 7f8d53e00640 1 -- 172.21.15.102:0/3882005387 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f8d4c09bc40 con 0x7f8d5c1015b0 2024-04-25T01:51:59.299 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:51:59.299 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T01:51:59.299 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 1 2024-04-25T01:51:59.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8d400a2b90 msgr2=0x7f8d400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:59.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8d400a2b90 0x7f8d400a5050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c198d80 tx=0x7f8d44005f90 comp rx=0 tx=0).stop 2024-04-25T01:51:59.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c1015b0 msgr2=0x7f8d5c197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:51:59.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c1015b0 0x7f8d5c197870 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00e9c0 tx=0x7f8d4c00ee80 comp rx=0 tx=0).stop 2024-04-25T01:51:59.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 shutdown_connections 2024-04-25T01:51:59.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8d400a2b90 0x7f8d400a5050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d5c103160 0x7f8d5c19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d5c1027a0 0x7f8d5c197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 --2- 172.21.15.102:0/3882005387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d5c1015b0 0x7f8d5c197870 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:51:59.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 >> 172.21.15.102:0/3882005387 conn(0x7f8d5c0fcb20 msgr2=0x7f8d5c105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:51:59.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 shutdown_connections 2024-04-25T01:51:59.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:51:59.302+0000 7f8d61000640 1 -- 172.21.15.102:0/3882005387 wait complete. 2024-04-25T01:52:00.096 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:59 smithi193 ceph-mon[32981]: Deploying daemon mds.l on smithi193 2024-04-25T01:52:00.096 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:59 smithi193 ceph-mon[32981]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:00.096 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:59 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3639780457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:52:00.096 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:51:59 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3882005387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:00.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:59 smithi102 ceph-mon[27796]: Deploying daemon mds.l on smithi193 2024-04-25T01:52:00.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:59 smithi102 ceph-mon[27796]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:00.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:59 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3639780457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:52:00.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:51:59 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3882005387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:00.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:59 smithi110 ceph-mon[32848]: Deploying daemon mds.l on smithi193 2024-04-25T01:52:00.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:59 smithi110 ceph-mon[32848]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:00.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:59 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3639780457' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:52:00.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:51:59 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3882005387' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:00.495 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 01:52:00 smithi193 systemd[1]: Starting Ceph mds.l for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:00.754 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 01:52:00 smithi193 podman[63045]: 2024-04-25 01:52:00.398627356 +0000 UTC m=+0.017724364 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:00.754 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 01:52:00 smithi193 podman[63045]: 2024-04-25 01:52:00.524367534 +0000 UTC m=+0.143464573 container create e5133fdb413beaf25aa2b8e090f8ba11f72ac8b13e58f136d6d22db49ad5d544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:52:00.754 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 01:52:00 smithi193 podman[63045]: 2024-04-25 01:52:00.743291935 +0000 UTC m=+0.362388928 container init e5133fdb413beaf25aa2b8e090f8ba11f72ac8b13e58f136d6d22db49ad5d544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T01:52:00.754 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 01:52:00 smithi193 podman[63045]: 2024-04-25 01:52:00.745776083 +0000 UTC m=+0.364873079 container start e5133fdb413beaf25aa2b8e090f8ba11f72ac8b13e58f136d6d22db49ad5d544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-25T01:52:01.254 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 01:52:00 smithi193 ceph-mds[63064]: -- 172.21.15.193:0/3156606883 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a45d70ba40 con 0x55a45d6de800 2024-04-25T01:52:01.254 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 01:52:00 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-l[63060]: starting mds.l at 2024-04-25T01:52:01.254 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 01:52:00 smithi193 bash[63045]: e5133fdb413beaf25aa2b8e090f8ba11f72ac8b13e58f136d6d22db49ad5d544 2024-04-25T01:52:01.254 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 01:52:00 smithi193 systemd[1]: Started Ceph mds.l for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:01.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.535+0000 7fb577200640 1 -- 172.21.15.102:0/2875331887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5780ffe30 msgr2=0x7fb578102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.535+0000 7fb577200640 1 --2- 172.21.15.102:0/2875331887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5780ffe30 0x7fb578102260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb568009900 tx=0x7fb56802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.536+0000 7fb577200640 1 -- 172.21.15.102:0/2875331887 shutdown_connections 2024-04-25T01:52:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.536+0000 7fb577200640 1 --2- 172.21.15.102:0/2875331887 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb578105150 0x7fb5781055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.536+0000 7fb577200640 1 --2- 172.21.15.102:0/2875331887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5781027a0 0x7fb578104c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.536+0000 7fb577200640 1 --2- 172.21.15.102:0/2875331887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5780ffe30 0x7fb578102260 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.536+0000 7fb577200640 1 -- 172.21.15.102:0/2875331887 >> 172.21.15.102:0/2875331887 conn(0x7fb5780fba10 msgr2=0x7fb5780fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:01.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.536+0000 7fb577200640 1 -- 172.21.15.102:0/2875331887 shutdown_connections 2024-04-25T01:52:01.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.536+0000 7fb577200640 1 -- 172.21.15.102:0/2875331887 wait complete. 2024-04-25T01:52:01.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb577200640 1 Processor -- start 2024-04-25T01:52:01.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb577200640 1 -- start start 2024-04-25T01:52:01.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb577200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5780ffe30 0x7fb57819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:01.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb577200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781027a0 0x7fb57819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:01.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb577200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb578105150 0x7fb5781a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:01.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb577200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb578113880 con 0x7fb578105150 2024-04-25T01:52:01.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb577200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb578113700 con 0x7fb5781027a0 2024-04-25T01:52:01.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb577200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb578113a00 con 0x7fb5780ffe30 2024-04-25T01:52:01.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb56d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781027a0 0x7fb57819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb576800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb578105150 0x7fb5781a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb575e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5780ffe30 0x7fb57819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb56d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781027a0 0x7fb57819e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:35388/0 (socket says 172.21.15.102:35388) 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb575e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5780ffe30 0x7fb57819df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46240/0 (socket says 172.21.15.102:46240) 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.537+0000 7fb56d400640 1 -- 172.21.15.102:0/3494969309 learned_addr learned my addr 172.21.15.102:0/3494969309 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb575e00640 1 -- 172.21.15.102:0/3494969309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781027a0 msgr2=0x7fb57819e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb575e00640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781027a0 0x7fb57819e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb575e00640 1 -- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb578105150 msgr2=0x7fb5781a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb575e00640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb578105150 0x7fb5781a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb575e00640 1 -- 172.21.15.102:0/3494969309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5781a5b00 con 0x7fb5780ffe30 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb576800640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb578105150 0x7fb5781a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb56d400640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781027a0 0x7fb57819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:01.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb575e00640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5780ffe30 0x7fb57819df30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb56802a910 tx=0x7fb568032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:01.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb56f400640 1 -- 172.21.15.102:0/3494969309 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56803f030 con 0x7fb5780ffe30 2024-04-25T01:52:01.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5781a5d90 con 0x7fb5780ffe30 2024-04-25T01:52:01.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb56f400640 1 -- 172.21.15.102:0/3494969309 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb568003ce0 con 0x7fb5780ffe30 2024-04-25T01:52:01.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.538+0000 7fb56f400640 1 -- 172.21.15.102:0/3494969309 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb568035070 con 0x7fb5780ffe30 2024-04-25T01:52:01.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.539+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5781a6190 con 0x7fb5780ffe30 2024-04-25T01:52:01.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.540+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5780fd500 con 0x7fb5780ffe30 2024-04-25T01:52:01.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.541+0000 7fb56f400640 1 -- 172.21.15.102:0/3494969309 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb568042030 con 0x7fb5780ffe30 2024-04-25T01:52:01.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.543+0000 7fb56f400640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb55c0a2b90 0x7fb55c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:01.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.543+0000 7fb56f400640 1 -- 172.21.15.102:0/3494969309 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb5680ef570 con 0x7fb5780ffe30 2024-04-25T01:52:01.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.543+0000 7fb56d400640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb55c0a2b90 0x7fb55c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:01.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.544+0000 7fb56d400640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb55c0a2b90 0x7fb55c0a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb57819f3f0 tx=0x7fb560005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:01.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.545+0000 7fb56f400640 1 -- 172.21.15.102:0/3494969309 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb5680bc280 con 0x7fb5780ffe30 2024-04-25T01:52:01.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.785+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb57802d170 con 0x7fb5780ffe30 2024-04-25T01:52:01.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.786+0000 7fb56f400640 1 -- 172.21.15.102:0/3494969309 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fb5680c0580 con 0x7fb5780ffe30 2024-04-25T01:52:01.788 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:01.789 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T01:52:01.789 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 2 2024-04-25T01:52:01.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.789+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb55c0a2b90 msgr2=0x7fb55c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:01.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.789+0000 7fb577200640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb55c0a2b90 0x7fb55c0a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb57819f3f0 tx=0x7fb560005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:01.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5780ffe30 msgr2=0x7fb57819df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:01.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5780ffe30 0x7fb57819df30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb56802a910 tx=0x7fb568032ce0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 shutdown_connections 2024-04-25T01:52:01.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb55c0a2b90 0x7fb55c0a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb578105150 0x7fb5781a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5781027a0 0x7fb57819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 --2- 172.21.15.102:0/3494969309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5780ffe30 0x7fb57819df30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:01.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 >> 172.21.15.102:0/3494969309 conn(0x7fb5780fba10 msgr2=0x7fb578103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:01.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 shutdown_connections 2024-04-25T01:52:01.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:01.790+0000 7fb577200640 1 -- 172.21.15.102:0/3494969309 wait complete. 2024-04-25T01:52:01.937 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:01.938 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.193:6832/3303248011,v1:172.21.15.193:6833/3303248011] up:boot 2024-04-25T01:52:01.938 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: fsmap 1 up:standby 2024-04-25T01:52:01.939 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:52:01.939 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:01.939 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:01.939 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:01.939 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:01.940 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:01.940 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:01 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.193:6832/3303248011,v1:172.21.15.193:6833/3303248011] up:boot 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: fsmap 1 up:standby 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:02.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:01 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.193:6832/3303248011,v1:172.21.15.193:6833/3303248011] up:boot 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: fsmap 1 up:standby 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:01 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:02.968 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:02 smithi110 ceph-mon[32848]: Deploying daemon mds.k on smithi110 2024-04-25T01:52:02.969 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:02 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3494969309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:03.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:02 smithi102 ceph-mon[27796]: Deploying daemon mds.k on smithi110 2024-04-25T01:52:03.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:02 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3494969309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:03.236 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 01:52:03 smithi110 systemd[1]: Starting Ceph mds.k for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:03.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:02 smithi193 ceph-mon[32981]: Deploying daemon mds.k on smithi110 2024-04-25T01:52:03.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:02 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3494969309' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:03.628 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 01:52:03 smithi110 podman[63148]: 2024-04-25 01:52:03.268503436 +0000 UTC m=+0.018128320 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:03.628 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 01:52:03 smithi110 podman[63148]: 2024-04-25 01:52:03.385458458 +0000 UTC m=+0.135083318 container create 91adea0f106ba43564e994c8f2d56aa363c3f502958990436eeebbe1e16ea8f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T01:52:03.629 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 01:52:03 smithi110 podman[63148]: 2024-04-25 01:52:03.604699428 +0000 UTC m=+0.354324287 container init 91adea0f106ba43564e994c8f2d56aa363c3f502958990436eeebbe1e16ea8f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T01:52:03.629 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 01:52:03 smithi110 podman[63148]: 2024-04-25 01:52:03.607292332 +0000 UTC m=+0.356917193 container start 91adea0f106ba43564e994c8f2d56aa363c3f502958990436eeebbe1e16ea8f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T01:52:03.817 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:03.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:03 smithi110 ceph-mon[32848]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:03.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:03 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:03.986 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:03 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:03.987 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 01:52:03 smithi110 ceph-mds[63167]: -- 172.21.15.110:0/2885453503 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae9e43da40 con 0x55ae9e410c00 2024-04-25T01:52:03.987 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 01:52:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-k[63163]: starting mds.k at 2024-04-25T01:52:03.987 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 01:52:03 smithi110 bash[63148]: 91adea0f106ba43564e994c8f2d56aa363c3f502958990436eeebbe1e16ea8f5 2024-04-25T01:52:03.987 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 01:52:03 smithi110 systemd[1]: Started Ceph mds.k for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.018+0000 7f6c9c800640 1 -- 172.21.15.102:0/3467909335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c94102990 msgr2=0x7f6c94102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.018+0000 7f6c9c800640 1 --2- 172.21.15.102:0/3467909335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c94102990 0x7f6c94102e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009900 tx=0x7f6c8402a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.018+0000 7f6c9c800640 1 -- 172.21.15.102:0/3467909335 shutdown_connections 2024-04-25T01:52:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.018+0000 7f6c9c800640 1 --2- 172.21.15.102:0/3467909335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c94103350 0x7f6c94109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.018+0000 7f6c9c800640 1 --2- 172.21.15.102:0/3467909335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c94102990 0x7f6c94102e10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.018+0000 7f6c9c800640 1 --2- 172.21.15.102:0/3467909335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c941017a0 0x7f6c94101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.018+0000 7f6c9c800640 1 -- 172.21.15.102:0/3467909335 >> 172.21.15.102:0/3467909335 conn(0x7f6c940fccf0 msgr2=0x7f6c940ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:04.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.018+0000 7f6c9c800640 1 -- 172.21.15.102:0/3467909335 shutdown_connections 2024-04-25T01:52:04.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.018+0000 7f6c9c800640 1 -- 172.21.15.102:0/3467909335 wait complete. 2024-04-25T01:52:04.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.019+0000 7f6c9c800640 1 Processor -- start 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.019+0000 7f6c9c800640 1 -- start start 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.019+0000 7f6c9c800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c941017a0 0x7f6c940754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.019+0000 7f6c9c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c94102990 0x7f6c94073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.019+0000 7f6c9c800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c94103350 0x7f6c94074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.019+0000 7f6c9c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9410d450 con 0x7f6c94102990 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.019+0000 7f6c9c800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9410d2d0 con 0x7f6c94103350 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.019+0000 7f6c9c800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9410d5d0 con 0x7f6c941017a0 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.019+0000 7f6c99e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c941017a0 0x7f6c940754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c99e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c941017a0 0x7f6c940754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56290/0 (socket says 172.21.15.102:56290) 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c99e00640 1 -- 172.21.15.102:0/2203403170 learned_addr learned my addr 172.21.15.102:0/2203403170 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c99400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c94102990 0x7f6c94073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:04.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c9a800640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c94103350 0x7f6c94074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c99e00640 1 -- 172.21.15.102:0/2203403170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c94103350 msgr2=0x7f6c94074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c99e00640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c94103350 0x7f6c94074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c99e00640 1 -- 172.21.15.102:0/2203403170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c94102990 msgr2=0x7f6c94073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c99e00640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c94102990 0x7f6c94073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c99e00640 1 -- 172.21.15.102:0/2203403170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c94074900 con 0x7f6c941017a0 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c99e00640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c941017a0 0x7f6c940754f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800e9c0 tx=0x7f6c8800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c8ea00640 1 -- 172.21.15.102:0/2203403170 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800cde0 con 0x7f6c941017a0 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.020+0000 7f6c8ea00640 1 -- 172.21.15.102:0/2203403170 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c88010910 con 0x7f6c941017a0 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.021+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c941a1bc0 con 0x7f6c941017a0 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.021+0000 7f6c8ea00640 1 -- 172.21.15.102:0/2203403170 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c880189c0 con 0x7f6c941017a0 2024-04-25T01:52:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.021+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c941a20e0 con 0x7f6c941017a0 2024-04-25T01:52:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.022+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c94108ac0 con 0x7f6c941017a0 2024-04-25T01:52:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.024+0000 7f6c8ea00640 1 -- 172.21.15.102:0/2203403170 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6c88010ac0 con 0x7f6c941017a0 2024-04-25T01:52:04.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.025+0000 7f6c8ea00640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6c7c0a2b90 0x7f6c7c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:04.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.025+0000 7f6c8ea00640 1 -- 172.21.15.102:0/2203403170 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f6c880cf190 con 0x7f6c941017a0 2024-04-25T01:52:04.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.025+0000 7f6c99400640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6c7c0a2b90 0x7f6c7c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:04.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.026+0000 7f6c99400640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6c7c0a2b90 0x7f6c7c0a5050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6c84031440 tx=0x7f6c840315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:04.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.029+0000 7f6c8ea00640 1 -- 172.21.15.102:0/2203403170 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6c88053740 con 0x7f6c941017a0 2024-04-25T01:52:04.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:03 smithi102 ceph-mon[27796]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:04.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:03 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:04.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:03 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:04.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:03 smithi193 ceph-mon[32981]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:04.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:03 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:04.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:03 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:04.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.278+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c94062340 con 0x7f6c941017a0 2024-04-25T01:52:04.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.279+0000 7f6c8ea00640 1 -- 172.21.15.102:0/2203403170 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f6c8809c4f0 con 0x7f6c941017a0 2024-04-25T01:52:04.279 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:04.279 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:52:04.279 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 3 2024-04-25T01:52:04.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6c7c0a2b90 msgr2=0x7f6c7c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:04.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6c7c0a2b90 0x7f6c7c0a5050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6c84031440 tx=0x7f6c840315b0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c941017a0 msgr2=0x7f6c940754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:04.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c941017a0 0x7f6c940754f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800e9c0 tx=0x7f6c8800ee80 comp rx=0 tx=0).stop 2024-04-25T01:52:04.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 shutdown_connections 2024-04-25T01:52:04.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6c7c0a2b90 0x7f6c7c0a5050 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c94103350 0x7f6c94074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c94102990 0x7f6c94073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 --2- 172.21.15.102:0/2203403170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c941017a0 0x7f6c940754f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 >> 172.21.15.102:0/2203403170 conn(0x7f6c940fccf0 msgr2=0x7f6c94105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 shutdown_connections 2024-04-25T01:52:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:04.282+0000 7f6c9c800640 1 -- 172.21.15.102:0/2203403170 wait complete. 2024-04-25T01:52:05.109 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:05.109 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:04 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.110:6832/3320624567,v1:172.21.15.110:6833/3320624567] up:boot 2024-04-25T01:52:05.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:04 smithi102 ceph-mon[27796]: fsmap 2 up:standby 2024-04-25T01:52:05.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:52:05.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:05.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:05.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:04 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:05.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:04 smithi102 ceph-mon[27796]: Deploying daemon mds.j on smithi102 2024-04-25T01:52:05.110 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:04 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2203403170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:05.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:05.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:04 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.110:6832/3320624567,v1:172.21.15.110:6833/3320624567] up:boot 2024-04-25T01:52:05.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:04 smithi110 ceph-mon[32848]: fsmap 2 up:standby 2024-04-25T01:52:05.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:52:05.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:05.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:05.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:04 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:05.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:04 smithi110 ceph-mon[32848]: Deploying daemon mds.j on smithi102 2024-04-25T01:52:05.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:04 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2203403170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:05.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:05.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:04 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.110:6832/3320624567,v1:172.21.15.110:6833/3320624567] up:boot 2024-04-25T01:52:05.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:04 smithi193 ceph-mon[32981]: fsmap 2 up:standby 2024-04-25T01:52:05.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:52:05.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:05.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:05.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:04 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:05.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:04 smithi193 ceph-mon[32981]: Deploying daemon mds.j on smithi102 2024-04-25T01:52:05.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:04 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2203403170' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:05.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:05 smithi102 ceph-mon[27796]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:06.005 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:05 smithi110 ceph-mon[32848]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:06.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:05 smithi193 ceph-mon[32981]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:06.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:06.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.554+0000 7fd11fe00640 1 -- 172.21.15.102:0/2318524527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd120103190 msgr2=0x7fd120109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:06.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.554+0000 7fd11fe00640 1 --2- 172.21.15.102:0/2318524527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd120103190 0x7fd120109a20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd118009ca0 tx=0x7fd11802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.555+0000 7fd11fe00640 1 -- 172.21.15.102:0/2318524527 shutdown_connections 2024-04-25T01:52:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.555+0000 7fd11fe00640 1 --2- 172.21.15.102:0/2318524527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd120103190 0x7fd120109a20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.555+0000 7fd11fe00640 1 --2- 172.21.15.102:0/2318524527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1201027d0 0x7fd120102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.555+0000 7fd11fe00640 1 --2- 172.21.15.102:0/2318524527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1201015e0 0x7fd120101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.555+0000 7fd11fe00640 1 -- 172.21.15.102:0/2318524527 >> 172.21.15.102:0/2318524527 conn(0x7fd1200fcb50 msgr2=0x7fd1200fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.555+0000 7fd11fe00640 1 -- 172.21.15.102:0/2318524527 shutdown_connections 2024-04-25T01:52:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.556+0000 7fd11fe00640 1 -- 172.21.15.102:0/2318524527 wait complete. 2024-04-25T01:52:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.557+0000 7fd11fe00640 1 Processor -- start 2024-04-25T01:52:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.557+0000 7fd11fe00640 1 -- start start 2024-04-25T01:52:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.558+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1201015e0 0x7fd1201a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.558+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1201027d0 0x7fd1201a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.558+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd120103190 0x7fd1201adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.558+0000 7fd11fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd12010d290 con 0x7fd120103190 2024-04-25T01:52:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.558+0000 7fd11fe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd12010d110 con 0x7fd1201027d0 2024-04-25T01:52:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.558+0000 7fd11fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd12010d410 con 0x7fd1201015e0 2024-04-25T01:52:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.559+0000 7fd11f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd120103190 0x7fd1201adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.559+0000 7fd11f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd120103190 0x7fd1201adc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59146/0 (socket says 172.21.15.102:59146) 2024-04-25T01:52:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.559+0000 7fd11f400640 1 -- 172.21.15.102:0/1947428758 learned_addr learned my addr 172.21.15.102:0/1947428758 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.559+0000 7fd11f400640 1 -- 172.21.15.102:0/1947428758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1201015e0 msgr2=0x7fd1201a66a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:52:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.559+0000 7fd11f400640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1201015e0 0x7fd1201a66a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.559+0000 7fd11f400640 1 -- 172.21.15.102:0/1947428758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1201027d0 msgr2=0x7fd1201a6be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:52:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.559+0000 7fd11f400640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1201027d0 0x7fd1201a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.559+0000 7fd11f400640 1 -- 172.21.15.102:0/1947428758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1201ae2c0 con 0x7fd120103190 2024-04-25T01:52:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.560+0000 7fd11f400640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd120103190 0x7fd1201adc50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd11802f540 tx=0x7fd118031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.561+0000 7fd117400640 1 -- 172.21.15.102:0/1947428758 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd118003b70 con 0x7fd120103190 2024-04-25T01:52:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.561+0000 7fd117400640 1 -- 172.21.15.102:0/1947428758 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd118003d10 con 0x7fd120103190 2024-04-25T01:52:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.561+0000 7fd117400640 1 -- 172.21.15.102:0/1947428758 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11802fb50 con 0x7fd120103190 2024-04-25T01:52:06.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.562+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1201ae550 con 0x7fd120103190 2024-04-25T01:52:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.562+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1201aea20 con 0x7fd120103190 2024-04-25T01:52:06.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.570+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd120068480 con 0x7fd120103190 2024-04-25T01:52:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.571+0000 7fd117400640 1 -- 172.21.15.102:0/1947428758 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd11802fcf0 con 0x7fd120103190 2024-04-25T01:52:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.573+0000 7fd117400640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1040a2c10 0x7fd1040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.573+0000 7fd117400640 1 -- 172.21.15.102:0/1947428758 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fd1180f0310 con 0x7fd120103190 2024-04-25T01:52:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.577+0000 7fd11ea00640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1040a2c10 0x7fd1040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:06.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.579+0000 7fd11ea00640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1040a2c10 0x7fd1040a50d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd120102630 tx=0x7fd110005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:06.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.582+0000 7fd117400640 1 -- 172.21.15.102:0/1947428758 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd1180bd710 con 0x7fd120103190 2024-04-25T01:52:06.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.829+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd12002d300 con 0x7fd120103190 2024-04-25T01:52:06.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.830+0000 7fd117400640 1 -- 172.21.15.102:0/1947428758 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fd1180bd010 con 0x7fd120103190 2024-04-25T01:52:06.830 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:06.831 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:52:06.831 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 3 2024-04-25T01:52:06.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.832+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1040a2c10 msgr2=0x7fd1040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.832+0000 7fd11fe00640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1040a2c10 0x7fd1040a50d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd120102630 tx=0x7fd110005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.832+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd120103190 msgr2=0x7fd1201adc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.832+0000 7fd11fe00640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd120103190 0x7fd1201adc50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd11802f540 tx=0x7fd118031750 comp rx=0 tx=0).stop 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.833+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 shutdown_connections 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.833+0000 7fd11fe00640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd1040a2c10 0x7fd1040a50d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.833+0000 7fd11fe00640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd120103190 0x7fd1201adc50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.833+0000 7fd11fe00640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1201027d0 0x7fd1201a6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.833+0000 7fd11fe00640 1 --2- 172.21.15.102:0/1947428758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1201015e0 0x7fd1201a66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.833+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 >> 172.21.15.102:0/1947428758 conn(0x7fd1200fcb50 msgr2=0x7fd120103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.833+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 shutdown_connections 2024-04-25T01:52:06.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:06.833+0000 7fd11fe00640 1 -- 172.21.15.102:0/1947428758 wait complete. 2024-04-25T01:52:07.455 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 01:52:07 smithi102 podman[68350]: 2024-04-25 01:52:07.209115834 +0000 UTC m=+0.159151147 container create fcb3f2944bffd85d138393f1f2abdaabf63de3a14e5b2a2ec7050e132f424ea1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-d2c45d9) 2024-04-25T01:52:07.455 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 01:52:07 smithi102 podman[68350]: 2024-04-25 01:52:07.454513241 +0000 UTC m=+0.404548561 container init fcb3f2944bffd85d138393f1f2abdaabf63de3a14e5b2a2ec7050e132f424ea1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T01:52:07.828 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 01:52:07 smithi102 podman[68350]: 2024-04-25 01:52:07.465452771 +0000 UTC m=+0.415488066 container start fcb3f2944bffd85d138393f1f2abdaabf63de3a14e5b2a2ec7050e132f424ea1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T01:52:07.828 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-mds[68369]: -- 172.21.15.102:0/3412557383 <== mon.0 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5585ae37fa40 con 0x5585ae353000 2024-04-25T01:52:07.828 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j[68365]: starting mds.j at 2024-04-25T01:52:07.828 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 01:52:07 smithi102 bash[68350]: fcb3f2944bffd85d138393f1f2abdaabf63de3a14e5b2a2ec7050e132f424ea1 2024-04-25T01:52:07.828 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 01:52:07 smithi102 systemd[1]: Started Ceph mds.j for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:07.828 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-mon[27796]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:07.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1947428758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:07.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:07.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:07.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:07.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:07.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:07.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:07 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:08.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:07 smithi110 ceph-mon[32848]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:08.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:07 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1947428758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:08.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:08.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:08.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:08.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:08.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:08.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:07 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:07 smithi193 ceph-mon[32981]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:07 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1947428758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:07 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:08.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:09.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:08 smithi102 ceph-mon[27796]: Deploying daemon mds.d on smithi102 2024-04-25T01:52:09.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:08 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.102:6834/2866248981,v1:172.21.15.102:6835/2866248981] up:boot 2024-04-25T01:52:09.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:08 smithi102 ceph-mon[27796]: fsmap 3 up:standby 2024-04-25T01:52:09.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:08 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:52:09.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:08 smithi102 ceph-mon[27796]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:09.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.072+0000 7f3f1fe00640 1 -- 172.21.15.102:0/1693098858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 msgr2=0x7f3f20073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.072+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/1693098858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 0x7f3f20073b70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009900 tx=0x7f3f1002a490 comp rx=0 tx=0).stop 2024-04-25T01:52:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.072+0000 7f3f1fe00640 1 -- 172.21.15.102:0/1693098858 shutdown_connections 2024-04-25T01:52:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.072+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/1693098858 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f20109440 0x7f3f2010b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.072+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/1693098858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f20074140 0x7f3f200745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.072+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/1693098858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 0x7f3f20073b70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.072+0000 7f3f1fe00640 1 -- 172.21.15.102:0/1693098858 >> 172.21.15.102:0/1693098858 conn(0x7f3f200fcb20 msgr2=0x7f3f200fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.072+0000 7f3f1fe00640 1 -- 172.21.15.102:0/1693098858 shutdown_connections 2024-04-25T01:52:09.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.072+0000 7f3f1fe00640 1 -- 172.21.15.102:0/1693098858 wait complete. 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.073+0000 7f3f1fe00640 1 Processor -- start 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.073+0000 7f3f1fe00640 1 -- start start 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f20074140 0x7f3f20199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 0x7f3f2019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f20109440 0x7f3f201a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2010f470 con 0x7f3f20075720 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1fe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2010f2f0 con 0x7f3f20109440 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2010f5f0 con 0x7f3f20074140 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 0x7f3f2019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 0x7f3f2019a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59164/0 (socket says 172.21.15.102:59164) 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f20074140 0x7f3f20199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1e000640 1 -- 172.21.15.102:0/98258467 learned_addr learned my addr 172.21.15.102:0/98258467 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.074+0000 7f3f1f400640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f20109440 0x7f3f201a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1e000640 1 -- 172.21.15.102:0/98258467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f20074140 msgr2=0x7f3f20199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1e000640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f20074140 0x7f3f20199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1e000640 1 -- 172.21.15.102:0/98258467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f20109440 msgr2=0x7f3f201a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1e000640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f20109440 0x7f3f201a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1e000640 1 -- 172.21.15.102:0/98258467 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f201a1730 con 0x7f3f20075720 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1ea00640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f20074140 0x7f3f20199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1f400640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f20109440 0x7f3f201a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1e000640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 0x7f3f2019a050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00e9f0 tx=0x7f3f0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f17400640 1 -- 172.21.15.102:0/98258467 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c00cd80 con 0x7f3f20075720 2024-04-25T01:52:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f17400640 1 -- 172.21.15.102:0/98258467 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f0c005500 con 0x7f3f20075720 2024-04-25T01:52:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f201a1a20 con 0x7f3f20075720 2024-04-25T01:52:09.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f17400640 1 -- 172.21.15.102:0/98258467 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c010690 con 0x7f3f20075720 2024-04-25T01:52:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.075+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f201a1f20 con 0x7f3f20075720 2024-04-25T01:52:09.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.077+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f20108200 con 0x7f3f20075720 2024-04-25T01:52:09.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.082+0000 7f3f17400640 1 -- 172.21.15.102:0/98258467 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3f0c0046c0 con 0x7f3f20075720 2024-04-25T01:52:09.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.083+0000 7f3f17400640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3f040a2b90 0x7f3f040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:09.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.083+0000 7f3f17400640 1 -- 172.21.15.102:0/98258467 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f3f0c0cef80 con 0x7f3f20075720 2024-04-25T01:52:09.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.083+0000 7f3f1ea00640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3f040a2b90 0x7f3f040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:09.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.084+0000 7f3f1ea00640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3f040a2b90 0x7f3f040a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3f10031440 tx=0x7f3f100315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:09.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.088+0000 7f3f17400640 1 -- 172.21.15.102:0/98258467 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3f0c09c380 con 0x7f3f20075720 2024-04-25T01:52:09.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:08 smithi110 ceph-mon[32848]: Deploying daemon mds.d on smithi102 2024-04-25T01:52:09.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:08 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.102:6834/2866248981,v1:172.21.15.102:6835/2866248981] up:boot 2024-04-25T01:52:09.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:08 smithi110 ceph-mon[32848]: fsmap 3 up:standby 2024-04-25T01:52:09.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:08 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:52:09.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:08 smithi110 ceph-mon[32848]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:09.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:08 smithi193 ceph-mon[32981]: Deploying daemon mds.d on smithi102 2024-04-25T01:52:09.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:08 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.102:6834/2866248981,v1:172.21.15.102:6835/2866248981] up:boot 2024-04-25T01:52:09.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:08 smithi193 ceph-mon[32981]: fsmap 3 up:standby 2024-04-25T01:52:09.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:08 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:52:09.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:08 smithi193 ceph-mon[32981]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:09.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.331+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f2002d380 con 0x7f3f20075720 2024-04-25T01:52:09.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.332+0000 7f3f17400640 1 -- 172.21.15.102:0/98258467 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7f3f0c09bc80 con 0x7f3f20075720 2024-04-25T01:52:09.333 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:09.333 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:52:09.333 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 4 2024-04-25T01:52:09.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3f040a2b90 msgr2=0x7f3f040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3f040a2b90 0x7f3f040a5050 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3f10031440 tx=0x7f3f100315b0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 msgr2=0x7f3f2019a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:09.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 0x7f3f2019a050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00e9f0 tx=0x7f3f0c00edf0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 shutdown_connections 2024-04-25T01:52:09.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3f040a2b90 0x7f3f040a5050 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f20109440 0x7f3f201a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f20075720 0x7f3f2019a050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 --2- 172.21.15.102:0/98258467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f20074140 0x7f3f20199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:09.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 >> 172.21.15.102:0/98258467 conn(0x7f3f200fcb20 msgr2=0x7f3f201053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:09.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 shutdown_connections 2024-04-25T01:52:09.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:09.335+0000 7f3f1fe00640 1 -- 172.21.15.102:0/98258467 wait complete. 2024-04-25T01:52:09.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:09 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/98258467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:10.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:09 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/98258467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:09 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/98258467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:10.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:10 smithi102 ceph-mon[27796]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:11.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:10 smithi110 ceph-mon[32848]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:11.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:10 smithi193 ceph-mon[32981]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:11.316 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 01:52:10 smithi102 systemd[1]: Starting Ceph mds.d for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:11.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:11.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.567+0000 7f1747e00640 1 -- 172.21.15.102:0/809388055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740109640 msgr2=0x7f174010bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:11.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.567+0000 7f1747e00640 1 --2- 172.21.15.102:0/809388055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740109640 0x7f174010bad0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1730009860 tx=0x7f173002a3d0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.567+0000 7f1747e00640 1 -- 172.21.15.102:0/809388055 shutdown_connections 2024-04-25T01:52:11.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.567+0000 7f1747e00640 1 --2- 172.21.15.102:0/809388055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740109640 0x7f174010bad0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.567+0000 7f1747e00640 1 --2- 172.21.15.102:0/809388055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1740074120 0x7f1740074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.567+0000 7f1747e00640 1 --2- 172.21.15.102:0/809388055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1740075700 0x7f1740073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.567+0000 7f1747e00640 1 -- 172.21.15.102:0/809388055 >> 172.21.15.102:0/809388055 conn(0x7f17400fcd40 msgr2=0x7f17400ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:11.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.567+0000 7f1747e00640 1 -- 172.21.15.102:0/809388055 shutdown_connections 2024-04-25T01:52:11.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.567+0000 7f1747e00640 1 -- 172.21.15.102:0/809388055 wait complete. 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.568+0000 7f1747e00640 1 Processor -- start 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.568+0000 7f1747e00640 1 -- start start 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1747e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1740074120 0x7f1740199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1747e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740075700 0x7f174019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1747e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1740109640 0x7f17401a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1747e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f174010f6c0 con 0x7f1740109640 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1747e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f174010f540 con 0x7f1740075700 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1747e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f174010f840 con 0x7f1740074120 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1744a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740075700 0x7f174019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1745400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1740074120 0x7f1740199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1745e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1740109640 0x7f17401a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:11.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1744a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740075700 0x7f174019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42434/0 (socket says 172.21.15.102:42434) 2024-04-25T01:52:11.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1744a00640 1 -- 172.21.15.102:0/1171354289 learned_addr learned my addr 172.21.15.102:0/1171354289 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:11.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.569+0000 7f1745400640 1 -- 172.21.15.102:0/1171354289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740075700 msgr2=0x7f174019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:11.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f1745400640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740075700 0x7f174019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f1745400640 1 -- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1740109640 msgr2=0x7f17401a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:11.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f1745400640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1740109640 0x7f17401a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f1745400640 1 -- 172.21.15.102:0/1171354289 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17401a1880 con 0x7f1740074120 2024-04-25T01:52:11.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f1745e00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1740109640 0x7f17401a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:11.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f1744a00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740075700 0x7f174019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:11.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f1745400640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1740074120 0x7f1740199c70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1738009fd0 tx=0x7f173800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:11.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f173e000640 1 -- 172.21.15.102:0/1171354289 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738003970 con 0x7f1740074120 2024-04-25T01:52:11.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f173e000640 1 -- 172.21.15.102:0/1171354289 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1738004020 con 0x7f1740074120 2024-04-25T01:52:11.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f173e000640 1 -- 172.21.15.102:0/1171354289 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173800ba60 con 0x7f1740074120 2024-04-25T01:52:11.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17401a1b70 con 0x7f1740074120 2024-04-25T01:52:11.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.570+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17401a2070 con 0x7f1740074120 2024-04-25T01:52:11.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.571+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1740108400 con 0x7f1740074120 2024-04-25T01:52:11.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.574+0000 7f173e000640 1 -- 172.21.15.102:0/1171354289 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f17380041d0 con 0x7f1740074120 2024-04-25T01:52:11.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.576+0000 7f173e000640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17240a2b90 0x7f17240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:11.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.576+0000 7f173e000640 1 -- 172.21.15.102:0/1171354289 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f17380cf490 con 0x7f1740074120 2024-04-25T01:52:11.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.576+0000 7f1744a00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17240a2b90 0x7f17240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:11.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.577+0000 7f1744a00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17240a2b90 0x7f17240a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f172c009e10 tx=0x7f172c0091c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:11.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.578+0000 7f173e000640 1 -- 172.21.15.102:0/1171354289 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1738024020 con 0x7f1740074120 2024-04-25T01:52:11.612 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 01:52:11 smithi102 podman[68694]: 2024-04-25 01:52:11.221325311 +0000 UTC m=+0.033535601 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:11.612 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 01:52:11 smithi102 podman[68694]: 2024-04-25 01:52:11.377655223 +0000 UTC m=+0.189865563 container create 404ae44b30b463e94cd20f23f322ab6977871fbf73f03ea755e7d5a48034ace4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:52:11.612 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 01:52:11 smithi102 podman[68694]: 2024-04-25 01:52:11.612665374 +0000 UTC m=+0.424875706 container init 404ae44b30b463e94cd20f23f322ab6977871fbf73f03ea755e7d5a48034ace4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T01:52:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.828+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f174002d420 con 0x7f1740074120 2024-04-25T01:52:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.829+0000 7f173e000640 1 -- 172.21.15.102:0/1171354289 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7f173809c7f0 con 0x7f1740074120 2024-04-25T01:52:11.830 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:11.830 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:52:11.830 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 4 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17240a2b90 msgr2=0x7f17240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17240a2b90 0x7f17240a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f172c009e10 tx=0x7f172c0091c0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1740074120 msgr2=0x7f1740199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1740074120 0x7f1740199c70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1738009fd0 tx=0x7f173800c880 comp rx=0 tx=0).stop 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 shutdown_connections 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f17240a2b90 0x7f17240a5050 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1740109640 0x7f17401a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1740075700 0x7f174019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 --2- 172.21.15.102:0/1171354289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1740074120 0x7f1740199c70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 >> 172.21.15.102:0/1171354289 conn(0x7f17400fcd40 msgr2=0x7f17401055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 shutdown_connections 2024-04-25T01:52:11.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:11.832+0000 7f1747e00640 1 -- 172.21.15.102:0/1171354289 wait complete. 2024-04-25T01:52:11.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:11 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1171354289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:11.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:11 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:11.923 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 01:52:11 smithi102 podman[68694]: 2024-04-25 01:52:11.622029198 +0000 UTC m=+0.434239494 container start 404ae44b30b463e94cd20f23f322ab6977871fbf73f03ea755e7d5a48034ace4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T01:52:11.924 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 01:52:11 smithi102 ceph-mds[68752]: -- 172.21.15.102:0/2797706632 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d6aa2f9a40 con 0x55d6aa2cc800 2024-04-25T01:52:11.924 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 01:52:11 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d[68729]: starting mds.d at 2024-04-25T01:52:11.924 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 01:52:11 smithi102 bash[68694]: 404ae44b30b463e94cd20f23f322ab6977871fbf73f03ea755e7d5a48034ace4 2024-04-25T01:52:11.924 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 01:52:11 smithi102 systemd[1]: Started Ceph mds.d for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:12.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:11 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1171354289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:12.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:11 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:12.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:11 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1171354289' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:12.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:11 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:13.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:13.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.102:6836/3271665645,v1:172.21.15.102:6837/3271665645] up:boot 2024-04-25T01:52:13.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: fsmap 4 up:standby 2024-04-25T01:52:13.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:52:13.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:13.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:13.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:13.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:13.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: Deploying daemon mds.e on smithi110 2024-04-25T01:52:13.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:12 smithi102 ceph-mon[27796]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:13.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:13.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.102:6836/3271665645,v1:172.21.15.102:6837/3271665645] up:boot 2024-04-25T01:52:13.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: fsmap 4 up:standby 2024-04-25T01:52:13.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:52:13.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:13.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:13.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:13.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:13.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: Deploying daemon mds.e on smithi110 2024-04-25T01:52:13.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:12 smithi110 ceph-mon[32848]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.102:6836/3271665645,v1:172.21.15.102:6837/3271665645] up:boot 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: fsmap 4 up:standby 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: Deploying daemon mds.e on smithi110 2024-04-25T01:52:13.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:12 smithi193 ceph-mon[32981]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:13.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:14.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.067+0000 7f6557000640 1 -- 172.21.15.102:0/3687251617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65501039c0 msgr2=0x7f6550103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.067+0000 7f6557000640 1 --2- 172.21.15.102:0/3687251617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65501039c0 0x7f6550103de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6540009900 tx=0x7f654002a400 comp rx=0 tx=0).stop 2024-04-25T01:52:14.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.067+0000 7f6557000640 1 -- 172.21.15.102:0/3687251617 shutdown_connections 2024-04-25T01:52:14.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.067+0000 7f6557000640 1 --2- 172.21.15.102:0/3687251617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6550105570 0x7f655010be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.067+0000 7f6557000640 1 --2- 172.21.15.102:0/3687251617 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550104bb0 0x7f6550105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.067+0000 7f6557000640 1 --2- 172.21.15.102:0/3687251617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65501039c0 0x7f6550103de0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.067+0000 7f6557000640 1 -- 172.21.15.102:0/3687251617 >> 172.21.15.102:0/3687251617 conn(0x7f65500fef30 msgr2=0x7f6550101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:14.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.068+0000 7f6557000640 1 -- 172.21.15.102:0/3687251617 shutdown_connections 2024-04-25T01:52:14.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.068+0000 7f6557000640 1 -- 172.21.15.102:0/3687251617 wait complete. 2024-04-25T01:52:14.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.068+0000 7f6557000640 1 Processor -- start 2024-04-25T01:52:14.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f6557000640 1 -- start start 2024-04-25T01:52:14.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f6557000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65501039c0 0x7f6550199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:14.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f6557000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550104bb0 0x7f655019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:14.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f6557000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6550105570 0x7f65501a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:14.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f6557000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f655010f640 con 0x7f65501039c0 2024-04-25T01:52:14.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f6557000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f655010f4c0 con 0x7f6550104bb0 2024-04-25T01:52:14.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f6557000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f655010f7c0 con 0x7f6550105570 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f654f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550104bb0 0x7f655019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f654fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65501039c0 0x7f6550199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f654f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550104bb0 0x7f655019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:57966/0 (socket says 172.21.15.102:57966) 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f654f400640 1 -- 172.21.15.102:0/3791991653 learned_addr learned my addr 172.21.15.102:0/3791991653 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.069+0000 7f6555000640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6550105570 0x7f65501a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654f400640 1 -- 172.21.15.102:0/3791991653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6550105570 msgr2=0x7f65501a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654f400640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6550105570 0x7f65501a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654f400640 1 -- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65501039c0 msgr2=0x7f6550199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654f400640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65501039c0 0x7f6550199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654f400640 1 -- 172.21.15.102:0/3791991653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65501a18e0 con 0x7f6550104bb0 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654fe00640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65501039c0 0x7f6550199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f6555000640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6550105570 0x7f65501a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:14.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654f400640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550104bb0 0x7f655019a1e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f653c00e9f0 tx=0x7f653c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654cc00640 1 -- 172.21.15.102:0/3791991653 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653c00cd80 con 0x7f6550104bb0 2024-04-25T01:52:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654cc00640 1 -- 172.21.15.102:0/3791991653 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f653c010910 con 0x7f6550104bb0 2024-04-25T01:52:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65501a1bd0 con 0x7f6550104bb0 2024-04-25T01:52:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f654cc00640 1 -- 172.21.15.102:0/3791991653 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f653c018990 con 0x7f6550104bb0 2024-04-25T01:52:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.070+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65501a20f0 con 0x7f6550104bb0 2024-04-25T01:52:14.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.071+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f655010ad00 con 0x7f6550104bb0 2024-04-25T01:52:14.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.074+0000 7f654cc00640 1 -- 172.21.15.102:0/3791991653 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f653c010ac0 con 0x7f6550104bb0 2024-04-25T01:52:14.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.076+0000 7f654cc00640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f65340a2b90 0x7f65340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:14.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.076+0000 7f654cc00640 1 -- 172.21.15.102:0/3791991653 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f653c0cf3e0 con 0x7f6550104bb0 2024-04-25T01:52:14.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.076+0000 7f654fe00640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f65340a2b90 0x7f65340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:14.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.077+0000 7f654cc00640 1 -- 172.21.15.102:0/3791991653 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f653c09c740 con 0x7f6550104bb0 2024-04-25T01:52:14.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.077+0000 7f654fe00640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f65340a2b90 0x7f65340a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6540005ab0 tx=0x7f6540000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:14.238 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 01:52:14 smithi110 systemd[1]: Starting Ceph mds.e for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:14.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.319+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f655002d170 con 0x7f6550104bb0 2024-04-25T01:52:14.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.320+0000 7f654cc00640 1 -- 172.21.15.102:0/3791991653 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3880 (secure 0 0 0) 0x7f653c09c040 con 0x7f6550104bb0 2024-04-25T01:52:14.321 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:14.321 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T01:52:14.321 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 5 2024-04-25T01:52:14.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f65340a2b90 msgr2=0x7f65340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f65340a2b90 0x7f65340a5050 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6540005ab0 tx=0x7f6540000bc0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550104bb0 msgr2=0x7f655019a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550104bb0 0x7f655019a1e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f653c00e9f0 tx=0x7f653c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 shutdown_connections 2024-04-25T01:52:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f65340a2b90 0x7f65340a5050 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6550105570 0x7f65501a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550104bb0 0x7f655019a1e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 --2- 172.21.15.102:0/3791991653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65501039c0 0x7f6550199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 >> 172.21.15.102:0/3791991653 conn(0x7f65500fef30 msgr2=0x7f6550107ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 shutdown_connections 2024-04-25T01:52:14.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:14.323+0000 7f6557000640 1 -- 172.21.15.102:0/3791991653 wait complete. 2024-04-25T01:52:14.565 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:14 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3791991653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:14.565 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 01:52:14 smithi110 podman[63469]: 2024-04-25 01:52:14.25508066 +0000 UTC m=+0.106397915 container create 29a3871a9cba1292d90092f0b4c57621415b5341fbf6dabba3e9673f81e66d8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-04-25T01:52:14.565 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 01:52:14 smithi110 podman[63469]: 2024-04-25 01:52:14.166417472 +0000 UTC m=+0.017734734 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:14.565 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 01:52:14 smithi110 podman[63469]: 2024-04-25 01:52:14.466171262 +0000 UTC m=+0.317488520 container init 29a3871a9cba1292d90092f0b4c57621415b5341fbf6dabba3e9673f81e66d8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T01:52:14.565 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 01:52:14 smithi110 podman[63469]: 2024-04-25 01:52:14.468724276 +0000 UTC m=+0.320041529 container start 29a3871a9cba1292d90092f0b4c57621415b5341fbf6dabba3e9673f81e66d8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T01:52:14.566 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 01:52:14 smithi110 ceph-mds[63488]: -- 172.21.15.110:0/3092389678 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fbbce27a40 con 0x55fbbcdfa800 2024-04-25T01:52:14.566 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 01:52:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e[63484]: starting mds.e at 2024-04-25T01:52:14.566 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 01:52:14 smithi110 bash[63469]: 29a3871a9cba1292d90092f0b4c57621415b5341fbf6dabba3e9673f81e66d8f 2024-04-25T01:52:14.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:14 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3791991653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:14.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:14 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3791991653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:14.986 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 01:52:14 smithi110 systemd[1]: Started Ceph mds.e for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:15.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:15.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:15.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:15.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:15.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:15.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:15.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: Deploying daemon mds.f on smithi193 2024-04-25T01:52:15.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:15.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.110:6834/4057730311,v1:172.21.15.110:6835/4057730311] up:boot 2024-04-25T01:52:15.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: fsmap 5 up:standby 2024-04-25T01:52:15.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:15 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:52:15.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:15.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:15.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:15.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:15.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:15.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:15.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: Deploying daemon mds.f on smithi193 2024-04-25T01:52:15.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:15.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.110:6834/4057730311,v1:172.21.15.110:6835/4057730311] up:boot 2024-04-25T01:52:15.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: fsmap 5 up:standby 2024-04-25T01:52:15.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:15 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: Deploying daemon mds.f on smithi193 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.110:6834/4057730311,v1:172.21.15.110:6835/4057730311] up:boot 2024-04-25T01:52:16.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: fsmap 5 up:standby 2024-04-25T01:52:16.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:15 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:52:16.345 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:16.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.557+0000 7f7d86000640 1 -- 172.21.15.102:0/3060631276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d801049b0 msgr2=0x7f7d80104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.557+0000 7f7d86000640 1 --2- 172.21.15.102:0/3060631276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d801049b0 0x7f7d80104e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009900 tx=0x7f7d6c02a400 comp rx=0 tx=0).stop 2024-04-25T01:52:16.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.557+0000 7f7d86000640 1 -- 172.21.15.102:0/3060631276 shutdown_connections 2024-04-25T01:52:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.557+0000 7f7d86000640 1 --2- 172.21.15.102:0/3060631276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d80105370 0x7f7d8010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.557+0000 7f7d86000640 1 --2- 172.21.15.102:0/3060631276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d801049b0 0x7f7d80104e30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.557+0000 7f7d86000640 1 --2- 172.21.15.102:0/3060631276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d801037c0 0x7f7d80103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.557+0000 7f7d86000640 1 -- 172.21.15.102:0/3060631276 >> 172.21.15.102:0/3060631276 conn(0x7f7d800fed50 msgr2=0x7f7d80101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.557+0000 7f7d86000640 1 -- 172.21.15.102:0/3060631276 shutdown_connections 2024-04-25T01:52:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.558+0000 7f7d86000640 1 -- 172.21.15.102:0/3060631276 wait complete. 2024-04-25T01:52:16.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.558+0000 7f7d86000640 1 Processor -- start 2024-04-25T01:52:16.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d86000640 1 -- start start 2024-04-25T01:52:16.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d86000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d801037c0 0x7f7d80199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:16.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d86000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d801049b0 0x7f7d8019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:16.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d86000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d80105370 0x7f7d801a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:16.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d86000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8010f490 con 0x7f7d801037c0 2024-04-25T01:52:16.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d86000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8010f310 con 0x7f7d801049b0 2024-04-25T01:52:16.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d86000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8010f610 con 0x7f7d80105370 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d84c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d801037c0 0x7f7d80199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d84c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d801037c0 0x7f7d80199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59420/0 (socket says 172.21.15.102:59420) 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d84c00640 1 -- 172.21.15.102:0/2957800296 learned_addr learned my addr 172.21.15.102:0/2957800296 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d85600640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d80105370 0x7f7d801a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.559+0000 7f7d7fe00640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d801049b0 0x7f7d8019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d84c00640 1 -- 172.21.15.102:0/2957800296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d80105370 msgr2=0x7f7d801a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d84c00640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d80105370 0x7f7d801a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d84c00640 1 -- 172.21.15.102:0/2957800296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d801049b0 msgr2=0x7f7d8019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d84c00640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d801049b0 0x7f7d8019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d84c00640 1 -- 172.21.15.102:0/2957800296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d801a1760 con 0x7f7d801037c0 2024-04-25T01:52:16.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d84c00640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d801037c0 0x7f7d80199b40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400e9c0 tx=0x7f7d7400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:16.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d7d600640 1 -- 172.21.15.102:0/2957800296 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400ccc0 con 0x7f7d801037c0 2024-04-25T01:52:16.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d7d600640 1 -- 172.21.15.102:0/2957800296 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d740035f0 con 0x7f7d801037c0 2024-04-25T01:52:16.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d801a1a50 con 0x7f7d801037c0 2024-04-25T01:52:16.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d7d600640 1 -- 172.21.15.102:0/2957800296 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400ce60 con 0x7f7d801037c0 2024-04-25T01:52:16.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.560+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d801a1f20 con 0x7f7d801037c0 2024-04-25T01:52:16.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.562+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d8010aa80 con 0x7f7d801037c0 2024-04-25T01:52:16.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.563+0000 7f7d7d600640 1 -- 172.21.15.102:0/2957800296 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7d740037a0 con 0x7f7d801037c0 2024-04-25T01:52:16.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.565+0000 7f7d7d600640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d680a2b90 0x7f7d680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.566+0000 7f7d7fe00640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d680a2b90 0x7f7d680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.566+0000 7f7d7d600640 1 -- 172.21.15.102:0/2957800296 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f7d740cefa0 con 0x7f7d801037c0 2024-04-25T01:52:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.566+0000 7f7d7fe00640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d680a2b90 0x7f7d680a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c005ab0 tx=0x7f7d6c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:16.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.570+0000 7f7d7d600640 1 -- 172.21.15.102:0/2957800296 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7d7409c300 con 0x7f7d801037c0 2024-04-25T01:52:16.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.826+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d80062340 con 0x7f7d801037c0 2024-04-25T01:52:16.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.827+0000 7f7d7d600640 1 -- 172.21.15.102:0/2957800296 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4708 (secure 0 0 0) 0x7f7d7409bc00 con 0x7f7d801037c0 2024-04-25T01:52:16.828 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:16.828 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T01:52:16.828 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 6 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d680a2b90 msgr2=0x7f7d680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d680a2b90 0x7f7d680a5050 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c005ab0 tx=0x7f7d6c0315b0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d801037c0 msgr2=0x7f7d80199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d801037c0 0x7f7d80199b40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400e9c0 tx=0x7f7d7400edc0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 shutdown_connections 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7d680a2b90 0x7f7d680a5050 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d80105370 0x7f7d801a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d801049b0 0x7f7d8019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 --2- 172.21.15.102:0/2957800296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d801037c0 0x7f7d80199b40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.830+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 >> 172.21.15.102:0/2957800296 conn(0x7f7d800fed50 msgr2=0x7f7d80107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:16.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.831+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 shutdown_connections 2024-04-25T01:52:16.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:16.831+0000 7f7d86000640 1 -- 172.21.15.102:0/2957800296 wait complete. 2024-04-25T01:52:17.004 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 01:52:16 smithi193 systemd[1]: Starting Ceph mds.f for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:17.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 01:52:17 smithi193 podman[63346]: 2024-04-25 01:52:16.997557798 +0000 UTC m=+0.018240198 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:17.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 01:52:17 smithi193 podman[63346]: 2024-04-25 01:52:17.100827656 +0000 UTC m=+0.121510047 container create 899614771ba16e559701d6daf974ce753d2384acfe2a7ee478e771b7a6a9e658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-25T01:52:17.377 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 01:52:17 smithi193 podman[63346]: 2024-04-25 01:52:17.278037592 +0000 UTC m=+0.298719986 container init 899614771ba16e559701d6daf974ce753d2384acfe2a7ee478e771b7a6a9e658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T01:52:17.378 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 01:52:17 smithi193 podman[63346]: 2024-04-25 01:52:17.280584018 +0000 UTC m=+0.301266418 container start 899614771ba16e559701d6daf974ce753d2384acfe2a7ee478e771b7a6a9e658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T01:52:17.378 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-mds[63365]: -- 172.21.15.193:0/1803558705 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d58f8a7a40 con 0x55d58f87a800 2024-04-25T01:52:17.378 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f[63361]: starting mds.f at 2024-04-25T01:52:17.378 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 01:52:17 smithi193 bash[63346]: 899614771ba16e559701d6daf974ce753d2384acfe2a7ee478e771b7a6a9e658 2024-04-25T01:52:17.754 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 01:52:17 smithi193 systemd[1]: Started Ceph mds.f for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:18.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-mon[32981]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:18.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2957800296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:18.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:18.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:18.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:18.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:18.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:18.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:17 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:18.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:17 smithi102 ceph-mon[27796]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:18.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:17 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2957800296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:18.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:18.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:18.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:18.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:18.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:18.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:17 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:18.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:17 smithi110 ceph-mon[32848]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:17 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2957800296' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:18.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:17 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:18.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:19.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.061+0000 7f0d43400640 1 -- 172.21.15.102:0/205665473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c076100 msgr2=0x7f0d3c076520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.061+0000 7f0d43400640 1 --2- 172.21.15.102:0/205665473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c076100 0x7f0d3c076520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0d20009900 tx=0x7f0d2002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.061+0000 7f0d43400640 1 -- 172.21.15.102:0/205665473 shutdown_connections 2024-04-25T01:52:19.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.061+0000 7f0d43400640 1 --2- 172.21.15.102:0/205665473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d3c077ce0 0x7f0d3c078580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.061+0000 7f0d43400640 1 --2- 172.21.15.102:0/205665473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c077340 0x7f0d3c0777a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.061+0000 7f0d43400640 1 --2- 172.21.15.102:0/205665473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c076100 0x7f0d3c076520 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.062+0000 7f0d43400640 1 -- 172.21.15.102:0/205665473 >> 172.21.15.102:0/205665473 conn(0x7f0d3c0fef30 msgr2=0x7f0d3c101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:19.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.062+0000 7f0d43400640 1 -- 172.21.15.102:0/205665473 shutdown_connections 2024-04-25T01:52:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.062+0000 7f0d43400640 1 -- 172.21.15.102:0/205665473 wait complete. 2024-04-25T01:52:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.063+0000 7f0d43400640 1 Processor -- start 2024-04-25T01:52:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.063+0000 7f0d43400640 1 -- start start 2024-04-25T01:52:19.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.063+0000 7f0d43400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c076100 0x7f0d3c19e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:19.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.063+0000 7f0d43400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c077340 0x7f0d3c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:19.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.063+0000 7f0d43400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d3c077ce0 0x7f0d3c1a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:19.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d43400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c113ab0 con 0x7f0d3c076100 2024-04-25T01:52:19.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d43400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c113930 con 0x7f0d3c077ce0 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d43400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c113c30 con 0x7f0d3c077340 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d40a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c076100 0x7f0d3c19e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d40a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c076100 0x7f0d3c19e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59440/0 (socket says 172.21.15.102:59440) 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d40a00640 1 -- 172.21.15.102:0/3336340709 learned_addr learned my addr 172.21.15.102:0/3336340709 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d33e00640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c077340 0x7f0d3c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d41400640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d3c077ce0 0x7f0d3c1a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d40a00640 1 -- 172.21.15.102:0/3336340709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c077340 msgr2=0x7f0d3c19e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d40a00640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c077340 0x7f0d3c19e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d40a00640 1 -- 172.21.15.102:0/3336340709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d3c077ce0 msgr2=0x7f0d3c1a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d40a00640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d3c077ce0 0x7f0d3c1a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d40a00640 1 -- 172.21.15.102:0/3336340709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c1a5d50 con 0x7f0d3c076100 2024-04-25T01:52:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.064+0000 7f0d40a00640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c076100 0x7f0d3c19e130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0d20009230 tx=0x7f0d20031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.065+0000 7f0d3a000640 1 -- 172.21.15.102:0/3336340709 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2003f030 con 0x7f0d3c076100 2024-04-25T01:52:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.065+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3c1a5fe0 con 0x7f0d3c076100 2024-04-25T01:52:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.065+0000 7f0d3a000640 1 -- 172.21.15.102:0/3336340709 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d20003ad0 con 0x7f0d3c076100 2024-04-25T01:52:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.065+0000 7f0d3a000640 1 -- 172.21.15.102:0/3336340709 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d20035070 con 0x7f0d3c076100 2024-04-25T01:52:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.065+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d3c1a6460 con 0x7f0d3c076100 2024-04-25T01:52:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.066+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d3c1104e0 con 0x7f0d3c076100 2024-04-25T01:52:19.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.068+0000 7f0d3a000640 1 -- 172.21.15.102:0/3336340709 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0d20042030 con 0x7f0d3c076100 2024-04-25T01:52:19.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.070+0000 7f0d3a000640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d180a2b90 0x7f0d180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:19.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.070+0000 7f0d3a000640 1 -- 172.21.15.102:0/3336340709 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f0d200ef150 con 0x7f0d3c076100 2024-04-25T01:52:19.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.070+0000 7f0d33e00640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d180a2b90 0x7f0d180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:19.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.071+0000 7f0d33e00640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d180a2b90 0x7f0d180a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c19f640 tx=0x7f0d28009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:19.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.075+0000 7f0d3a000640 1 -- 172.21.15.102:0/3336340709 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0d200bc4b0 con 0x7f0d3c076100 2024-04-25T01:52:19.092 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:18 smithi193 ceph-mon[32981]: Deploying daemon mds.c on smithi193 2024-04-25T01:52:19.092 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:18 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.193:6834/1885428891,v1:172.21.15.193:6835/1885428891] up:boot 2024-04-25T01:52:19.092 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:18 smithi193 ceph-mon[32981]: fsmap 6 up:standby 2024-04-25T01:52:19.092 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:18 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:52:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:18 smithi102 ceph-mon[27796]: Deploying daemon mds.c on smithi193 2024-04-25T01:52:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:18 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.193:6834/1885428891,v1:172.21.15.193:6835/1885428891] up:boot 2024-04-25T01:52:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:18 smithi102 ceph-mon[27796]: fsmap 6 up:standby 2024-04-25T01:52:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:18 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:52:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:18 smithi110 ceph-mon[32848]: Deploying daemon mds.c on smithi193 2024-04-25T01:52:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:18 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.193:6834/1885428891,v1:172.21.15.193:6835/1885428891] up:boot 2024-04-25T01:52:19.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:18 smithi110 ceph-mon[32848]: fsmap 6 up:standby 2024-04-25T01:52:19.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:18 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:52:19.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.309+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d3c02d170 con 0x7f0d3c076100 2024-04-25T01:52:19.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.310+0000 7f0d3a000640 1 -- 172.21.15.102:0/3336340709 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5536 (secure 0 0 0) 0x7f0d200bbdb0 con 0x7f0d3c076100 2024-04-25T01:52:19.310 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:19.310 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T01:52:19.311 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 7 2024-04-25T01:52:19.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d180a2b90 msgr2=0x7f0d180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d180a2b90 0x7f0d180a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c19f640 tx=0x7f0d28009110 comp rx=0 tx=0).stop 2024-04-25T01:52:19.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c076100 msgr2=0x7f0d3c19e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:19.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c076100 0x7f0d3c19e130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0d20009230 tx=0x7f0d20031b60 comp rx=0 tx=0).stop 2024-04-25T01:52:19.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 shutdown_connections 2024-04-25T01:52:19.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0d180a2b90 0x7f0d180a5050 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d3c077ce0 0x7f0d3c1a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d3c077340 0x7f0d3c19e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 --2- 172.21.15.102:0/3336340709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c076100 0x7f0d3c19e130 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:19.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.312+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 >> 172.21.15.102:0/3336340709 conn(0x7f0d3c0fef30 msgr2=0x7f0d3c100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:19.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.313+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 shutdown_connections 2024-04-25T01:52:19.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:19.313+0000 7f0d43400640 1 -- 172.21.15.102:0/3336340709 wait complete. 2024-04-25T01:52:19.754 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 01:52:19 smithi193 systemd[1]: Starting Ceph mds.c for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:20.063 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:19 smithi193 ceph-mon[32981]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:20.063 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:19 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3336340709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:20.064 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 01:52:19 smithi193 podman[63652]: 2024-04-25 01:52:19.708003846 +0000 UTC m=+0.017841312 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:20.064 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 01:52:19 smithi193 podman[63652]: 2024-04-25 01:52:19.820060902 +0000 UTC m=+0.129898350 container create ae9e095acf8cfe39a80327b3da3abe35ec13ce3e33eb4a61849111e46a266410 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux ) 2024-04-25T01:52:20.064 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 01:52:20 smithi193 podman[63652]: 2024-04-25 01:52:20.038949094 +0000 UTC m=+0.348786543 container init ae9e095acf8cfe39a80327b3da3abe35ec13ce3e33eb4a61849111e46a266410 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T01:52:20.064 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 01:52:20 smithi193 podman[63652]: 2024-04-25 01:52:20.041413216 +0000 UTC m=+0.351250667 container start ae9e095acf8cfe39a80327b3da3abe35ec13ce3e33eb4a61849111e46a266410 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T01:52:20.064 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 01:52:20 smithi193 ceph-mds[63670]: -- 172.21.15.193:0/849877099 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dd0b44fa40 con 0x55dd0b422800 2024-04-25T01:52:20.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:19 smithi102 ceph-mon[27796]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:20.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:19 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3336340709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:20.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:19 smithi110 ceph-mon[32848]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:20.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:19 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3336340709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:20.504 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 01:52:20 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c[63666]: starting mds.c at 2024-04-25T01:52:20.504 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 01:52:20 smithi193 bash[63652]: ae9e095acf8cfe39a80327b3da3abe35ec13ce3e33eb4a61849111e46a266410 2024-04-25T01:52:20.504 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 01:52:20 smithi193 systemd[1]: Started Ceph mds.c for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:21.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:21.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:21.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:21.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:21.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:21.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:21.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:21.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: Deploying daemon mds.h on smithi110 2024-04-25T01:52:21.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:21.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.193:6836/3613019309,v1:172.21.15.193:6837/3613019309] up:boot 2024-04-25T01:52:21.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: fsmap 7 up:standby 2024-04-25T01:52:21.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:21 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:52:21.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:21.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:21.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:21.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:21.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:21.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:21.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: Deploying daemon mds.h on smithi110 2024-04-25T01:52:21.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:21.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.193:6836/3613019309,v1:172.21.15.193:6837/3613019309] up:boot 2024-04-25T01:52:21.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: fsmap 7 up:standby 2024-04-25T01:52:21.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:21 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:52:21.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.546+0000 7f233e200640 1 -- 172.21.15.102:0/564844258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23381039c0 msgr2=0x7f2338103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.546+0000 7f233e200640 1 --2- 172.21.15.102:0/564844258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23381039c0 0x7f2338103de0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2328009900 tx=0x7f232802a410 comp rx=0 tx=0).stop 2024-04-25T01:52:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.546+0000 7f233e200640 1 -- 172.21.15.102:0/564844258 shutdown_connections 2024-04-25T01:52:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.546+0000 7f233e200640 1 --2- 172.21.15.102:0/564844258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2338105570 0x7f233810be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.546+0000 7f233e200640 1 --2- 172.21.15.102:0/564844258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2338104bb0 0x7f2338105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.546+0000 7f233e200640 1 --2- 172.21.15.102:0/564844258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23381039c0 0x7f2338103de0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.546+0000 7f233e200640 1 -- 172.21.15.102:0/564844258 >> 172.21.15.102:0/564844258 conn(0x7f23380fef30 msgr2=0x7f2338101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.546+0000 7f233e200640 1 -- 172.21.15.102:0/564844258 shutdown_connections 2024-04-25T01:52:21.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.546+0000 7f233e200640 1 -- 172.21.15.102:0/564844258 wait complete. 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.547+0000 7f233e200640 1 Processor -- start 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.547+0000 7f233e200640 1 -- start start 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f233e200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23381039c0 0x7f2338199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f233e200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2338104bb0 0x7f233819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f233e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2338105570 0x7f23381a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f233e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f233810f610 con 0x7f2338105570 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f233e200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f233810f490 con 0x7f2338104bb0 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f233e200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f233810f790 con 0x7f23381039c0 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f2336a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2338104bb0 0x7f233819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f2337e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2338105570 0x7f23381a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f2336a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2338104bb0 0x7f233819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58032/0 (socket says 172.21.15.102:58032) 2024-04-25T01:52:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f2336a00640 1 -- 172.21.15.102:0/96408776 learned_addr learned my addr 172.21.15.102:0/96408776 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.548+0000 7f2337400640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23381039c0 0x7f2338199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f2336a00640 1 -- 172.21.15.102:0/96408776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23381039c0 msgr2=0x7f2338199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f2336a00640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23381039c0 0x7f2338199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f2336a00640 1 -- 172.21.15.102:0/96408776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2338105570 msgr2=0x7f23381a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f2336a00640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2338105570 0x7f23381a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f2336a00640 1 -- 172.21.15.102:0/96408776 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23381a1920 con 0x7f2338104bb0 2024-04-25T01:52:21.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f2336a00640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2338104bb0 0x7f233819a250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f232000e9f0 tx=0x7f232000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:21.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f232fe00640 1 -- 172.21.15.102:0/96408776 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232000cd80 con 0x7f2338104bb0 2024-04-25T01:52:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f232fe00640 1 -- 172.21.15.102:0/96408776 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2320010910 con 0x7f2338104bb0 2024-04-25T01:52:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f232fe00640 1 -- 172.21.15.102:0/96408776 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320018990 con 0x7f2338104bb0 2024-04-25T01:52:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23381a1c10 con 0x7f2338104bb0 2024-04-25T01:52:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.549+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23381a21b0 con 0x7f2338104bb0 2024-04-25T01:52:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.550+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f233810ad60 con 0x7f2338104bb0 2024-04-25T01:52:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.552+0000 7f232fe00640 1 -- 172.21.15.102:0/96408776 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f2320010ac0 con 0x7f2338104bb0 2024-04-25T01:52:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.553+0000 7f232fe00640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f231c0a2c10 0x7f231c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:21.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.554+0000 7f232fe00640 1 -- 172.21.15.102:0/96408776 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f23200ceff0 con 0x7f2338104bb0 2024-04-25T01:52:21.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.554+0000 7f2337400640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f231c0a2c10 0x7f231c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:21.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.554+0000 7f2337400640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f231c0a2c10 0x7f231c0a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f23280314e0 tx=0x7f2328000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:21.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.558+0000 7f232fe00640 1 -- 172.21.15.102:0/96408776 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f232009c350 con 0x7f2338104bb0 2024-04-25T01:52:21.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:21.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:21.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:21.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:21.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:21.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:21.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: Deploying daemon mds.h on smithi110 2024-04-25T01:52:21.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:21.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.193:6836/3613019309,v1:172.21.15.193:6837/3613019309] up:boot 2024-04-25T01:52:21.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: fsmap 7 up:standby 2024-04-25T01:52:21.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:21 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:52:21.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.796+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2338062340 con 0x7f2338104bb0 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.797+0000 7f232fe00640 1 -- 172.21.15.102:0/96408776 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6364 (secure 0 0 0) 0x7f232009bc50 con 0x7f2338104bb0 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:21.796 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T01:52:21.797 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-04-25T01:52:21.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f231c0a2c10 msgr2=0x7f231c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f231c0a2c10 0x7f231c0a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f23280314e0 tx=0x7f2328000bc0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2338104bb0 msgr2=0x7f233819a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:21.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2338104bb0 0x7f233819a250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f232000e9f0 tx=0x7f232000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 shutdown_connections 2024-04-25T01:52:21.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f231c0a2c10 0x7f231c0a50d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2338105570 0x7f23381a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2338104bb0 0x7f233819a250 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 --2- 172.21.15.102:0/96408776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23381039c0 0x7f2338199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:21.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 >> 172.21.15.102:0/96408776 conn(0x7f23380fef30 msgr2=0x7f2338107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:21.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 shutdown_connections 2024-04-25T01:52:21.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:21.800+0000 7f233e200640 1 -- 172.21.15.102:0/96408776 wait complete. 2024-04-25T01:52:22.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:22 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/96408776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.487 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 01:52:22 smithi110 systemd[1]: Starting Ceph mds.h for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:22.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:22 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/96408776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:22 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/96408776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:22.768 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 01:52:22 smithi110 podman[63783]: 2024-04-25 01:52:22.422041173 +0000 UTC m=+0.018004508 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:22.769 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 01:52:22 smithi110 podman[63783]: 2024-04-25 01:52:22.542553382 +0000 UTC m=+0.138516758 container create 57e29242caf0576f53b9952ef158d3d1a934b390bf09de917d1add751325e3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T01:52:23.236 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 01:52:22 smithi110 podman[63783]: 2024-04-25 01:52:22.770108453 +0000 UTC m=+0.366071788 container init 57e29242caf0576f53b9952ef158d3d1a934b390bf09de917d1add751325e3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-25T01:52:23.236 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 01:52:22 smithi110 podman[63783]: 2024-04-25 01:52:22.772581958 +0000 UTC m=+0.368545295 container start 57e29242caf0576f53b9952ef158d3d1a934b390bf09de917d1add751325e3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9) 2024-04-25T01:52:23.237 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 01:52:22 smithi110 ceph-mds[63803]: -- 172.21.15.110:0/3765389494 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5640ffe41a40 con 0x5640ffe15000 2024-04-25T01:52:23.237 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 01:52:22 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h[63799]: starting mds.h at 2024-04-25T01:52:23.237 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 01:52:22 smithi110 bash[63783]: 57e29242caf0576f53b9952ef158d3d1a934b390bf09de917d1add751325e3d0 2024-04-25T01:52:23.237 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 01:52:22 smithi110 systemd[1]: Started Ceph mds.h for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:23.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:23 smithi193 ceph-mon[32981]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:23.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:23 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:23.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:23 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:23.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:23 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:23.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:23 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:23.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:23 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:23.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:23 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:23.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:23 smithi102 ceph-mon[27796]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:23.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:23.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:23.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:23.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:23.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:23.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:23 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:23.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:23 smithi110 ceph-mon[32848]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:23.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:23 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:23.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:23 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:23.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:23 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:23.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:23 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:23.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:23 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:23.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:23 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:23.823 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:24.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.033+0000 7f1b6aa00640 1 -- 172.21.15.102:0/1567233579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 msgr2=0x7f1b640777a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.033+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/1567233579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 0x7f1b640777a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009900 tx=0x7f1b4c02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.033+0000 7f1b6aa00640 1 -- 172.21.15.102:0/1567233579 shutdown_connections 2024-04-25T01:52:24.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.033+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/1567233579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b64077ce0 0x7f1b64078580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.033+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/1567233579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 0x7f1b640777a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.033+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/1567233579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b64076100 0x7f1b64076520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.033+0000 7f1b6aa00640 1 -- 172.21.15.102:0/1567233579 >> 172.21.15.102:0/1567233579 conn(0x7f1b640fef30 msgr2=0x7f1b64101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:24.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.033+0000 7f1b6aa00640 1 -- 172.21.15.102:0/1567233579 shutdown_connections 2024-04-25T01:52:24.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.033+0000 7f1b6aa00640 1 -- 172.21.15.102:0/1567233579 wait complete. 2024-04-25T01:52:24.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.034+0000 7f1b6aa00640 1 Processor -- start 2024-04-25T01:52:24.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.034+0000 7f1b6aa00640 1 -- start start 2024-04-25T01:52:24.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b6aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b64076100 0x7f1b6419e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:24.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b6aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 0x7f1b6419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:24.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b6aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b64077ce0 0x7f1b641a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:24.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b6aa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1b64113a50 con 0x7f1b64077340 2024-04-25T01:52:24.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b6aa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1b641138d0 con 0x7f1b64077ce0 2024-04-25T01:52:24.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b6aa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1b64113bd0 con 0x7f1b64076100 2024-04-25T01:52:24.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b64076100 0x7f1b6419e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:24.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 0x7f1b6419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:24.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 0x7f1b6419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41088/0 (socket says 172.21.15.102:41088) 2024-04-25T01:52:24.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63400640 1 -- 172.21.15.102:0/2466215955 learned_addr learned my addr 172.21.15.102:0/2466215955 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:24.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b64076100 0x7f1b6419e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52292/0 (socket says 172.21.15.102:52292) 2024-04-25T01:52:24.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b68a00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b64077ce0 0x7f1b641a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:24.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63400640 1 -- 172.21.15.102:0/2466215955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b64076100 msgr2=0x7f1b6419e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63400640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b64076100 0x7f1b6419e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63400640 1 -- 172.21.15.102:0/2466215955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b64077ce0 msgr2=0x7f1b641a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63400640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b64077ce0 0x7f1b641a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.035+0000 7f1b63400640 1 -- 172.21.15.102:0/2466215955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b641a5cd0 con 0x7f1b64077340 2024-04-25T01:52:24.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.036+0000 7f1b63400640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 0x7f1b6419e5f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009230 tx=0x7f1b4c031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:24.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.036+0000 7f1b60c00640 1 -- 172.21.15.102:0/2466215955 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c03f030 con 0x7f1b64077340 2024-04-25T01:52:24.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.036+0000 7f1b60c00640 1 -- 172.21.15.102:0/2466215955 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b4c003ad0 con 0x7f1b64077340 2024-04-25T01:52:24.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.036+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b641a5f60 con 0x7f1b64077340 2024-04-25T01:52:24.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.036+0000 7f1b60c00640 1 -- 172.21.15.102:0/2466215955 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c035070 con 0x7f1b64077340 2024-04-25T01:52:24.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.036+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b641a63c0 con 0x7f1b64077340 2024-04-25T01:52:24.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.037+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b641104e0 con 0x7f1b64077340 2024-04-25T01:52:24.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.039+0000 7f1b60c00640 1 -- 172.21.15.102:0/2466215955 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1b4c042030 con 0x7f1b64077340 2024-04-25T01:52:24.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.041+0000 7f1b60c00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1b480a2b90 0x7f1b480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:24.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.041+0000 7f1b60c00640 1 -- 172.21.15.102:0/2466215955 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f1b4c0ef120 con 0x7f1b64077340 2024-04-25T01:52:24.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.041+0000 7f1b63e00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1b480a2b90 0x7f1b480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:24.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.042+0000 7f1b63e00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1b480a2b90 0x7f1b480a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b54005fd0 tx=0x7f1b54009760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:24.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.045+0000 7f1b60c00640 1 -- 172.21.15.102:0/2466215955 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b4c0bbe30 con 0x7f1b64077340 2024-04-25T01:52:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.284+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b640623e0 con 0x7f1b64077340 2024-04-25T01:52:24.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.285+0000 7f1b60c00640 1 -- 172.21.15.102:0/2466215955 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7192 (secure 0 0 0) 0x7f1b4c0c0130 con 0x7f1b64077340 2024-04-25T01:52:24.284 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:24.284 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T01:52:24.284 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 9 2024-04-25T01:52:24.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.287+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1b480a2b90 msgr2=0x7f1b480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.287+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1b480a2b90 0x7f1b480a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b54005fd0 tx=0x7f1b54009760 comp rx=0 tx=0).stop 2024-04-25T01:52:24.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.287+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 msgr2=0x7f1b6419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.287+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 0x7f1b6419e5f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009230 tx=0x7f1b4c031b60 comp rx=0 tx=0).stop 2024-04-25T01:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.288+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 shutdown_connections 2024-04-25T01:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.288+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1b480a2b90 0x7f1b480a5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.288+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b64077ce0 0x7f1b641a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.288+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b64077340 0x7f1b6419e5f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.288+0000 7f1b6aa00640 1 --2- 172.21.15.102:0/2466215955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b64076100 0x7f1b6419e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.288+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 >> 172.21.15.102:0/2466215955 conn(0x7f1b640fef30 msgr2=0x7f1b64100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.288+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 shutdown_connections 2024-04-25T01:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:24.288+0000 7f1b6aa00640 1 -- 172.21.15.102:0/2466215955 wait complete. 2024-04-25T01:52:24.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:24 smithi193 ceph-mon[32981]: Deploying daemon mds.i on smithi193 2024-04-25T01:52:24.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:24 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] up:boot 2024-04-25T01:52:24.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:24 smithi193 ceph-mon[32981]: fsmap 8 up:standby 2024-04-25T01:52:24.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:24 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:52:24.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:24 smithi102 ceph-mon[27796]: Deploying daemon mds.i on smithi193 2024-04-25T01:52:24.672 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:24 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] up:boot 2024-04-25T01:52:24.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:24 smithi102 ceph-mon[27796]: fsmap 8 up:standby 2024-04-25T01:52:24.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:24 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:52:24.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:24 smithi110 ceph-mon[32848]: Deploying daemon mds.i on smithi193 2024-04-25T01:52:24.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:24 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] up:boot 2024-04-25T01:52:24.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:24 smithi110 ceph-mon[32848]: fsmap 8 up:standby 2024-04-25T01:52:24.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:24 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:52:25.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:25 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2466215955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:25.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:25 smithi102 ceph-mon[27796]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:25.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:25 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2466215955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:25.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:25 smithi193 ceph-mon[32981]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:25.506 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 01:52:25 smithi193 systemd[1]: Starting Ceph mds.i for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:25.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:25 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2466215955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:25.736 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:25 smithi110 ceph-mon[32848]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:25.990 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 01:52:25 smithi193 podman[63959]: 2024-04-25 01:52:25.575124795 +0000 UTC m=+0.017645116 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:25.991 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 01:52:25 smithi193 podman[63959]: 2024-04-25 01:52:25.681020646 +0000 UTC m=+0.123540954 container create 8e755f7c76b79892d0a59a783c67a14cbaa5f4fb888aa4a60697d4949dd6c41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T01:52:25.991 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 01:52:25 smithi193 podman[63959]: 2024-04-25 01:52:25.891839246 +0000 UTC m=+0.334359551 container init 8e755f7c76b79892d0a59a783c67a14cbaa5f4fb888aa4a60697d4949dd6c41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-25T01:52:25.991 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 01:52:25 smithi193 podman[63959]: 2024-04-25 01:52:25.894232771 +0000 UTC m=+0.336753085 container start 8e755f7c76b79892d0a59a783c67a14cbaa5f4fb888aa4a60697d4949dd6c41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-04-25T01:52:25.991 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 01:52:25 smithi193 ceph-mds[63977]: -- 172.21.15.193:0/3607198183 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56336e663a40 con 0x56336e636c00 2024-04-25T01:52:25.991 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 01:52:25 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: starting mds.i at 2024-04-25T01:52:25.991 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 01:52:25 smithi193 bash[63959]: 8e755f7c76b79892d0a59a783c67a14cbaa5f4fb888aa4a60697d4949dd6c41b 2024-04-25T01:52:26.254 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 01:52:25 smithi193 systemd[1]: Started Ceph mds.i for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:26.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:26.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.493+0000 7f0ef2400640 1 -- 172.21.15.102:0/3430939881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0eec102980 msgr2=0x7f0eec102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.493+0000 7f0ef2400640 1 --2- 172.21.15.102:0/3430939881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0eec102980 0x7f0eec102e00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009900 tx=0x7f0ed802a410 comp rx=0 tx=0).stop 2024-04-25T01:52:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.493+0000 7f0ef2400640 1 -- 172.21.15.102:0/3430939881 shutdown_connections 2024-04-25T01:52:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.493+0000 7f0ef2400640 1 --2- 172.21.15.102:0/3430939881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0eec103340 0x7f0eec109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.493+0000 7f0ef2400640 1 --2- 172.21.15.102:0/3430939881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0eec102980 0x7f0eec102e00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.493+0000 7f0ef2400640 1 --2- 172.21.15.102:0/3430939881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0eec101790 0x7f0eec101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.493+0000 7f0ef2400640 1 -- 172.21.15.102:0/3430939881 >> 172.21.15.102:0/3430939881 conn(0x7f0eec0fcd40 msgr2=0x7f0eec0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.494+0000 7f0ef2400640 1 -- 172.21.15.102:0/3430939881 shutdown_connections 2024-04-25T01:52:26.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.494+0000 7f0ef2400640 1 -- 172.21.15.102:0/3430939881 wait complete. 2024-04-25T01:52:26.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0ef2400640 1 Processor -- start 2024-04-25T01:52:26.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0ef2400640 1 -- start start 2024-04-25T01:52:26.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0ef2400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0eec101790 0x7f0eec197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0ef2400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0eec102980 0x7f0eec197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0ef2400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0eec103340 0x7f0eec19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0ef2400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec10d460 con 0x7f0eec101790 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0ef2400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec10d2e0 con 0x7f0eec103340 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0ef2400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec10d5e0 con 0x7f0eec102980 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0eebe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0eec103340 0x7f0eec19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0eeb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0eec101790 0x7f0eec197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0eeaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0eec102980 0x7f0eec197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0eebe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0eec103340 0x7f0eec19f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32878/0 (socket says 172.21.15.102:32878) 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.495+0000 7f0eebe00640 1 -- 172.21.15.102:0/2035767248 learned_addr learned my addr 172.21.15.102:0/2035767248 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.496+0000 7f0eeaa00640 1 -- 172.21.15.102:0/2035767248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0eec103340 msgr2=0x7f0eec19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.496+0000 7f0eeaa00640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0eec103340 0x7f0eec19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.496+0000 7f0eeaa00640 1 -- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0eec101790 msgr2=0x7f0eec197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.496+0000 7f0eeaa00640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0eec101790 0x7f0eec197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.496+0000 7f0eeaa00640 1 -- 172.21.15.102:0/2035767248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eec19f6a0 con 0x7f0eec102980 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.496+0000 7f0eeb400640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0eec101790 0x7f0eec197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.496+0000 7f0eeaa00640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0eec102980 0x7f0eec197fc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ed802a930 tx=0x7f0ed8032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.497+0000 7f0ee3e00640 1 -- 172.21.15.102:0/2035767248 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed803f030 con 0x7f0eec102980 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.497+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eec19f930 con 0x7f0eec102980 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.497+0000 7f0ee3e00640 1 -- 172.21.15.102:0/2035767248 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ed8003ce0 con 0x7f0eec102980 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.497+0000 7f0ee3e00640 1 -- 172.21.15.102:0/2035767248 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8035070 con 0x7f0eec102980 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.497+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eec19fe40 con 0x7f0eec102980 2024-04-25T01:52:26.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.498+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eec108a50 con 0x7f0eec102980 2024-04-25T01:52:26.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.501+0000 7f0ee3e00640 1 -- 172.21.15.102:0/2035767248 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0ed8042030 con 0x7f0eec102980 2024-04-25T01:52:26.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.502+0000 7f0ee3e00640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0ed00a2b90 0x7f0ed00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:26.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.502+0000 7f0ee3e00640 1 -- 172.21.15.102:0/2035767248 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f0ed80eefd0 con 0x7f0eec102980 2024-04-25T01:52:26.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.502+0000 7f0eeb400640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0ed00a2b90 0x7f0ed00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:26.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.503+0000 7f0eeb400640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0ed00a2b90 0x7f0ed00a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0eec1027e0 tx=0x7f0edc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:26.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.506+0000 7f0ee3e00640 1 -- 172.21.15.102:0/2035767248 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0ed80bbce0 con 0x7f0eec102980 2024-04-25T01:52:26.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.745+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0eec062340 con 0x7f0eec102980 2024-04-25T01:52:26.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.746+0000 7f0ee3e00640 1 -- 172.21.15.102:0/2035767248 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8022 (secure 0 0 0) 0x7f0ed80bffe0 con 0x7f0eec102980 2024-04-25T01:52:26.746 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:26.746 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:52:26.746 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0ed00a2b90 msgr2=0x7f0ed00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0ed00a2b90 0x7f0ed00a5050 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0eec1027e0 tx=0x7f0edc005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0eec102980 msgr2=0x7f0eec197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0eec102980 0x7f0eec197fc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ed802a930 tx=0x7f0ed8032ce0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 shutdown_connections 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0ed00a2b90 0x7f0ed00a5050 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0eec103340 0x7f0eec19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0eec102980 0x7f0eec197fc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 --2- 172.21.15.102:0/2035767248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0eec101790 0x7f0eec197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:26.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.749+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 >> 172.21.15.102:0/2035767248 conn(0x7f0eec0fcd40 msgr2=0x7f0eec105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:26.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.750+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 shutdown_connections 2024-04-25T01:52:26.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:26.750+0000 7f0ef2400640 1 -- 172.21.15.102:0/2035767248 wait complete. 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: Deploying daemon mds.g on smithi102 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] up:boot 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: fsmap 9 up:standby 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:52:27.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:27.321 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:27 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2035767248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:27.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: Deploying daemon mds.g on smithi102 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] up:boot 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: fsmap 9 up:standby 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:27.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:27 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2035767248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:27.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: Deploying daemon mds.g on smithi102 2024-04-25T01:52:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] up:boot 2024-04-25T01:52:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: fsmap 9 up:standby 2024-04-25T01:52:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:52:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:27.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:27 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2035767248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:28.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:28.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:28.998+0000 7fae0ae00640 1 -- 172.21.15.102:0/1273506634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae04108740 msgr2=0x7fae0410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:28.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:28.998+0000 7fae0ae00640 1 --2- 172.21.15.102:0/1273506634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae04108740 0x7fae0410ab70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009900 tx=0x7fadf402a440 comp rx=0 tx=0).stop 2024-04-25T01:52:28.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:28.998+0000 7fae0ae00640 1 -- 172.21.15.102:0/1273506634 shutdown_connections 2024-04-25T01:52:28.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:28.998+0000 7fae0ae00640 1 --2- 172.21.15.102:0/1273506634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae0410b530 0x7fae041143d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:28.998+0000 7fae0ae00640 1 --2- 172.21.15.102:0/1273506634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae0410b0b0 0x7fae04111e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:28.998+0000 7fae0ae00640 1 --2- 172.21.15.102:0/1273506634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae04108740 0x7fae0410ab70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:28.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:28.998+0000 7fae0ae00640 1 -- 172.21.15.102:0/1273506634 >> 172.21.15.102:0/1273506634 conn(0x7fae0406e5e0 msgr2=0x7fae04070a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:28.999+0000 7fae0ae00640 1 -- 172.21.15.102:0/1273506634 shutdown_connections 2024-04-25T01:52:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:28.999+0000 7fae0ae00640 1 -- 172.21.15.102:0/1273506634 wait complete. 2024-04-25T01:52:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.000+0000 7fae0ae00640 1 Processor -- start 2024-04-25T01:52:29.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.000+0000 7fae0ae00640 1 -- start start 2024-04-25T01:52:29.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.001+0000 7fae0ae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae04108740 0x7fae041a2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:29.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.001+0000 7fae0ae00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae0410b0b0 0x7fae041a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:29.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.001+0000 7fae0ae00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae0410b530 0x7fae041a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:29.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.001+0000 7fae0ae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fae04117ee0 con 0x7fae04108740 2024-04-25T01:52:29.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.001+0000 7fae0ae00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae04117d60 con 0x7fae0410b0b0 2024-04-25T01:52:29.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.001+0000 7fae0ae00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fae04118060 con 0x7fae0410b530 2024-04-25T01:52:29.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.002+0000 7fae03400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae0410b0b0 0x7fae041a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:29.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.002+0000 7fae03e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae04108740 0x7fae041a2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:29.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.002+0000 7fae03400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae0410b0b0 0x7fae041a2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32888/0 (socket says 172.21.15.102:32888) 2024-04-25T01:52:29.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.002+0000 7fae03400640 1 -- 172.21.15.102:0/2077583756 learned_addr learned my addr 172.21.15.102:0/2077583756 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:29.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.002+0000 7fae08e00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae0410b530 0x7fae041a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:29.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.003+0000 7fae03400640 1 -- 172.21.15.102:0/2077583756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae0410b530 msgr2=0x7fae041a9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:29.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.003+0000 7fae03400640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae0410b530 0x7fae041a9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.003+0000 7fae03400640 1 -- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae04108740 msgr2=0x7fae041a2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.003+0000 7fae03400640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae04108740 0x7fae041a2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.003+0000 7fae03400640 1 -- 172.21.15.102:0/2077583756 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae041aa1b0 con 0x7fae0410b0b0 2024-04-25T01:52:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.003+0000 7fae03e00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae04108740 0x7fae041a2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.003+0000 7fae03400640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae0410b0b0 0x7fae041a2ad0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b470 tx=0x7fadf000b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.004+0000 7fae00c00640 1 -- 172.21.15.102:0/2077583756 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0015020 con 0x7fae0410b0b0 2024-04-25T01:52:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.004+0000 7fae00c00640 1 -- 172.21.15.102:0/2077583756 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadf0011d10 con 0x7fae0410b0b0 2024-04-25T01:52:29.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.004+0000 7fae00c00640 1 -- 172.21.15.102:0/2077583756 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf000f2e0 con 0x7fae0410b0b0 2024-04-25T01:52:29.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.004+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae041aa4a0 con 0x7fae0410b0b0 2024-04-25T01:52:29.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.004+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae041aa9a0 con 0x7fae0410b0b0 2024-04-25T01:52:29.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.005+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae04113590 con 0x7fae0410b0b0 2024-04-25T01:52:29.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.009+0000 7fae00c00640 1 -- 172.21.15.102:0/2077583756 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fadf0018020 con 0x7fae0410b0b0 2024-04-25T01:52:29.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.010+0000 7fae00c00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fade80a2c10 0x7fade80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:29.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.011+0000 7fae00c00640 1 -- 172.21.15.102:0/2077583756 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fadf00ce980 con 0x7fae0410b0b0 2024-04-25T01:52:29.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.011+0000 7fae03e00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fade80a2c10 0x7fade80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:29.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.012+0000 7fae03e00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fade80a2c10 0x7fade80a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005ab0 tx=0x7fadf4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:29.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.012+0000 7fae00c00640 1 -- 172.21.15.102:0/2077583756 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fadf001f020 con 0x7fae0410b0b0 2024-04-25T01:52:29.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.261+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae0402d170 con 0x7fae0410b0b0 2024-04-25T01:52:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.262+0000 7fae00c00640 1 -- 172.21.15.102:0/2077583756 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8022 (secure 0 0 0) 0x7fadf009bce0 con 0x7fae0410b0b0 2024-04-25T01:52:29.262 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:29.262 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:52:29.263 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 10 2024-04-25T01:52:29.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fade80a2c10 msgr2=0x7fade80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:29.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fade80a2c10 0x7fade80a50d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005ab0 tx=0x7fadf4005a20 comp rx=0 tx=0).stop 2024-04-25T01:52:29.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae0410b0b0 msgr2=0x7fae041a2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:29.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae0410b0b0 0x7fae041a2ad0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b470 tx=0x7fadf000b870 comp rx=0 tx=0).stop 2024-04-25T01:52:29.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 shutdown_connections 2024-04-25T01:52:29.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fade80a2c10 0x7fade80a50d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae0410b530 0x7fae041a9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae0410b0b0 0x7fae041a2ad0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 --2- 172.21.15.102:0/2077583756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae04108740 0x7fae041a2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:29.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 >> 172.21.15.102:0/2077583756 conn(0x7fae0406e5e0 msgr2=0x7fae040707f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:29.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.264+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 shutdown_connections 2024-04-25T01:52:29.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:29.265+0000 7fae0ae00640 1 -- 172.21.15.102:0/2077583756 wait complete. 2024-04-25T01:52:29.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:29 smithi102 ceph-mon[27796]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:29.877 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:29 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2077583756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:30.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:29 smithi110 ceph-mon[32848]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:30.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:29 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2077583756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:30.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:29 smithi193 ceph-mon[32981]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:30.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:29 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2077583756' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:30.423 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 25 01:52:30 smithi102 systemd[1]: Started Ceph mds.g for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:31.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:31.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:31.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:31.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:31.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:31.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:31.445 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:31.446 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: Deploying daemon mds.a on smithi102 2024-04-25T01:52:31.446 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:31.446 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.102:6838/4134663811,v1:172.21.15.102:6839/4134663811] up:boot 2024-04-25T01:52:31.446 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: fsmap 10 up:standby 2024-04-25T01:52:31.446 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:31 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:52:31.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: Deploying daemon mds.a on smithi102 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.102:6838/4134663811,v1:172.21.15.102:6839/4134663811] up:boot 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: fsmap 10 up:standby 2024-04-25T01:52:31.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:31 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:52:31.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:31.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:31.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:31.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:31.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:31.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:31.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: Deploying daemon mds.a on smithi102 2024-04-25T01:52:31.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:31.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.102:6838/4134663811,v1:172.21.15.102:6839/4134663811] up:boot 2024-04-25T01:52:31.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: fsmap 10 up:standby 2024-04-25T01:52:31.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:31 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:52:31.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.508+0000 7f3fee000640 1 -- 172.21.15.102:0/2877838915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe81039c0 msgr2=0x7f3fe8103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.508+0000 7f3fee000640 1 --2- 172.21.15.102:0/2877838915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe81039c0 0x7f3fe8103de0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009900 tx=0x7f3fd802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.508+0000 7f3fee000640 1 -- 172.21.15.102:0/2877838915 shutdown_connections 2024-04-25T01:52:31.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.508+0000 7f3fee000640 1 --2- 172.21.15.102:0/2877838915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fe8105570 0x7f3fe810be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.508+0000 7f3fee000640 1 --2- 172.21.15.102:0/2877838915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fe8104bb0 0x7f3fe8105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.508+0000 7f3fee000640 1 --2- 172.21.15.102:0/2877838915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe81039c0 0x7f3fe8103de0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.508+0000 7f3fee000640 1 -- 172.21.15.102:0/2877838915 >> 172.21.15.102:0/2877838915 conn(0x7f3fe80fef30 msgr2=0x7f3fe8101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:31.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.509+0000 7f3fee000640 1 -- 172.21.15.102:0/2877838915 shutdown_connections 2024-04-25T01:52:31.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.509+0000 7f3fee000640 1 -- 172.21.15.102:0/2877838915 wait complete. 2024-04-25T01:52:31.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.510+0000 7f3fee000640 1 Processor -- start 2024-04-25T01:52:31.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.510+0000 7f3fee000640 1 -- start start 2024-04-25T01:52:31.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.510+0000 7f3fee000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fe81039c0 0x7f3fe8199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.510+0000 7f3fee000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fe8104bb0 0x7f3fe819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.510+0000 7f3fee000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe8105570 0x7f3fe81a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.510+0000 7f3fee000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe810f680 con 0x7f3fe8104bb0 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.510+0000 7f3fee000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe810f500 con 0x7f3fe81039c0 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.510+0000 7f3fee000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe810f800 con 0x7f3fe8105570 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fe81039c0 0x7f3fe8199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fdea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fe8104bb0 0x7f3fe819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe8105570 0x7f3fe81a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fe81039c0 0x7f3fe8199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32908/0 (socket says 172.21.15.102:32908) 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7400640 1 -- 172.21.15.102:0/2801913380 learned_addr learned my addr 172.21.15.102:0/2801913380 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7400640 1 -- 172.21.15.102:0/2801913380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe8105570 msgr2=0x7f3fe81a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7400640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe8105570 0x7f3fe81a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7400640 1 -- 172.21.15.102:0/2801913380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fe8104bb0 msgr2=0x7f3fe819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7400640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fe8104bb0 0x7f3fe819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7400640 1 -- 172.21.15.102:0/2801913380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe81a1920 con 0x7f3fe81039c0 2024-04-25T01:52:31.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7e00640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe8105570 0x7f3fe81a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:31.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.511+0000 7f3fe7400640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fe81039c0 0x7f3fe8199d10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8031440 tx=0x7f3fd8032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:31.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.512+0000 7f3fe4c00640 1 -- 172.21.15.102:0/2801913380 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd803f030 con 0x7f3fe81039c0 2024-04-25T01:52:31.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.512+0000 7f3fe4c00640 1 -- 172.21.15.102:0/2801913380 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fd8003ce0 con 0x7f3fe81039c0 2024-04-25T01:52:31.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.512+0000 7f3fe4c00640 1 -- 172.21.15.102:0/2801913380 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8035070 con 0x7f3fe81039c0 2024-04-25T01:52:31.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.512+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe81a1bb0 con 0x7f3fe81039c0 2024-04-25T01:52:31.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.512+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe81a2050 con 0x7f3fe81039c0 2024-04-25T01:52:31.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.513+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fe810ad30 con 0x7f3fe81039c0 2024-04-25T01:52:31.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.515+0000 7f3fe4c00640 1 -- 172.21.15.102:0/2801913380 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3fd8042030 con 0x7f3fe81039c0 2024-04-25T01:52:31.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.517+0000 7f3fe4c00640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3fcc0a2c10 0x7f3fcc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:31.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.517+0000 7f3fe4c00640 1 -- 172.21.15.102:0/2801913380 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f3fd80ef480 con 0x7f3fe81039c0 2024-04-25T01:52:31.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.517+0000 7f3fdea00640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3fcc0a2c10 0x7f3fcc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:31.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.518+0000 7f3fdea00640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3fcc0a2c10 0x7f3fcc0a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009bd0 tx=0x7f3fd0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:31.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.519+0000 7f3fe4c00640 1 -- 172.21.15.102:0/2801913380 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3fd80bc2b0 con 0x7f3fe81039c0 2024-04-25T01:52:31.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.767+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fe802d170 con 0x7f3fe81039c0 2024-04-25T01:52:31.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.769+0000 7f3fe4c00640 1 -- 172.21.15.102:0/2801913380 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8851 (secure 0 0 0) 0x7f3fd80c05b0 con 0x7f3fe81039c0 2024-04-25T01:52:31.769 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:31.769 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:52:31.769 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 11 2024-04-25T01:52:31.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3fcc0a2c10 msgr2=0x7f3fcc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3fcc0a2c10 0x7f3fcc0a50d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009bd0 tx=0x7f3fd0005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:31.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fe81039c0 msgr2=0x7f3fe8199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:31.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fe81039c0 0x7f3fe8199d10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8031440 tx=0x7f3fd8032ce0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 shutdown_connections 2024-04-25T01:52:31.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f3fcc0a2c10 0x7f3fcc0a50d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe8105570 0x7f3fe81a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3fe8104bb0 0x7f3fe819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 --2- 172.21.15.102:0/2801913380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fe81039c0 0x7f3fe8199d10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:31.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 >> 172.21.15.102:0/2801913380 conn(0x7f3fe80fef30 msgr2=0x7f3fe8107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:31.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 shutdown_connections 2024-04-25T01:52:31.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:31.772+0000 7f3fee000640 1 -- 172.21.15.102:0/2801913380 wait complete. 2024-04-25T01:52:32.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:32 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2801913380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:32.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:32 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2801913380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:32.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:32 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2801913380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:33.405 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:33 smithi102 ceph-mon[27796]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:33.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:33 smithi110 ceph-mon[32848]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:33.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:33 smithi193 ceph-mon[32981]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:33.672 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 01:52:33 smithi102 systemd[1]: Starting Ceph mds.a for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:33.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.008+0000 7fd017600640 1 -- 172.21.15.102:0/3073634863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010105550 msgr2=0x7fd01010bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.008+0000 7fd017600640 1 --2- 172.21.15.102:0/3073634863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010105550 0x7fd01010bde0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd008009ca0 tx=0x7fd00802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.008+0000 7fd017600640 1 -- 172.21.15.102:0/3073634863 shutdown_connections 2024-04-25T01:52:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.008+0000 7fd017600640 1 --2- 172.21.15.102:0/3073634863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010105550 0x7fd01010bde0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.008+0000 7fd017600640 1 --2- 172.21.15.102:0/3073634863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd010104b90 0x7fd010105010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.008+0000 7fd017600640 1 --2- 172.21.15.102:0/3073634863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0101039a0 0x7fd010103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.008+0000 7fd017600640 1 -- 172.21.15.102:0/3073634863 >> 172.21.15.102:0/3073634863 conn(0x7fd0100fef30 msgr2=0x7fd010101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.009+0000 7fd017600640 1 -- 172.21.15.102:0/3073634863 shutdown_connections 2024-04-25T01:52:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.009+0000 7fd017600640 1 -- 172.21.15.102:0/3073634863 wait complete. 2024-04-25T01:52:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.009+0000 7fd017600640 1 Processor -- start 2024-04-25T01:52:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd017600640 1 -- start start 2024-04-25T01:52:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd017600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0101039a0 0x7fd010199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd017600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010104b90 0x7fd01019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd017600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd010105550 0x7fd0101a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd017600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd01010f600 con 0x7fd010104b90 2024-04-25T01:52:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd017600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd01010f480 con 0x7fd0101039a0 2024-04-25T01:52:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd017600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd01010f780 con 0x7fd010105550 2024-04-25T01:52:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd00fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010104b90 0x7fd01019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd00fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010104b90 0x7fd01019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43210/0 (socket says 172.21.15.102:43210) 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd00fe00640 1 -- 172.21.15.102:0/1407338954 learned_addr learned my addr 172.21.15.102:0/1407338954 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd014c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0101039a0 0x7fd010199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd015600640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd010105550 0x7fd0101a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.010+0000 7fd00fe00640 1 -- 172.21.15.102:0/1407338954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd010105550 msgr2=0x7fd0101a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd00fe00640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd010105550 0x7fd0101a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd00fe00640 1 -- 172.21.15.102:0/1407338954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0101039a0 msgr2=0x7fd010199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd00fe00640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0101039a0 0x7fd010199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd00fe00640 1 -- 172.21.15.102:0/1407338954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0101a1870 con 0x7fd010104b90 2024-04-25T01:52:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd015600640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd010105550 0x7fd0101a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd00fe00640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010104b90 0x7fd01019a1a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00e9f0 tx=0x7fcffc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd00d600640 1 -- 172.21.15.102:0/1407338954 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc00cd80 con 0x7fd010104b90 2024-04-25T01:52:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd00d600640 1 -- 172.21.15.102:0/1407338954 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcffc010910 con 0x7fd010104b90 2024-04-25T01:52:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd00d600640 1 -- 172.21.15.102:0/1407338954 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc018990 con 0x7fd010104b90 2024-04-25T01:52:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0101a1b60 con 0x7fd010104b90 2024-04-25T01:52:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.011+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0101a2060 con 0x7fd010104b90 2024-04-25T01:52:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.012+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd01010acc0 con 0x7fd010104b90 2024-04-25T01:52:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.014+0000 7fd00d600640 1 -- 172.21.15.102:0/1407338954 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fcffc010ac0 con 0x7fd010104b90 2024-04-25T01:52:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.016+0000 7fd00d600640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcff80a2c10 0x7fcff80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:34.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.016+0000 7fd00d600640 1 -- 172.21.15.102:0/1407338954 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fcffc0ceff0 con 0x7fd010104b90 2024-04-25T01:52:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.016+0000 7fd014c00640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcff80a2c10 0x7fcff80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.017+0000 7fd014c00640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcff80a2c10 0x7fcff80a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd01019b130 tx=0x7fd004005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:34.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.020+0000 7fd00d600640 1 -- 172.21.15.102:0/1407338954 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcffc09c350 con 0x7fd010104b90 2024-04-25T01:52:34.087 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 01:52:33 smithi102 podman[69681]: 2024-04-25 01:52:33.678319304 +0000 UTC m=+0.035541607 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:34.087 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 01:52:33 smithi102 podman[69681]: 2024-04-25 01:52:33.792358465 +0000 UTC m=+0.149580802 container create dbf63e9e5b10d2923e70d793d29e111a626bf358382cc857b983bf79896f7317 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T01:52:34.087 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 01:52:34 smithi102 podman[69681]: 2024-04-25 01:52:34.087227699 +0000 UTC m=+0.444450053 container init dbf63e9e5b10d2923e70d793d29e111a626bf358382cc857b983bf79896f7317 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T01:52:34.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.280+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd01002d170 con 0x7fd010104b90 2024-04-25T01:52:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.282+0000 7fd00d600640 1 -- 172.21.15.102:0/1407338954 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9680 (secure 0 0 0) 0x7fcffc09bc50 con 0x7fd010104b90 2024-04-25T01:52:34.281 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:34.282 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:52:34.282 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 12 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcff80a2c10 msgr2=0x7fcff80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcff80a2c10 0x7fcff80a50d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd01019b130 tx=0x7fd004005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010104b90 msgr2=0x7fd01019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010104b90 0x7fd01019a1a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00e9f0 tx=0x7fcffc00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 shutdown_connections 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fcff80a2c10 0x7fcff80a50d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd010105550 0x7fd0101a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd010104b90 0x7fd01019a1a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 --2- 172.21.15.102:0/1407338954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0101039a0 0x7fd010199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.285+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 >> 172.21.15.102:0/1407338954 conn(0x7fd0100fef30 msgr2=0x7fd010107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.286+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 shutdown_connections 2024-04-25T01:52:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:34.286+0000 7fd017600640 1 -- 172.21.15.102:0/1407338954 wait complete. 2024-04-25T01:52:34.423 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 01:52:34 smithi102 podman[69681]: 2024-04-25 01:52:34.094368344 +0000 UTC m=+0.451590644 container start dbf63e9e5b10d2923e70d793d29e111a626bf358382cc857b983bf79896f7317 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-25T01:52:34.423 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 01:52:34 smithi102 ceph-mds[69738]: -- 172.21.15.102:0/2273577513 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f7376a3a40 con 0x55f737676800 2024-04-25T01:52:34.423 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 01:52:34 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a[69715]: starting mds.a at 2024-04-25T01:52:34.423 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 01:52:34 smithi102 bash[69681]: dbf63e9e5b10d2923e70d793d29e111a626bf358382cc857b983bf79896f7317 2024-04-25T01:52:34.423 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 01:52:34 smithi102 systemd[1]: Started Ceph mds.a for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:35.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.102:6840/3724889216,v1:172.21.15.102:6841/3724889216] up:boot 2024-04-25T01:52:35.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: fsmap 11 up:standby 2024-04-25T01:52:35.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:52:35.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1407338954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:35.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:35.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:35.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:35.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:35.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:35.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:35.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: Deploying daemon mds.b on smithi110 2024-04-25T01:52:35.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:35 smithi193 ceph-mon[32981]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:35.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.102:6840/3724889216,v1:172.21.15.102:6841/3724889216] up:boot 2024-04-25T01:52:35.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: fsmap 11 up:standby 2024-04-25T01:52:35.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:52:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1407338954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: Deploying daemon mds.b on smithi110 2024-04-25T01:52:35.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:35 smithi102 ceph-mon[27796]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.102:6840/3724889216,v1:172.21.15.102:6841/3724889216] up:boot 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: fsmap 11 up:standby 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1407338954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T01:52:35.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:35.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: Deploying daemon mds.b on smithi110 2024-04-25T01:52:35.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:35 smithi110 ceph-mon[32848]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:36.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:36.486 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:36 smithi110 systemd[1]: Starting Ceph mds.b for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T01:52:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.521+0000 7f9617e00640 1 -- 172.21.15.102:0/3683372066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9618075780 msgr2=0x7f9618073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.521+0000 7f9617e00640 1 --2- 172.21.15.102:0/3683372066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9618075780 0x7f9618073bd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9608009930 tx=0x7f960802a420 comp rx=0 tx=0).stop 2024-04-25T01:52:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.521+0000 7f9617e00640 1 -- 172.21.15.102:0/3683372066 shutdown_connections 2024-04-25T01:52:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.521+0000 7f9617e00640 1 --2- 172.21.15.102:0/3683372066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f961810db80 0x7f961810ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.521+0000 7f9617e00640 1 --2- 172.21.15.102:0/3683372066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96180741a0 0x7f9618074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.521+0000 7f9617e00640 1 --2- 172.21.15.102:0/3683372066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9618075780 0x7f9618073bd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.521+0000 7f9617e00640 1 -- 172.21.15.102:0/3683372066 >> 172.21.15.102:0/3683372066 conn(0x7f96180fcb20 msgr2=0x7f96180fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.521+0000 7f9617e00640 1 -- 172.21.15.102:0/3683372066 shutdown_connections 2024-04-25T01:52:36.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.521+0000 7f9617e00640 1 -- 172.21.15.102:0/3683372066 wait complete. 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.522+0000 7f9617e00640 1 Processor -- start 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.522+0000 7f9617e00640 1 -- start start 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96180741a0 0x7f961819df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9618075780 0x7f961819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f961810db80 0x7f96181a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9618113870 con 0x7f9618075780 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f96181136f0 con 0x7f961810db80 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f96181139f0 con 0x7f96180741a0 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f961810db80 0x7f96181a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9616000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9618075780 0x7f961819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f961810db80 0x7f96181a54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53584/0 (socket says 172.21.15.102:53584) 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617400640 1 -- 172.21.15.102:0/1507182057 learned_addr learned my addr 172.21.15.102:0/1507182057 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9616a00640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96180741a0 0x7f961819df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617400640 1 -- 172.21.15.102:0/1507182057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96180741a0 msgr2=0x7f961819df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617400640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96180741a0 0x7f961819df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617400640 1 -- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9618075780 msgr2=0x7f961819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617400640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9618075780 0x7f961819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9617400640 1 -- 172.21.15.102:0/1507182057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96181a5b40 con 0x7f961810db80 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.523+0000 7f9616000640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9618075780 0x7f961819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.524+0000 7f9617400640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f961810db80 0x7f96181a54d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f961000eb10 tx=0x7f961000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.524+0000 7f960f400640 1 -- 172.21.15.102:0/1507182057 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610004b20 con 0x7f961810db80 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.524+0000 7f960f400640 1 -- 172.21.15.102:0/1507182057 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9610004cc0 con 0x7f961810db80 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.524+0000 7f960f400640 1 -- 172.21.15.102:0/1507182057 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9610010760 con 0x7f961810db80 2024-04-25T01:52:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.524+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96181a5e30 con 0x7f961810db80 2024-04-25T01:52:36.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.524+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96181a62a0 con 0x7f961810db80 2024-04-25T01:52:36.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.525+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9618103d50 con 0x7f961810db80 2024-04-25T01:52:36.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.527+0000 7f960f400640 1 -- 172.21.15.102:0/1507182057 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9610005020 con 0x7f961810db80 2024-04-25T01:52:36.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.528+0000 7f960f400640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f95fc0a2b90 0x7f95fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:36.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.528+0000 7f960f400640 1 -- 172.21.15.102:0/1507182057 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f96100cf2e0 con 0x7f961810db80 2024-04-25T01:52:36.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.529+0000 7f9616a00640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f95fc0a2b90 0x7f95fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:36.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.529+0000 7f9616a00640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f95fc0a2b90 0x7f95fc0a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9608009900 tx=0x7f96080315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.532+0000 7f960f400640 1 -- 172.21.15.102:0/1507182057 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f961009c640 con 0x7f961810db80 2024-04-25T01:52:36.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.773+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f961802d170 con 0x7f961810db80 2024-04-25T01:52:36.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.776+0000 7f960f400640 1 -- 172.21.15.102:0/1507182057 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9680 (secure 0 0 0) 0x7f961009bf40 con 0x7f961810db80 2024-04-25T01:52:36.778 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:36.778 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T01:52:36.779 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 12 2024-04-25T01:52:36.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f95fc0a2b90 msgr2=0x7f95fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f95fc0a2b90 0x7f95fc0a5050 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9608009900 tx=0x7f96080315b0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f961810db80 msgr2=0x7f96181a54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:36.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f961810db80 0x7f96181a54d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f961000eb10 tx=0x7f961000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 shutdown_connections 2024-04-25T01:52:36.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f95fc0a2b90 0x7f95fc0a5050 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f961810db80 0x7f96181a54d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9618075780 0x7f961819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 --2- 172.21.15.102:0/1507182057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96180741a0 0x7f961819df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:36.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.779+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 >> 172.21.15.102:0/1507182057 conn(0x7f96180fcb20 msgr2=0x7f9618100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:36.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.780+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 shutdown_connections 2024-04-25T01:52:36.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:36.780+0000 7f9617e00640 1 -- 172.21.15.102:0/1507182057 wait complete. 2024-04-25T01:52:36.856 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:36 smithi110 podman[64091]: 2024-04-25 01:52:36.537948428 +0000 UTC m=+0.106031499 container create 0735d4cfb609f307069163e7dc3e02358a11b1f60f2d43ce3aa650ff24207ffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9) 2024-04-25T01:52:36.856 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:36 smithi110 podman[64091]: 2024-04-25 01:52:36.450360898 +0000 UTC m=+0.018443979 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T01:52:36.856 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:36 smithi110 podman[64091]: 2024-04-25 01:52:36.75689996 +0000 UTC m=+0.324983033 container init 0735d4cfb609f307069163e7dc3e02358a11b1f60f2d43ce3aa650ff24207ffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T01:52:36.856 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:36 smithi110 podman[64091]: 2024-04-25 01:52:36.759359495 +0000 UTC m=+0.327442569 container start 0735d4cfb609f307069163e7dc3e02358a11b1f60f2d43ce3aa650ff24207ffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T01:52:36.856 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:36 smithi110 ceph-mds[64110]: -- 172.21.15.110:0/3989253107 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5621fc997a40 con 0x5621fc96ac00 2024-04-25T01:52:36.857 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:36 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b[64106]: starting mds.b at 2024-04-25T01:52:36.857 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:36 smithi110 bash[64091]: 0735d4cfb609f307069163e7dc3e02358a11b1f60f2d43ce3aa650ff24207ffa 2024-04-25T01:52:37.156 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:36 smithi110 systemd[1]: Started Ceph mds.b for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T01:52:38.102 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:37 smithi110 ceph-mon[32848]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:38.102 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:37 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1507182057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:38.102 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.102 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.103 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.103 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.103 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.103 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:38.103 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:37 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:37 smithi102 ceph-mon[27796]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:38.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:37 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1507182057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:38.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:38.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:37 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:37 smithi193 ceph-mon[32981]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:38.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:37 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1507182057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:38.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:38.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:37 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:38.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:38.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:38 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] up:boot 2024-04-25T01:52:38.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:38 smithi102 ceph-mon[27796]: fsmap 12 up:standby 2024-04-25T01:52:38.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:38 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:52:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.002+0000 7f744a400640 1 -- 172.21.15.102:0/2197972406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 msgr2=0x7f7444103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.002+0000 7f744a400640 1 --2- 172.21.15.102:0/2197972406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 0x7f7444103de0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7434009900 tx=0x7f743402a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.002+0000 7f744a400640 1 -- 172.21.15.102:0/2197972406 shutdown_connections 2024-04-25T01:52:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.002+0000 7f744a400640 1 --2- 172.21.15.102:0/2197972406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7444105570 0x7f744410be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.002+0000 7f744a400640 1 --2- 172.21.15.102:0/2197972406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7444104bb0 0x7f7444105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.002+0000 7f744a400640 1 --2- 172.21.15.102:0/2197972406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 0x7f7444103de0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.002+0000 7f744a400640 1 -- 172.21.15.102:0/2197972406 >> 172.21.15.102:0/2197972406 conn(0x7f74440fef30 msgr2=0x7f7444101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.002+0000 7f744a400640 1 -- 172.21.15.102:0/2197972406 shutdown_connections 2024-04-25T01:52:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.003+0000 7f744a400640 1 -- 172.21.15.102:0/2197972406 wait complete. 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.004+0000 7f744a400640 1 Processor -- start 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.004+0000 7f744a400640 1 -- start start 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.004+0000 7f744a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 0x7f7444199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.004+0000 7f744a400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7444104bb0 0x7f744419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.004+0000 7f744a400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7444105570 0x7f74441a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.004+0000 7f744a400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f744410f660 con 0x7f7444104bb0 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.004+0000 7f744a400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f744410f4e0 con 0x7f7444105570 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.004+0000 7f744a400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f744410f7e0 con 0x7f74441039c0 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 0x7f7444199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 0x7f7444199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46174/0 (socket says 172.21.15.102:46174) 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443400640 1 -- 172.21.15.102:0/3077232131 learned_addr learned my addr 172.21.15.102:0/3077232131 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443e00640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7444105570 0x7f74441a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f743aa00640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7444104bb0 0x7f744419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443400640 1 -- 172.21.15.102:0/3077232131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7444105570 msgr2=0x7f74441a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443400640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7444105570 0x7f74441a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443400640 1 -- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7444104bb0 msgr2=0x7f744419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443400640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7444104bb0 0x7f744419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443400640 1 -- 172.21.15.102:0/3077232131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74441a1950 con 0x7f74441039c0 2024-04-25T01:52:39.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.005+0000 7f7443400640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 0x7f7444199d40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7434031440 tx=0x7f7434032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.008+0000 7f7440c00640 1 -- 172.21.15.102:0/3077232131 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743403f030 con 0x7f74441039c0 2024-04-25T01:52:39.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.008+0000 7f7440c00640 1 -- 172.21.15.102:0/3077232131 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7434003ce0 con 0x7f74441039c0 2024-04-25T01:52:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.008+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74441a1be0 con 0x7f74441039c0 2024-04-25T01:52:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.008+0000 7f7440c00640 1 -- 172.21.15.102:0/3077232131 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434035070 con 0x7f74441039c0 2024-04-25T01:52:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.008+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74441a2120 con 0x7f74441039c0 2024-04-25T01:52:39.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.009+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f744410ace0 con 0x7f74441039c0 2024-04-25T01:52:39.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.011+0000 7f7440c00640 1 -- 172.21.15.102:0/3077232131 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7434042030 con 0x7f74441039c0 2024-04-25T01:52:39.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.012+0000 7f7440c00640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f74280a2c10 0x7f74280a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.013+0000 7f7440c00640 1 -- 172.21.15.102:0/3077232131 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f74340eee10 con 0x7f74441039c0 2024-04-25T01:52:39.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.013+0000 7f743aa00640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f74280a2c10 0x7f74280a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.014+0000 7f743aa00640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f74280a2c10 0x7f74280a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f742c009bd0 tx=0x7f742c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:39.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.020+0000 7f7440c00640 1 -- 172.21.15.102:0/3077232131 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f74340f2030 con 0x7f74441039c0 2024-04-25T01:52:39.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:38 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] up:boot 2024-04-25T01:52:39.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:38 smithi110 ceph-mon[32848]: fsmap 12 up:standby 2024-04-25T01:52:39.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:38 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:52:39.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:38 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] up:boot 2024-04-25T01:52:39.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:38 smithi193 ceph-mon[32981]: fsmap 12 up:standby 2024-04-25T01:52:39.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:38 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:52:39.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.284+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f744402d170 con 0x7f74441039c0 2024-04-25T01:52:39.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.285+0000 7f7440c00640 1 -- 172.21.15.102:0/3077232131 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10509 (secure 0 0 0) 0x7f74340bbb20 con 0x7f74441039c0 2024-04-25T01:52:39.285 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:39.285 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24520,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24587,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T01:52:39.285 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 13 2024-04-25T01:52:39.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.287+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f74280a2c10 msgr2=0x7f74280a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.287+0000 7f744a400640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f74280a2c10 0x7f74280a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f742c009bd0 tx=0x7f742c005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:39.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.287+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 msgr2=0x7f7444199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.287+0000 7f744a400640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 0x7f7444199d40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7434031440 tx=0x7f7434032ce0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.288+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 shutdown_connections 2024-04-25T01:52:39.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.288+0000 7f744a400640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f74280a2c10 0x7f74280a50d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.288+0000 7f744a400640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7444105570 0x7f74441a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.288+0000 7f744a400640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7444104bb0 0x7f744419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.288+0000 7f744a400640 1 --2- 172.21.15.102:0/3077232131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74441039c0 0x7f7444199d40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.288+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 >> 172.21.15.102:0/3077232131 conn(0x7f74440fef30 msgr2=0x7f7444107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:39.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.288+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 shutdown_connections 2024-04-25T01:52:39.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.288+0000 7f744a400640 1 -- 172.21.15.102:0/3077232131 wait complete. 2024-04-25T01:52:39.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:52:39.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.491+0000 7fbc82a00640 1 -- 172.21.15.102:0/595403382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c104b90 msgr2=0x7fbc7c105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.491+0000 7fbc82a00640 1 --2- 172.21.15.102:0/595403382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c104b90 0x7fbc7c105010 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbc68009900 tx=0x7fbc6802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.491+0000 7fbc82a00640 1 -- 172.21.15.102:0/595403382 shutdown_connections 2024-04-25T01:52:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.491+0000 7fbc82a00640 1 --2- 172.21.15.102:0/595403382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c105550 0x7fbc7c10bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.491+0000 7fbc82a00640 1 --2- 172.21.15.102:0/595403382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c104b90 0x7fbc7c105010 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.491+0000 7fbc82a00640 1 --2- 172.21.15.102:0/595403382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c1039a0 0x7fbc7c103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.491+0000 7fbc82a00640 1 -- 172.21.15.102:0/595403382 >> 172.21.15.102:0/595403382 conn(0x7fbc7c0fef30 msgr2=0x7fbc7c101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.491+0000 7fbc82a00640 1 -- 172.21.15.102:0/595403382 shutdown_connections 2024-04-25T01:52:39.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.491+0000 7fbc82a00640 1 -- 172.21.15.102:0/595403382 wait complete. 2024-04-25T01:52:39.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.492+0000 7fbc82a00640 1 Processor -- start 2024-04-25T01:52:39.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.492+0000 7fbc82a00640 1 -- start start 2024-04-25T01:52:39.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.492+0000 7fbc82a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c1039a0 0x7fbc7c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.492+0000 7fbc82a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c104b90 0x7fbc7c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.492+0000 7fbc82a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c105550 0x7fbc7c1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.492+0000 7fbc82a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10f650 con 0x7fbc7c1039a0 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.492+0000 7fbc82a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10f4d0 con 0x7fbc7c104b90 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.492+0000 7fbc82a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10f7d0 con 0x7fbc7c105550 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc7be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c1039a0 0x7fbc7c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc80a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c105550 0x7fbc7c1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc7b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c104b90 0x7fbc7c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc7be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c1039a0 0x7fbc7c199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43280/0 (socket says 172.21.15.102:43280) 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc7be00640 1 -- 172.21.15.102:0/4253013848 learned_addr learned my addr 172.21.15.102:0/4253013848 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc80a00640 1 -- 172.21.15.102:0/4253013848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c104b90 msgr2=0x7fbc7c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc80a00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c104b90 0x7fbc7c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc80a00640 1 -- 172.21.15.102:0/4253013848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c1039a0 msgr2=0x7fbc7c199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc80a00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c1039a0 0x7fbc7c199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc80a00640 1 -- 172.21.15.102:0/4253013848 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c1a1920 con 0x7fbc7c105550 2024-04-25T01:52:39.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.493+0000 7fbc80a00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c105550 0x7fbc7c1a1280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009fd0 tx=0x7fbc7400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:39.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.494+0000 7fbc78c00640 1 -- 172.21.15.102:0/4253013848 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7400ce20 con 0x7fbc7c105550 2024-04-25T01:52:39.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.494+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c1a1c10 con 0x7fbc7c105550 2024-04-25T01:52:39.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.494+0000 7fbc78c00640 1 -- 172.21.15.102:0/4253013848 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc74010430 con 0x7fbc7c105550 2024-04-25T01:52:39.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.494+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c1a2160 con 0x7fbc7c105550 2024-04-25T01:52:39.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.494+0000 7fbc78c00640 1 -- 172.21.15.102:0/4253013848 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7400f620 con 0x7fbc7c105550 2024-04-25T01:52:39.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.495+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c10acc0 con 0x7fbc7c105550 2024-04-25T01:52:39.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.497+0000 7fbc78c00640 1 -- 172.21.15.102:0/4253013848 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbc74010950 con 0x7fbc7c105550 2024-04-25T01:52:39.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.499+0000 7fbc78c00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbc600a2b90 0x7fbc600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.499+0000 7fbc78c00640 1 -- 172.21.15.102:0/4253013848 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fbc740cefe0 con 0x7fbc7c105550 2024-04-25T01:52:39.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.499+0000 7fbc7be00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbc600a2b90 0x7fbc600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.500+0000 7fbc7be00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbc600a2b90 0x7fbc600a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c19b1e0 tx=0x7fbc6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:39.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.503+0000 7fbc78c00640 1 -- 172.21.15.102:0/4253013848 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbc7409c360 con 0x7fbc7c105550 2024-04-25T01:52:39.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.728+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc7c062340 con 0x7fbc7c105550 2024-04-25T01:52:39.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.730+0000 7fbc78c00640 1 -- 172.21.15.102:0/4253013848 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18490 (secure 0 0 0) 0x7fbc7409bc60 con 0x7fbc7c105550 2024-04-25T01:52:39.729 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:39.730 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":83,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:51:43.497262+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbc600a2b90 msgr2=0x7fbc600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbc600a2b90 0x7fbc600a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c19b1e0 tx=0x7fbc6c005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c105550 msgr2=0x7fbc7c1a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c105550 0x7fbc7c1a1280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbc74009fd0 tx=0x7fbc7400ef10 comp rx=0 tx=0).stop 2024-04-25T01:52:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 shutdown_connections 2024-04-25T01:52:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbc600a2b90 0x7fbc600a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc7c105550 0x7fbc7c1a1280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc7c104b90 0x7fbc7c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 --2- 172.21.15.102:0/4253013848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc7c1039a0 0x7fbc7c199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.732+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 >> 172.21.15.102:0/4253013848 conn(0x7fbc7c0fef30 msgr2=0x7fbc7c107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:39.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.733+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 shutdown_connections 2024-04-25T01:52:39.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.733+0000 7fbc82a00640 1 -- 172.21.15.102:0/4253013848 wait complete. 2024-04-25T01:52:39.751 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:52:39.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:52:39.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.937+0000 7f4324a00640 1 -- 172.21.15.102:0/333902586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4320102200 msgr2=0x7f43201045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.937+0000 7f4324a00640 1 --2- 172.21.15.102:0/333902586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4320102200 0x7f43201045f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4310009ca0 tx=0x7f431002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.937+0000 7f4324a00640 1 -- 172.21.15.102:0/333902586 shutdown_connections 2024-04-25T01:52:39.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.937+0000 7f4324a00640 1 --2- 172.21.15.102:0/333902586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4320107460 0x7f43201098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.937+0000 7f4324a00640 1 --2- 172.21.15.102:0/333902586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4320104b30 0x7f4320106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.937+0000 7f4324a00640 1 --2- 172.21.15.102:0/333902586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4320102200 0x7f43201045f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.937+0000 7f4324a00640 1 -- 172.21.15.102:0/333902586 >> 172.21.15.102:0/333902586 conn(0x7f43200fbbf0 msgr2=0x7f43200fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:39.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.937+0000 7f4324a00640 1 -- 172.21.15.102:0/333902586 shutdown_connections 2024-04-25T01:52:39.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.937+0000 7f4324a00640 1 -- 172.21.15.102:0/333902586 wait complete. 2024-04-25T01:52:39.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.938+0000 7f4324a00640 1 Processor -- start 2024-04-25T01:52:39.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.938+0000 7f4324a00640 1 -- start start 2024-04-25T01:52:39.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.938+0000 7f4324a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4320102200 0x7f4320197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.938+0000 7f4324a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4320104b30 0x7f4320198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.938+0000 7f4324a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4320107460 0x7f432019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f4324a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f432010d450 con 0x7f4320102200 2024-04-25T01:52:39.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f4324a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f432010d2d0 con 0x7f4320104b30 2024-04-25T01:52:39.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f4324a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f432010d5d0 con 0x7f4320107460 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4320102200 0x7f4320197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4320107460 0x7f432019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4320107460 0x7f432019f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46210/0 (socket says 172.21.15.102:46210) 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431e800640 1 -- 172.21.15.102:0/688653930 learned_addr learned my addr 172.21.15.102:0/688653930 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f4315400640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4320104b30 0x7f4320198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431e800640 1 -- 172.21.15.102:0/688653930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4320104b30 msgr2=0x7f4320198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431e800640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4320104b30 0x7f4320198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431e800640 1 -- 172.21.15.102:0/688653930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4320102200 msgr2=0x7f4320197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431e800640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4320102200 0x7f4320197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431e800640 1 -- 172.21.15.102:0/688653930 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432019f750 con 0x7f4320107460 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.939+0000 7f431e800640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4320107460 0x7f432019f0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f431800e9b0 tx=0x7f431800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.940+0000 7f4317400640 1 -- 172.21.15.102:0/688653930 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431800cd80 con 0x7f4320107460 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.940+0000 7f4317400640 1 -- 172.21.15.102:0/688653930 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43180035f0 con 0x7f4320107460 2024-04-25T01:52:39.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.940+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f432019fa40 con 0x7f4320107460 2024-04-25T01:52:39.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.940+0000 7f4317400640 1 -- 172.21.15.102:0/688653930 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4318010670 con 0x7f4320107460 2024-04-25T01:52:39.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.940+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f432019ff40 con 0x7f4320107460 2024-04-25T01:52:39.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.941+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43200fd6e0 con 0x7f4320107460 2024-04-25T01:52:39.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.943+0000 7f4317400640 1 -- 172.21.15.102:0/688653930 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f43180037a0 con 0x7f4320107460 2024-04-25T01:52:39.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.944+0000 7f4317400640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f43040a2b90 0x7f43040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:39.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.944+0000 7f4317400640 1 -- 172.21.15.102:0/688653930 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f43180cef30 con 0x7f4320107460 2024-04-25T01:52:39.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.944+0000 7f431de00640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f43040a2b90 0x7f43040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:39.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.945+0000 7f431de00640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f43040a2b90 0x7f43040a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4310005e30 tx=0x7f4310005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:39.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:39.948+0000 7f4317400640 1 -- 172.21.15.102:0/688653930 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f431809c290 con 0x7f4320107460 2024-04-25T01:52:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.155+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4320062340 con 0x7f4320107460 2024-04-25T01:52:40.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.156+0000 7f4317400640 1 -- 172.21.15.102:0/688653930 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f431809bb90 con 0x7f4320107460 2024-04-25T01:52:40.155 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:52:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f43040a2b90 msgr2=0x7f43040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f43040a2b90 0x7f43040a5050 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4310005e30 tx=0x7f4310005f50 comp rx=0 tx=0).stop 2024-04-25T01:52:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4320107460 msgr2=0x7f432019f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4320107460 0x7f432019f0e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f431800e9b0 tx=0x7f431800ee70 comp rx=0 tx=0).stop 2024-04-25T01:52:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 shutdown_connections 2024-04-25T01:52:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f43040a2b90 0x7f43040a5050 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4320107460 0x7f432019f0e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4320104b30 0x7f4320198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 --2- 172.21.15.102:0/688653930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4320102200 0x7f4320197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.158+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 >> 172.21.15.102:0/688653930 conn(0x7f43200fbbf0 msgr2=0x7f4320102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.159+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 shutdown_connections 2024-04-25T01:52:40.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.159+0000 7f4324a00640 1 -- 172.21.15.102:0/688653930 wait complete. 2024-04-25T01:52:40.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:39 smithi102 ceph-mon[27796]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:40.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:39 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:39 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3077232131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:40.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:39 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/4253013848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:40.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T01:52:40.236 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:39 smithi110 ceph-mon[32848]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:40.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:39 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:39 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3077232131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:40.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:39 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/4253013848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:40.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:39 smithi193 ceph-mon[32981]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:40.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:39 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:40.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:39 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3077232131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:40.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:39 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/4253013848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.365+0000 7fc159600640 1 -- 172.21.15.102:0/2524845300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc154103190 msgr2=0x7fc154109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.365+0000 7fc159600640 1 --2- 172.21.15.102:0/2524845300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc154103190 0x7fc154109a20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009be0 tx=0x7fc14c02a6c0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.365+0000 7fc159600640 1 -- 172.21.15.102:0/2524845300 shutdown_connections 2024-04-25T01:52:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.365+0000 7fc159600640 1 --2- 172.21.15.102:0/2524845300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc154103190 0x7fc154109a20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.365+0000 7fc159600640 1 --2- 172.21.15.102:0/2524845300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1541027d0 0x7fc154102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.365+0000 7fc159600640 1 --2- 172.21.15.102:0/2524845300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1541015e0 0x7fc154101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.365+0000 7fc159600640 1 -- 172.21.15.102:0/2524845300 >> 172.21.15.102:0/2524845300 conn(0x7fc1540fcb50 msgr2=0x7fc1540fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:40.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.365+0000 7fc159600640 1 -- 172.21.15.102:0/2524845300 shutdown_connections 2024-04-25T01:52:40.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.365+0000 7fc159600640 1 -- 172.21.15.102:0/2524845300 wait complete. 2024-04-25T01:52:40.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.366+0000 7fc159600640 1 Processor -- start 2024-04-25T01:52:40.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.366+0000 7fc159600640 1 -- start start 2024-04-25T01:52:40.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc159600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1541015e0 0x7fc154199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:40.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc159600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1541027d0 0x7fc154199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:40.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc159600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc154103190 0x7fc1541a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:40.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc159600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc15410d220 con 0x7fc1541015e0 2024-04-25T01:52:40.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc159600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc15410d0a0 con 0x7fc154103190 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc159600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc15410d3a0 con 0x7fc1541027d0 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc153400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1541027d0 0x7fc154199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc153400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1541027d0 0x7fc154199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46234/0 (socket says 172.21.15.102:46234) 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc158c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc154103190 0x7fc1541a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc153e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1541015e0 0x7fc154199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc153400640 1 -- 172.21.15.102:0/1810462098 learned_addr learned my addr 172.21.15.102:0/1810462098 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc153400640 1 -- 172.21.15.102:0/1810462098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc154103190 msgr2=0x7fc1541a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc153400640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc154103190 0x7fc1541a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc153400640 1 -- 172.21.15.102:0/1810462098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1541015e0 msgr2=0x7fc154199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc153400640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1541015e0 0x7fc154199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.367+0000 7fc153400640 1 -- 172.21.15.102:0/1810462098 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1541a16c0 con 0x7fc1541027d0 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.368+0000 7fc158c00640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc154103190 0x7fc1541a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:40.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.368+0000 7fc153400640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1541027d0 0x7fc154199fe0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc14000e9f0 tx=0x7fc14000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:40.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.368+0000 7fc150c00640 1 -- 172.21.15.102:0/1810462098 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14000ccc0 con 0x7fc1541027d0 2024-04-25T01:52:40.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.368+0000 7fc150c00640 1 -- 172.21.15.102:0/1810462098 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc140010910 con 0x7fc1541027d0 2024-04-25T01:52:40.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.368+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1541a19b0 con 0x7fc1541027d0 2024-04-25T01:52:40.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.368+0000 7fc150c00640 1 -- 172.21.15.102:0/1810462098 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1400188a0 con 0x7fc1541027d0 2024-04-25T01:52:40.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.368+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1541a1f00 con 0x7fc1541027d0 2024-04-25T01:52:40.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.369+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1541a2130 con 0x7fc1541027d0 2024-04-25T01:52:40.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.372+0000 7fc150c00640 1 -- 172.21.15.102:0/1810462098 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc140010ac0 con 0x7fc1541027d0 2024-04-25T01:52:40.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.373+0000 7fc150c00640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc1380a2b90 0x7fc1380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:40.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.373+0000 7fc150c00640 1 -- 172.21.15.102:0/1810462098 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc1400cf030 con 0x7fc1541027d0 2024-04-25T01:52:40.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.374+0000 7fc153e00640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc1380a2b90 0x7fc1380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:40.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.374+0000 7fc153e00640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc1380a2b90 0x7fc1380a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc154102630 tx=0x7fc144009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:40.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.377+0000 7fc150c00640 1 -- 172.21.15.102:0/1810462098 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc14009c390 con 0x7fc1541027d0 2024-04-25T01:52:40.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.585+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc1541a24c0 con 0x7fc1541027d0 2024-04-25T01:52:40.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.586+0000 7fc150c00640 1 -- 172.21.15.102:0/1810462098 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7fc14009bc90 con 0x7fc1541027d0 2024-04-25T01:52:40.585 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:40.585 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-04-25T01:52:40.585 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T01:52:40.585 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-04-25T01:52:40.585 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-04-25T01:52:40.586 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:52:40.586 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-04-25T01:52:40.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc1380a2b90 msgr2=0x7fc1380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc1380a2b90 0x7fc1380a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc154102630 tx=0x7fc144009260 comp rx=0 tx=0).stop 2024-04-25T01:52:40.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1541027d0 msgr2=0x7fc154199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1541027d0 0x7fc154199fe0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc14000e9f0 tx=0x7fc14000edf0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 shutdown_connections 2024-04-25T01:52:40.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc1380a2b90 0x7fc1380a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc154103190 0x7fc1541a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1541027d0 0x7fc154199fe0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 --2- 172.21.15.102:0/1810462098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1541015e0 0x7fc154199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.588+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 >> 172.21.15.102:0/1810462098 conn(0x7fc1540fcb50 msgr2=0x7fc154073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:40.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.589+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 shutdown_connections 2024-04-25T01:52:40.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.589+0000 7fc159600640 1 -- 172.21.15.102:0/1810462098 wait complete. 2024-04-25T01:52:40.607 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T01:52:40.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T01:52:40.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.810+0000 7fc8e1000640 1 -- 172.21.15.102:0/3949570564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc075780 msgr2=0x7fc8dc073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.810+0000 7fc8e1000640 1 --2- 172.21.15.102:0/3949570564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc075780 0x7fc8dc073bd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009900 tx=0x7fc8cc02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.810+0000 7fc8e1000640 1 -- 172.21.15.102:0/3949570564 shutdown_connections 2024-04-25T01:52:40.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.810+0000 7fc8e1000640 1 --2- 172.21.15.102:0/3949570564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8dc109590 0x7fc8dc10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.810+0000 7fc8e1000640 1 --2- 172.21.15.102:0/3949570564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc0741a0 0x7fc8dc074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.810+0000 7fc8e1000640 1 --2- 172.21.15.102:0/3949570564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc075780 0x7fc8dc073bd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.810+0000 7fc8e1000640 1 -- 172.21.15.102:0/3949570564 >> 172.21.15.102:0/3949570564 conn(0x7fc8dc0fcc40 msgr2=0x7fc8dc0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:40.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.811+0000 7fc8e1000640 1 -- 172.21.15.102:0/3949570564 shutdown_connections 2024-04-25T01:52:40.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.811+0000 7fc8e1000640 1 -- 172.21.15.102:0/3949570564 wait complete. 2024-04-25T01:52:40.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.811+0000 7fc8e1000640 1 Processor -- start 2024-04-25T01:52:40.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8e1000640 1 -- start start 2024-04-25T01:52:40.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8e1000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc0741a0 0x7fc8dc199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:40.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8e1000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc075780 0x7fc8dc19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:40.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8e1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8dc109590 0x7fc8dc1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:40.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8e1000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc10f4f0 con 0x7fc8dc109590 2024-04-25T01:52:40.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8e1000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc10f370 con 0x7fc8dc075780 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8e1000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc10f670 con 0x7fc8dc0741a0 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8db400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc0741a0 0x7fc8dc199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8db400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc0741a0 0x7fc8dc199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46252/0 (socket says 172.21.15.102:46252) 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8db400640 1 -- 172.21.15.102:0/68780600 learned_addr learned my addr 172.21.15.102:0/68780600 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8dbe00640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8dc109590 0x7fc8dc1a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.812+0000 7fc8daa00640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc075780 0x7fc8dc19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8db400640 1 -- 172.21.15.102:0/68780600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc075780 msgr2=0x7fc8dc19a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8db400640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc075780 0x7fc8dc19a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8db400640 1 -- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8dc109590 msgr2=0x7fc8dc1a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8db400640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8dc109590 0x7fc8dc1a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8db400640 1 -- 172.21.15.102:0/68780600 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8dc1a1770 con 0x7fc8dc0741a0 2024-04-25T01:52:40.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8db400640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc0741a0 0x7fc8dc199b50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc031440 tx=0x7fc8cc032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:40.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8d3e00640 1 -- 172.21.15.102:0/68780600 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc03f030 con 0x7fc8dc0741a0 2024-04-25T01:52:40.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8d3e00640 1 -- 172.21.15.102:0/68780600 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8cc003ce0 con 0x7fc8dc0741a0 2024-04-25T01:52:40.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8dc1a1a00 con 0x7fc8dc0741a0 2024-04-25T01:52:40.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8d3e00640 1 -- 172.21.15.102:0/68780600 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc035070 con 0x7fc8dc0741a0 2024-04-25T01:52:40.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.813+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8dc1a1ef0 con 0x7fc8dc0741a0 2024-04-25T01:52:40.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.815+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8dc1082c0 con 0x7fc8dc0741a0 2024-04-25T01:52:40.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.816+0000 7fc8d3e00640 1 -- 172.21.15.102:0/68780600 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc8cc042030 con 0x7fc8dc0741a0 2024-04-25T01:52:40.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.818+0000 7fc8d3e00640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc8c00a2b90 0x7fc8c00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:40.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.818+0000 7fc8d3e00640 1 -- 172.21.15.102:0/68780600 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc8cc0ef0a0 con 0x7fc8dc0741a0 2024-04-25T01:52:40.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.818+0000 7fc8daa00640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc8c00a2b90 0x7fc8c00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:40.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.818+0000 7fc8daa00640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc8c00a2b90 0x7fc8c00a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80095f0 tx=0x7fc8c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:40.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:40.821+0000 7fc8d3e00640 1 -- 172.21.15.102:0/68780600 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc8cc0bbdb0 con 0x7fc8dc0741a0 2024-04-25T01:52:41.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:41.038+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc8dc062340 con 0x7fc8dc0741a0 2024-04-25T01:52:41.279 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:41 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:41.279 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:41 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:41.279 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:41 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:41.279 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:41 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:41.279 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:41 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:41.279 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:41 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/688653930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:52:41.279 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:41 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:41.279 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:41 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1810462098' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:52:41.279 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:41 smithi110 ceph-mon[32848]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:41.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:41 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:41.287 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:41 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:41.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:41 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:41.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:41 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:41.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:41 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:41.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:41 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/688653930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:52:41.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:41 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:41.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:41 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1810462098' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:52:41.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:41 smithi193 ceph-mon[32981]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:41.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:41 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:41.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:41 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:41.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:41 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:41.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:41 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:41.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:41 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:41.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:41 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/688653930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:52:41.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:41 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:52:41.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:41 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1810462098' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T01:52:41.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:41 smithi102 ceph-mon[27796]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:42.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.053+0000 7fc8d3e00640 1 -- 172.21.15.102:0/68780600 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7fc8cc0c00b0 con 0x7fc8dc0741a0 2024-04-25T01:52:42.054 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-04-25T01:52:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc8c00a2b90 msgr2=0x7fc8c00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc8c00a2b90 0x7fc8c00a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80095f0 tx=0x7fc8c8005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc0741a0 msgr2=0x7fc8dc199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc0741a0 0x7fc8dc199b50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc031440 tx=0x7fc8cc032ce0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 shutdown_connections 2024-04-25T01:52:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fc8c00a2b90 0x7fc8c00a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8dc109590 0x7fc8dc1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc075780 0x7fc8dc19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 --2- 172.21.15.102:0/68780600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc0741a0 0x7fc8dc199b50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.059+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 >> 172.21.15.102:0/68780600 conn(0x7fc8dc0fcc40 msgr2=0x7fc8dc105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:42.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.060+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 shutdown_connections 2024-04-25T01:52:42.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.060+0000 7fc8e1000640 1 -- 172.21.15.102:0/68780600 wait complete. 2024-04-25T01:52:42.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T01:52:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.267+0000 7f397a000640 1 -- 172.21.15.102:0/4027540424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39741039a0 msgr2=0x7f3974103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.267+0000 7f397a000640 1 --2- 172.21.15.102:0/4027540424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39741039a0 0x7f3974103dc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3964009900 tx=0x7f396402a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.267+0000 7f397a000640 1 -- 172.21.15.102:0/4027540424 shutdown_connections 2024-04-25T01:52:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.267+0000 7f397a000640 1 --2- 172.21.15.102:0/4027540424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3974105550 0x7f397410bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.267+0000 7f397a000640 1 --2- 172.21.15.102:0/4027540424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3974104b90 0x7f3974105010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.267+0000 7f397a000640 1 --2- 172.21.15.102:0/4027540424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39741039a0 0x7f3974103dc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.267+0000 7f397a000640 1 -- 172.21.15.102:0/4027540424 >> 172.21.15.102:0/4027540424 conn(0x7f39740feef0 msgr2=0x7f3974101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.267+0000 7f397a000640 1 -- 172.21.15.102:0/4027540424 shutdown_connections 2024-04-25T01:52:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.267+0000 7f397a000640 1 -- 172.21.15.102:0/4027540424 wait complete. 2024-04-25T01:52:42.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.268+0000 7f397a000640 1 Processor -- start 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.268+0000 7f397a000640 1 -- start start 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.268+0000 7f397a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39741039a0 0x7f3974199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.268+0000 7f397a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3974104b90 0x7f397419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.268+0000 7f397a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3974105550 0x7f39741a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.268+0000 7f397a000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f397410f660 con 0x7f39741039a0 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.268+0000 7f397a000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f397410f4e0 con 0x7f3974104b90 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.268+0000 7f397a000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f397410f7e0 con 0x7f3974105550 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3972a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3974104b90 0x7f397419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3973400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39741039a0 0x7f3974199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3972a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3974104b90 0x7f397419a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53706/0 (socket says 172.21.15.102:53706) 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3972a00640 1 -- 172.21.15.102:0/373635351 learned_addr learned my addr 172.21.15.102:0/373635351 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:42.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3973e00640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3974105550 0x7f39741a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3972a00640 1 -- 172.21.15.102:0/373635351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3974105550 msgr2=0x7f39741a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3972a00640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3974105550 0x7f39741a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3972a00640 1 -- 172.21.15.102:0/373635351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39741039a0 msgr2=0x7f3974199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3972a00640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39741039a0 0x7f3974199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3972a00640 1 -- 172.21.15.102:0/373635351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39741a1930 con 0x7f3974104b90 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f3972a00640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3974104b90 0x7f397419a260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f396000e9f0 tx=0x7f396000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f396be00640 1 -- 172.21.15.102:0/373635351 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396000cd80 con 0x7f3974104b90 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.269+0000 7f396be00640 1 -- 172.21.15.102:0/373635351 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3960005500 con 0x7f3974104b90 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.270+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39741a1c20 con 0x7f3974104b90 2024-04-25T01:52:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.270+0000 7f396be00640 1 -- 172.21.15.102:0/373635351 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960010690 con 0x7f3974104b90 2024-04-25T01:52:42.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.270+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39741a2140 con 0x7f3974104b90 2024-04-25T01:52:42.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.271+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f397410acc0 con 0x7f3974104b90 2024-04-25T01:52:42.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.273+0000 7f396be00640 1 -- 172.21.15.102:0/373635351 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f39600046c0 con 0x7f3974104b90 2024-04-25T01:52:42.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.274+0000 7f396be00640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39580a2c10 0x7f39580a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:42.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.275+0000 7f396be00640 1 -- 172.21.15.102:0/373635351 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f39600cf1c0 con 0x7f3974104b90 2024-04-25T01:52:42.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.275+0000 7f3973400640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39580a2c10 0x7f39580a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:42.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.275+0000 7f3973400640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39580a2c10 0x7f39580a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f396402a9e0 tx=0x7f39640315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:42.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.278+0000 7f396be00640 1 -- 172.21.15.102:0/373635351 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f396009c3b0 con 0x7f3974104b90 2024-04-25T01:52:42.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:42 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/68780600' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:52:42.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:42 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:52:42.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:42.483+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f397402d170 con 0x7f3974104b90 2024-04-25T01:52:42.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:42 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/68780600' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:52:42.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:42 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:52:42.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:42 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/68780600' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:52:42.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:42 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T01:52:43.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.059+0000 7f396be00640 1 -- 172.21.15.102:0/373635351 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v85) ==== 170+0+0 (secure 0 0 0) 0x7f3960086070 con 0x7f3974104b90 2024-04-25T01:52:43.060 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-04-25T01:52:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.064+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39580a2c10 msgr2=0x7f39580a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.064+0000 7f397a000640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39580a2c10 0x7f39580a50d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f396402a9e0 tx=0x7f39640315b0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.064+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3974104b90 msgr2=0x7f397419a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.064+0000 7f397a000640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3974104b90 0x7f397419a260 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f396000e9f0 tx=0x7f396000edf0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.065+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 shutdown_connections 2024-04-25T01:52:43.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.065+0000 7f397a000640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39580a2c10 0x7f39580a50d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.065+0000 7f397a000640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3974105550 0x7f39741a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.065+0000 7f397a000640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3974104b90 0x7f397419a260 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.065+0000 7f397a000640 1 --2- 172.21.15.102:0/373635351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39741039a0 0x7f3974199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.065+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 >> 172.21.15.102:0/373635351 conn(0x7f39740feef0 msgr2=0x7f3974107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.066+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 shutdown_connections 2024-04-25T01:52:43.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.066+0000 7f397a000640 1 -- 172.21.15.102:0/373635351 wait complete. 2024-04-25T01:52:43.097 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T01:52:43.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.267+0000 7fec05000640 1 -- 172.21.15.102:0/1890609676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00104b90 msgr2=0x7fec00105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.267+0000 7fec05000640 1 --2- 172.21.15.102:0/1890609676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00104b90 0x7fec00105010 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7febec009900 tx=0x7febec02a410 comp rx=0 tx=0).stop 2024-04-25T01:52:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.267+0000 7fec05000640 1 -- 172.21.15.102:0/1890609676 shutdown_connections 2024-04-25T01:52:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.267+0000 7fec05000640 1 --2- 172.21.15.102:0/1890609676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec00105550 0x7fec0010bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.267+0000 7fec05000640 1 --2- 172.21.15.102:0/1890609676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00104b90 0x7fec00105010 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.267+0000 7fec05000640 1 --2- 172.21.15.102:0/1890609676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec001039a0 0x7fec00103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.267+0000 7fec05000640 1 -- 172.21.15.102:0/1890609676 >> 172.21.15.102:0/1890609676 conn(0x7fec000fef30 msgr2=0x7fec00101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.267+0000 7fec05000640 1 -- 172.21.15.102:0/1890609676 shutdown_connections 2024-04-25T01:52:43.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.267+0000 7fec05000640 1 -- 172.21.15.102:0/1890609676 wait complete. 2024-04-25T01:52:43.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7fec05000640 1 Processor -- start 2024-04-25T01:52:43.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7fec05000640 1 -- start start 2024-04-25T01:52:43.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7fec05000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec001039a0 0x7fec00199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:43.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7fec05000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00104b90 0x7fec0019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:43.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7fec05000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec00105550 0x7fec001a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:43.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7fec05000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fec0010f670 con 0x7fec00105550 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7fec05000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fec0010f4f0 con 0x7fec001039a0 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7fec05000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec0010f7f0 con 0x7fec00104b90 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7febfe800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec00105550 0x7fec001a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7febfd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00104b90 0x7fec0019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7febfe800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec00105550 0x7fec001a12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58546/0 (socket says 172.21.15.102:58546) 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7febfd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00104b90 0x7fec0019a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:54688/0 (socket says 172.21.15.102:54688) 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.268+0000 7febfe800640 1 -- 172.21.15.102:0/2677475572 learned_addr learned my addr 172.21.15.102:0/2677475572 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7febfde00640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec001039a0 0x7fec00199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7febfe800640 1 -- 172.21.15.102:0/2677475572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00104b90 msgr2=0x7fec0019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7febfe800640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00104b90 0x7fec0019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7febfe800640 1 -- 172.21.15.102:0/2677475572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec001039a0 msgr2=0x7fec00199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7febfe800640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec001039a0 0x7fec00199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:43.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7febfe800640 1 -- 172.21.15.102:0/2677475572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec001a1950 con 0x7fec00105550 2024-04-25T01:52:43.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7febfe800640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec00105550 0x7fec001a12b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7febf800eb10 tx=0x7febf800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:43.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7febf6a00640 1 -- 172.21.15.102:0/2677475572 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8004b20 con 0x7fec00105550 2024-04-25T01:52:43.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec001a1c40 con 0x7fec00105550 2024-04-25T01:52:43.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.269+0000 7febf6a00640 1 -- 172.21.15.102:0/2677475572 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf8004cc0 con 0x7fec00105550 2024-04-25T01:52:43.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.270+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec001a2190 con 0x7fec00105550 2024-04-25T01:52:43.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.270+0000 7febf6a00640 1 -- 172.21.15.102:0/2677475572 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8010700 con 0x7fec00105550 2024-04-25T01:52:43.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.271+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec0010ac60 con 0x7fec00105550 2024-04-25T01:52:43.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.273+0000 7febf6a00640 1 -- 172.21.15.102:0/2677475572 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7febf8005020 con 0x7fec00105550 2024-04-25T01:52:43.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.274+0000 7febf6a00640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7febe40a2b90 0x7febe40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:43.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.274+0000 7febf6a00640 1 -- 172.21.15.102:0/2677475572 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7febf80cf970 con 0x7fec00105550 2024-04-25T01:52:43.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.274+0000 7febfde00640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7febe40a2b90 0x7febe40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:43.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.275+0000 7febfde00640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7febe40a2b90 0x7febe40a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fec0019b210 tx=0x7febf0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:43.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.277+0000 7febf6a00640 1 -- 172.21.15.102:0/2677475572 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7febf809c9e0 con 0x7fec00105550 2024-04-25T01:52:43.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:52:43.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/373635351' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: pgmap v314: 33 pgs: 7 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:43.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:52:43.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:52:43.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/373635351' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:52:43.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:52:43.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: pgmap v314: 33 pgs: 7 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:43.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:43.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:43.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T01:52:43.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T01:52:43.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/373635351' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:52:43.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T01:52:43.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: pgmap v314: 33 pgs: 7 creating+peering, 1 active+clean, 25 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:43.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:43.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:52:43.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:52:43.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:43.523+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fec0002d170 con 0x7fec00105550 2024-04-25T01:52:44.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.095+0000 7febf6a00640 1 -- 172.21.15.102:0/2677475572 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7febf809c2e0 con 0x7fec00105550 2024-04-25T01:52:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T01:52:44.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.102+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7febe40a2b90 msgr2=0x7febe40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.102+0000 7fec05000640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7febe40a2b90 0x7febe40a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fec0019b210 tx=0x7febf0005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:44.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.102+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec00105550 msgr2=0x7fec001a12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.102+0000 7fec05000640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec00105550 0x7fec001a12b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7febf800eb10 tx=0x7febf800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.104+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 shutdown_connections 2024-04-25T01:52:44.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.104+0000 7fec05000640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7febe40a2b90 0x7febe40a5050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.104+0000 7fec05000640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec00105550 0x7fec001a12b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.104+0000 7fec05000640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec00104b90 0x7fec0019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.104+0000 7fec05000640 1 --2- 172.21.15.102:0/2677475572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec001039a0 0x7fec00199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.104+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 >> 172.21.15.102:0/2677475572 conn(0x7fec000fef30 msgr2=0x7fec00107e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.104+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 shutdown_connections 2024-04-25T01:52:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.105+0000 7fec05000640 1 -- 172.21.15.102:0/2677475572 wait complete. 2024-04-25T01:52:44.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:52:44.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.309+0000 7f009bc00640 1 -- 172.21.15.102:0/3503175327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094105540 msgr2=0x7f009410bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.309+0000 7f009bc00640 1 --2- 172.21.15.102:0/3503175327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094105540 0x7f009410bdd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f008c009ca0 tx=0x7f008c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.310+0000 7f009bc00640 1 -- 172.21.15.102:0/3503175327 shutdown_connections 2024-04-25T01:52:44.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.310+0000 7f009bc00640 1 --2- 172.21.15.102:0/3503175327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094105540 0x7f009410bdd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.310+0000 7f009bc00640 1 --2- 172.21.15.102:0/3503175327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0094104b80 0x7f0094105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.310+0000 7f009bc00640 1 --2- 172.21.15.102:0/3503175327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0094103990 0x7f0094103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.310+0000 7f009bc00640 1 -- 172.21.15.102:0/3503175327 >> 172.21.15.102:0/3503175327 conn(0x7f00940fef00 msgr2=0x7f0094101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:44.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.310+0000 7f009bc00640 1 -- 172.21.15.102:0/3503175327 shutdown_connections 2024-04-25T01:52:44.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.310+0000 7f009bc00640 1 -- 172.21.15.102:0/3503175327 wait complete. 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f009bc00640 1 Processor -- start 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f009bc00640 1 -- start start 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f009bc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0094103990 0x7f0094199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f009bc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094104b80 0x7f009419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f009bc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0094105540 0x7f00941a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f009bc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f009410f660 con 0x7f0094104b80 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f009bc00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f009410f4e0 con 0x7f0094105540 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f009bc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f009410f7e0 con 0x7f0094103990 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f008be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094104b80 0x7f009419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.311+0000 7f008be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094104b80 0x7f009419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58560/0 (socket says 172.21.15.102:58560) 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f008be00640 1 -- 172.21.15.102:0/2440505407 learned_addr learned my addr 172.21.15.102:0/2440505407 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:44.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f0099200640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0094103990 0x7f0094199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f0099c00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0094105540 0x7f00941a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f008be00640 1 -- 172.21.15.102:0/2440505407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0094103990 msgr2=0x7f0094199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f008be00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0094103990 0x7f0094199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f008be00640 1 -- 172.21.15.102:0/2440505407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0094105540 msgr2=0x7f00941a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f008be00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0094105540 0x7f00941a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f008be00640 1 -- 172.21.15.102:0/2440505407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00941a18b0 con 0x7f0094104b80 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f0099c00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0094105540 0x7f00941a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f008be00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094104b80 0x7f009419a1b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f007c00e9f0 tx=0x7f007c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.312+0000 7f0092000640 1 -- 172.21.15.102:0/2440505407 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c00cd80 con 0x7f0094104b80 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.313+0000 7f0092000640 1 -- 172.21.15.102:0/2440505407 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007c010910 con 0x7f0094104b80 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.313+0000 7f0092000640 1 -- 172.21.15.102:0/2440505407 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c018990 con 0x7f0094104b80 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.313+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00941a1ba0 con 0x7f0094104b80 2024-04-25T01:52:44.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.313+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00941a20a0 con 0x7f0094104b80 2024-04-25T01:52:44.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.314+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f009410acb0 con 0x7f0094104b80 2024-04-25T01:52:44.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.316+0000 7f0092000640 1 -- 172.21.15.102:0/2440505407 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f007c010ac0 con 0x7f0094104b80 2024-04-25T01:52:44.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.317+0000 7f0092000640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f00780a2c10 0x7f00780a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:44.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.317+0000 7f0092000640 1 -- 172.21.15.102:0/2440505407 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f007c0cf2b0 con 0x7f0094104b80 2024-04-25T01:52:44.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.318+0000 7f0099200640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f00780a2c10 0x7f00780a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:44.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.318+0000 7f0099200640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f00780a2c10 0x7f00780a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f009419ab40 tx=0x7f0084005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:44.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.322+0000 7f0092000640 1 -- 172.21.15.102:0/2440505407 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f007c09c2c0 con 0x7f0094104b80 2024-04-25T01:52:44.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:44 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:52:44.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:44 smithi102 ceph-mon[27796]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:52:44.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:44 smithi102 ceph-mon[27796]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:52:44.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:44 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:44.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:44 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2677475572' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:52:44.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:44 smithi102 ceph-mon[27796]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:52:44.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:44 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:52:44.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:44 smithi110 ceph-mon[32848]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:52:44.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:44 smithi110 ceph-mon[32848]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:52:44.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:44 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:44.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:44 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2677475572' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:52:44.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:44 smithi110 ceph-mon[32848]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:52:44.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:44 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T01:52:44.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:44 smithi193 ceph-mon[32981]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T01:52:44.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:44 smithi193 ceph-mon[32981]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T01:52:44.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:44 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:52:44.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:44 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2677475572' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T01:52:44.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:44 smithi193 ceph-mon[32981]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T01:52:44.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.520+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0094062440 con 0x7f0094104b80 2024-04-25T01:52:44.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.524+0000 7f0092000640 1 -- 172.21.15.102:0/2440505407 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22266 (secure 0 0 0) 0x7f007c09bbc0 con 0x7f0094104b80 2024-04-25T01:52:44.525 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":86,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:52:43.525368+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:44.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.527+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f00780a2c10 msgr2=0x7f00780a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.527+0000 7f009bc00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f00780a2c10 0x7f00780a50d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f009419ab40 tx=0x7f0084005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:44.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.527+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094104b80 msgr2=0x7f009419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.527+0000 7f009bc00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094104b80 0x7f009419a1b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f007c00e9f0 tx=0x7f007c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.527+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 shutdown_connections 2024-04-25T01:52:44.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.527+0000 7f009bc00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f00780a2c10 0x7f00780a50d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.527+0000 7f009bc00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0094105540 0x7f00941a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.527+0000 7f009bc00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0094104b80 0x7f009419a1b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.527+0000 7f009bc00640 1 --2- 172.21.15.102:0/2440505407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0094103990 0x7f0094199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.528+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 >> 172.21.15.102:0/2440505407 conn(0x7f00940fef00 msgr2=0x7f0094107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:44.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.528+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 shutdown_connections 2024-04-25T01:52:44.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.528+0000 7f009bc00640 1 -- 172.21.15.102:0/2440505407 wait complete. 2024-04-25T01:52:44.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:52:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.737+0000 7fea7ba00640 1 -- 172.21.15.102:0/2111839113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea740741a0 msgr2=0x7fea74074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.737+0000 7fea7ba00640 1 --2- 172.21.15.102:0/2111839113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea740741a0 0x7fea74074620 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fea60009930 tx=0x7fea6002a420 comp rx=0 tx=0).stop 2024-04-25T01:52:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.737+0000 7fea7ba00640 1 -- 172.21.15.102:0/2111839113 shutdown_connections 2024-04-25T01:52:44.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.737+0000 7fea7ba00640 1 --2- 172.21.15.102:0/2111839113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea7410dd80 0x7fea74110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.737+0000 7fea7ba00640 1 --2- 172.21.15.102:0/2111839113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea740741a0 0x7fea74074620 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.737+0000 7fea7ba00640 1 --2- 172.21.15.102:0/2111839113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea74075780 0x7fea74073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.737+0000 7fea7ba00640 1 -- 172.21.15.102:0/2111839113 >> 172.21.15.102:0/2111839113 conn(0x7fea740fcd20 msgr2=0x7fea740ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:44.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.738+0000 7fea7ba00640 1 -- 172.21.15.102:0/2111839113 shutdown_connections 2024-04-25T01:52:44.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.738+0000 7fea7ba00640 1 -- 172.21.15.102:0/2111839113 wait complete. 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.738+0000 7fea7ba00640 1 Processor -- start 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea7ba00640 1 -- start start 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea7ba00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740741a0 0x7fea7419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea7ba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea74075780 0x7fea7419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea7ba00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea7410dd80 0x7fea741a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea7ba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fea74113a70 con 0x7fea74075780 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea7ba00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fea741138f0 con 0x7fea740741a0 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea7ba00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea74113bf0 con 0x7fea7410dd80 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea73e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea74075780 0x7fea7419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea73e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea74075780 0x7fea7419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58590/0 (socket says 172.21.15.102:58590) 2024-04-25T01:52:44.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea73e00640 1 -- 172.21.15.102:0/937788834 learned_addr learned my addr 172.21.15.102:0/937788834 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea79000640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740741a0 0x7fea7419e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.739+0000 7fea79a00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea7410dd80 0x7fea741a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.740+0000 7fea79000640 1 -- 172.21.15.102:0/937788834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea7410dd80 msgr2=0x7fea741a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.740+0000 7fea79000640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea7410dd80 0x7fea741a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.740+0000 7fea79000640 1 -- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea74075780 msgr2=0x7fea7419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.740+0000 7fea79000640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea74075780 0x7fea7419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.740+0000 7fea79000640 1 -- 172.21.15.102:0/937788834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea741a5d10 con 0x7fea740741a0 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.740+0000 7fea73e00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea74075780 0x7fea7419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.740+0000 7fea79000640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740741a0 0x7fea7419e140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fea6800e9f0 tx=0x7fea6800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.740+0000 7fea71600640 1 -- 172.21.15.102:0/937788834 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6800ccc0 con 0x7fea740741a0 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.740+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea741a5fa0 con 0x7fea740741a0 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.741+0000 7fea71600640 1 -- 172.21.15.102:0/937788834 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea68010910 con 0x7fea740741a0 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.741+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea741a64b0 con 0x7fea740741a0 2024-04-25T01:52:44.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.741+0000 7fea71600640 1 -- 172.21.15.102:0/937788834 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea68018960 con 0x7fea740741a0 2024-04-25T01:52:44.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.742+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea540052c0 con 0x7fea740741a0 2024-04-25T01:52:44.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.744+0000 7fea71600640 1 -- 172.21.15.102:0/937788834 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fea68010ac0 con 0x7fea740741a0 2024-04-25T01:52:44.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.745+0000 7fea71600640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fea5c0a2c10 0x7fea5c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:44.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.745+0000 7fea71600640 1 -- 172.21.15.102:0/937788834 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fea680cf580 con 0x7fea740741a0 2024-04-25T01:52:44.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.745+0000 7fea73e00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fea5c0a2c10 0x7fea5c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:44.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.746+0000 7fea73e00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fea5c0a2c10 0x7fea5c0a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea60000c00 tx=0x7fea600315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:44.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.748+0000 7fea71600640 1 -- 172.21.15.102:0/937788834 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fea6809bf40 con 0x7fea740741a0 2024-04-25T01:52:44.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.940+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea54005880 con 0x7fea740741a0 2024-04-25T01:52:44.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.942+0000 7fea71600640 1 -- 172.21.15.102:0/937788834 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22266 (secure 0 0 0) 0x7fea680a0240 con 0x7fea740741a0 2024-04-25T01:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:44.942 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":86,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:52:43.525368+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:44.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.944+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fea5c0a2c10 msgr2=0x7fea5c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.944+0000 7fea7ba00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fea5c0a2c10 0x7fea5c0a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea60000c00 tx=0x7fea600315b0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740741a0 msgr2=0x7fea7419e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:44.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740741a0 0x7fea7419e140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fea6800e9f0 tx=0x7fea6800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 shutdown_connections 2024-04-25T01:52:44.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fea5c0a2c10 0x7fea5c0a50d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea7410dd80 0x7fea741a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea74075780 0x7fea7419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 --2- 172.21.15.102:0/937788834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea740741a0 0x7fea7419e140 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:44.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 >> 172.21.15.102:0/937788834 conn(0x7fea740fcd20 msgr2=0x7fea74101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:44.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 shutdown_connections 2024-04-25T01:52:44.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:44.945+0000 7fea7ba00640 1 -- 172.21.15.102:0/937788834 wait complete. 2024-04-25T01:52:44.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T01:52:45.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.154+0000 7ff403e00640 1 -- 172.21.15.102:0/4261805204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff404102860 msgr2=0x7ff404102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.154+0000 7ff403e00640 1 --2- 172.21.15.102:0/4261805204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff404102860 0x7ff404102ce0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009900 tx=0x7ff3f002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.154+0000 7ff403e00640 1 -- 172.21.15.102:0/4261805204 shutdown_connections 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.154+0000 7ff403e00640 1 --2- 172.21.15.102:0/4261805204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404103220 0x7ff404109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.154+0000 7ff403e00640 1 --2- 172.21.15.102:0/4261805204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff404102860 0x7ff404102ce0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.154+0000 7ff403e00640 1 --2- 172.21.15.102:0/4261805204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff404101670 0x7ff404101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.154+0000 7ff403e00640 1 -- 172.21.15.102:0/4261805204 >> 172.21.15.102:0/4261805204 conn(0x7ff4040fcbe0 msgr2=0x7ff4040ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.154+0000 7ff403e00640 1 -- 172.21.15.102:0/4261805204 shutdown_connections 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.154+0000 7ff403e00640 1 -- 172.21.15.102:0/4261805204 wait complete. 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.155+0000 7ff403e00640 1 Processor -- start 2024-04-25T01:52:45.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.155+0000 7ff403e00640 1 -- start start 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.155+0000 7ff403e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff404101670 0x7ff404197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.155+0000 7ff403e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff404102860 0x7ff404197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff403e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404103220 0x7ff40419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff403e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff40410d340 con 0x7ff404101670 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff403e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff40410d1c0 con 0x7ff404103220 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff403e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff40410d4c0 con 0x7ff404102860 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff402000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff404102860 0x7ff404197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff402000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff404102860 0x7ff404197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:54756/0 (socket says 172.21.15.102:54756) 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff402000640 1 -- 172.21.15.102:0/2124356900 learned_addr learned my addr 172.21.15.102:0/2124356900 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff402a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff404101670 0x7ff404197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff403400640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404103220 0x7ff40419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:45.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff402000640 1 -- 172.21.15.102:0/2124356900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404103220 msgr2=0x7ff40419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff402000640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404103220 0x7ff40419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff402000640 1 -- 172.21.15.102:0/2124356900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff404101670 msgr2=0x7ff404197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff402000640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff404101670 0x7ff404197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.156+0000 7ff402000640 1 -- 172.21.15.102:0/2124356900 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40419f570 con 0x7ff404102860 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.157+0000 7ff402000640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff404102860 0x7ff404197e90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0031440 tx=0x7ff3f0032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.157+0000 7ff3fb400640 1 -- 172.21.15.102:0/2124356900 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f003f030 con 0x7ff404102860 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.157+0000 7ff3fb400640 1 -- 172.21.15.102:0/2124356900 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3f0003ce0 con 0x7ff404102860 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.157+0000 7ff3fb400640 1 -- 172.21.15.102:0/2124356900 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0035070 con 0x7ff404102860 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.157+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40419f800 con 0x7ff404102860 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.157+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff40419fcf0 con 0x7ff404102860 2024-04-25T01:52:45.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.158+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff404108990 con 0x7ff404102860 2024-04-25T01:52:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.160+0000 7ff3fb400640 1 -- 172.21.15.102:0/2124356900 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff3f0042030 con 0x7ff404102860 2024-04-25T01:52:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.161+0000 7ff3fb400640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e80a2b90 0x7ff3e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.161+0000 7ff3fb400640 1 -- 172.21.15.102:0/2124356900 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff3f00ef420 con 0x7ff404102860 2024-04-25T01:52:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.161+0000 7ff402a00640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e80a2b90 0x7ff3e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:45.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.162+0000 7ff402a00640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e80a2b90 0x7ff3e80a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff4041026c0 tx=0x7ff3f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:45.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.165+0000 7ff3fb400640 1 -- 172.21.15.102:0/2124356900 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff3f00bbde0 con 0x7ff404102860 2024-04-25T01:52:45.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:45 smithi102 ceph-mon[27796]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:52:45.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:45 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2677475572' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:52:45.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:45 smithi102 ceph-mon[27796]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:52:45.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:45 smithi102 ceph-mon[27796]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:52:45.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:45 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2440505407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:45.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:45 smithi102 ceph-mon[27796]: pgmap v317: 97 pgs: 10 creating+peering, 1 active+clean, 86 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:45.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:45 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/937788834' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:45.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:45.403+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff40402d170 con 0x7ff404102860 2024-04-25T01:52:45.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:45 smithi110 ceph-mon[32848]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:52:45.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:45 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2677475572' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:52:45.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:45 smithi110 ceph-mon[32848]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:52:45.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:45 smithi110 ceph-mon[32848]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:52:45.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:45 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2440505407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:45.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:45 smithi110 ceph-mon[32848]: pgmap v317: 97 pgs: 10 creating+peering, 1 active+clean, 86 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:45.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:45 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/937788834' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:45.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:45 smithi193 ceph-mon[32981]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T01:52:45.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:45 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2677475572' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T01:52:45.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:45 smithi193 ceph-mon[32981]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T01:52:45.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:45 smithi193 ceph-mon[32981]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T01:52:45.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:45 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2440505407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:45.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:45 smithi193 ceph-mon[32981]: pgmap v317: 97 pgs: 10 creating+peering, 1 active+clean, 86 unknown; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:52:45.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:45 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/937788834' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.113+0000 7ff3fb400640 1 -- 172.21.15.102:0/2124356900 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7ff3f00c00e0 con 0x7ff404102860 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e80a2b90 msgr2=0x7ff3e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e80a2b90 0x7ff3e80a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff4041026c0 tx=0x7ff3f4005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff404102860 msgr2=0x7ff404197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff404102860 0x7ff404197e90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0031440 tx=0x7ff3f0032ce0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 shutdown_connections 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e80a2b90 0x7ff3e80a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff404103220 0x7ff40419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff404102860 0x7ff404197e90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 --2- 172.21.15.102:0/2124356900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff404101670 0x7ff404197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 >> 172.21.15.102:0/2124356900 conn(0x7ff4040fcbe0 msgr2=0x7ff404105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 shutdown_connections 2024-04-25T01:52:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.116+0000 7ff403e00640 1 -- 172.21.15.102:0/2124356900 wait complete. 2024-04-25T01:52:46.135 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-25T01:52:46.135 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-25T01:52:46.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.321+0000 7fe7b6800640 1 -- 172.21.15.102:0/1641536819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 msgr2=0x7fe7b0073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.321+0000 7fe7b6800640 1 --2- 172.21.15.102:0/1641536819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 0x7fe7b0073bd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009930 tx=0x7fe7a402a420 comp rx=0 tx=0).stop 2024-04-25T01:52:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.322+0000 7fe7b6800640 1 -- 172.21.15.102:0/1641536819 shutdown_connections 2024-04-25T01:52:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.322+0000 7fe7b6800640 1 --2- 172.21.15.102:0/1641536819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7b010dca0 0x7fe7b0110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.322+0000 7fe7b6800640 1 --2- 172.21.15.102:0/1641536819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7b00741a0 0x7fe7b0074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.322+0000 7fe7b6800640 1 --2- 172.21.15.102:0/1641536819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 0x7fe7b0073bd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.322+0000 7fe7b6800640 1 -- 172.21.15.102:0/1641536819 >> 172.21.15.102:0/1641536819 conn(0x7fe7b00fcc40 msgr2=0x7fe7b00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.322+0000 7fe7b6800640 1 -- 172.21.15.102:0/1641536819 shutdown_connections 2024-04-25T01:52:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.322+0000 7fe7b6800640 1 -- 172.21.15.102:0/1641536819 wait complete. 2024-04-25T01:52:46.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.323+0000 7fe7b6800640 1 Processor -- start 2024-04-25T01:52:46.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.323+0000 7fe7b6800640 1 -- start start 2024-04-25T01:52:46.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.323+0000 7fe7b6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7b00741a0 0x7fe7b019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:46.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.323+0000 7fe7b6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 0x7fe7b019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:46.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.323+0000 7fe7b6800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7b010dca0 0x7fe7b01a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.323+0000 7fe7b6800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b0113900 con 0x7fe7b0075780 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.323+0000 7fe7b6800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b0113780 con 0x7fe7b010dca0 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.323+0000 7fe7b6800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b0113a80 con 0x7fe7b00741a0 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 0x7fe7b019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7b00741a0 0x7fe7b019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 0x7fe7b019e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58650/0 (socket says 172.21.15.102:58650) 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b4a00640 1 -- 172.21.15.102:0/3134148703 learned_addr learned my addr 172.21.15.102:0/3134148703 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b5e00640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7b010dca0 0x7fe7b01a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b5400640 1 -- 172.21.15.102:0/3134148703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7b010dca0 msgr2=0x7fe7b01a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b5400640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7b010dca0 0x7fe7b01a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b5400640 1 -- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 msgr2=0x7fe7b019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b5400640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 0x7fe7b019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b5400640 1 -- 172.21.15.102:0/3134148703 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b01a5bd0 con 0x7fe7b00741a0 2024-04-25T01:52:46.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b4a00640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 0x7fe7b019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.324+0000 7fe7b5400640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7b00741a0 0x7fe7b019dfb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe7a402a940 tx=0x7fe7a4032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.325+0000 7fe7ae000640 1 -- 172.21.15.102:0/3134148703 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a403f030 con 0x7fe7b00741a0 2024-04-25T01:52:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.325+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b01a5e60 con 0x7fe7b00741a0 2024-04-25T01:52:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.325+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b01a62c0 con 0x7fe7b00741a0 2024-04-25T01:52:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.325+0000 7fe7ae000640 1 -- 172.21.15.102:0/3134148703 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7a4003ce0 con 0x7fe7b00741a0 2024-04-25T01:52:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.325+0000 7fe7ae000640 1 -- 172.21.15.102:0/3134148703 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4049060 con 0x7fe7b00741a0 2024-04-25T01:52:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.327+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b0103e70 con 0x7fe7b00741a0 2024-04-25T01:52:46.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.328+0000 7fe7ae000640 1 -- 172.21.15.102:0/3134148703 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe7a4042030 con 0x7fe7b00741a0 2024-04-25T01:52:46.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.329+0000 7fe7ae000640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe7980a2b90 0x7fe7980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:46.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.330+0000 7fe7ae000640 1 -- 172.21.15.102:0/3134148703 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fe7a40eeff0 con 0x7fe7b00741a0 2024-04-25T01:52:46.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.330+0000 7fe7b4a00640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe7980a2b90 0x7fe7980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:46.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.330+0000 7fe7b4a00640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe7980a2b90 0x7fe7980a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7b019f4c0 tx=0x7fe79c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:46.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.334+0000 7fe7ae000640 1 -- 172.21.15.102:0/3134148703 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe7a40bb9b0 con 0x7fe7b00741a0 2024-04-25T01:52:46.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:46 smithi102 ceph-mon[27796]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:52:46.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:46 smithi102 ceph-mon[27796]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:52:46.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:46 smithi102 ceph-mon[27796]: Cluster is now healthy 2024-04-25T01:52:46.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:46 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] up:active 2024-04-25T01:52:46.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:46 smithi102 ceph-mon[27796]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:46.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:46 smithi102 ceph-mon[27796]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:46.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:46 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2124356900' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:52:46.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:46 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:52:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:46 smithi110 ceph-mon[32848]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:52:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:46 smithi110 ceph-mon[32848]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:52:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:46 smithi110 ceph-mon[32848]: Cluster is now healthy 2024-04-25T01:52:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:46 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] up:active 2024-04-25T01:52:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:46 smithi110 ceph-mon[32848]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:46 smithi110 ceph-mon[32848]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:46 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2124356900' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:52:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:46 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:52:46.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:46 smithi193 ceph-mon[32981]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T01:52:46.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:46 smithi193 ceph-mon[32981]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T01:52:46.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:46 smithi193 ceph-mon[32981]: Cluster is now healthy 2024-04-25T01:52:46.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:46 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] up:active 2024-04-25T01:52:46.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:46 smithi193 ceph-mon[32981]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:46.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:46 smithi193 ceph-mon[32981]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:46.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:46 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2124356900' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:52:46.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:46 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T01:52:46.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:46.578+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fe7b002d390 con 0x7fe7b00741a0 2024-04-25T01:52:47.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.053+0000 7fe7ae000640 1 -- 172.21.15.102:0/3134148703 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fe7a40bfcb0 con 0x7fe7b00741a0 2024-04-25T01:52:47.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.058+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe7980a2b90 msgr2=0x7fe7980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:47.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.058+0000 7fe7b6800640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe7980a2b90 0x7fe7980a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7b019f4c0 tx=0x7fe79c005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:47.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.058+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7b00741a0 msgr2=0x7fe7b019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:47.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.058+0000 7fe7b6800640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7b00741a0 0x7fe7b019dfb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe7a402a940 tx=0x7fe7a4032ce0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.059+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 shutdown_connections 2024-04-25T01:52:47.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.059+0000 7fe7b6800640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe7980a2b90 0x7fe7980a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.059+0000 7fe7b6800640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7b010dca0 0x7fe7b01a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.059+0000 7fe7b6800640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7b0075780 0x7fe7b019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.059+0000 7fe7b6800640 1 --2- 172.21.15.102:0/3134148703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7b00741a0 0x7fe7b019dfb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.059+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 >> 172.21.15.102:0/3134148703 conn(0x7fe7b00fcc40 msgr2=0x7fe7b0101040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:47.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.060+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 shutdown_connections 2024-04-25T01:52:47.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.060+0000 7fe7b6800640 1 -- 172.21.15.102:0/3134148703 wait complete. 2024-04-25T01:52:47.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T01:52:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.271+0000 7f7ba3e00640 1 -- 172.21.15.102:0/540619396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c10bb30 msgr2=0x7f7b9c1123c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.271+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/540619396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c10bb30 0x7f7b9c1123c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009be0 tx=0x7f7b9402a6c0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.271+0000 7f7ba3e00640 1 -- 172.21.15.102:0/540619396 shutdown_connections 2024-04-25T01:52:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.271+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/540619396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c10bb30 0x7f7b9c1123c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.271+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/540619396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9c10b170 0x7f7b9c10b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.271+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/540619396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b9c109f80 0x7f7b9c10a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.271+0000 7f7ba3e00640 1 -- 172.21.15.102:0/540619396 >> 172.21.15.102:0/540619396 conn(0x7f7b9c070730 msgr2=0x7f7b9c072b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.271+0000 7f7ba3e00640 1 -- 172.21.15.102:0/540619396 shutdown_connections 2024-04-25T01:52:47.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.271+0000 7f7ba3e00640 1 -- 172.21.15.102:0/540619396 wait complete. 2024-04-25T01:52:47.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.272+0000 7f7ba3e00640 1 Processor -- start 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.272+0000 7f7ba3e00640 1 -- start start 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.272+0000 7f7ba3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b9c109f80 0x7f7b9c06ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.272+0000 7f7ba3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c10b170 0x7f7b9c06b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.272+0000 7f7ba3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9c10bb30 0x7f7b9c06b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.272+0000 7f7ba3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c115c10 con 0x7f7b9c10b170 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.272+0000 7f7ba3e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c115a90 con 0x7f7b9c10bb30 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.272+0000 7f7ba3e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c115d90 con 0x7f7b9c109f80 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b9c109f80 0x7f7b9c06ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c10b170 0x7f7b9c06b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b9c109f80 0x7f7b9c06ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:54788/0 (socket says 172.21.15.102:54788) 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1400640 1 -- 172.21.15.102:0/3695088362 learned_addr learned my addr 172.21.15.102:0/3695088362 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:47.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1e00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9c10bb30 0x7f7b9c06b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1400640 1 -- 172.21.15.102:0/3695088362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9c10bb30 msgr2=0x7f7b9c06b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1400640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9c10bb30 0x7f7b9c06b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1400640 1 -- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c10b170 msgr2=0x7f7b9c06b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1400640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c10b170 0x7f7b9c06b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1400640 1 -- 172.21.15.102:0/3695088362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9c06be40 con 0x7f7b9c109f80 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba0a00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c10b170 0x7f7b9c06b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.273+0000 7f7ba1400640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b9c109f80 0x7f7b9c06ca30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000e9a0 tx=0x7f7b9000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.274+0000 7f7b9a000640 1 -- 172.21.15.102:0/3695088362 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9000cd80 con 0x7f7b9c109f80 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.274+0000 7f7b9a000640 1 -- 172.21.15.102:0/3695088362 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b90010910 con 0x7f7b9c109f80 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.274+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b9c1aa360 con 0x7f7b9c109f80 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.274+0000 7f7b9a000640 1 -- 172.21.15.102:0/3695088362 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90018a30 con 0x7f7b9c109f80 2024-04-25T01:52:47.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.274+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b9c1aa8b0 con 0x7f7b9c109f80 2024-04-25T01:52:47.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.275+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b9c1112a0 con 0x7f7b9c109f80 2024-04-25T01:52:47.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.277+0000 7f7b9a000640 1 -- 172.21.15.102:0/3695088362 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7b90010ac0 con 0x7f7b9c109f80 2024-04-25T01:52:47.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.278+0000 7f7b9a000640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7b840a2b90 0x7f7b840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:47.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.278+0000 7f7b9a000640 1 -- 172.21.15.102:0/3695088362 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f7b900cf320 con 0x7f7b9c109f80 2024-04-25T01:52:47.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.278+0000 7f7ba0a00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7b840a2b90 0x7f7b840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:47.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.279+0000 7f7ba0a00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7b840a2b90 0x7f7b840a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c06c7a0 tx=0x7f7b88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:47.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.283+0000 7f7b9a000640 1 -- 172.21.15.102:0/3695088362 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b90053740 con 0x7f7b9c109f80 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3134148703' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: pgmap v319: 97 pgs: 10 creating+peering, 44 active+clean, 43 unknown; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 3 op/s 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:52:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:47 smithi102 ceph-mon[27796]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:52:47.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:52:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3134148703' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:52:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:52:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: pgmap v319: 97 pgs: 10 creating+peering, 44 active+clean, 43 unknown; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 3 op/s 2024-04-25T01:52:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:52:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:52:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T01:52:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:52:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-mon[32848]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:52:47.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T01:52:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T01:52:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3134148703' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:52:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T01:52:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: pgmap v319: 97 pgs: 10 creating+peering, 44 active+clean, 43 unknown; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 3 op/s 2024-04-25T01:52:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T01:52:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T01:52:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T01:52:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T01:52:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:47 smithi193 ceph-mon[32981]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T01:52:47.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:47.524+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f7b9c02d170 con 0x7f7b9c109f80 2024-04-25T01:52:48.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.071+0000 7f7b9a000640 1 -- 172.21.15.102:0/3695088362 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f7b9009c330 con 0x7f7b9c109f80 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7b840a2b90 msgr2=0x7f7b840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7b840a2b90 0x7f7b840a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c06c7a0 tx=0x7f7b88005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b9c109f80 msgr2=0x7f7b9c06ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b9c109f80 0x7f7b9c06ca30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000e9a0 tx=0x7f7b9000ee60 comp rx=0 tx=0).stop 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 shutdown_connections 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7b840a2b90 0x7f7b840a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9c10bb30 0x7f7b9c06b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c10b170 0x7f7b9c06b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 --2- 172.21.15.102:0/3695088362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b9c109f80 0x7f7b9c06ca30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 >> 172.21.15.102:0/3695088362 conn(0x7f7b9c070730 msgr2=0x7f7b9c10e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 shutdown_connections 2024-04-25T01:52:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.075+0000 7f7ba3e00640 1 -- 172.21.15.102:0/3695088362 wait complete. 2024-04-25T01:52:48.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T01:52:48.117 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 01:52:47 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b[64106]: 2024-04-25T01:52:47.793+0000 7f8825b52700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T01:52:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.286+0000 7f6441000640 1 -- 172.21.15.102:0/3528540593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c1039a0 msgr2=0x7f643c103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.286+0000 7f6441000640 1 --2- 172.21.15.102:0/3528540593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c1039a0 0x7f643c103dc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f642c009900 tx=0x7f642c02a410 comp rx=0 tx=0).stop 2024-04-25T01:52:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.286+0000 7f6441000640 1 -- 172.21.15.102:0/3528540593 shutdown_connections 2024-04-25T01:52:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.286+0000 7f6441000640 1 --2- 172.21.15.102:0/3528540593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c105550 0x7f643c10bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.286+0000 7f6441000640 1 --2- 172.21.15.102:0/3528540593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f643c104b90 0x7f643c105010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.286+0000 7f6441000640 1 --2- 172.21.15.102:0/3528540593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c1039a0 0x7f643c103dc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.286+0000 7f6441000640 1 -- 172.21.15.102:0/3528540593 >> 172.21.15.102:0/3528540593 conn(0x7f643c0fef30 msgr2=0x7f643c101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.286+0000 7f6441000640 1 -- 172.21.15.102:0/3528540593 shutdown_connections 2024-04-25T01:52:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.286+0000 7f6441000640 1 -- 172.21.15.102:0/3528540593 wait complete. 2024-04-25T01:52:48.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.287+0000 7f6441000640 1 Processor -- start 2024-04-25T01:52:48.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.287+0000 7f6441000640 1 -- start start 2024-04-25T01:52:48.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.287+0000 7f6441000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1039a0 0x7f643c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.287+0000 7f6441000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f643c104b90 0x7f643c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.287+0000 7f6441000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c105550 0x7f643c1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.287+0000 7f6441000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f643c10f660 con 0x7f643c1039a0 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.287+0000 7f6441000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f643c10f4e0 con 0x7f643c105550 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.287+0000 7f6441000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f643c10f7e0 con 0x7f643c104b90 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f643a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c105550 0x7f643c1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f6431400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f643c104b90 0x7f643c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f643a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c105550 0x7f643c1a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54376/0 (socket says 172.21.15.102:54376) 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f643a800640 1 -- 172.21.15.102:0/827216372 learned_addr learned my addr 172.21.15.102:0/827216372 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f6439e00640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1039a0 0x7f643c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f643a800640 1 -- 172.21.15.102:0/827216372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f643c104b90 msgr2=0x7f643c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f643a800640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f643c104b90 0x7f643c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f643a800640 1 -- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1039a0 msgr2=0x7f643c199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:48.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f643a800640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1039a0 0x7f643c199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:48.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f643a800640 1 -- 172.21.15.102:0/827216372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643c1a1920 con 0x7f643c105550 2024-04-25T01:52:48.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.288+0000 7f643a800640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c105550 0x7f643c1a1280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f643400eb10 tx=0x7f643400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:48.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.289+0000 7f6433400640 1 -- 172.21.15.102:0/827216372 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434004b20 con 0x7f643c105550 2024-04-25T01:52:48.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.289+0000 7f6433400640 1 -- 172.21.15.102:0/827216372 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6434004cc0 con 0x7f643c105550 2024-04-25T01:52:48.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.289+0000 7f6433400640 1 -- 172.21.15.102:0/827216372 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434010760 con 0x7f643c105550 2024-04-25T01:52:48.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.289+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f643c1a1c10 con 0x7f643c105550 2024-04-25T01:52:48.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.289+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f643c1a20e0 con 0x7f643c105550 2024-04-25T01:52:48.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.290+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f643c10ad60 con 0x7f643c105550 2024-04-25T01:52:48.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.292+0000 7f6433400640 1 -- 172.21.15.102:0/827216372 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6434005020 con 0x7f643c105550 2024-04-25T01:52:48.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.293+0000 7f6433400640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f64200a2c10 0x7f64200a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:48.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.293+0000 7f6433400640 1 -- 172.21.15.102:0/827216372 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f64340cf550 con 0x7f643c105550 2024-04-25T01:52:48.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.293+0000 7f6439e00640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f64200a2c10 0x7f64200a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:48.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.294+0000 7f6439e00640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f64200a2c10 0x7f64200a50d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f642c02aa00 tx=0x7f642c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:48.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.297+0000 7f6433400640 1 -- 172.21.15.102:0/827216372 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f643409c560 con 0x7f643c105550 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3695088362' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] up:active 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: Cluster is now healthy 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:52:48.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:48 smithi102 ceph-mon[27796]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:52:48.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3695088362' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:52:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:52:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:52:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] up:active 2024-04-25T01:52:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:52:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:52:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: Cluster is now healthy 2024-04-25T01:52:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:52:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:52:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:48 smithi110 ceph-mon[32848]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:52:48.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3695088362' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:52:48.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T01:52:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T01:52:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] up:active 2024-04-25T01:52:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T01:52:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T01:52:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: Cluster is now healthy 2024-04-25T01:52:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T01:52:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-25T01:52:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:48 smithi193 ceph-mon[32981]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-25T01:52:48.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:48.538+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f643c062340 con 0x7f643c105550 2024-04-25T01:52:49.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.083+0000 7f6433400640 1 -- 172.21.15.102:0/827216372 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f643409be60 con 0x7f643c105550 2024-04-25T01:52:49.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.087+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f64200a2c10 msgr2=0x7f64200a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.087+0000 7f6441000640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f64200a2c10 0x7f64200a50d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f642c02aa00 tx=0x7f642c0315b0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.087+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c105550 msgr2=0x7f643c1a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.087+0000 7f6441000640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c105550 0x7f643c1a1280 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f643400eb10 tx=0x7f643400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.088+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 shutdown_connections 2024-04-25T01:52:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.088+0000 7f6441000640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f64200a2c10 0x7f64200a50d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.088+0000 7f6441000640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f643c105550 0x7f643c1a1280 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.088+0000 7f6441000640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f643c104b90 0x7f643c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.088+0000 7f6441000640 1 --2- 172.21.15.102:0/827216372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f643c1039a0 0x7f643c199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.088+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 >> 172.21.15.102:0/827216372 conn(0x7f643c0fef30 msgr2=0x7f643c107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.088+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 shutdown_connections 2024-04-25T01:52:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.088+0000 7f6441000640 1 -- 172.21.15.102:0/827216372 wait complete. 2024-04-25T01:52:49.114 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-25T01:52:49.114 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T01:52:49.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.316+0000 7fb3cb000640 1 -- 172.21.15.102:0/305794429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4102970 msgr2=0x7fb3c4102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.316+0000 7fb3cb000640 1 --2- 172.21.15.102:0/305794429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4102970 0x7fb3c4102df0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009900 tx=0x7fb3ac02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.316+0000 7fb3cb000640 1 -- 172.21.15.102:0/305794429 shutdown_connections 2024-04-25T01:52:49.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.316+0000 7fb3cb000640 1 --2- 172.21.15.102:0/305794429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3c4103330 0x7fb3c4109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.316+0000 7fb3cb000640 1 --2- 172.21.15.102:0/305794429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4102970 0x7fb3c4102df0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.316+0000 7fb3cb000640 1 --2- 172.21.15.102:0/305794429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3c4101780 0x7fb3c4101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.316+0000 7fb3cb000640 1 -- 172.21.15.102:0/305794429 >> 172.21.15.102:0/305794429 conn(0x7fb3c40fccf0 msgr2=0x7fb3c40ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:49.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.316+0000 7fb3cb000640 1 -- 172.21.15.102:0/305794429 shutdown_connections 2024-04-25T01:52:49.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.316+0000 7fb3cb000640 1 -- 172.21.15.102:0/305794429 wait complete. 2024-04-25T01:52:49.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.317+0000 7fb3cb000640 1 Processor -- start 2024-04-25T01:52:49.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.317+0000 7fb3cb000640 1 -- start start 2024-04-25T01:52:49.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3cb000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3c4101780 0x7fb3c4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:49.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3cb000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3c4102970 0x7fb3c4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:49.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3cb000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4103330 0x7fb3c419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3cb000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c410d450 con 0x7fb3c4101780 2024-04-25T01:52:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3cb000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c410d2d0 con 0x7fb3c4103330 2024-04-25T01:52:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3cb000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c410d5d0 con 0x7fb3c4102970 2024-04-25T01:52:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c9000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4103330 0x7fb3c419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3c4101780 0x7fb3c4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c9000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4103330 0x7fb3c419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54406/0 (socket says 172.21.15.102:54406) 2024-04-25T01:52:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3c4101780 0x7fb3c4197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58714/0 (socket says 172.21.15.102:58714) 2024-04-25T01:52:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c9000640 1 -- 172.21.15.102:0/2779672536 learned_addr learned my addr 172.21.15.102:0/2779672536 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3bbe00640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3c4102970 0x7fb3c4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c9000640 1 -- 172.21.15.102:0/2779672536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3c4102970 msgr2=0x7fb3c4198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c9000640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3c4102970 0x7fb3c4198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c9000640 1 -- 172.21.15.102:0/2779672536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3c4101780 msgr2=0x7fb3c4197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c9000640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3c4101780 0x7fb3c4197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.318+0000 7fb3c9000640 1 -- 172.21.15.102:0/2779672536 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c419f6e0 con 0x7fb3c4103330 2024-04-25T01:52:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.319+0000 7fb3c9000640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4103330 0x7fb3c419f070 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00eb10 tx=0x7fb3bc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.319+0000 7fb3c1600640 1 -- 172.21.15.102:0/2779672536 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc004b20 con 0x7fb3c4103330 2024-04-25T01:52:49.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.319+0000 7fb3c1600640 1 -- 172.21.15.102:0/2779672536 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3bc004cc0 con 0x7fb3c4103330 2024-04-25T01:52:49.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.319+0000 7fb3c1600640 1 -- 172.21.15.102:0/2779672536 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc010760 con 0x7fb3c4103330 2024-04-25T01:52:49.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.319+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c419f9d0 con 0x7fb3c4103330 2024-04-25T01:52:49.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.319+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c419fef0 con 0x7fb3c4103330 2024-04-25T01:52:49.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.320+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3c4108b20 con 0x7fb3c4103330 2024-04-25T01:52:49.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.322+0000 7fb3c1600640 1 -- 172.21.15.102:0/2779672536 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb3bc005020 con 0x7fb3c4103330 2024-04-25T01:52:49.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.323+0000 7fb3c1600640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb3a80a2c10 0x7fb3a80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:49.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.323+0000 7fb3c1600640 1 -- 172.21.15.102:0/2779672536 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fb3bc0cf4c0 con 0x7fb3c4103330 2024-04-25T01:52:49.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.324+0000 7fb3c3e00640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb3a80a2c10 0x7fb3a80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:49.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.324+0000 7fb3c3e00640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb3a80a2c10 0x7fb3a80a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3c41027d0 tx=0x7fb3b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:49.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.327+0000 7fb3c1600640 1 -- 172.21.15.102:0/2779672536 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb3bc09c4d0 con 0x7fb3c4103330 2024-04-25T01:52:49.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/827216372' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:52:49.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:52:49.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-mon[27796]: pgmap v320: 97 pgs: 97 active+clean; 580 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 9 op/s 2024-04-25T01:52:49.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:52:49.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] up:active 2024-04-25T01:52:49.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:52:49.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:52:49.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:49 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/827216372' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:52:49.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:49 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:52:49.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:49 smithi110 ceph-mon[32848]: pgmap v320: 97 pgs: 97 active+clean; 580 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 9 op/s 2024-04-25T01:52:49.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:49 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:52:49.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:49 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] up:active 2024-04-25T01:52:49.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:49 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:52:49.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:49 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:52:49.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:49 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/827216372' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:52:49.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:49 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T01:52:49.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:49 smithi193 ceph-mon[32981]: pgmap v320: 97 pgs: 97 active+clean; 580 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s wr, 9 op/s 2024-04-25T01:52:49.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:49 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T01:52:49.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:49 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] up:active 2024-04-25T01:52:49.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:49 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-25T01:52:49.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:49 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:52:49.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.553+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb3c4061b80 con 0x7fb3a80a2c10 2024-04-25T01:52:49.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.582+0000 7fb3c1600640 1 -- 172.21.15.102:0/2779672536 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb3c4061b80 con 0x7fb3a80a2c10 2024-04-25T01:52:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb3a80a2c10 msgr2=0x7fb3a80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb3a80a2c10 0x7fb3a80a50d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3c41027d0 tx=0x7fb3b4005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:49.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4103330 msgr2=0x7fb3c419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4103330 0x7fb3c419f070 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00eb10 tx=0x7fb3bc00c2d0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 shutdown_connections 2024-04-25T01:52:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb3a80a2c10 0x7fb3a80a50d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c4103330 0x7fb3c419f070 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3c4102970 0x7fb3c4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 --2- 172.21.15.102:0/2779672536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3c4101780 0x7fb3c4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 >> 172.21.15.102:0/2779672536 conn(0x7fb3c40fccf0 msgr2=0x7fb3c4105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 shutdown_connections 2024-04-25T01:52:49.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.584+0000 7fb3cb000640 1 -- 172.21.15.102:0/2779672536 wait complete. 2024-04-25T01:52:49.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T01:52:49.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.783+0000 7f31e6600640 1 -- 172.21.15.102:0/167659972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0105430 msgr2=0x7f31e010bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.783+0000 7f31e6600640 1 --2- 172.21.15.102:0/167659972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0105430 0x7f31e010bcc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009ca0 tx=0x7f31d802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.783+0000 7f31e6600640 1 -- 172.21.15.102:0/167659972 shutdown_connections 2024-04-25T01:52:49.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.783+0000 7f31e6600640 1 --2- 172.21.15.102:0/167659972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0105430 0x7f31e010bcc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.783+0000 7f31e6600640 1 --2- 172.21.15.102:0/167659972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31e0104a70 0x7f31e0104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.783+0000 7f31e6600640 1 --2- 172.21.15.102:0/167659972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31e0103880 0x7f31e0103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.783+0000 7f31e6600640 1 -- 172.21.15.102:0/167659972 >> 172.21.15.102:0/167659972 conn(0x7f31e00fedf0 msgr2=0x7f31e0101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:49.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.783+0000 7f31e6600640 1 -- 172.21.15.102:0/167659972 shutdown_connections 2024-04-25T01:52:49.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.783+0000 7f31e6600640 1 -- 172.21.15.102:0/167659972 wait complete. 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.784+0000 7f31e6600640 1 Processor -- start 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.784+0000 7f31e6600640 1 -- start start 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.784+0000 7f31e6600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0103880 0x7f31e0199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e6600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31e0104a70 0x7f31e019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e6600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31e0105430 0x7f31e01a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e6600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f31e010f550 con 0x7f31e0105430 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e6600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f31e010f3d0 con 0x7f31e0103880 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e6600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f31e010f6d0 con 0x7f31e0104a70 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0103880 0x7f31e0199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0103880 0x7f31e0199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54442/0 (socket says 172.21.15.102:54442) 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5200640 1 -- 172.21.15.102:0/1253691808 learned_addr learned my addr 172.21.15.102:0/1253691808 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5c00640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31e0105430 0x7f31e01a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:49.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31dfe00640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31e0104a70 0x7f31e019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5200640 1 -- 172.21.15.102:0/1253691808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31e0104a70 msgr2=0x7f31e019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5200640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31e0104a70 0x7f31e019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5200640 1 -- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31e0105430 msgr2=0x7f31e01a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5200640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31e0105430 0x7f31e01a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5200640 1 -- 172.21.15.102:0/1253691808 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e01a1760 con 0x7f31e0103880 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.785+0000 7f31e5200640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0103880 0x7f31e0199b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f31d400e9c0 tx=0x7f31d400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.786+0000 7f31dd600640 1 -- 172.21.15.102:0/1253691808 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d400cd80 con 0x7f31e0103880 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.786+0000 7f31dd600640 1 -- 172.21.15.102:0/1253691808 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31d40035f0 con 0x7f31e0103880 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.786+0000 7f31dd600640 1 -- 172.21.15.102:0/1253691808 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4010690 con 0x7f31e0103880 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.786+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e01a1a50 con 0x7f31e0103880 2024-04-25T01:52:49.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.786+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e01a1fa0 con 0x7f31e0103880 2024-04-25T01:52:49.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.787+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e010ab40 con 0x7f31e0103880 2024-04-25T01:52:49.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.789+0000 7f31dd600640 1 -- 172.21.15.102:0/1253691808 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f31d40037a0 con 0x7f31e0103880 2024-04-25T01:52:49.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.790+0000 7f31dd600640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f31c80a2b90 0x7f31c80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.790+0000 7f31dd600640 1 -- 172.21.15.102:0/1253691808 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f31d40cf370 con 0x7f31e0103880 2024-04-25T01:52:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.790+0000 7f31dfe00640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f31c80a2b90 0x7f31c80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.791+0000 7f31dfe00640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f31c80a2b90 0x7f31c80a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f31e019b050 tx=0x7f31cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:49.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:49.794+0000 7f31dd600640 1 -- 172.21.15.102:0/1253691808 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f31d409c380 con 0x7f31e0103880 2024-04-25T01:52:49.923 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:52:49.562+0000 7fd1fb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:49.923 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:52:49.562+0000 7fd1fb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:49.923 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:52:49.562+0000 7fd1fb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:49.923 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:52:49.562+0000 7fd1fb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:49.923 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:52:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:52:49.562+0000 7fd1fb000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:52:50.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.020+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f31e0061b80 con 0x7f31c80a2b90 2024-04-25T01:52:50.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.053+0000 7f31dd600640 1 -- 172.21.15.102:0/1253691808 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f31e0061b80 con 0x7f31c80a2b90 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f31c80a2b90 msgr2=0x7f31c80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f31c80a2b90 0x7f31c80a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f31e019b050 tx=0x7f31cc005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0103880 msgr2=0x7f31e0199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0103880 0x7f31e0199b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f31d400e9c0 tx=0x7f31d400ee80 comp rx=0 tx=0).stop 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 shutdown_connections 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f31c80a2b90 0x7f31c80a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31e0105430 0x7f31e01a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31e0104a70 0x7f31e019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 --2- 172.21.15.102:0/1253691808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31e0103880 0x7f31e0199b40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 >> 172.21.15.102:0/1253691808 conn(0x7f31e00fedf0 msgr2=0x7f31e0107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:50.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 shutdown_connections 2024-04-25T01:52:50.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.056+0000 7f31e6600640 1 -- 172.21.15.102:0/1253691808 wait complete. 2024-04-25T01:52:50.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T01:52:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.280+0000 7fb6bda00640 1 -- 172.21.15.102:0/1637906187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8102840 msgr2=0x7fb6b8102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.280+0000 7fb6bda00640 1 --2- 172.21.15.102:0/1637906187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8102840 0x7fb6b8102cc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0009900 tx=0x7fb6a002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.280+0000 7fb6bda00640 1 -- 172.21.15.102:0/1637906187 shutdown_connections 2024-04-25T01:52:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.280+0000 7fb6bda00640 1 --2- 172.21.15.102:0/1637906187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6b8103200 0x7fb6b8109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.280+0000 7fb6bda00640 1 --2- 172.21.15.102:0/1637906187 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8102840 0x7fb6b8102cc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.280+0000 7fb6bda00640 1 --2- 172.21.15.102:0/1637906187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6b8101650 0x7fb6b8101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.280+0000 7fb6bda00640 1 -- 172.21.15.102:0/1637906187 >> 172.21.15.102:0/1637906187 conn(0x7fb6b80fcbe0 msgr2=0x7fb6b80ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.280+0000 7fb6bda00640 1 -- 172.21.15.102:0/1637906187 shutdown_connections 2024-04-25T01:52:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.280+0000 7fb6bda00640 1 -- 172.21.15.102:0/1637906187 wait complete. 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.281+0000 7fb6bda00640 1 Processor -- start 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6bda00640 1 -- start start 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6bda00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8101650 0x7fb6b80754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6bda00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6b8102840 0x7fb6b8073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6bda00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6b8103200 0x7fb6b8074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6bda00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b810d2e0 con 0x7fb6b8103200 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6bda00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b810d160 con 0x7fb6b8101650 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6bda00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b810d460 con 0x7fb6b8102840 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6b7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8101650 0x7fb6b80754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6b7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8101650 0x7fb6b80754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54468/0 (socket says 172.21.15.102:54468) 2024-04-25T01:52:50.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6bd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6b8103200 0x7fb6b8074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.282+0000 7fb6b7e00640 1 -- 172.21.15.102:0/3906903066 learned_addr learned my addr 172.21.15.102:0/3906903066 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6afe00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6b8102840 0x7fb6b8073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6b7e00640 1 -- 172.21.15.102:0/3906903066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6b8102840 msgr2=0x7fb6b8073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6b7e00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6b8102840 0x7fb6b8073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6b7e00640 1 -- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6b8103200 msgr2=0x7fb6b8074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6b7e00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6b8103200 0x7fb6b8074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6b7e00640 1 -- 172.21.15.102:0/3906903066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b8074900 con 0x7fb6b8101650 2024-04-25T01:52:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6afe00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6b8102840 0x7fb6b8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6bd000640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6b8103200 0x7fb6b8074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6b7e00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8101650 0x7fb6b80754f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800e9c0 tx=0x7fb6a800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6b5600640 1 -- 172.21.15.102:0/3906903066 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a800cd80 con 0x7fb6b8101650 2024-04-25T01:52:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6b5600640 1 -- 172.21.15.102:0/3906903066 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6a80035f0 con 0x7fb6b8101650 2024-04-25T01:52:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6b5600640 1 -- 172.21.15.102:0/3906903066 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a8010690 con 0x7fb6b8101650 2024-04-25T01:52:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.283+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b81a1aa0 con 0x7fb6b8101650 2024-04-25T01:52:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.284+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6b81a1f70 con 0x7fb6b8101650 2024-04-25T01:52:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.285+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6b80688d0 con 0x7fb6b8101650 2024-04-25T01:52:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.286+0000 7fb6b5600640 1 -- 172.21.15.102:0/3906903066 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb6a80037a0 con 0x7fb6b8101650 2024-04-25T01:52:50.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.287+0000 7fb6b5600640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6980a2c10 0x7fb6980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.288+0000 7fb6b5600640 1 -- 172.21.15.102:0/3906903066 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fb6a80cf350 con 0x7fb6b8101650 2024-04-25T01:52:50.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.288+0000 7fb6afe00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6980a2c10 0x7fb6980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.288+0000 7fb6afe00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6980a2c10 0x7fb6980a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0000c00 tx=0x7fb6a00095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:50.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.291+0000 7fb6b5600640 1 -- 172.21.15.102:0/3906903066 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb6a809c360 con 0x7fb6b8101650 2024-04-25T01:52:50.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:50 smithi102 ceph-mon[27796]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:50.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:50 smithi110 ceph-mon[32848]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:50.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:50 smithi193 ceph-mon[32981]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:50.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.522+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fb6b8061c40 con 0x7fb6980a2c10 2024-04-25T01:52:50.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.540+0000 7fb6b5600640 1 -- 172.21.15.102:0/3906903066 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fb6b8061c40 con 0x7fb6980a2c10 2024-04-25T01:52:50.541 INFO:teuthology.orchestra.run.smithi102.stdout:/volumes/qa/sv_0/b798e3d1-32f3-48e9-b02e-e5e15977d118 2024-04-25T01:52:50.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6980a2c10 msgr2=0x7fb6980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6980a2c10 0x7fb6980a50d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0000c00 tx=0x7fb6a00095f0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8101650 msgr2=0x7fb6b80754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8101650 0x7fb6b80754f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800e9c0 tx=0x7fb6a800ee80 comp rx=0 tx=0).stop 2024-04-25T01:52:50.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 shutdown_connections 2024-04-25T01:52:50.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6980a2c10 0x7fb6980a50d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6b8103200 0x7fb6b8074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6b8102840 0x7fb6b8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 --2- 172.21.15.102:0/3906903066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6b8101650 0x7fb6b80754f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.542+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 >> 172.21.15.102:0/3906903066 conn(0x7fb6b80fcbe0 msgr2=0x7fb6b8105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:50.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.543+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 shutdown_connections 2024-04-25T01:52:50.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.543+0000 7fb6bda00640 1 -- 172.21.15.102:0/3906903066 wait complete. 2024-04-25T01:52:50.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T01:52:50.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.774+0000 7f7cbac00640 1 -- 172.21.15.102:0/3665346065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4107540 msgr2=0x7f7cb4109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.774+0000 7f7cbac00640 1 --2- 172.21.15.102:0/3665346065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4107540 0x7f7cb4109930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009be0 tx=0x7f7cac02a6d0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.774+0000 7f7cbac00640 1 -- 172.21.15.102:0/3665346065 shutdown_connections 2024-04-25T01:52:50.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.774+0000 7f7cbac00640 1 --2- 172.21.15.102:0/3665346065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4107540 0x7f7cb4109930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.774+0000 7f7cbac00640 1 --2- 172.21.15.102:0/3665346065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb4104c10 0x7f7cb4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.774+0000 7f7cbac00640 1 --2- 172.21.15.102:0/3665346065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb41022e0 0x7f7cb41046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.774+0000 7f7cbac00640 1 -- 172.21.15.102:0/3665346065 >> 172.21.15.102:0/3665346065 conn(0x7f7cb40fbc10 msgr2=0x7f7cb40fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:50.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.774+0000 7f7cbac00640 1 -- 172.21.15.102:0/3665346065 shutdown_connections 2024-04-25T01:52:50.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.774+0000 7f7cbac00640 1 -- 172.21.15.102:0/3665346065 wait complete. 2024-04-25T01:52:50.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.775+0000 7f7cbac00640 1 Processor -- start 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.775+0000 7f7cbac00640 1 -- start start 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.775+0000 7f7cbac00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb41022e0 0x7f7cb4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.775+0000 7f7cbac00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb4104c10 0x7f7cb4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cbac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4107540 0x7f7cb419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cbac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb410d3e0 con 0x7f7cb4107540 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cbac00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb410d260 con 0x7f7cb41022e0 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cbac00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb410d560 con 0x7f7cb4104c10 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb41022e0 0x7f7cb4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cabe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb4104c10 0x7f7cb4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4107540 0x7f7cb419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb41022e0 0x7f7cb4197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54482/0 (socket says 172.21.15.102:54482) 2024-04-25T01:52:50.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb3e00640 1 -- 172.21.15.102:0/715881584 learned_addr learned my addr 172.21.15.102:0/715881584 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb3e00640 1 -- 172.21.15.102:0/715881584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb4104c10 msgr2=0x7f7cb4198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb3e00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb4104c10 0x7f7cb4198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb3e00640 1 -- 172.21.15.102:0/715881584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4107540 msgr2=0x7f7cb419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb3e00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4107540 0x7f7cb419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb3e00640 1 -- 172.21.15.102:0/715881584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb419f6e0 con 0x7f7cb41022e0 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.776+0000 7f7cb3e00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb41022e0 0x7f7cb4197ac0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400ea60 tx=0x7f7ca400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.777+0000 7f7cb1600640 1 -- 172.21.15.102:0/715881584 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca400cde0 con 0x7f7cb41022e0 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.777+0000 7f7cb1600640 1 -- 172.21.15.102:0/715881584 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca40035f0 con 0x7f7cb41022e0 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.777+0000 7f7cb1600640 1 -- 172.21.15.102:0/715881584 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca4010670 con 0x7f7cb41022e0 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.777+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb419f9d0 con 0x7f7cb41022e0 2024-04-25T01:52:50.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.777+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb419fed0 con 0x7f7cb41022e0 2024-04-25T01:52:50.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.778+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb40fd700 con 0x7f7cb41022e0 2024-04-25T01:52:50.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.780+0000 7f7cb1600640 1 -- 172.21.15.102:0/715881584 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7ca40037a0 con 0x7f7cb41022e0 2024-04-25T01:52:50.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.781+0000 7f7cb1600640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c980a2c10 0x7f7c980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:50.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.781+0000 7f7cb1600640 1 -- 172.21.15.102:0/715881584 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f7ca40cf220 con 0x7f7cb41022e0 2024-04-25T01:52:50.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.781+0000 7f7cabe00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c980a2c10 0x7f7c980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:50.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.782+0000 7f7cabe00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c980a2c10 0x7f7c980a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4198fd0 tx=0x7f7c9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:50.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:50.785+0000 7f7cb1600640 1 -- 172.21.15.102:0/715881584 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7ca409c230 con 0x7f7cb41022e0 2024-04-25T01:52:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.014+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7cb4061b80 con 0x7f7c980a2c10 2024-04-25T01:52:51.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.051+0000 7f7cb1600640 1 -- 172.21.15.102:0/715881584 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7cb4061b80 con 0x7f7c980a2c10 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c980a2c10 msgr2=0x7f7c980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c980a2c10 0x7f7c980a50d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4198fd0 tx=0x7f7c9c009260 comp rx=0 tx=0).stop 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb41022e0 msgr2=0x7f7cb4197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb41022e0 0x7f7cb4197ac0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7ca400ea60 tx=0x7f7ca400ef20 comp rx=0 tx=0).stop 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 shutdown_connections 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c980a2c10 0x7f7c980a50d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4107540 0x7f7cb419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb4104c10 0x7f7cb4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 --2- 172.21.15.102:0/715881584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb41022e0 0x7f7cb4197ac0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 >> 172.21.15.102:0/715881584 conn(0x7f7cb40fbc10 msgr2=0x7f7cb4102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:51.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 shutdown_connections 2024-04-25T01:52:51.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.054+0000 7f7cbac00640 1 -- 172.21.15.102:0/715881584 wait complete. 2024-04-25T01:52:51.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T01:52:51.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.273+0000 7f5526a00640 1 -- 172.21.15.102:0/4060872777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 msgr2=0x7f55201080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.273+0000 7f5526a00640 1 --2- 172.21.15.102:0/4060872777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 0x7f55201080a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5514009900 tx=0x7f551402a410 comp rx=0 tx=0).stop 2024-04-25T01:52:51.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.273+0000 7f5526a00640 1 -- 172.21.15.102:0/4060872777 shutdown_connections 2024-04-25T01:52:51.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.273+0000 7f5526a00640 1 --2- 172.21.15.102:0/4060872777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5520109830 0x7f55201100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.273+0000 7f5526a00640 1 --2- 172.21.15.102:0/4060872777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5520108e70 0x7f55201092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.273+0000 7f5526a00640 1 --2- 172.21.15.102:0/4060872777 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 0x7f55201080a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.273+0000 7f5526a00640 1 -- 172.21.15.102:0/4060872777 >> 172.21.15.102:0/4060872777 conn(0x7f5520076c10 msgr2=0x7f5520079050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:51.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.273+0000 7f5526a00640 1 -- 172.21.15.102:0/4060872777 shutdown_connections 2024-04-25T01:52:51.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.273+0000 7f5526a00640 1 -- 172.21.15.102:0/4060872777 wait complete. 2024-04-25T01:52:51.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5526a00640 1 Processor -- start 2024-04-25T01:52:51.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5526a00640 1 -- start start 2024-04-25T01:52:51.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5526a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 0x7f552019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:51.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5526a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5520108e70 0x7f552019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:51.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5526a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5520109830 0x7f55201a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:51.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5526a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5520113950 con 0x7f5520108e70 2024-04-25T01:52:51.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5526a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f55201137d0 con 0x7f5520107c80 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5526a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5520113ad0 con 0x7f5520109830 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5525600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 0x7f552019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5524c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5520108e70 0x7f552019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5525600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 0x7f552019df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54500/0 (socket says 172.21.15.102:54500) 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5525600640 1 -- 172.21.15.102:0/4227190918 learned_addr learned my addr 172.21.15.102:0/4227190918 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.275+0000 7f5526000640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5520109830 0x7f55201a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f5525600640 1 -- 172.21.15.102:0/4227190918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5520109830 msgr2=0x7f55201a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f5525600640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5520109830 0x7f55201a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f5525600640 1 -- 172.21.15.102:0/4227190918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5520108e70 msgr2=0x7f552019e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f5525600640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5520108e70 0x7f552019e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f5525600640 1 -- 172.21.15.102:0/4227190918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55201a5b80 con 0x7f5520107c80 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f5526000640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5520109830 0x7f55201a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:52:51.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f5525600640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 0x7f552019df60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5514031440 tx=0x7f5514032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:51.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f551e000640 1 -- 172.21.15.102:0/4227190918 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551403f030 con 0x7f5520107c80 2024-04-25T01:52:51.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f551e000640 1 -- 172.21.15.102:0/4227190918 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5514003ce0 con 0x7f5520107c80 2024-04-25T01:52:51.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f551e000640 1 -- 172.21.15.102:0/4227190918 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514035070 con 0x7f5520107c80 2024-04-25T01:52:51.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.276+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55201a5e10 con 0x7f5520107c80 2024-04-25T01:52:51.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.277+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55201a6220 con 0x7f5520107c80 2024-04-25T01:52:51.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.278+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f552010efa0 con 0x7f5520107c80 2024-04-25T01:52:51.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.279+0000 7f551e000640 1 -- 172.21.15.102:0/4227190918 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5514042030 con 0x7f5520107c80 2024-04-25T01:52:51.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.281+0000 7f551e000640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f55080a2c20 0x7f55080a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:51.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.281+0000 7f551e000640 1 -- 172.21.15.102:0/4227190918 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f55140efa00 con 0x7f5520107c80 2024-04-25T01:52:51.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.281+0000 7f5524c00640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f55080a2c20 0x7f55080a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:51.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.281+0000 7f5524c00640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f55080a2c20 0x7f55080a50e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f552019f470 tx=0x7f550c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:51.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.284+0000 7f551e000640 1 -- 172.21.15.102:0/4227190918 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f55140bc8b0 con 0x7f5520107c80 2024-04-25T01:52:51.422 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:51 smithi102 ceph-mon[27796]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:51.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:51 smithi102 ceph-mon[27796]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:51.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:51 smithi102 ceph-mon[27796]: pgmap v321: 97 pgs: 97 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 267 B/s rd, 3.3 KiB/s wr, 9 op/s 2024-04-25T01:52:51.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:51 smithi110 ceph-mon[32848]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:51.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:51 smithi110 ceph-mon[32848]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:51.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:51 smithi110 ceph-mon[32848]: pgmap v321: 97 pgs: 97 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 267 B/s rd, 3.3 KiB/s wr, 9 op/s 2024-04-25T01:52:51.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:51 smithi193 ceph-mon[32981]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:51.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:51 smithi193 ceph-mon[32981]: from='client.24689 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:51.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:51 smithi193 ceph-mon[32981]: pgmap v321: 97 pgs: 97 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 267 B/s rd, 3.3 KiB/s wr, 9 op/s 2024-04-25T01:52:51.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.517+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f5520061c10 con 0x7f55080a2c20 2024-04-25T01:52:51.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.526+0000 7f551e000640 1 -- 172.21.15.102:0/4227190918 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f5520061c10 con 0x7f55080a2c20 2024-04-25T01:52:51.526 INFO:teuthology.orchestra.run.smithi102.stdout:/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76 2024-04-25T01:52:51.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f55080a2c20 msgr2=0x7f55080a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f55080a2c20 0x7f55080a50e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f552019f470 tx=0x7f550c005f90 comp rx=0 tx=0).stop 2024-04-25T01:52:51.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 msgr2=0x7f552019df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 0x7f552019df60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5514031440 tx=0x7f5514032ce0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 shutdown_connections 2024-04-25T01:52:51.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f55080a2c20 0x7f55080a50e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5520109830 0x7f55201a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5520108e70 0x7f552019e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 --2- 172.21.15.102:0/4227190918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5520107c80 0x7f552019df60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 >> 172.21.15.102:0/4227190918 conn(0x7f5520076c10 msgr2=0x7f552010c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:51.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.528+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 shutdown_connections 2024-04-25T01:52:51.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.529+0000 7f5526a00640 1 -- 172.21.15.102:0/4227190918 wait complete. 2024-04-25T01:52:51.549 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:52:51.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.744+0000 7ff3fbe00640 1 -- 172.21.15.102:0/3104068759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3fc077340 msgr2=0x7ff3fc0777a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.744+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/3104068759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3fc077340 0x7ff3fc0777a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8009900 tx=0x7ff3e802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.745+0000 7ff3fbe00640 1 -- 172.21.15.102:0/3104068759 shutdown_connections 2024-04-25T01:52:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.745+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/3104068759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3fc077ce0 0x7ff3fc078580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.745+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/3104068759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3fc077340 0x7ff3fc0777a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.745+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/3104068759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3fc076100 0x7ff3fc076520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.745+0000 7ff3fbe00640 1 -- 172.21.15.102:0/3104068759 >> 172.21.15.102:0/3104068759 conn(0x7ff3fc0fed50 msgr2=0x7ff3fc101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:51.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.745+0000 7ff3fbe00640 1 -- 172.21.15.102:0/3104068759 shutdown_connections 2024-04-25T01:52:51.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.745+0000 7ff3fbe00640 1 -- 172.21.15.102:0/3104068759 wait complete. 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fbe00640 1 Processor -- start 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fbe00640 1 -- start start 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3fc076100 0x7ff3fc19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3fc077340 0x7ff3fc19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3fc077ce0 0x7ff3fc1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc113850 con 0x7ff3fc077ce0 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fbe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc1136d0 con 0x7ff3fc076100 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fbe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc1139d0 con 0x7ff3fc077340 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3fc077ce0 0x7ff3fc1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3fc077ce0 0x7ff3fc1a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58800/0 (socket says 172.21.15.102:58800) 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fb400640 1 -- 172.21.15.102:0/2584880386 learned_addr learned my addr 172.21.15.102:0/2584880386 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3faa00640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3fc076100 0x7ff3fc19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:51.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.746+0000 7ff3fa000640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3fc077340 0x7ff3fc19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3fb400640 1 -- 172.21.15.102:0/2584880386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3fc077340 msgr2=0x7ff3fc19e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3fb400640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3fc077340 0x7ff3fc19e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3fb400640 1 -- 172.21.15.102:0/2584880386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3fc076100 msgr2=0x7ff3fc19de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3fb400640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3fc076100 0x7ff3fc19de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3fb400640 1 -- 172.21.15.102:0/2584880386 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3fc1a5a90 con 0x7ff3fc077ce0 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3fb400640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3fc077ce0 0x7ff3fc1a5420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400eb10 tx=0x7ff3f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3f3400640 1 -- 172.21.15.102:0/2584880386 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4004b20 con 0x7ff3fc077ce0 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3fc1a5d80 con 0x7ff3fc077ce0 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3f3400640 1 -- 172.21.15.102:0/2584880386 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3f4004cc0 con 0x7ff3fc077ce0 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.747+0000 7ff3f3400640 1 -- 172.21.15.102:0/2584880386 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f4010760 con 0x7ff3fc077ce0 2024-04-25T01:52:51.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.748+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3fc1a6240 con 0x7ff3fc077ce0 2024-04-25T01:52:51.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.749+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3fc110280 con 0x7ff3fc077ce0 2024-04-25T01:52:51.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.751+0000 7ff3f3400640 1 -- 172.21.15.102:0/2584880386 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff3f4010900 con 0x7ff3fc077ce0 2024-04-25T01:52:51.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.752+0000 7ff3f3400640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e00a2c20 0x7ff3e00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:51.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.752+0000 7ff3f3400640 1 -- 172.21.15.102:0/2584880386 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff3f40cf8a0 con 0x7ff3fc077ce0 2024-04-25T01:52:51.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.752+0000 7ff3faa00640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e00a2c20 0x7ff3e00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:51.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.753+0000 7ff3faa00640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e00a2c20 0x7ff3e00a50e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00bbe0 tx=0x7ff3ec00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:51.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.755+0000 7ff3f3400640 1 -- 172.21.15.102:0/2584880386 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff3f4053740 con 0x7ff3fc077ce0 2024-04-25T01:52:51.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.995+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3fc0623e0 con 0x7ff3fc077ce0 2024-04-25T01:52:51.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:51.997+0000 7ff3f3400640 1 -- 172.21.15.102:0/2584880386 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11896 (secure 0 0 0) 0x7ff3f409c750 con 0x7ff3fc077ce0 2024-04-25T01:52:51.998 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:51.998 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:52:49.089206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T01:52:51.999 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 22 2024-04-25T01:52:52.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e00a2c20 msgr2=0x7ff3e00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e00a2c20 0x7ff3e00a50e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec00bbe0 tx=0x7ff3ec00b390 comp rx=0 tx=0).stop 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3fc077ce0 msgr2=0x7ff3fc1a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3fc077ce0 0x7ff3fc1a5420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff3f400eb10 tx=0x7ff3f400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 shutdown_connections 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff3e00a2c20 0x7ff3e00a50e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3fc077ce0 0x7ff3fc1a5420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3fc077340 0x7ff3fc19e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 --2- 172.21.15.102:0/2584880386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3fc076100 0x7ff3fc19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 >> 172.21.15.102:0/2584880386 conn(0x7ff3fc0fed50 msgr2=0x7ff3fc1003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 shutdown_connections 2024-04-25T01:52:52.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.000+0000 7ff3fbe00640 1 -- 172.21.15.102:0/2584880386 wait complete. 2024-04-25T01:52:52.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:52:52.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.222+0000 7fab9b200640 1 -- 172.21.15.102:0/2512810674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9c075780 msgr2=0x7fab9c073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.222+0000 7fab9b200640 1 --2- 172.21.15.102:0/2512810674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9c075780 0x7fab9c073bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fab88009860 tx=0x7fab8802a340 comp rx=0 tx=0).stop 2024-04-25T01:52:52.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.223+0000 7fab9b200640 1 -- 172.21.15.102:0/2512810674 shutdown_connections 2024-04-25T01:52:52.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.223+0000 7fab9b200640 1 --2- 172.21.15.102:0/2512810674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab9c1094a0 0x7fab9c10b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.223+0000 7fab9b200640 1 --2- 172.21.15.102:0/2512810674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0741a0 0x7fab9c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.223+0000 7fab9b200640 1 --2- 172.21.15.102:0/2512810674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9c075780 0x7fab9c073bd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.223+0000 7fab9b200640 1 -- 172.21.15.102:0/2512810674 >> 172.21.15.102:0/2512810674 conn(0x7fab9c0fcb50 msgr2=0x7fab9c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:52.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.223+0000 7fab9b200640 1 -- 172.21.15.102:0/2512810674 shutdown_connections 2024-04-25T01:52:52.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.223+0000 7fab9b200640 1 -- 172.21.15.102:0/2512810674 wait complete. 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.224+0000 7fab9b200640 1 Processor -- start 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.224+0000 7fab9b200640 1 -- start start 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.224+0000 7fab9b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0741a0 0x7fab9c199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.224+0000 7fab9b200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9c075780 0x7fab9c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.224+0000 7fab9b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab9c1094a0 0x7fab9c1a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.224+0000 7fab9b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c10f440 con 0x7fab9c0741a0 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.224+0000 7fab9b200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c10f2c0 con 0x7fab9c075780 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.224+0000 7fab9b200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c10f5c0 con 0x7fab9c1094a0 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0741a0 0x7fab9c199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9c075780 0x7fab9c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0741a0 0x7fab9c199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58818/0 (socket says 172.21.15.102:58818) 2024-04-25T01:52:52.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99e00640 1 -- 172.21.15.102:0/3503039888 learned_addr learned my addr 172.21.15.102:0/3503039888 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:52.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9c075780 0x7fab9c199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54540/0 (socket says 172.21.15.102:54540) 2024-04-25T01:52:52.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab9a800640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab9c1094a0 0x7fab9c1a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:52.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99e00640 1 -- 172.21.15.102:0/3503039888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab9c1094a0 msgr2=0x7fab9c1a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99e00640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab9c1094a0 0x7fab9c1a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99e00640 1 -- 172.21.15.102:0/3503039888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9c075780 msgr2=0x7fab9c199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99e00640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9c075780 0x7fab9c199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99e00640 1 -- 172.21.15.102:0/3503039888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c1a16d0 con 0x7fab9c0741a0 2024-04-25T01:52:52.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.225+0000 7fab99e00640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0741a0 0x7fab9c199ab0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fab88035040 tx=0x7fab88033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:52.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.226+0000 7fab92a00640 1 -- 172.21.15.102:0/3503039888 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88037040 con 0x7fab9c0741a0 2024-04-25T01:52:52.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.226+0000 7fab92a00640 1 -- 172.21.15.102:0/3503039888 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab88030850 con 0x7fab9c0741a0 2024-04-25T01:52:52.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.226+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c1a1960 con 0x7fab9c0741a0 2024-04-25T01:52:52.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.226+0000 7fab92a00640 1 -- 172.21.15.102:0/3503039888 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88004920 con 0x7fab9c0741a0 2024-04-25T01:52:52.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.226+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c1a1e50 con 0x7fab9c0741a0 2024-04-25T01:52:52.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.227+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c1081d0 con 0x7fab9c0741a0 2024-04-25T01:52:52.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.229+0000 7fab92a00640 1 -- 172.21.15.102:0/3503039888 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fab88004050 con 0x7fab9c0741a0 2024-04-25T01:52:52.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.231+0000 7fab92a00640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab800a2c20 0x7fab800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:52.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.232+0000 7fab92a00640 1 -- 172.21.15.102:0/3503039888 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fab880eee10 con 0x7fab9c0741a0 2024-04-25T01:52:52.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.232+0000 7fab99400640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab800a2c20 0x7fab800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:52.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.232+0000 7fab99400640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab800a2c20 0x7fab800a50e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab9c19afc0 tx=0x7fab8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:52.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.237+0000 7fab92a00640 1 -- 172.21.15.102:0/3503039888 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fab880bbed0 con 0x7fab9c0741a0 2024-04-25T01:52:52.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:52 smithi102 ceph-mon[27796]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:52.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:52 smithi102 ceph-mon[27796]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-25T01:52:52.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:52 smithi102 ceph-mon[27796]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:52.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:52 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2584880386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:52.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.442+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab9c062340 con 0x7fab9c0741a0 2024-04-25T01:52:52.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.446+0000 7fab92a00640 1 -- 172.21.15.102:0/3503039888 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22253 (secure 0 0 0) 0x7fab880bb7d0 con 0x7fab9c0741a0 2024-04-25T01:52:52.447 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:52.447 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":87,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:52:45.058627+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:52:52.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab800a2c20 msgr2=0x7fab800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab800a2c20 0x7fab800a50e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab9c19afc0 tx=0x7fab8c009260 comp rx=0 tx=0).stop 2024-04-25T01:52:52.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0741a0 msgr2=0x7fab9c199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0741a0 0x7fab9c199ab0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fab88035040 tx=0x7fab88033d90 comp rx=0 tx=0).stop 2024-04-25T01:52:52.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 shutdown_connections 2024-04-25T01:52:52.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab800a2c20 0x7fab800a50e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab9c1094a0 0x7fab9c1a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9c075780 0x7fab9c199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 --2- 172.21.15.102:0/3503039888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9c0741a0 0x7fab9c199ab0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.449+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 >> 172.21.15.102:0/3503039888 conn(0x7fab9c0fcb50 msgr2=0x7fab9c1053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:52.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.450+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 shutdown_connections 2024-04-25T01:52:52.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.450+0000 7fab9b200640 1 -- 172.21.15.102:0/3503039888 wait complete. 2024-04-25T01:52:52.469 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T01:52:52.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:52:52.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:52 smithi110 ceph-mon[32848]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:52.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:52 smithi110 ceph-mon[32848]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-25T01:52:52.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:52 smithi110 ceph-mon[32848]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:52.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:52 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2584880386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:52 smithi193 ceph-mon[32981]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:52.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:52 smithi193 ceph-mon[32981]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-25T01:52:52.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:52 smithi193 ceph-mon[32981]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:52:52.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:52 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2584880386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:52:52.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.672+0000 7f9e66c00640 1 -- 172.21.15.102:0/1755027659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e60104bb0 msgr2=0x7f9e60105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.672+0000 7f9e66c00640 1 --2- 172.21.15.102:0/1755027659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e60104bb0 0x7f9e60105030 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009860 tx=0x7f9e4802a340 comp rx=0 tx=0).stop 2024-04-25T01:52:52.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.673+0000 7f9e66c00640 1 -- 172.21.15.102:0/1755027659 shutdown_connections 2024-04-25T01:52:52.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.673+0000 7f9e66c00640 1 --2- 172.21.15.102:0/1755027659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e60105570 0x7f9e6010be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.673+0000 7f9e66c00640 1 --2- 172.21.15.102:0/1755027659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e60104bb0 0x7f9e60105030 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.673+0000 7f9e66c00640 1 --2- 172.21.15.102:0/1755027659 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e601039c0 0x7f9e60103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.673+0000 7f9e66c00640 1 -- 172.21.15.102:0/1755027659 >> 172.21.15.102:0/1755027659 conn(0x7f9e600fef30 msgr2=0x7f9e60101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:52.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.673+0000 7f9e66c00640 1 -- 172.21.15.102:0/1755027659 shutdown_connections 2024-04-25T01:52:52.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.673+0000 7f9e66c00640 1 -- 172.21.15.102:0/1755027659 wait complete. 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.674+0000 7f9e66c00640 1 Processor -- start 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.674+0000 7f9e66c00640 1 -- start start 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.674+0000 7f9e66c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e601039c0 0x7f9e60199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.674+0000 7f9e66c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e60104bb0 0x7f9e6019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.674+0000 7f9e66c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e60105570 0x7f9e601a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.674+0000 7f9e66c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6010f640 con 0x7f9e60104bb0 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.674+0000 7f9e66c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6010f4c0 con 0x7f9e60105570 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.674+0000 7f9e66c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6010f7c0 con 0x7f9e601039c0 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e5f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e60104bb0 0x7f9e6019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e5f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e60104bb0 0x7f9e6019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58830/0 (socket says 172.21.15.102:58830) 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e5f400640 1 -- 172.21.15.102:0/4261218703 learned_addr learned my addr 172.21.15.102:0/4261218703 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:52.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e5fe00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e601039c0 0x7f9e60199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:52.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e64c00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e60105570 0x7f9e601a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:52.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e5f400640 1 -- 172.21.15.102:0/4261218703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e601039c0 msgr2=0x7f9e60199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e5f400640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e601039c0 0x7f9e60199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e5f400640 1 -- 172.21.15.102:0/4261218703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e60105570 msgr2=0x7f9e601a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e5f400640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e60105570 0x7f9e601a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.675+0000 7f9e5f400640 1 -- 172.21.15.102:0/4261218703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e601a1910 con 0x7f9e60104bb0 2024-04-25T01:52:52.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.676+0000 7f9e5f400640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e60104bb0 0x7f9e6019a240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9e48035040 tx=0x7f9e48033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:52.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.676+0000 7f9e5cc00640 1 -- 172.21.15.102:0/4261218703 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48037040 con 0x7f9e60104bb0 2024-04-25T01:52:52.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.676+0000 7f9e5cc00640 1 -- 172.21.15.102:0/4261218703 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e48030850 con 0x7f9e60104bb0 2024-04-25T01:52:52.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.676+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e601a1ba0 con 0x7f9e60104bb0 2024-04-25T01:52:52.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.676+0000 7f9e5cc00640 1 -- 172.21.15.102:0/4261218703 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48004920 con 0x7f9e60104bb0 2024-04-25T01:52:52.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.676+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e601a2090 con 0x7f9e60104bb0 2024-04-25T01:52:52.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.678+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e6010ace0 con 0x7f9e60104bb0 2024-04-25T01:52:52.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.679+0000 7f9e5cc00640 1 -- 172.21.15.102:0/4261218703 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f9e48004050 con 0x7f9e60104bb0 2024-04-25T01:52:52.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.681+0000 7f9e5cc00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f9e440a2c20 0x7f9e440a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:52.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.682+0000 7f9e5cc00640 1 -- 172.21.15.102:0/4261218703 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f9e480eefa0 con 0x7f9e60104bb0 2024-04-25T01:52:52.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.682+0000 7f9e5fe00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f9e440a2c20 0x7f9e440a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:52.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.682+0000 7f9e5fe00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f9e440a2c20 0x7f9e440a50e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9e6019b1d0 tx=0x7f9e50009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:52.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.685+0000 7f9e5cc00640 1 -- 172.21.15.102:0/4261218703 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9e480bbf60 con 0x7f9e60104bb0 2024-04-25T01:52:52.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.880+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9e60061b80 con 0x7f9e440a2c20 2024-04-25T01:52:52.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.894+0000 7f9e5cc00640 1 -- 172.21.15.102:0/4261218703 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258738 (secure 0 0 0) 0x7f9e60061b80 con 0x7f9e440a2c20 2024-04-25T01:52:52.899 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-25T01:52:52.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f9e440a2c20 msgr2=0x7f9e440a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f9e440a2c20 0x7f9e440a50e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9e6019b1d0 tx=0x7f9e50009260 comp rx=0 tx=0).stop 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e60104bb0 msgr2=0x7f9e6019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e60104bb0 0x7f9e6019a240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9e48035040 tx=0x7f9e48033d90 comp rx=0 tx=0).stop 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 shutdown_connections 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f9e440a2c20 0x7f9e440a50e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e60105570 0x7f9e601a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e60104bb0 0x7f9e6019a240 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 --2- 172.21.15.102:0/4261218703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e601039c0 0x7f9e60199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 >> 172.21.15.102:0/4261218703 conn(0x7f9e600fef30 msgr2=0x7f9e60107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 shutdown_connections 2024-04-25T01:52:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:52.901+0000 7f9e66c00640 1 -- 172.21.15.102:0/4261218703 wait complete. 2024-04-25T01:52:52.922 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-04-25T01:52:52.723279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632174,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":169,"num_read_kb":179,"num_write":319,"num_write_kb":3016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":257,"ondisk_log_size":257,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374548,"kb_used_data":12912,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567276,"statfs":{"total":1151940427776,"available":1151556890624,"internally_reserved":0,"allocated":13221888,"data_stored":7155939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":370193723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41806,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":33,"num_read_kb":60,"num_write":138,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.661225"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092625+0000","last_change":"2024-04-25T01:52:44.102143+0000","last_active":"2024-04-25T01:52:46.092625+0000","last_peered":"2024-04-25T01:52:46.092625+0000","last_clean":"2024-04-25T01:52:46.092625+0000","last_became_active":"2024-04-25T01:52:44.101810+0000","last_became_peered":"2024-04-25T01:52:44.101810+0000","last_unstale":"2024-04-25T01:52:46.092625+0000","last_undegraded":"2024-04-25T01:52:46.092625+0000","last_fullsized":"2024-04-25T01:52:46.092625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:02.356294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064344+0000","last_change":"2024-04-25T01:52:44.103074+0000","last_active":"2024-04-25T01:52:45.064344+0000","last_peered":"2024-04-25T01:52:45.064344+0000","last_clean":"2024-04-25T01:52:45.064344+0000","last_became_active":"2024-04-25T01:52:44.102103+0000","last_became_peered":"2024-04-25T01:52:44.102103+0000","last_unstale":"2024-04-25T01:52:45.064344+0000","last_undegraded":"2024-04-25T01:52:45.064344+0000","last_fullsized":"2024-04-25T01:52:45.064344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:36:43.016756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064304+0000","last_change":"2024-04-25T01:52:44.101467+0000","last_active":"2024-04-25T01:52:45.064304+0000","last_peered":"2024-04-25T01:52:45.064304+0000","last_clean":"2024-04-25T01:52:45.064304+0000","last_became_active":"2024-04-25T01:52:44.100911+0000","last_became_peered":"2024-04-25T01:52:44.100911+0000","last_unstale":"2024-04-25T01:52:45.064304+0000","last_undegraded":"2024-04-25T01:52:45.064304+0000","last_fullsized":"2024-04-25T01:52:45.064304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:58.533213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091759+0000","last_change":"2024-04-25T01:52:44.088592+0000","last_active":"2024-04-25T01:52:46.091759+0000","last_peered":"2024-04-25T01:52:46.091759+0000","last_clean":"2024-04-25T01:52:46.091759+0000","last_became_active":"2024-04-25T01:52:44.087651+0000","last_became_peered":"2024-04-25T01:52:44.087651+0000","last_unstale":"2024-04-25T01:52:46.091759+0000","last_undegraded":"2024-04-25T01:52:46.091759+0000","last_fullsized":"2024-04-25T01:52:46.091759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:52.240818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063402+0000","last_change":"2024-04-25T01:52:44.095033+0000","last_active":"2024-04-25T01:52:45.063402+0000","last_peered":"2024-04-25T01:52:45.063402+0000","last_clean":"2024-04-25T01:52:45.063402+0000","last_became_active":"2024-04-25T01:52:44.094719+0000","last_became_peered":"2024-04-25T01:52:44.094719+0000","last_unstale":"2024-04-25T01:52:45.063402+0000","last_undegraded":"2024-04-25T01:52:45.063402+0000","last_fullsized":"2024-04-25T01:52:45.063402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:52.422544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.077136+0000","last_change":"2024-04-25T01:52:44.102574+0000","last_active":"2024-04-25T01:52:45.077136+0000","last_peered":"2024-04-25T01:52:45.077136+0000","last_clean":"2024-04-25T01:52:45.077136+0000","last_became_active":"2024-04-25T01:52:44.102158+0000","last_became_peered":"2024-04-25T01:52:44.102158+0000","last_unstale":"2024-04-25T01:52:45.077136+0000","last_undegraded":"2024-04-25T01:52:45.077136+0000","last_fullsized":"2024-04-25T01:52:45.077136+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:20.931392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063622+0000","last_change":"2024-04-25T01:52:44.107662+0000","last_active":"2024-04-25T01:52:45.063622+0000","last_peered":"2024-04-25T01:52:45.063622+0000","last_clean":"2024-04-25T01:52:45.063622+0000","last_became_active":"2024-04-25T01:52:44.107304+0000","last_became_peered":"2024-04-25T01:52:44.107304+0000","last_unstale":"2024-04-25T01:52:45.063622+0000","last_undegraded":"2024-04-25T01:52:45.063622+0000","last_fullsized":"2024-04-25T01:52:45.063622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:42:54.636950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356615+0000","last_change":"2024-04-25T01:52:44.100273+0000","last_active":"2024-04-25T01:52:45.356615+0000","last_peered":"2024-04-25T01:52:45.356615+0000","last_clean":"2024-04-25T01:52:45.356615+0000","last_became_active":"2024-04-25T01:52:44.098223+0000","last_became_peered":"2024-04-25T01:52:44.098223+0000","last_unstale":"2024-04-25T01:52:45.356615+0000","last_undegraded":"2024-04-25T01:52:45.356615+0000","last_fullsized":"2024-04-25T01:52:45.356615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:05.003265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469299+0000","last_change":"2024-04-25T01:52:44.089658+0000","last_active":"2024-04-25T01:52:45.469299+0000","last_peered":"2024-04-25T01:52:45.469299+0000","last_clean":"2024-04-25T01:52:45.469299+0000","last_became_active":"2024-04-25T01:52:44.089232+0000","last_became_peered":"2024-04-25T01:52:44.089232+0000","last_unstale":"2024-04-25T01:52:45.469299+0000","last_undegraded":"2024-04-25T01:52:45.469299+0000","last_fullsized":"2024-04-25T01:52:45.469299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:54.385782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.077492+0000","last_change":"2024-04-25T01:52:44.086660+0000","last_active":"2024-04-25T01:52:45.077492+0000","last_peered":"2024-04-25T01:52:45.077492+0000","last_clean":"2024-04-25T01:52:45.077492+0000","last_became_active":"2024-04-25T01:52:44.086306+0000","last_became_peered":"2024-04-25T01:52:44.086306+0000","last_unstale":"2024-04-25T01:52:45.077492+0000","last_undegraded":"2024-04-25T01:52:45.077492+0000","last_fullsized":"2024-04-25T01:52:45.077492+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:33:20.726616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064170+0000","last_change":"2024-04-25T01:52:44.090239+0000","last_active":"2024-04-25T01:52:45.064170+0000","last_peered":"2024-04-25T01:52:45.064170+0000","last_clean":"2024-04-25T01:52:45.064170+0000","last_became_active":"2024-04-25T01:52:44.089736+0000","last_became_peered":"2024-04-25T01:52:44.089736+0000","last_unstale":"2024-04-25T01:52:45.064170+0000","last_undegraded":"2024-04-25T01:52:45.064170+0000","last_fullsized":"2024-04-25T01:52:45.064170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:22.954011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091726+0000","last_change":"2024-04-25T01:52:44.088878+0000","last_active":"2024-04-25T01:52:46.091726+0000","last_peered":"2024-04-25T01:52:46.091726+0000","last_clean":"2024-04-25T01:52:46.091726+0000","last_became_active":"2024-04-25T01:52:44.088250+0000","last_became_peered":"2024-04-25T01:52:44.088250+0000","last_unstale":"2024-04-25T01:52:46.091726+0000","last_undegraded":"2024-04-25T01:52:46.091726+0000","last_fullsized":"2024-04-25T01:52:46.091726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:15.490131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092718+0000","last_change":"2024-04-25T01:52:44.094675+0000","last_active":"2024-04-25T01:52:46.092718+0000","last_peered":"2024-04-25T01:52:46.092718+0000","last_clean":"2024-04-25T01:52:46.092718+0000","last_became_active":"2024-04-25T01:52:44.094219+0000","last_became_peered":"2024-04-25T01:52:44.094219+0000","last_unstale":"2024-04-25T01:52:46.092718+0000","last_undegraded":"2024-04-25T01:52:46.092718+0000","last_fullsized":"2024-04-25T01:52:46.092718+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:26.160156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063520+0000","last_change":"2024-04-25T01:52:44.111131+0000","last_active":"2024-04-25T01:52:45.063520+0000","last_peered":"2024-04-25T01:52:45.063520+0000","last_clean":"2024-04-25T01:52:45.063520+0000","last_became_active":"2024-04-25T01:52:44.105670+0000","last_became_peered":"2024-04-25T01:52:44.105670+0000","last_unstale":"2024-04-25T01:52:45.063520+0000","last_undegraded":"2024-04-25T01:52:45.063520+0000","last_fullsized":"2024-04-25T01:52:45.063520+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:00.377724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063359+0000","last_change":"2024-04-25T01:52:44.109707+0000","last_active":"2024-04-25T01:52:45.063359+0000","last_peered":"2024-04-25T01:52:45.063359+0000","last_clean":"2024-04-25T01:52:45.063359+0000","last_became_active":"2024-04-25T01:52:44.108550+0000","last_became_peered":"2024-04-25T01:52:44.108550+0000","last_unstale":"2024-04-25T01:52:45.063359+0000","last_undegraded":"2024-04-25T01:52:45.063359+0000","last_fullsized":"2024-04-25T01:52:45.063359+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:46.296789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.468826+0000","last_change":"2024-04-25T01:52:44.104504+0000","last_active":"2024-04-25T01:52:45.468826+0000","last_peered":"2024-04-25T01:52:45.468826+0000","last_clean":"2024-04-25T01:52:45.468826+0000","last_became_active":"2024-04-25T01:52:44.104180+0000","last_became_peered":"2024-04-25T01:52:44.104180+0000","last_unstale":"2024-04-25T01:52:45.468826+0000","last_undegraded":"2024-04-25T01:52:45.468826+0000","last_fullsized":"2024-04-25T01:52:45.468826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:17.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092802+0000","last_change":"2024-04-25T01:52:44.107550+0000","last_active":"2024-04-25T01:52:46.092802+0000","last_peered":"2024-04-25T01:52:46.092802+0000","last_clean":"2024-04-25T01:52:46.092802+0000","last_became_active":"2024-04-25T01:52:44.107195+0000","last_became_peered":"2024-04-25T01:52:44.107195+0000","last_unstale":"2024-04-25T01:52:46.092802+0000","last_undegraded":"2024-04-25T01:52:46.092802+0000","last_fullsized":"2024-04-25T01:52:46.092802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:59:42.220575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092172+0000","last_change":"2024-04-25T01:52:44.101165+0000","last_active":"2024-04-25T01:52:46.092172+0000","last_peered":"2024-04-25T01:52:46.092172+0000","last_clean":"2024-04-25T01:52:46.092172+0000","last_became_active":"2024-04-25T01:52:44.100386+0000","last_became_peered":"2024-04-25T01:52:44.100386+0000","last_unstale":"2024-04-25T01:52:46.092172+0000","last_undegraded":"2024-04-25T01:52:46.092172+0000","last_fullsized":"2024-04-25T01:52:46.092172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:30.922707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092985+0000","last_change":"2024-04-25T01:52:44.116694+0000","last_active":"2024-04-25T01:52:46.092985+0000","last_peered":"2024-04-25T01:52:46.092985+0000","last_clean":"2024-04-25T01:52:46.092985+0000","last_became_active":"2024-04-25T01:52:44.116326+0000","last_became_peered":"2024-04-25T01:52:44.116326+0000","last_unstale":"2024-04-25T01:52:46.092985+0000","last_undegraded":"2024-04-25T01:52:46.092985+0000","last_fullsized":"2024-04-25T01:52:46.092985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:16.329899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.093471+0000","last_change":"2024-04-25T01:52:43.072358+0000","last_active":"2024-04-25T01:52:48.093471+0000","last_peered":"2024-04-25T01:52:48.093471+0000","last_clean":"2024-04-25T01:52:48.093471+0000","last_became_active":"2024-04-25T01:52:43.071815+0000","last_became_peered":"2024-04-25T01:52:43.071815+0000","last_unstale":"2024-04-25T01:52:48.093471+0000","last_undegraded":"2024-04-25T01:52:48.093471+0000","last_fullsized":"2024-04-25T01:52:48.093471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:15.295744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.357381+0000","last_change":"2024-04-25T01:52:44.094509+0000","last_active":"2024-04-25T01:52:45.357381+0000","last_peered":"2024-04-25T01:52:45.357381+0000","last_clean":"2024-04-25T01:52:45.357381+0000","last_became_active":"2024-04-25T01:52:44.093955+0000","last_became_peered":"2024-04-25T01:52:44.093955+0000","last_unstale":"2024-04-25T01:52:45.357381+0000","last_undegraded":"2024-04-25T01:52:45.357381+0000","last_fullsized":"2024-04-25T01:52:45.357381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:37.701835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.357441+0000","last_change":"2024-04-25T01:52:43.094435+0000","last_active":"2024-04-25T01:52:45.357441+0000","last_peered":"2024-04-25T01:52:45.357441+0000","last_clean":"2024-04-25T01:52:45.357441+0000","last_became_active":"2024-04-25T01:52:43.094018+0000","last_became_peered":"2024-04-25T01:52:43.094018+0000","last_unstale":"2024-04-25T01:52:45.357441+0000","last_undegraded":"2024-04-25T01:52:45.357441+0000","last_fullsized":"2024-04-25T01:52:45.357441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:36.781014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064164+0000","last_change":"2024-04-25T01:52:44.104347+0000","last_active":"2024-04-25T01:52:45.064164+0000","last_peered":"2024-04-25T01:52:45.064164+0000","last_clean":"2024-04-25T01:52:45.064164+0000","last_became_active":"2024-04-25T01:52:44.103803+0000","last_became_peered":"2024-04-25T01:52:44.103803+0000","last_unstale":"2024-04-25T01:52:45.064164+0000","last_undegraded":"2024-04-25T01:52:45.064164+0000","last_fullsized":"2024-04-25T01:52:45.064164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:01.784893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.086026+0000","last_change":"2024-04-25T01:52:43.070902+0000","last_active":"2024-04-25T01:52:48.086026+0000","last_peered":"2024-04-25T01:52:48.086026+0000","last_clean":"2024-04-25T01:52:48.086026+0000","last_became_active":"2024-04-25T01:52:43.070313+0000","last_became_peered":"2024-04-25T01:52:43.070313+0000","last_unstale":"2024-04-25T01:52:48.086026+0000","last_undegraded":"2024-04-25T01:52:48.086026+0000","last_fullsized":"2024-04-25T01:52:48.086026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:10.677320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469073+0000","last_change":"2024-04-25T01:52:44.093293+0000","last_active":"2024-04-25T01:52:45.469073+0000","last_peered":"2024-04-25T01:52:45.469073+0000","last_clean":"2024-04-25T01:52:45.469073+0000","last_became_active":"2024-04-25T01:52:44.092932+0000","last_became_peered":"2024-04-25T01:52:44.092932+0000","last_unstale":"2024-04-25T01:52:45.469073+0000","last_undegraded":"2024-04-25T01:52:45.469073+0000","last_fullsized":"2024-04-25T01:52:45.469073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:58:37.705594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063605+0000","last_change":"2024-04-25T01:52:43.069743+0000","last_active":"2024-04-25T01:52:45.063605+0000","last_peered":"2024-04-25T01:52:45.063605+0000","last_clean":"2024-04-25T01:52:45.063605+0000","last_became_active":"2024-04-25T01:52:43.068947+0000","last_became_peered":"2024-04-25T01:52:43.068947+0000","last_unstale":"2024-04-25T01:52:45.063605+0000","last_undegraded":"2024-04-25T01:52:45.063605+0000","last_fullsized":"2024-04-25T01:52:45.063605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:02:52.582176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215774+0000","last_change":"2024-04-25T01:52:44.091398+0000","last_active":"2024-04-25T01:52:45.215774+0000","last_peered":"2024-04-25T01:52:45.215774+0000","last_clean":"2024-04-25T01:52:45.215774+0000","last_became_active":"2024-04-25T01:52:44.091061+0000","last_became_peered":"2024-04-25T01:52:44.091061+0000","last_unstale":"2024-04-25T01:52:45.215774+0000","last_undegraded":"2024-04-25T01:52:45.215774+0000","last_fullsized":"2024-04-25T01:52:45.215774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:54.858769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:51.105148+0000","last_change":"2024-04-25T01:52:43.068450+0000","last_active":"2024-04-25T01:52:51.105148+0000","last_peered":"2024-04-25T01:52:51.105148+0000","last_clean":"2024-04-25T01:52:51.105148+0000","last_became_active":"2024-04-25T01:52:43.067881+0000","last_became_peered":"2024-04-25T01:52:43.067881+0000","last_unstale":"2024-04-25T01:52:51.105148+0000","last_undegraded":"2024-04-25T01:52:51.105148+0000","last_fullsized":"2024-04-25T01:52:51.105148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:12.739321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063670+0000","last_change":"2024-04-25T01:52:44.101743+0000","last_active":"2024-04-25T01:52:45.063670+0000","last_peered":"2024-04-25T01:52:45.063670+0000","last_clean":"2024-04-25T01:52:45.063670+0000","last_became_active":"2024-04-25T01:52:44.101351+0000","last_became_peered":"2024-04-25T01:52:44.101351+0000","last_unstale":"2024-04-25T01:52:45.063670+0000","last_undegraded":"2024-04-25T01:52:45.063670+0000","last_fullsized":"2024-04-25T01:52:45.063670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:54:24.104819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.087944+0000","last_change":"2024-04-25T01:52:43.075880+0000","last_active":"2024-04-25T01:52:48.087944+0000","last_peered":"2024-04-25T01:52:48.087944+0000","last_clean":"2024-04-25T01:52:48.087944+0000","last_became_active":"2024-04-25T01:52:43.075522+0000","last_became_peered":"2024-04-25T01:52:43.075522+0000","last_unstale":"2024-04-25T01:52:48.087944+0000","last_undegraded":"2024-04-25T01:52:48.087944+0000","last_fullsized":"2024-04-25T01:52:48.087944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:26.218774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215753+0000","last_change":"2024-04-25T01:52:44.116772+0000","last_active":"2024-04-25T01:52:45.215753+0000","last_peered":"2024-04-25T01:52:45.215753+0000","last_clean":"2024-04-25T01:52:45.215753+0000","last_became_active":"2024-04-25T01:52:44.116408+0000","last_became_peered":"2024-04-25T01:52:44.116408+0000","last_unstale":"2024-04-25T01:52:45.215753+0000","last_undegraded":"2024-04-25T01:52:45.215753+0000","last_fullsized":"2024-04-25T01:52:45.215753+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:21.988425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.096835+0000","last_change":"2024-04-25T01:52:43.076964+0000","last_active":"2024-04-25T01:52:48.096835+0000","last_peered":"2024-04-25T01:52:48.096835+0000","last_clean":"2024-04-25T01:52:48.096835+0000","last_became_active":"2024-04-25T01:52:43.073915+0000","last_became_peered":"2024-04-25T01:52:43.073915+0000","last_unstale":"2024-04-25T01:52:48.096835+0000","last_undegraded":"2024-04-25T01:52:48.096835+0000","last_fullsized":"2024-04-25T01:52:48.096835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:03.321920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356871+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:52:45.356871+0000","last_peered":"2024-04-25T01:52:45.356871+0000","last_clean":"2024-04-25T01:52:45.356871+0000","last_became_active":"2024-04-25T01:52:44.088198+0000","last_became_peered":"2024-04-25T01:52:44.088198+0000","last_unstale":"2024-04-25T01:52:45.356871+0000","last_undegraded":"2024-04-25T01:52:45.356871+0000","last_fullsized":"2024-04-25T01:52:45.356871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:30.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215295+0000","last_change":"2024-04-25T01:52:43.069853+0000","last_active":"2024-04-25T01:52:45.215295+0000","last_peered":"2024-04-25T01:52:45.215295+0000","last_clean":"2024-04-25T01:52:45.215295+0000","last_became_active":"2024-04-25T01:52:43.069526+0000","last_became_peered":"2024-04-25T01:52:43.069526+0000","last_unstale":"2024-04-25T01:52:45.215295+0000","last_undegraded":"2024-04-25T01:52:45.215295+0000","last_fullsized":"2024-04-25T01:52:45.215295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:35:03.272610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091612+0000","last_change":"2024-04-25T01:52:44.114296+0000","last_active":"2024-04-25T01:52:46.091612+0000","last_peered":"2024-04-25T01:52:46.091612+0000","last_clean":"2024-04-25T01:52:46.091612+0000","last_became_active":"2024-04-25T01:52:44.113958+0000","last_became_peered":"2024-04-25T01:52:44.113958+0000","last_unstale":"2024-04-25T01:52:46.091612+0000","last_undegraded":"2024-04-25T01:52:46.091612+0000","last_fullsized":"2024-04-25T01:52:46.091612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:45.068080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:50.102913+0000","last_change":"2024-04-25T01:52:43.071388+0000","last_active":"2024-04-25T01:52:50.102913+0000","last_peered":"2024-04-25T01:52:50.102913+0000","last_clean":"2024-04-25T01:52:50.102913+0000","last_became_active":"2024-04-25T01:52:43.071005+0000","last_became_peered":"2024-04-25T01:52:43.071005+0000","last_unstale":"2024-04-25T01:52:50.102913+0000","last_undegraded":"2024-04-25T01:52:50.102913+0000","last_fullsized":"2024-04-25T01:52:50.102913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:56:59.580458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469294+0000","last_change":"2024-04-25T01:52:44.106106+0000","last_active":"2024-04-25T01:52:45.469294+0000","last_peered":"2024-04-25T01:52:45.469294+0000","last_clean":"2024-04-25T01:52:45.469294+0000","last_became_active":"2024-04-25T01:52:44.105760+0000","last_became_peered":"2024-04-25T01:52:44.105760+0000","last_unstale":"2024-04-25T01:52:45.469294+0000","last_undegraded":"2024-04-25T01:52:45.469294+0000","last_fullsized":"2024-04-25T01:52:45.469294+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:24.626753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.061022+0000","last_change":"2024-04-25T01:52:43.068926+0000","last_active":"2024-04-25T01:52:47.061022+0000","last_peered":"2024-04-25T01:52:47.061022+0000","last_clean":"2024-04-25T01:52:47.061022+0000","last_became_active":"2024-04-25T01:52:43.067356+0000","last_became_peered":"2024-04-25T01:52:43.067356+0000","last_unstale":"2024-04-25T01:52:47.061022+0000","last_undegraded":"2024-04-25T01:52:47.061022+0000","last_fullsized":"2024-04-25T01:52:47.061022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:19.134633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064251+0000","last_change":"2024-04-25T01:52:44.108299+0000","last_active":"2024-04-25T01:52:45.064251+0000","last_peered":"2024-04-25T01:52:45.064251+0000","last_clean":"2024-04-25T01:52:45.064251+0000","last_became_active":"2024-04-25T01:52:44.107573+0000","last_became_peered":"2024-04-25T01:52:44.107573+0000","last_unstale":"2024-04-25T01:52:45.064251+0000","last_undegraded":"2024-04-25T01:52:45.064251+0000","last_fullsized":"2024-04-25T01:52:45.064251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.804579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.063044+0000","last_change":"2024-04-25T01:52:43.079263+0000","last_active":"2024-04-25T01:52:47.063044+0000","last_peered":"2024-04-25T01:52:47.063044+0000","last_clean":"2024-04-25T01:52:47.063044+0000","last_became_active":"2024-04-25T01:52:43.078459+0000","last_became_peered":"2024-04-25T01:52:43.078459+0000","last_unstale":"2024-04-25T01:52:47.063044+0000","last_undegraded":"2024-04-25T01:52:47.063044+0000","last_fullsized":"2024-04-25T01:52:47.063044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:59.349897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469345+0000","last_change":"2024-04-25T01:52:44.088044+0000","last_active":"2024-04-25T01:52:45.469345+0000","last_peered":"2024-04-25T01:52:45.469345+0000","last_clean":"2024-04-25T01:52:45.469345+0000","last_became_active":"2024-04-25T01:52:44.087682+0000","last_became_peered":"2024-04-25T01:52:44.087682+0000","last_unstale":"2024-04-25T01:52:45.469345+0000","last_undegraded":"2024-04-25T01:52:45.469345+0000","last_fullsized":"2024-04-25T01:52:45.469345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:08.933923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'19","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:51.107239+0000","last_change":"2024-04-25T01:52:43.068205+0000","last_active":"2024-04-25T01:52:51.107239+0000","last_peered":"2024-04-25T01:52:51.107239+0000","last_clean":"2024-04-25T01:52:51.107239+0000","last_became_active":"2024-04-25T01:52:43.067866+0000","last_became_peered":"2024-04-25T01:52:43.067866+0000","last_unstale":"2024-04-25T01:52:51.107239+0000","last_undegraded":"2024-04-25T01:52:51.107239+0000","last_fullsized":"2024-04-25T01:52:51.107239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:04.133809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":41,"num_write":20,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063510+0000","last_change":"2024-04-25T01:52:44.091547+0000","last_active":"2024-04-25T01:52:45.063510+0000","last_peered":"2024-04-25T01:52:45.063510+0000","last_clean":"2024-04-25T01:52:45.063510+0000","last_became_active":"2024-04-25T01:52:44.091167+0000","last_became_peered":"2024-04-25T01:52:44.091167+0000","last_unstale":"2024-04-25T01:52:45.063510+0000","last_undegraded":"2024-04-25T01:52:45.063510+0000","last_fullsized":"2024-04-25T01:52:45.063510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:24.559694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.095568+0000","last_change":"2024-04-25T01:52:43.088385+0000","last_active":"2024-04-25T01:52:48.095568+0000","last_peered":"2024-04-25T01:52:48.095568+0000","last_clean":"2024-04-25T01:52:48.095568+0000","last_became_active":"2024-04-25T01:52:43.087982+0000","last_became_peered":"2024-04-25T01:52:43.087982+0000","last_unstale":"2024-04-25T01:52:48.095568+0000","last_undegraded":"2024-04-25T01:52:48.095568+0000","last_fullsized":"2024-04-25T01:52:48.095568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:28.944427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092103+0000","last_change":"2024-04-25T01:52:44.099420+0000","last_active":"2024-04-25T01:52:46.092103+0000","last_peered":"2024-04-25T01:52:46.092103+0000","last_clean":"2024-04-25T01:52:46.092103+0000","last_became_active":"2024-04-25T01:52:44.099045+0000","last_became_peered":"2024-04-25T01:52:44.099045+0000","last_unstale":"2024-04-25T01:52:46.092103+0000","last_undegraded":"2024-04-25T01:52:46.092103+0000","last_fullsized":"2024-04-25T01:52:46.092103+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:28.377091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.063640+0000","last_change":"2024-04-25T01:52:43.086252+0000","last_active":"2024-04-25T01:52:47.063640+0000","last_peered":"2024-04-25T01:52:47.063640+0000","last_clean":"2024-04-25T01:52:47.063640+0000","last_became_active":"2024-04-25T01:52:43.085892+0000","last_became_peered":"2024-04-25T01:52:43.085892+0000","last_unstale":"2024-04-25T01:52:47.063640+0000","last_undegraded":"2024-04-25T01:52:47.063640+0000","last_fullsized":"2024-04-25T01:52:47.063640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:25:26.395705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063826+0000","last_change":"2024-04-25T01:52:44.097163+0000","last_active":"2024-04-25T01:52:45.063826+0000","last_peered":"2024-04-25T01:52:45.063826+0000","last_clean":"2024-04-25T01:52:45.063826+0000","last_became_active":"2024-04-25T01:52:44.096758+0000","last_became_peered":"2024-04-25T01:52:44.096758+0000","last_unstale":"2024-04-25T01:52:45.063826+0000","last_undegraded":"2024-04-25T01:52:45.063826+0000","last_fullsized":"2024-04-25T01:52:45.063826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:22:47.394855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091245+0000","last_change":"2024-04-25T01:52:43.069024+0000","last_active":"2024-04-25T01:52:46.091245+0000","last_peered":"2024-04-25T01:52:46.091245+0000","last_clean":"2024-04-25T01:52:46.091245+0000","last_became_active":"2024-04-25T01:52:43.068058+0000","last_became_peered":"2024-04-25T01:52:43.068058+0000","last_unstale":"2024-04-25T01:52:46.091245+0000","last_undegraded":"2024-04-25T01:52:46.091245+0000","last_fullsized":"2024-04-25T01:52:46.091245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:43.126131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064254+0000","last_change":"2024-04-25T01:52:44.091829+0000","last_active":"2024-04-25T01:52:45.064254+0000","last_peered":"2024-04-25T01:52:45.064254+0000","last_clean":"2024-04-25T01:52:45.064254+0000","last_became_active":"2024-04-25T01:52:44.091528+0000","last_became_peered":"2024-04-25T01:52:44.091528+0000","last_unstale":"2024-04-25T01:52:45.064254+0000","last_undegraded":"2024-04-25T01:52:45.064254+0000","last_fullsized":"2024-04-25T01:52:45.064254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:39.585296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.095590+0000","last_change":"2024-04-25T01:52:43.078464+0000","last_active":"2024-04-25T01:52:48.095590+0000","last_peered":"2024-04-25T01:52:48.095590+0000","last_clean":"2024-04-25T01:52:48.095590+0000","last_became_active":"2024-04-25T01:52:43.078134+0000","last_became_peered":"2024-04-25T01:52:43.078134+0000","last_unstale":"2024-04-25T01:52:48.095590+0000","last_undegraded":"2024-04-25T01:52:48.095590+0000","last_fullsized":"2024-04-25T01:52:48.095590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:49.709382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215767+0000","last_change":"2024-04-25T01:52:44.094696+0000","last_active":"2024-04-25T01:52:45.215767+0000","last_peered":"2024-04-25T01:52:45.215767+0000","last_clean":"2024-04-25T01:52:45.215767+0000","last_became_active":"2024-04-25T01:52:44.094371+0000","last_became_peered":"2024-04-25T01:52:44.094371+0000","last_unstale":"2024-04-25T01:52:45.215767+0000","last_undegraded":"2024-04-25T01:52:45.215767+0000","last_fullsized":"2024-04-25T01:52:45.215767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:20.035674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:49.098997+0000","last_change":"2024-04-25T01:52:43.071275+0000","last_active":"2024-04-25T01:52:49.098997+0000","last_peered":"2024-04-25T01:52:49.098997+0000","last_clean":"2024-04-25T01:52:49.098997+0000","last_became_active":"2024-04-25T01:52:43.070958+0000","last_became_peered":"2024-04-25T01:52:43.070958+0000","last_unstale":"2024-04-25T01:52:49.098997+0000","last_undegraded":"2024-04-25T01:52:49.098997+0000","last_fullsized":"2024-04-25T01:52:49.098997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:23:55.284943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092489+0000","last_change":"2024-04-25T01:52:44.092742+0000","last_active":"2024-04-25T01:52:46.092489+0000","last_peered":"2024-04-25T01:52:46.092489+0000","last_clean":"2024-04-25T01:52:46.092489+0000","last_became_active":"2024-04-25T01:52:44.092345+0000","last_became_peered":"2024-04-25T01:52:44.092345+0000","last_unstale":"2024-04-25T01:52:46.092489+0000","last_undegraded":"2024-04-25T01:52:46.092489+0000","last_fullsized":"2024-04-25T01:52:46.092489+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:14.325814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091892+0000","last_change":"2024-04-25T01:52:43.090057+0000","last_active":"2024-04-25T01:52:46.091892+0000","last_peered":"2024-04-25T01:52:46.091892+0000","last_clean":"2024-04-25T01:52:46.091892+0000","last_became_active":"2024-04-25T01:52:43.089280+0000","last_became_peered":"2024-04-25T01:52:43.089280+0000","last_unstale":"2024-04-25T01:52:46.091892+0000","last_undegraded":"2024-04-25T01:52:46.091892+0000","last_fullsized":"2024-04-25T01:52:46.091892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:41.048908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.155153+0000","last_change":"2024-04-25T01:52:44.097651+0000","last_active":"2024-04-25T01:52:46.155153+0000","last_peered":"2024-04-25T01:52:46.155153+0000","last_clean":"2024-04-25T01:52:46.155153+0000","last_became_active":"2024-04-25T01:52:44.097254+0000","last_became_peered":"2024-04-25T01:52:44.097254+0000","last_unstale":"2024-04-25T01:52:46.155153+0000","last_undegraded":"2024-04-25T01:52:46.155153+0000","last_fullsized":"2024-04-25T01:52:46.155153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:15.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.855239+0000","last_change":"2024-04-25T01:52:43.076925+0000","last_active":"2024-04-25T01:52:46.855239+0000","last_peered":"2024-04-25T01:52:46.855239+0000","last_clean":"2024-04-25T01:52:46.855239+0000","last_became_active":"2024-04-25T01:52:43.076514+0000","last_became_peered":"2024-04-25T01:52:43.076514+0000","last_unstale":"2024-04-25T01:52:46.855239+0000","last_undegraded":"2024-04-25T01:52:46.855239+0000","last_fullsized":"2024-04-25T01:52:46.855239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:31:17.305379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.077325+0000","last_change":"2024-04-25T01:52:44.103731+0000","last_active":"2024-04-25T01:52:45.077325+0000","last_peered":"2024-04-25T01:52:45.077325+0000","last_clean":"2024-04-25T01:52:45.077325+0000","last_became_active":"2024-04-25T01:52:44.103173+0000","last_became_peered":"2024-04-25T01:52:44.103173+0000","last_unstale":"2024-04-25T01:52:45.077325+0000","last_undegraded":"2024-04-25T01:52:45.077325+0000","last_fullsized":"2024-04-25T01:52:45.077325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:43.617579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215705+0000","last_change":"2024-04-25T01:52:44.090847+0000","last_active":"2024-04-25T01:52:45.215705+0000","last_peered":"2024-04-25T01:52:45.215705+0000","last_clean":"2024-04-25T01:52:45.215705+0000","last_became_active":"2024-04-25T01:52:44.090506+0000","last_became_peered":"2024-04-25T01:52:44.090506+0000","last_unstale":"2024-04-25T01:52:45.215705+0000","last_undegraded":"2024-04-25T01:52:45.215705+0000","last_fullsized":"2024-04-25T01:52:45.215705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:24.539726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:50.101485+0000","last_change":"2024-04-25T01:52:43.090469+0000","last_active":"2024-04-25T01:52:50.101485+0000","last_peered":"2024-04-25T01:52:50.101485+0000","last_clean":"2024-04-25T01:52:50.101485+0000","last_became_active":"2024-04-25T01:52:43.090159+0000","last_became_peered":"2024-04-25T01:52:43.090159+0000","last_unstale":"2024-04-25T01:52:50.101485+0000","last_undegraded":"2024-04-25T01:52:50.101485+0000","last_fullsized":"2024-04-25T01:52:50.101485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:42:45.532017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356488+0000","last_change":"2024-04-25T01:52:44.093527+0000","last_active":"2024-04-25T01:52:45.356488+0000","last_peered":"2024-04-25T01:52:45.356488+0000","last_clean":"2024-04-25T01:52:45.356488+0000","last_became_active":"2024-04-25T01:52:44.092729+0000","last_became_peered":"2024-04-25T01:52:44.092729+0000","last_unstale":"2024-04-25T01:52:45.356488+0000","last_undegraded":"2024-04-25T01:52:45.356488+0000","last_fullsized":"2024-04-25T01:52:45.356488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:32:50.668352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.063648+0000","last_change":"2024-04-25T01:52:43.075323+0000","last_active":"2024-04-25T01:52:47.063648+0000","last_peered":"2024-04-25T01:52:47.063648+0000","last_clean":"2024-04-25T01:52:47.063648+0000","last_became_active":"2024-04-25T01:52:43.074876+0000","last_became_peered":"2024-04-25T01:52:43.074876+0000","last_unstale":"2024-04-25T01:52:47.063648+0000","last_undegraded":"2024-04-25T01:52:47.063648+0000","last_fullsized":"2024-04-25T01:52:47.063648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:47.472920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":350,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063387+0000","last_change":"2024-04-25T01:51:20.173285+0000","last_active":"2024-04-25T01:52:45.063387+0000","last_peered":"2024-04-25T01:52:45.063387+0000","last_clean":"2024-04-25T01:52:45.063387+0000","last_became_active":"2024-04-25T01:51:20.172811+0000","last_became_peered":"2024-04-25T01:51:20.172811+0000","last_unstale":"2024-04-25T01:52:45.063387+0000","last_undegraded":"2024-04-25T01:52:45.063387+0000","last_fullsized":"2024-04-25T01:52:45.063387+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_clean_scrub_stamp":"2024-04-25T01:47:31.712321+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:33.708082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091610+0000","last_change":"2024-04-25T01:52:44.087379+0000","last_active":"2024-04-25T01:52:46.091610+0000","last_peered":"2024-04-25T01:52:46.091610+0000","last_clean":"2024-04-25T01:52:46.091610+0000","last_became_active":"2024-04-25T01:52:44.084983+0000","last_became_peered":"2024-04-25T01:52:44.084983+0000","last_unstale":"2024-04-25T01:52:46.091610+0000","last_undegraded":"2024-04-25T01:52:46.091610+0000","last_fullsized":"2024-04-25T01:52:46.091610+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:05.572166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.087287+0000","last_change":"2024-04-25T01:52:43.072565+0000","last_active":"2024-04-25T01:52:48.087287+0000","last_peered":"2024-04-25T01:52:48.087287+0000","last_clean":"2024-04-25T01:52:48.087287+0000","last_became_active":"2024-04-25T01:52:43.072193+0000","last_became_peered":"2024-04-25T01:52:43.072193+0000","last_unstale":"2024-04-25T01:52:48.087287+0000","last_undegraded":"2024-04-25T01:52:48.087287+0000","last_fullsized":"2024-04-25T01:52:48.087287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:07.261859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.063640+0000","last_change":"2024-04-25T01:52:43.088955+0000","last_active":"2024-04-25T01:52:47.063640+0000","last_peered":"2024-04-25T01:52:47.063640+0000","last_clean":"2024-04-25T01:52:47.063640+0000","last_became_active":"2024-04-25T01:52:43.088542+0000","last_became_peered":"2024-04-25T01:52:43.088542+0000","last_unstale":"2024-04-25T01:52:47.063640+0000","last_undegraded":"2024-04-25T01:52:47.063640+0000","last_fullsized":"2024-04-25T01:52:47.063640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:48.427719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092808+0000","last_change":"2024-04-25T01:52:44.104892+0000","last_active":"2024-04-25T01:52:46.092808+0000","last_peered":"2024-04-25T01:52:46.092808+0000","last_clean":"2024-04-25T01:52:46.092808+0000","last_became_active":"2024-04-25T01:52:44.103802+0000","last_became_peered":"2024-04-25T01:52:44.103802+0000","last_unstale":"2024-04-25T01:52:46.092808+0000","last_undegraded":"2024-04-25T01:52:46.092808+0000","last_fullsized":"2024-04-25T01:52:46.092808+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:16.445968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.092939+0000","last_change":"2024-04-25T01:52:43.087143+0000","last_active":"2024-04-25T01:52:48.092939+0000","last_peered":"2024-04-25T01:52:48.092939+0000","last_clean":"2024-04-25T01:52:48.092939+0000","last_became_active":"2024-04-25T01:52:43.085987+0000","last_became_peered":"2024-04-25T01:52:43.085987+0000","last_unstale":"2024-04-25T01:52:48.092939+0000","last_undegraded":"2024-04-25T01:52:48.092939+0000","last_fullsized":"2024-04-25T01:52:48.092939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:17.936671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.357570+0000","last_change":"2024-04-25T01:52:44.092342+0000","last_active":"2024-04-25T01:52:45.357570+0000","last_peered":"2024-04-25T01:52:45.357570+0000","last_clean":"2024-04-25T01:52:45.357570+0000","last_became_active":"2024-04-25T01:52:44.091991+0000","last_became_peered":"2024-04-25T01:52:44.091991+0000","last_unstale":"2024-04-25T01:52:45.357570+0000","last_undegraded":"2024-04-25T01:52:45.357570+0000","last_fullsized":"2024-04-25T01:52:45.357570+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:22.147284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.061672+0000","last_change":"2024-04-25T01:52:43.077972+0000","last_active":"2024-04-25T01:52:47.061672+0000","last_peered":"2024-04-25T01:52:47.061672+0000","last_clean":"2024-04-25T01:52:47.061672+0000","last_became_active":"2024-04-25T01:52:43.077655+0000","last_became_peered":"2024-04-25T01:52:43.077655+0000","last_unstale":"2024-04-25T01:52:47.061672+0000","last_undegraded":"2024-04-25T01:52:47.061672+0000","last_fullsized":"2024-04-25T01:52:47.061672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:24.820910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064151+0000","last_change":"2024-04-25T01:52:44.119560+0000","last_active":"2024-04-25T01:52:45.064151+0000","last_peered":"2024-04-25T01:52:45.064151+0000","last_clean":"2024-04-25T01:52:45.064151+0000","last_became_active":"2024-04-25T01:52:44.119106+0000","last_became_peered":"2024-04-25T01:52:44.119106+0000","last_unstale":"2024-04-25T01:52:45.064151+0000","last_undegraded":"2024-04-25T01:52:45.064151+0000","last_fullsized":"2024-04-25T01:52:45.064151+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:50.702393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.064882+0000","last_change":"2024-04-25T01:52:43.075783+0000","last_active":"2024-04-25T01:52:47.064882+0000","last_peered":"2024-04-25T01:52:47.064882+0000","last_clean":"2024-04-25T01:52:47.064882+0000","last_became_active":"2024-04-25T01:52:43.075462+0000","last_became_peered":"2024-04-25T01:52:43.075462+0000","last_unstale":"2024-04-25T01:52:47.064882+0000","last_undegraded":"2024-04-25T01:52:47.064882+0000","last_fullsized":"2024-04-25T01:52:47.064882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:07.077047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063269+0000","last_change":"2024-04-25T01:52:44.090695+0000","last_active":"2024-04-25T01:52:45.063269+0000","last_peered":"2024-04-25T01:52:45.063269+0000","last_clean":"2024-04-25T01:52:45.063269+0000","last_became_active":"2024-04-25T01:52:44.090203+0000","last_became_peered":"2024-04-25T01:52:44.090203+0000","last_unstale":"2024-04-25T01:52:45.063269+0000","last_undegraded":"2024-04-25T01:52:45.063269+0000","last_fullsized":"2024-04-25T01:52:45.063269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:58.104569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.068717+0000","last_change":"2024-04-25T01:52:43.086057+0000","last_active":"2024-04-25T01:52:47.068717+0000","last_peered":"2024-04-25T01:52:47.068717+0000","last_clean":"2024-04-25T01:52:47.068717+0000","last_became_active":"2024-04-25T01:52:43.085475+0000","last_became_peered":"2024-04-25T01:52:43.085475+0000","last_unstale":"2024-04-25T01:52:47.068717+0000","last_undegraded":"2024-04-25T01:52:47.068717+0000","last_fullsized":"2024-04-25T01:52:47.068717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:42.237414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092187+0000","last_change":"2024-04-25T01:52:44.106128+0000","last_active":"2024-04-25T01:52:46.092187+0000","last_peered":"2024-04-25T01:52:46.092187+0000","last_clean":"2024-04-25T01:52:46.092187+0000","last_became_active":"2024-04-25T01:52:44.105805+0000","last_became_peered":"2024-04-25T01:52:44.105805+0000","last_unstale":"2024-04-25T01:52:46.092187+0000","last_undegraded":"2024-04-25T01:52:46.092187+0000","last_fullsized":"2024-04-25T01:52:46.092187+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:41.901070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092497+0000","last_change":"2024-04-25T01:52:43.088347+0000","last_active":"2024-04-25T01:52:46.092497+0000","last_peered":"2024-04-25T01:52:46.092497+0000","last_clean":"2024-04-25T01:52:46.092497+0000","last_became_active":"2024-04-25T01:52:43.087932+0000","last_became_peered":"2024-04-25T01:52:43.087932+0000","last_unstale":"2024-04-25T01:52:46.092497+0000","last_undegraded":"2024-04-25T01:52:46.092497+0000","last_fullsized":"2024-04-25T01:52:46.092497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:26.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356784+0000","last_change":"2024-04-25T01:52:44.108686+0000","last_active":"2024-04-25T01:52:45.356784+0000","last_peered":"2024-04-25T01:52:45.356784+0000","last_clean":"2024-04-25T01:52:45.356784+0000","last_became_active":"2024-04-25T01:52:44.107732+0000","last_became_peered":"2024-04-25T01:52:44.107732+0000","last_unstale":"2024-04-25T01:52:45.356784+0000","last_undegraded":"2024-04-25T01:52:45.356784+0000","last_fullsized":"2024-04-25T01:52:45.356784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:14.820979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.061004+0000","last_change":"2024-04-25T01:52:43.069300+0000","last_active":"2024-04-25T01:52:47.061004+0000","last_peered":"2024-04-25T01:52:47.061004+0000","last_clean":"2024-04-25T01:52:47.061004+0000","last_became_active":"2024-04-25T01:52:43.068644+0000","last_became_peered":"2024-04-25T01:52:43.068644+0000","last_unstale":"2024-04-25T01:52:47.061004+0000","last_undegraded":"2024-04-25T01:52:47.061004+0000","last_fullsized":"2024-04-25T01:52:47.061004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:04.897638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064022+0000","last_change":"2024-04-25T01:52:44.100467+0000","last_active":"2024-04-25T01:52:45.064022+0000","last_peered":"2024-04-25T01:52:45.064022+0000","last_clean":"2024-04-25T01:52:45.064022+0000","last_became_active":"2024-04-25T01:52:44.099958+0000","last_became_peered":"2024-04-25T01:52:44.099958+0000","last_unstale":"2024-04-25T01:52:45.064022+0000","last_undegraded":"2024-04-25T01:52:45.064022+0000","last_fullsized":"2024-04-25T01:52:45.064022+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:29.919785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.095267+0000","last_change":"2024-04-25T01:52:43.090136+0000","last_active":"2024-04-25T01:52:48.095267+0000","last_peered":"2024-04-25T01:52:48.095267+0000","last_clean":"2024-04-25T01:52:48.095267+0000","last_became_active":"2024-04-25T01:52:43.089800+0000","last_became_peered":"2024-04-25T01:52:43.089800+0000","last_unstale":"2024-04-25T01:52:48.095267+0000","last_undegraded":"2024-04-25T01:52:48.095267+0000","last_fullsized":"2024-04-25T01:52:48.095267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:57:19.651527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063994+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:52:45.063994+0000","last_peered":"2024-04-25T01:52:45.063994+0000","last_clean":"2024-04-25T01:52:45.063994+0000","last_became_active":"2024-04-25T01:52:44.087911+0000","last_became_peered":"2024-04-25T01:52:44.087911+0000","last_unstale":"2024-04-25T01:52:45.063994+0000","last_undegraded":"2024-04-25T01:52:45.063994+0000","last_fullsized":"2024-04-25T01:52:45.063994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:29:38.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215585+0000","last_change":"2024-04-25T01:52:43.086083+0000","last_active":"2024-04-25T01:52:45.215585+0000","last_peered":"2024-04-25T01:52:45.215585+0000","last_clean":"2024-04-25T01:52:45.215585+0000","last_became_active":"2024-04-25T01:52:43.085613+0000","last_became_peered":"2024-04-25T01:52:43.085613+0000","last_unstale":"2024-04-25T01:52:45.215585+0000","last_undegraded":"2024-04-25T01:52:45.215585+0000","last_fullsized":"2024-04-25T01:52:45.215585+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:23.851812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.855271+0000","last_change":"2024-04-25T01:52:44.106435+0000","last_active":"2024-04-25T01:52:46.855271+0000","last_peered":"2024-04-25T01:52:46.855271+0000","last_clean":"2024-04-25T01:52:46.855271+0000","last_became_active":"2024-04-25T01:52:44.106074+0000","last_became_peered":"2024-04-25T01:52:44.106074+0000","last_unstale":"2024-04-25T01:52:46.855271+0000","last_undegraded":"2024-04-25T01:52:46.855271+0000","last_fullsized":"2024-04-25T01:52:46.855271+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:59.286266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092950+0000","last_change":"2024-04-25T01:52:43.088902+0000","last_active":"2024-04-25T01:52:46.092950+0000","last_peered":"2024-04-25T01:52:46.092950+0000","last_clean":"2024-04-25T01:52:46.092950+0000","last_became_active":"2024-04-25T01:52:43.088289+0000","last_became_peered":"2024-04-25T01:52:43.088289+0000","last_unstale":"2024-04-25T01:52:46.092950+0000","last_undegraded":"2024-04-25T01:52:46.092950+0000","last_fullsized":"2024-04-25T01:52:46.092950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:37:44.562369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064218+0000","last_change":"2024-04-25T01:52:44.094185+0000","last_active":"2024-04-25T01:52:45.064218+0000","last_peered":"2024-04-25T01:52:45.064218+0000","last_clean":"2024-04-25T01:52:45.064218+0000","last_became_active":"2024-04-25T01:52:44.093753+0000","last_became_peered":"2024-04-25T01:52:44.093753+0000","last_unstale":"2024-04-25T01:52:45.064218+0000","last_undegraded":"2024-04-25T01:52:45.064218+0000","last_fullsized":"2024-04-25T01:52:45.064218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:06.089670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092224+0000","last_change":"2024-04-25T01:52:44.093895+0000","last_active":"2024-04-25T01:52:46.092224+0000","last_peered":"2024-04-25T01:52:46.092224+0000","last_clean":"2024-04-25T01:52:46.092224+0000","last_became_active":"2024-04-25T01:52:44.093490+0000","last_became_peered":"2024-04-25T01:52:44.093490+0000","last_unstale":"2024-04-25T01:52:46.092224+0000","last_undegraded":"2024-04-25T01:52:46.092224+0000","last_fullsized":"2024-04-25T01:52:46.092224+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:53.848979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092345+0000","last_change":"2024-04-25T01:52:44.092500+0000","last_active":"2024-04-25T01:52:46.092345+0000","last_peered":"2024-04-25T01:52:46.092345+0000","last_clean":"2024-04-25T01:52:46.092345+0000","last_became_active":"2024-04-25T01:52:44.092044+0000","last_became_peered":"2024-04-25T01:52:44.092044+0000","last_unstale":"2024-04-25T01:52:46.092345+0000","last_undegraded":"2024-04-25T01:52:46.092345+0000","last_fullsized":"2024-04-25T01:52:46.092345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:28.279676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092445+0000","last_change":"2024-04-25T01:52:44.116668+0000","last_active":"2024-04-25T01:52:46.092445+0000","last_peered":"2024-04-25T01:52:46.092445+0000","last_clean":"2024-04-25T01:52:46.092445+0000","last_became_active":"2024-04-25T01:52:44.116295+0000","last_became_peered":"2024-04-25T01:52:44.116295+0000","last_unstale":"2024-04-25T01:52:46.092445+0000","last_undegraded":"2024-04-25T01:52:46.092445+0000","last_fullsized":"2024-04-25T01:52:46.092445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:49.109403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063737+0000","last_change":"2024-04-25T01:52:44.109950+0000","last_active":"2024-04-25T01:52:45.063737+0000","last_peered":"2024-04-25T01:52:45.063737+0000","last_clean":"2024-04-25T01:52:45.063737+0000","last_became_active":"2024-04-25T01:52:44.109636+0000","last_became_peered":"2024-04-25T01:52:44.109636+0000","last_unstale":"2024-04-25T01:52:45.063737+0000","last_undegraded":"2024-04-25T01:52:45.063737+0000","last_fullsized":"2024-04-25T01:52:45.063737+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:02:57.610067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092976+0000","last_change":"2024-04-25T01:52:44.105237+0000","last_active":"2024-04-25T01:52:46.092976+0000","last_peered":"2024-04-25T01:52:46.092976+0000","last_clean":"2024-04-25T01:52:46.092976+0000","last_became_active":"2024-04-25T01:52:44.104531+0000","last_became_peered":"2024-04-25T01:52:44.104531+0000","last_unstale":"2024-04-25T01:52:46.092976+0000","last_undegraded":"2024-04-25T01:52:46.092976+0000","last_fullsized":"2024-04-25T01:52:46.092976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.149119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092354+0000","last_change":"2024-04-25T01:52:44.095655+0000","last_active":"2024-04-25T01:52:46.092354+0000","last_peered":"2024-04-25T01:52:46.092354+0000","last_clean":"2024-04-25T01:52:46.092354+0000","last_became_active":"2024-04-25T01:52:44.095300+0000","last_became_peered":"2024-04-25T01:52:44.095300+0000","last_unstale":"2024-04-25T01:52:46.092354+0000","last_undegraded":"2024-04-25T01:52:46.092354+0000","last_fullsized":"2024-04-25T01:52:46.092354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:51.240170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.093105+0000","last_change":"2024-04-25T01:52:44.087382+0000","last_active":"2024-04-25T01:52:46.093105+0000","last_peered":"2024-04-25T01:52:46.093105+0000","last_clean":"2024-04-25T01:52:46.093105+0000","last_became_active":"2024-04-25T01:52:44.086815+0000","last_became_peered":"2024-04-25T01:52:44.086815+0000","last_unstale":"2024-04-25T01:52:46.093105+0000","last_undegraded":"2024-04-25T01:52:46.093105+0000","last_fullsized":"2024-04-25T01:52:46.093105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:52:13.682819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.357279+0000","last_change":"2024-04-25T01:52:44.091494+0000","last_active":"2024-04-25T01:52:45.357279+0000","last_peered":"2024-04-25T01:52:45.357279+0000","last_clean":"2024-04-25T01:52:45.357279+0000","last_became_active":"2024-04-25T01:52:44.091052+0000","last_became_peered":"2024-04-25T01:52:44.091052+0000","last_unstale":"2024-04-25T01:52:45.357279+0000","last_undegraded":"2024-04-25T01:52:45.357279+0000","last_fullsized":"2024-04-25T01:52:45.357279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:37:43.696713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215443+0000","last_change":"2024-04-25T01:52:44.114767+0000","last_active":"2024-04-25T01:52:45.215443+0000","last_peered":"2024-04-25T01:52:45.215443+0000","last_clean":"2024-04-25T01:52:45.215443+0000","last_became_active":"2024-04-25T01:52:44.114483+0000","last_became_peered":"2024-04-25T01:52:44.114483+0000","last_unstale":"2024-04-25T01:52:45.215443+0000","last_undegraded":"2024-04-25T01:52:45.215443+0000","last_fullsized":"2024-04-25T01:52:45.215443+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:14.439287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356227+0000","last_change":"2024-04-25T01:52:44.094424+0000","last_active":"2024-04-25T01:52:45.356227+0000","last_peered":"2024-04-25T01:52:45.356227+0000","last_clean":"2024-04-25T01:52:45.356227+0000","last_became_active":"2024-04-25T01:52:44.093074+0000","last_became_peered":"2024-04-25T01:52:44.093074+0000","last_unstale":"2024-04-25T01:52:45.356227+0000","last_undegraded":"2024-04-25T01:52:45.356227+0000","last_fullsized":"2024-04-25T01:52:45.356227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:20.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469062+0000","last_change":"2024-04-25T01:52:44.113465+0000","last_active":"2024-04-25T01:52:45.469062+0000","last_peered":"2024-04-25T01:52:45.469062+0000","last_clean":"2024-04-25T01:52:45.469062+0000","last_became_active":"2024-04-25T01:52:44.113168+0000","last_became_peered":"2024-04-25T01:52:44.113168+0000","last_unstale":"2024-04-25T01:52:45.469062+0000","last_undegraded":"2024-04-25T01:52:45.469062+0000","last_fullsized":"2024-04-25T01:52:45.469062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:42:47.758996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.093097+0000","last_change":"2024-04-25T01:52:44.104763+0000","last_active":"2024-04-25T01:52:46.093097+0000","last_peered":"2024-04-25T01:52:46.093097+0000","last_clean":"2024-04-25T01:52:46.093097+0000","last_became_active":"2024-04-25T01:52:44.104377+0000","last_became_peered":"2024-04-25T01:52:44.104377+0000","last_unstale":"2024-04-25T01:52:46.093097+0000","last_undegraded":"2024-04-25T01:52:46.093097+0000","last_fullsized":"2024-04-25T01:52:46.093097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:13.647020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092191+0000","last_change":"2024-04-25T01:52:44.088003+0000","last_active":"2024-04-25T01:52:46.092191+0000","last_peered":"2024-04-25T01:52:46.092191+0000","last_clean":"2024-04-25T01:52:46.092191+0000","last_became_active":"2024-04-25T01:52:44.087675+0000","last_became_peered":"2024-04-25T01:52:44.087675+0000","last_unstale":"2024-04-25T01:52:46.092191+0000","last_undegraded":"2024-04-25T01:52:46.092191+0000","last_fullsized":"2024-04-25T01:52:46.092191+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:58.066240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":41572,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":33,"num_read_kb":60,"num_write":136,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":78681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318289,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":445784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579924,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41048,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704104,"statfs":{"total":95995035648,"available":95953002496,"internally_reserved":0,"allocated":1589248,"data_stored":1058160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743451,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":446020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005089,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":445940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266727,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1037870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430251,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":445409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":453414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":446680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":448164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":445720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855414,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":445224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691889,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1037554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":20895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":9953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":39771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:52:52.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:52:53.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.169+0000 7fd797a00640 1 -- 172.21.15.102:0/105657986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7900ffff0 msgr2=0x7fd790100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.169+0000 7fd797a00640 1 --2- 172.21.15.102:0/105657986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7900ffff0 0x7fd790100410 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd784009900 tx=0x7fd78402a410 comp rx=0 tx=0).stop 2024-04-25T01:52:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.169+0000 7fd797a00640 1 -- 172.21.15.102:0/105657986 shutdown_connections 2024-04-25T01:52:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.169+0000 7fd797a00640 1 --2- 172.21.15.102:0/105657986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7901012f0 0x7fd790101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.169+0000 7fd797a00640 1 --2- 172.21.15.102:0/105657986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd790100950 0x7fd790100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.169+0000 7fd797a00640 1 --2- 172.21.15.102:0/105657986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7900ffff0 0x7fd790100410 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.169+0000 7fd797a00640 1 -- 172.21.15.102:0/105657986 >> 172.21.15.102:0/105657986 conn(0x7fd7900fbc30 msgr2=0x7fd7900fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.170+0000 7fd797a00640 1 -- 172.21.15.102:0/105657986 shutdown_connections 2024-04-25T01:52:53.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.170+0000 7fd797a00640 1 -- 172.21.15.102:0/105657986 wait complete. 2024-04-25T01:52:53.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.170+0000 7fd797a00640 1 Processor -- start 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd797a00640 1 -- start start 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd797a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7900ffff0 0x7fd790199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd797a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd790100950 0x7fd79019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd797a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7901012f0 0x7fd7901a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd797a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd79010f630 con 0x7fd7901012f0 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd797a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd79010f4b0 con 0x7fd790100950 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd797a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd79010f7b0 con 0x7fd7900ffff0 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd795a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7901012f0 0x7fd7901a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd78fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd790100950 0x7fd79019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd795a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7901012f0 0x7fd7901a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57912/0 (socket says 172.21.15.102:57912) 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd78fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd790100950 0x7fd79019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:55742/0 (socket says 172.21.15.102:55742) 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd795000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7900ffff0 0x7fd790199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:53.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.171+0000 7fd795a00640 1 -- 172.21.15.102:0/1618258661 learned_addr learned my addr 172.21.15.102:0/1618258661 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:53.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:53 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3503039888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:53.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:53 smithi102 ceph-mon[27796]: pgmap v322: 97 pgs: 97 active+clean; 617 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 9.0 KiB/s wr, 19 op/s 2024-04-25T01:52:53.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.172+0000 7fd78fe00640 1 -- 172.21.15.102:0/1618258661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7900ffff0 msgr2=0x7fd790199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:53.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.172+0000 7fd78fe00640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7900ffff0 0x7fd790199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.172+0000 7fd78fe00640 1 -- 172.21.15.102:0/1618258661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7901012f0 msgr2=0x7fd7901a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:53.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.172+0000 7fd78fe00640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7901012f0 0x7fd7901a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.172+0000 7fd78fe00640 1 -- 172.21.15.102:0/1618258661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7901a18b0 con 0x7fd790100950 2024-04-25T01:52:53.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.172+0000 7fd795000640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7900ffff0 0x7fd790199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.172+0000 7fd78fe00640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd790100950 0x7fd79019a1b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00e9f0 tx=0x7fd77c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.172+0000 7fd78d600640 1 -- 172.21.15.102:0/1618258661 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c00ccc0 con 0x7fd790100950 2024-04-25T01:52:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.173+0000 7fd78d600640 1 -- 172.21.15.102:0/1618258661 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd77c010910 con 0x7fd790100950 2024-04-25T01:52:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.173+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7901a1ba0 con 0x7fd790100950 2024-04-25T01:52:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.173+0000 7fd78d600640 1 -- 172.21.15.102:0/1618258661 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c018960 con 0x7fd790100950 2024-04-25T01:52:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.173+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7901a20c0 con 0x7fd790100950 2024-04-25T01:52:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.174+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd79010aed0 con 0x7fd790100950 2024-04-25T01:52:53.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.176+0000 7fd78d600640 1 -- 172.21.15.102:0/1618258661 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd77c010ac0 con 0x7fd790100950 2024-04-25T01:52:53.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.177+0000 7fd78d600640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7780a2c20 0x7fd7780a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:53.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.178+0000 7fd78d600640 1 -- 172.21.15.102:0/1618258661 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fd77c0cf490 con 0x7fd790100950 2024-04-25T01:52:53.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.178+0000 7fd795000640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7780a2c20 0x7fd7780a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:53.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.178+0000 7fd795000640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7780a2c20 0x7fd7780a50e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd784031440 tx=0x7fd7840315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:53.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.181+0000 7fd78d600640 1 -- 172.21.15.102:0/1618258661 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd77c09bcf0 con 0x7fd790100950 2024-04-25T01:52:53.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.378+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd790061b80 con 0x7fd7780a2c20 2024-04-25T01:52:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.402+0000 7fd78d600640 1 -- 172.21.15.102:0/1618258661 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258738 (secure 0 0 0) 0x7fd790061b80 con 0x7fd7780a2c20 2024-04-25T01:52:53.406 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:52:53.407 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.406+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7780a2c20 msgr2=0x7fd7780a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7780a2c20 0x7fd7780a50e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd784031440 tx=0x7fd7840315b0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd790100950 msgr2=0x7fd79019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd790100950 0x7fd79019a1b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00e9f0 tx=0x7fd77c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 shutdown_connections 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7780a2c20 0x7fd7780a50e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7901012f0 0x7fd7901a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd790100950 0x7fd79019a1b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 --2- 172.21.15.102:0/1618258661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7900ffff0 0x7fd790199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 >> 172.21.15.102:0/1618258661 conn(0x7fd7900fbc30 msgr2=0x7fd7900fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 shutdown_connections 2024-04-25T01:52:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:53.407+0000 7fd797a00640 1 -- 172.21.15.102:0/1618258661 wait complete. 2024-04-25T01:52:53.427 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-04-25T01:52:52.723279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632174,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":169,"num_read_kb":179,"num_write":319,"num_write_kb":3016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":257,"ondisk_log_size":257,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":374548,"kb_used_data":12912,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124567276,"statfs":{"total":1151940427776,"available":1151556890624,"internally_reserved":0,"allocated":13221888,"data_stored":7155939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":370193723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41806,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":33,"num_read_kb":60,"num_write":138,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.661225"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092625+0000","last_change":"2024-04-25T01:52:44.102143+0000","last_active":"2024-04-25T01:52:46.092625+0000","last_peered":"2024-04-25T01:52:46.092625+0000","last_clean":"2024-04-25T01:52:46.092625+0000","last_became_active":"2024-04-25T01:52:44.101810+0000","last_became_peered":"2024-04-25T01:52:44.101810+0000","last_unstale":"2024-04-25T01:52:46.092625+0000","last_undegraded":"2024-04-25T01:52:46.092625+0000","last_fullsized":"2024-04-25T01:52:46.092625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:02.356294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064344+0000","last_change":"2024-04-25T01:52:44.103074+0000","last_active":"2024-04-25T01:52:45.064344+0000","last_peered":"2024-04-25T01:52:45.064344+0000","last_clean":"2024-04-25T01:52:45.064344+0000","last_became_active":"2024-04-25T01:52:44.102103+0000","last_became_peered":"2024-04-25T01:52:44.102103+0000","last_unstale":"2024-04-25T01:52:45.064344+0000","last_undegraded":"2024-04-25T01:52:45.064344+0000","last_fullsized":"2024-04-25T01:52:45.064344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:36:43.016756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064304+0000","last_change":"2024-04-25T01:52:44.101467+0000","last_active":"2024-04-25T01:52:45.064304+0000","last_peered":"2024-04-25T01:52:45.064304+0000","last_clean":"2024-04-25T01:52:45.064304+0000","last_became_active":"2024-04-25T01:52:44.100911+0000","last_became_peered":"2024-04-25T01:52:44.100911+0000","last_unstale":"2024-04-25T01:52:45.064304+0000","last_undegraded":"2024-04-25T01:52:45.064304+0000","last_fullsized":"2024-04-25T01:52:45.064304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:58.533213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091759+0000","last_change":"2024-04-25T01:52:44.088592+0000","last_active":"2024-04-25T01:52:46.091759+0000","last_peered":"2024-04-25T01:52:46.091759+0000","last_clean":"2024-04-25T01:52:46.091759+0000","last_became_active":"2024-04-25T01:52:44.087651+0000","last_became_peered":"2024-04-25T01:52:44.087651+0000","last_unstale":"2024-04-25T01:52:46.091759+0000","last_undegraded":"2024-04-25T01:52:46.091759+0000","last_fullsized":"2024-04-25T01:52:46.091759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:52.240818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063402+0000","last_change":"2024-04-25T01:52:44.095033+0000","last_active":"2024-04-25T01:52:45.063402+0000","last_peered":"2024-04-25T01:52:45.063402+0000","last_clean":"2024-04-25T01:52:45.063402+0000","last_became_active":"2024-04-25T01:52:44.094719+0000","last_became_peered":"2024-04-25T01:52:44.094719+0000","last_unstale":"2024-04-25T01:52:45.063402+0000","last_undegraded":"2024-04-25T01:52:45.063402+0000","last_fullsized":"2024-04-25T01:52:45.063402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:52.422544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.077136+0000","last_change":"2024-04-25T01:52:44.102574+0000","last_active":"2024-04-25T01:52:45.077136+0000","last_peered":"2024-04-25T01:52:45.077136+0000","last_clean":"2024-04-25T01:52:45.077136+0000","last_became_active":"2024-04-25T01:52:44.102158+0000","last_became_peered":"2024-04-25T01:52:44.102158+0000","last_unstale":"2024-04-25T01:52:45.077136+0000","last_undegraded":"2024-04-25T01:52:45.077136+0000","last_fullsized":"2024-04-25T01:52:45.077136+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:20.931392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063622+0000","last_change":"2024-04-25T01:52:44.107662+0000","last_active":"2024-04-25T01:52:45.063622+0000","last_peered":"2024-04-25T01:52:45.063622+0000","last_clean":"2024-04-25T01:52:45.063622+0000","last_became_active":"2024-04-25T01:52:44.107304+0000","last_became_peered":"2024-04-25T01:52:44.107304+0000","last_unstale":"2024-04-25T01:52:45.063622+0000","last_undegraded":"2024-04-25T01:52:45.063622+0000","last_fullsized":"2024-04-25T01:52:45.063622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:42:54.636950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356615+0000","last_change":"2024-04-25T01:52:44.100273+0000","last_active":"2024-04-25T01:52:45.356615+0000","last_peered":"2024-04-25T01:52:45.356615+0000","last_clean":"2024-04-25T01:52:45.356615+0000","last_became_active":"2024-04-25T01:52:44.098223+0000","last_became_peered":"2024-04-25T01:52:44.098223+0000","last_unstale":"2024-04-25T01:52:45.356615+0000","last_undegraded":"2024-04-25T01:52:45.356615+0000","last_fullsized":"2024-04-25T01:52:45.356615+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:05.003265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469299+0000","last_change":"2024-04-25T01:52:44.089658+0000","last_active":"2024-04-25T01:52:45.469299+0000","last_peered":"2024-04-25T01:52:45.469299+0000","last_clean":"2024-04-25T01:52:45.469299+0000","last_became_active":"2024-04-25T01:52:44.089232+0000","last_became_peered":"2024-04-25T01:52:44.089232+0000","last_unstale":"2024-04-25T01:52:45.469299+0000","last_undegraded":"2024-04-25T01:52:45.469299+0000","last_fullsized":"2024-04-25T01:52:45.469299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:54.385782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.077492+0000","last_change":"2024-04-25T01:52:44.086660+0000","last_active":"2024-04-25T01:52:45.077492+0000","last_peered":"2024-04-25T01:52:45.077492+0000","last_clean":"2024-04-25T01:52:45.077492+0000","last_became_active":"2024-04-25T01:52:44.086306+0000","last_became_peered":"2024-04-25T01:52:44.086306+0000","last_unstale":"2024-04-25T01:52:45.077492+0000","last_undegraded":"2024-04-25T01:52:45.077492+0000","last_fullsized":"2024-04-25T01:52:45.077492+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:33:20.726616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064170+0000","last_change":"2024-04-25T01:52:44.090239+0000","last_active":"2024-04-25T01:52:45.064170+0000","last_peered":"2024-04-25T01:52:45.064170+0000","last_clean":"2024-04-25T01:52:45.064170+0000","last_became_active":"2024-04-25T01:52:44.089736+0000","last_became_peered":"2024-04-25T01:52:44.089736+0000","last_unstale":"2024-04-25T01:52:45.064170+0000","last_undegraded":"2024-04-25T01:52:45.064170+0000","last_fullsized":"2024-04-25T01:52:45.064170+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:22.954011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091726+0000","last_change":"2024-04-25T01:52:44.088878+0000","last_active":"2024-04-25T01:52:46.091726+0000","last_peered":"2024-04-25T01:52:46.091726+0000","last_clean":"2024-04-25T01:52:46.091726+0000","last_became_active":"2024-04-25T01:52:44.088250+0000","last_became_peered":"2024-04-25T01:52:44.088250+0000","last_unstale":"2024-04-25T01:52:46.091726+0000","last_undegraded":"2024-04-25T01:52:46.091726+0000","last_fullsized":"2024-04-25T01:52:46.091726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:15.490131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092718+0000","last_change":"2024-04-25T01:52:44.094675+0000","last_active":"2024-04-25T01:52:46.092718+0000","last_peered":"2024-04-25T01:52:46.092718+0000","last_clean":"2024-04-25T01:52:46.092718+0000","last_became_active":"2024-04-25T01:52:44.094219+0000","last_became_peered":"2024-04-25T01:52:44.094219+0000","last_unstale":"2024-04-25T01:52:46.092718+0000","last_undegraded":"2024-04-25T01:52:46.092718+0000","last_fullsized":"2024-04-25T01:52:46.092718+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:26.160156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063520+0000","last_change":"2024-04-25T01:52:44.111131+0000","last_active":"2024-04-25T01:52:45.063520+0000","last_peered":"2024-04-25T01:52:45.063520+0000","last_clean":"2024-04-25T01:52:45.063520+0000","last_became_active":"2024-04-25T01:52:44.105670+0000","last_became_peered":"2024-04-25T01:52:44.105670+0000","last_unstale":"2024-04-25T01:52:45.063520+0000","last_undegraded":"2024-04-25T01:52:45.063520+0000","last_fullsized":"2024-04-25T01:52:45.063520+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:00.377724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063359+0000","last_change":"2024-04-25T01:52:44.109707+0000","last_active":"2024-04-25T01:52:45.063359+0000","last_peered":"2024-04-25T01:52:45.063359+0000","last_clean":"2024-04-25T01:52:45.063359+0000","last_became_active":"2024-04-25T01:52:44.108550+0000","last_became_peered":"2024-04-25T01:52:44.108550+0000","last_unstale":"2024-04-25T01:52:45.063359+0000","last_undegraded":"2024-04-25T01:52:45.063359+0000","last_fullsized":"2024-04-25T01:52:45.063359+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:46.296789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.468826+0000","last_change":"2024-04-25T01:52:44.104504+0000","last_active":"2024-04-25T01:52:45.468826+0000","last_peered":"2024-04-25T01:52:45.468826+0000","last_clean":"2024-04-25T01:52:45.468826+0000","last_became_active":"2024-04-25T01:52:44.104180+0000","last_became_peered":"2024-04-25T01:52:44.104180+0000","last_unstale":"2024-04-25T01:52:45.468826+0000","last_undegraded":"2024-04-25T01:52:45.468826+0000","last_fullsized":"2024-04-25T01:52:45.468826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:17.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092802+0000","last_change":"2024-04-25T01:52:44.107550+0000","last_active":"2024-04-25T01:52:46.092802+0000","last_peered":"2024-04-25T01:52:46.092802+0000","last_clean":"2024-04-25T01:52:46.092802+0000","last_became_active":"2024-04-25T01:52:44.107195+0000","last_became_peered":"2024-04-25T01:52:44.107195+0000","last_unstale":"2024-04-25T01:52:46.092802+0000","last_undegraded":"2024-04-25T01:52:46.092802+0000","last_fullsized":"2024-04-25T01:52:46.092802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:59:42.220575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092172+0000","last_change":"2024-04-25T01:52:44.101165+0000","last_active":"2024-04-25T01:52:46.092172+0000","last_peered":"2024-04-25T01:52:46.092172+0000","last_clean":"2024-04-25T01:52:46.092172+0000","last_became_active":"2024-04-25T01:52:44.100386+0000","last_became_peered":"2024-04-25T01:52:44.100386+0000","last_unstale":"2024-04-25T01:52:46.092172+0000","last_undegraded":"2024-04-25T01:52:46.092172+0000","last_fullsized":"2024-04-25T01:52:46.092172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:30.922707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092985+0000","last_change":"2024-04-25T01:52:44.116694+0000","last_active":"2024-04-25T01:52:46.092985+0000","last_peered":"2024-04-25T01:52:46.092985+0000","last_clean":"2024-04-25T01:52:46.092985+0000","last_became_active":"2024-04-25T01:52:44.116326+0000","last_became_peered":"2024-04-25T01:52:44.116326+0000","last_unstale":"2024-04-25T01:52:46.092985+0000","last_undegraded":"2024-04-25T01:52:46.092985+0000","last_fullsized":"2024-04-25T01:52:46.092985+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:16.329899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.093471+0000","last_change":"2024-04-25T01:52:43.072358+0000","last_active":"2024-04-25T01:52:48.093471+0000","last_peered":"2024-04-25T01:52:48.093471+0000","last_clean":"2024-04-25T01:52:48.093471+0000","last_became_active":"2024-04-25T01:52:43.071815+0000","last_became_peered":"2024-04-25T01:52:43.071815+0000","last_unstale":"2024-04-25T01:52:48.093471+0000","last_undegraded":"2024-04-25T01:52:48.093471+0000","last_fullsized":"2024-04-25T01:52:48.093471+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:15.295744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.357381+0000","last_change":"2024-04-25T01:52:44.094509+0000","last_active":"2024-04-25T01:52:45.357381+0000","last_peered":"2024-04-25T01:52:45.357381+0000","last_clean":"2024-04-25T01:52:45.357381+0000","last_became_active":"2024-04-25T01:52:44.093955+0000","last_became_peered":"2024-04-25T01:52:44.093955+0000","last_unstale":"2024-04-25T01:52:45.357381+0000","last_undegraded":"2024-04-25T01:52:45.357381+0000","last_fullsized":"2024-04-25T01:52:45.357381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:37.701835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.357441+0000","last_change":"2024-04-25T01:52:43.094435+0000","last_active":"2024-04-25T01:52:45.357441+0000","last_peered":"2024-04-25T01:52:45.357441+0000","last_clean":"2024-04-25T01:52:45.357441+0000","last_became_active":"2024-04-25T01:52:43.094018+0000","last_became_peered":"2024-04-25T01:52:43.094018+0000","last_unstale":"2024-04-25T01:52:45.357441+0000","last_undegraded":"2024-04-25T01:52:45.357441+0000","last_fullsized":"2024-04-25T01:52:45.357441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:36.781014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064164+0000","last_change":"2024-04-25T01:52:44.104347+0000","last_active":"2024-04-25T01:52:45.064164+0000","last_peered":"2024-04-25T01:52:45.064164+0000","last_clean":"2024-04-25T01:52:45.064164+0000","last_became_active":"2024-04-25T01:52:44.103803+0000","last_became_peered":"2024-04-25T01:52:44.103803+0000","last_unstale":"2024-04-25T01:52:45.064164+0000","last_undegraded":"2024-04-25T01:52:45.064164+0000","last_fullsized":"2024-04-25T01:52:45.064164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:01.784893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.086026+0000","last_change":"2024-04-25T01:52:43.070902+0000","last_active":"2024-04-25T01:52:48.086026+0000","last_peered":"2024-04-25T01:52:48.086026+0000","last_clean":"2024-04-25T01:52:48.086026+0000","last_became_active":"2024-04-25T01:52:43.070313+0000","last_became_peered":"2024-04-25T01:52:43.070313+0000","last_unstale":"2024-04-25T01:52:48.086026+0000","last_undegraded":"2024-04-25T01:52:48.086026+0000","last_fullsized":"2024-04-25T01:52:48.086026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:10.677320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469073+0000","last_change":"2024-04-25T01:52:44.093293+0000","last_active":"2024-04-25T01:52:45.469073+0000","last_peered":"2024-04-25T01:52:45.469073+0000","last_clean":"2024-04-25T01:52:45.469073+0000","last_became_active":"2024-04-25T01:52:44.092932+0000","last_became_peered":"2024-04-25T01:52:44.092932+0000","last_unstale":"2024-04-25T01:52:45.469073+0000","last_undegraded":"2024-04-25T01:52:45.469073+0000","last_fullsized":"2024-04-25T01:52:45.469073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:58:37.705594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063605+0000","last_change":"2024-04-25T01:52:43.069743+0000","last_active":"2024-04-25T01:52:45.063605+0000","last_peered":"2024-04-25T01:52:45.063605+0000","last_clean":"2024-04-25T01:52:45.063605+0000","last_became_active":"2024-04-25T01:52:43.068947+0000","last_became_peered":"2024-04-25T01:52:43.068947+0000","last_unstale":"2024-04-25T01:52:45.063605+0000","last_undegraded":"2024-04-25T01:52:45.063605+0000","last_fullsized":"2024-04-25T01:52:45.063605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:02:52.582176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215774+0000","last_change":"2024-04-25T01:52:44.091398+0000","last_active":"2024-04-25T01:52:45.215774+0000","last_peered":"2024-04-25T01:52:45.215774+0000","last_clean":"2024-04-25T01:52:45.215774+0000","last_became_active":"2024-04-25T01:52:44.091061+0000","last_became_peered":"2024-04-25T01:52:44.091061+0000","last_unstale":"2024-04-25T01:52:45.215774+0000","last_undegraded":"2024-04-25T01:52:45.215774+0000","last_fullsized":"2024-04-25T01:52:45.215774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:54.858769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:51.105148+0000","last_change":"2024-04-25T01:52:43.068450+0000","last_active":"2024-04-25T01:52:51.105148+0000","last_peered":"2024-04-25T01:52:51.105148+0000","last_clean":"2024-04-25T01:52:51.105148+0000","last_became_active":"2024-04-25T01:52:43.067881+0000","last_became_peered":"2024-04-25T01:52:43.067881+0000","last_unstale":"2024-04-25T01:52:51.105148+0000","last_undegraded":"2024-04-25T01:52:51.105148+0000","last_fullsized":"2024-04-25T01:52:51.105148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:12.739321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063670+0000","last_change":"2024-04-25T01:52:44.101743+0000","last_active":"2024-04-25T01:52:45.063670+0000","last_peered":"2024-04-25T01:52:45.063670+0000","last_clean":"2024-04-25T01:52:45.063670+0000","last_became_active":"2024-04-25T01:52:44.101351+0000","last_became_peered":"2024-04-25T01:52:44.101351+0000","last_unstale":"2024-04-25T01:52:45.063670+0000","last_undegraded":"2024-04-25T01:52:45.063670+0000","last_fullsized":"2024-04-25T01:52:45.063670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:54:24.104819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.087944+0000","last_change":"2024-04-25T01:52:43.075880+0000","last_active":"2024-04-25T01:52:48.087944+0000","last_peered":"2024-04-25T01:52:48.087944+0000","last_clean":"2024-04-25T01:52:48.087944+0000","last_became_active":"2024-04-25T01:52:43.075522+0000","last_became_peered":"2024-04-25T01:52:43.075522+0000","last_unstale":"2024-04-25T01:52:48.087944+0000","last_undegraded":"2024-04-25T01:52:48.087944+0000","last_fullsized":"2024-04-25T01:52:48.087944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:26.218774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215753+0000","last_change":"2024-04-25T01:52:44.116772+0000","last_active":"2024-04-25T01:52:45.215753+0000","last_peered":"2024-04-25T01:52:45.215753+0000","last_clean":"2024-04-25T01:52:45.215753+0000","last_became_active":"2024-04-25T01:52:44.116408+0000","last_became_peered":"2024-04-25T01:52:44.116408+0000","last_unstale":"2024-04-25T01:52:45.215753+0000","last_undegraded":"2024-04-25T01:52:45.215753+0000","last_fullsized":"2024-04-25T01:52:45.215753+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:21.988425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.096835+0000","last_change":"2024-04-25T01:52:43.076964+0000","last_active":"2024-04-25T01:52:48.096835+0000","last_peered":"2024-04-25T01:52:48.096835+0000","last_clean":"2024-04-25T01:52:48.096835+0000","last_became_active":"2024-04-25T01:52:43.073915+0000","last_became_peered":"2024-04-25T01:52:43.073915+0000","last_unstale":"2024-04-25T01:52:48.096835+0000","last_undegraded":"2024-04-25T01:52:48.096835+0000","last_fullsized":"2024-04-25T01:52:48.096835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:03.321920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356871+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:52:45.356871+0000","last_peered":"2024-04-25T01:52:45.356871+0000","last_clean":"2024-04-25T01:52:45.356871+0000","last_became_active":"2024-04-25T01:52:44.088198+0000","last_became_peered":"2024-04-25T01:52:44.088198+0000","last_unstale":"2024-04-25T01:52:45.356871+0000","last_undegraded":"2024-04-25T01:52:45.356871+0000","last_fullsized":"2024-04-25T01:52:45.356871+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:30.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215295+0000","last_change":"2024-04-25T01:52:43.069853+0000","last_active":"2024-04-25T01:52:45.215295+0000","last_peered":"2024-04-25T01:52:45.215295+0000","last_clean":"2024-04-25T01:52:45.215295+0000","last_became_active":"2024-04-25T01:52:43.069526+0000","last_became_peered":"2024-04-25T01:52:43.069526+0000","last_unstale":"2024-04-25T01:52:45.215295+0000","last_undegraded":"2024-04-25T01:52:45.215295+0000","last_fullsized":"2024-04-25T01:52:45.215295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:35:03.272610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091612+0000","last_change":"2024-04-25T01:52:44.114296+0000","last_active":"2024-04-25T01:52:46.091612+0000","last_peered":"2024-04-25T01:52:46.091612+0000","last_clean":"2024-04-25T01:52:46.091612+0000","last_became_active":"2024-04-25T01:52:44.113958+0000","last_became_peered":"2024-04-25T01:52:44.113958+0000","last_unstale":"2024-04-25T01:52:46.091612+0000","last_undegraded":"2024-04-25T01:52:46.091612+0000","last_fullsized":"2024-04-25T01:52:46.091612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:45.068080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:50.102913+0000","last_change":"2024-04-25T01:52:43.071388+0000","last_active":"2024-04-25T01:52:50.102913+0000","last_peered":"2024-04-25T01:52:50.102913+0000","last_clean":"2024-04-25T01:52:50.102913+0000","last_became_active":"2024-04-25T01:52:43.071005+0000","last_became_peered":"2024-04-25T01:52:43.071005+0000","last_unstale":"2024-04-25T01:52:50.102913+0000","last_undegraded":"2024-04-25T01:52:50.102913+0000","last_fullsized":"2024-04-25T01:52:50.102913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:56:59.580458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469294+0000","last_change":"2024-04-25T01:52:44.106106+0000","last_active":"2024-04-25T01:52:45.469294+0000","last_peered":"2024-04-25T01:52:45.469294+0000","last_clean":"2024-04-25T01:52:45.469294+0000","last_became_active":"2024-04-25T01:52:44.105760+0000","last_became_peered":"2024-04-25T01:52:44.105760+0000","last_unstale":"2024-04-25T01:52:45.469294+0000","last_undegraded":"2024-04-25T01:52:45.469294+0000","last_fullsized":"2024-04-25T01:52:45.469294+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:24.626753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.061022+0000","last_change":"2024-04-25T01:52:43.068926+0000","last_active":"2024-04-25T01:52:47.061022+0000","last_peered":"2024-04-25T01:52:47.061022+0000","last_clean":"2024-04-25T01:52:47.061022+0000","last_became_active":"2024-04-25T01:52:43.067356+0000","last_became_peered":"2024-04-25T01:52:43.067356+0000","last_unstale":"2024-04-25T01:52:47.061022+0000","last_undegraded":"2024-04-25T01:52:47.061022+0000","last_fullsized":"2024-04-25T01:52:47.061022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:19.134633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064251+0000","last_change":"2024-04-25T01:52:44.108299+0000","last_active":"2024-04-25T01:52:45.064251+0000","last_peered":"2024-04-25T01:52:45.064251+0000","last_clean":"2024-04-25T01:52:45.064251+0000","last_became_active":"2024-04-25T01:52:44.107573+0000","last_became_peered":"2024-04-25T01:52:44.107573+0000","last_unstale":"2024-04-25T01:52:45.064251+0000","last_undegraded":"2024-04-25T01:52:45.064251+0000","last_fullsized":"2024-04-25T01:52:45.064251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.804579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.063044+0000","last_change":"2024-04-25T01:52:43.079263+0000","last_active":"2024-04-25T01:52:47.063044+0000","last_peered":"2024-04-25T01:52:47.063044+0000","last_clean":"2024-04-25T01:52:47.063044+0000","last_became_active":"2024-04-25T01:52:43.078459+0000","last_became_peered":"2024-04-25T01:52:43.078459+0000","last_unstale":"2024-04-25T01:52:47.063044+0000","last_undegraded":"2024-04-25T01:52:47.063044+0000","last_fullsized":"2024-04-25T01:52:47.063044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:59.349897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469345+0000","last_change":"2024-04-25T01:52:44.088044+0000","last_active":"2024-04-25T01:52:45.469345+0000","last_peered":"2024-04-25T01:52:45.469345+0000","last_clean":"2024-04-25T01:52:45.469345+0000","last_became_active":"2024-04-25T01:52:44.087682+0000","last_became_peered":"2024-04-25T01:52:44.087682+0000","last_unstale":"2024-04-25T01:52:45.469345+0000","last_undegraded":"2024-04-25T01:52:45.469345+0000","last_fullsized":"2024-04-25T01:52:45.469345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:08.933923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'19","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:51.107239+0000","last_change":"2024-04-25T01:52:43.068205+0000","last_active":"2024-04-25T01:52:51.107239+0000","last_peered":"2024-04-25T01:52:51.107239+0000","last_clean":"2024-04-25T01:52:51.107239+0000","last_became_active":"2024-04-25T01:52:43.067866+0000","last_became_peered":"2024-04-25T01:52:43.067866+0000","last_unstale":"2024-04-25T01:52:51.107239+0000","last_undegraded":"2024-04-25T01:52:51.107239+0000","last_fullsized":"2024-04-25T01:52:51.107239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:04.133809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38193,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":41,"num_write":20,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063510+0000","last_change":"2024-04-25T01:52:44.091547+0000","last_active":"2024-04-25T01:52:45.063510+0000","last_peered":"2024-04-25T01:52:45.063510+0000","last_clean":"2024-04-25T01:52:45.063510+0000","last_became_active":"2024-04-25T01:52:44.091167+0000","last_became_peered":"2024-04-25T01:52:44.091167+0000","last_unstale":"2024-04-25T01:52:45.063510+0000","last_undegraded":"2024-04-25T01:52:45.063510+0000","last_fullsized":"2024-04-25T01:52:45.063510+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:24.559694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.095568+0000","last_change":"2024-04-25T01:52:43.088385+0000","last_active":"2024-04-25T01:52:48.095568+0000","last_peered":"2024-04-25T01:52:48.095568+0000","last_clean":"2024-04-25T01:52:48.095568+0000","last_became_active":"2024-04-25T01:52:43.087982+0000","last_became_peered":"2024-04-25T01:52:43.087982+0000","last_unstale":"2024-04-25T01:52:48.095568+0000","last_undegraded":"2024-04-25T01:52:48.095568+0000","last_fullsized":"2024-04-25T01:52:48.095568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:28.944427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092103+0000","last_change":"2024-04-25T01:52:44.099420+0000","last_active":"2024-04-25T01:52:46.092103+0000","last_peered":"2024-04-25T01:52:46.092103+0000","last_clean":"2024-04-25T01:52:46.092103+0000","last_became_active":"2024-04-25T01:52:44.099045+0000","last_became_peered":"2024-04-25T01:52:44.099045+0000","last_unstale":"2024-04-25T01:52:46.092103+0000","last_undegraded":"2024-04-25T01:52:46.092103+0000","last_fullsized":"2024-04-25T01:52:46.092103+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:28.377091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.063640+0000","last_change":"2024-04-25T01:52:43.086252+0000","last_active":"2024-04-25T01:52:47.063640+0000","last_peered":"2024-04-25T01:52:47.063640+0000","last_clean":"2024-04-25T01:52:47.063640+0000","last_became_active":"2024-04-25T01:52:43.085892+0000","last_became_peered":"2024-04-25T01:52:43.085892+0000","last_unstale":"2024-04-25T01:52:47.063640+0000","last_undegraded":"2024-04-25T01:52:47.063640+0000","last_fullsized":"2024-04-25T01:52:47.063640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:25:26.395705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063826+0000","last_change":"2024-04-25T01:52:44.097163+0000","last_active":"2024-04-25T01:52:45.063826+0000","last_peered":"2024-04-25T01:52:45.063826+0000","last_clean":"2024-04-25T01:52:45.063826+0000","last_became_active":"2024-04-25T01:52:44.096758+0000","last_became_peered":"2024-04-25T01:52:44.096758+0000","last_unstale":"2024-04-25T01:52:45.063826+0000","last_undegraded":"2024-04-25T01:52:45.063826+0000","last_fullsized":"2024-04-25T01:52:45.063826+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:22:47.394855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091245+0000","last_change":"2024-04-25T01:52:43.069024+0000","last_active":"2024-04-25T01:52:46.091245+0000","last_peered":"2024-04-25T01:52:46.091245+0000","last_clean":"2024-04-25T01:52:46.091245+0000","last_became_active":"2024-04-25T01:52:43.068058+0000","last_became_peered":"2024-04-25T01:52:43.068058+0000","last_unstale":"2024-04-25T01:52:46.091245+0000","last_undegraded":"2024-04-25T01:52:46.091245+0000","last_fullsized":"2024-04-25T01:52:46.091245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:43.126131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064254+0000","last_change":"2024-04-25T01:52:44.091829+0000","last_active":"2024-04-25T01:52:45.064254+0000","last_peered":"2024-04-25T01:52:45.064254+0000","last_clean":"2024-04-25T01:52:45.064254+0000","last_became_active":"2024-04-25T01:52:44.091528+0000","last_became_peered":"2024-04-25T01:52:44.091528+0000","last_unstale":"2024-04-25T01:52:45.064254+0000","last_undegraded":"2024-04-25T01:52:45.064254+0000","last_fullsized":"2024-04-25T01:52:45.064254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:39.585296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.095590+0000","last_change":"2024-04-25T01:52:43.078464+0000","last_active":"2024-04-25T01:52:48.095590+0000","last_peered":"2024-04-25T01:52:48.095590+0000","last_clean":"2024-04-25T01:52:48.095590+0000","last_became_active":"2024-04-25T01:52:43.078134+0000","last_became_peered":"2024-04-25T01:52:43.078134+0000","last_unstale":"2024-04-25T01:52:48.095590+0000","last_undegraded":"2024-04-25T01:52:48.095590+0000","last_fullsized":"2024-04-25T01:52:48.095590+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:49.709382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215767+0000","last_change":"2024-04-25T01:52:44.094696+0000","last_active":"2024-04-25T01:52:45.215767+0000","last_peered":"2024-04-25T01:52:45.215767+0000","last_clean":"2024-04-25T01:52:45.215767+0000","last_became_active":"2024-04-25T01:52:44.094371+0000","last_became_peered":"2024-04-25T01:52:44.094371+0000","last_unstale":"2024-04-25T01:52:45.215767+0000","last_undegraded":"2024-04-25T01:52:45.215767+0000","last_fullsized":"2024-04-25T01:52:45.215767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:20.035674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:49.098997+0000","last_change":"2024-04-25T01:52:43.071275+0000","last_active":"2024-04-25T01:52:49.098997+0000","last_peered":"2024-04-25T01:52:49.098997+0000","last_clean":"2024-04-25T01:52:49.098997+0000","last_became_active":"2024-04-25T01:52:43.070958+0000","last_became_peered":"2024-04-25T01:52:43.070958+0000","last_unstale":"2024-04-25T01:52:49.098997+0000","last_undegraded":"2024-04-25T01:52:49.098997+0000","last_fullsized":"2024-04-25T01:52:49.098997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:23:55.284943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092489+0000","last_change":"2024-04-25T01:52:44.092742+0000","last_active":"2024-04-25T01:52:46.092489+0000","last_peered":"2024-04-25T01:52:46.092489+0000","last_clean":"2024-04-25T01:52:46.092489+0000","last_became_active":"2024-04-25T01:52:44.092345+0000","last_became_peered":"2024-04-25T01:52:44.092345+0000","last_unstale":"2024-04-25T01:52:46.092489+0000","last_undegraded":"2024-04-25T01:52:46.092489+0000","last_fullsized":"2024-04-25T01:52:46.092489+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:14.325814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091892+0000","last_change":"2024-04-25T01:52:43.090057+0000","last_active":"2024-04-25T01:52:46.091892+0000","last_peered":"2024-04-25T01:52:46.091892+0000","last_clean":"2024-04-25T01:52:46.091892+0000","last_became_active":"2024-04-25T01:52:43.089280+0000","last_became_peered":"2024-04-25T01:52:43.089280+0000","last_unstale":"2024-04-25T01:52:46.091892+0000","last_undegraded":"2024-04-25T01:52:46.091892+0000","last_fullsized":"2024-04-25T01:52:46.091892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:41.048908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.155153+0000","last_change":"2024-04-25T01:52:44.097651+0000","last_active":"2024-04-25T01:52:46.155153+0000","last_peered":"2024-04-25T01:52:46.155153+0000","last_clean":"2024-04-25T01:52:46.155153+0000","last_became_active":"2024-04-25T01:52:44.097254+0000","last_became_peered":"2024-04-25T01:52:44.097254+0000","last_unstale":"2024-04-25T01:52:46.155153+0000","last_undegraded":"2024-04-25T01:52:46.155153+0000","last_fullsized":"2024-04-25T01:52:46.155153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:15.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.855239+0000","last_change":"2024-04-25T01:52:43.076925+0000","last_active":"2024-04-25T01:52:46.855239+0000","last_peered":"2024-04-25T01:52:46.855239+0000","last_clean":"2024-04-25T01:52:46.855239+0000","last_became_active":"2024-04-25T01:52:43.076514+0000","last_became_peered":"2024-04-25T01:52:43.076514+0000","last_unstale":"2024-04-25T01:52:46.855239+0000","last_undegraded":"2024-04-25T01:52:46.855239+0000","last_fullsized":"2024-04-25T01:52:46.855239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:31:17.305379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.077325+0000","last_change":"2024-04-25T01:52:44.103731+0000","last_active":"2024-04-25T01:52:45.077325+0000","last_peered":"2024-04-25T01:52:45.077325+0000","last_clean":"2024-04-25T01:52:45.077325+0000","last_became_active":"2024-04-25T01:52:44.103173+0000","last_became_peered":"2024-04-25T01:52:44.103173+0000","last_unstale":"2024-04-25T01:52:45.077325+0000","last_undegraded":"2024-04-25T01:52:45.077325+0000","last_fullsized":"2024-04-25T01:52:45.077325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:43.617579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215705+0000","last_change":"2024-04-25T01:52:44.090847+0000","last_active":"2024-04-25T01:52:45.215705+0000","last_peered":"2024-04-25T01:52:45.215705+0000","last_clean":"2024-04-25T01:52:45.215705+0000","last_became_active":"2024-04-25T01:52:44.090506+0000","last_became_peered":"2024-04-25T01:52:44.090506+0000","last_unstale":"2024-04-25T01:52:45.215705+0000","last_undegraded":"2024-04-25T01:52:45.215705+0000","last_fullsized":"2024-04-25T01:52:45.215705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:24.539726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:50.101485+0000","last_change":"2024-04-25T01:52:43.090469+0000","last_active":"2024-04-25T01:52:50.101485+0000","last_peered":"2024-04-25T01:52:50.101485+0000","last_clean":"2024-04-25T01:52:50.101485+0000","last_became_active":"2024-04-25T01:52:43.090159+0000","last_became_peered":"2024-04-25T01:52:43.090159+0000","last_unstale":"2024-04-25T01:52:50.101485+0000","last_undegraded":"2024-04-25T01:52:50.101485+0000","last_fullsized":"2024-04-25T01:52:50.101485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:42:45.532017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356488+0000","last_change":"2024-04-25T01:52:44.093527+0000","last_active":"2024-04-25T01:52:45.356488+0000","last_peered":"2024-04-25T01:52:45.356488+0000","last_clean":"2024-04-25T01:52:45.356488+0000","last_became_active":"2024-04-25T01:52:44.092729+0000","last_became_peered":"2024-04-25T01:52:44.092729+0000","last_unstale":"2024-04-25T01:52:45.356488+0000","last_undegraded":"2024-04-25T01:52:45.356488+0000","last_fullsized":"2024-04-25T01:52:45.356488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:32:50.668352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.063648+0000","last_change":"2024-04-25T01:52:43.075323+0000","last_active":"2024-04-25T01:52:47.063648+0000","last_peered":"2024-04-25T01:52:47.063648+0000","last_clean":"2024-04-25T01:52:47.063648+0000","last_became_active":"2024-04-25T01:52:43.074876+0000","last_became_peered":"2024-04-25T01:52:43.074876+0000","last_unstale":"2024-04-25T01:52:47.063648+0000","last_undegraded":"2024-04-25T01:52:47.063648+0000","last_fullsized":"2024-04-25T01:52:47.063648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:47.472920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":350,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063387+0000","last_change":"2024-04-25T01:51:20.173285+0000","last_active":"2024-04-25T01:52:45.063387+0000","last_peered":"2024-04-25T01:52:45.063387+0000","last_clean":"2024-04-25T01:52:45.063387+0000","last_became_active":"2024-04-25T01:51:20.172811+0000","last_became_peered":"2024-04-25T01:51:20.172811+0000","last_unstale":"2024-04-25T01:52:45.063387+0000","last_undegraded":"2024-04-25T01:52:45.063387+0000","last_fullsized":"2024-04-25T01:52:45.063387+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_clean_scrub_stamp":"2024-04-25T01:47:31.712321+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:33.708082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.091610+0000","last_change":"2024-04-25T01:52:44.087379+0000","last_active":"2024-04-25T01:52:46.091610+0000","last_peered":"2024-04-25T01:52:46.091610+0000","last_clean":"2024-04-25T01:52:46.091610+0000","last_became_active":"2024-04-25T01:52:44.084983+0000","last_became_peered":"2024-04-25T01:52:44.084983+0000","last_unstale":"2024-04-25T01:52:46.091610+0000","last_undegraded":"2024-04-25T01:52:46.091610+0000","last_fullsized":"2024-04-25T01:52:46.091610+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:05.572166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.087287+0000","last_change":"2024-04-25T01:52:43.072565+0000","last_active":"2024-04-25T01:52:48.087287+0000","last_peered":"2024-04-25T01:52:48.087287+0000","last_clean":"2024-04-25T01:52:48.087287+0000","last_became_active":"2024-04-25T01:52:43.072193+0000","last_became_peered":"2024-04-25T01:52:43.072193+0000","last_unstale":"2024-04-25T01:52:48.087287+0000","last_undegraded":"2024-04-25T01:52:48.087287+0000","last_fullsized":"2024-04-25T01:52:48.087287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:07.261859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.063640+0000","last_change":"2024-04-25T01:52:43.088955+0000","last_active":"2024-04-25T01:52:47.063640+0000","last_peered":"2024-04-25T01:52:47.063640+0000","last_clean":"2024-04-25T01:52:47.063640+0000","last_became_active":"2024-04-25T01:52:43.088542+0000","last_became_peered":"2024-04-25T01:52:43.088542+0000","last_unstale":"2024-04-25T01:52:47.063640+0000","last_undegraded":"2024-04-25T01:52:47.063640+0000","last_fullsized":"2024-04-25T01:52:47.063640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:36:48.427719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092808+0000","last_change":"2024-04-25T01:52:44.104892+0000","last_active":"2024-04-25T01:52:46.092808+0000","last_peered":"2024-04-25T01:52:46.092808+0000","last_clean":"2024-04-25T01:52:46.092808+0000","last_became_active":"2024-04-25T01:52:44.103802+0000","last_became_peered":"2024-04-25T01:52:44.103802+0000","last_unstale":"2024-04-25T01:52:46.092808+0000","last_undegraded":"2024-04-25T01:52:46.092808+0000","last_fullsized":"2024-04-25T01:52:46.092808+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:16.445968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.092939+0000","last_change":"2024-04-25T01:52:43.087143+0000","last_active":"2024-04-25T01:52:48.092939+0000","last_peered":"2024-04-25T01:52:48.092939+0000","last_clean":"2024-04-25T01:52:48.092939+0000","last_became_active":"2024-04-25T01:52:43.085987+0000","last_became_peered":"2024-04-25T01:52:43.085987+0000","last_unstale":"2024-04-25T01:52:48.092939+0000","last_undegraded":"2024-04-25T01:52:48.092939+0000","last_fullsized":"2024-04-25T01:52:48.092939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:17.936671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.357570+0000","last_change":"2024-04-25T01:52:44.092342+0000","last_active":"2024-04-25T01:52:45.357570+0000","last_peered":"2024-04-25T01:52:45.357570+0000","last_clean":"2024-04-25T01:52:45.357570+0000","last_became_active":"2024-04-25T01:52:44.091991+0000","last_became_peered":"2024-04-25T01:52:44.091991+0000","last_unstale":"2024-04-25T01:52:45.357570+0000","last_undegraded":"2024-04-25T01:52:45.357570+0000","last_fullsized":"2024-04-25T01:52:45.357570+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:22.147284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.061672+0000","last_change":"2024-04-25T01:52:43.077972+0000","last_active":"2024-04-25T01:52:47.061672+0000","last_peered":"2024-04-25T01:52:47.061672+0000","last_clean":"2024-04-25T01:52:47.061672+0000","last_became_active":"2024-04-25T01:52:43.077655+0000","last_became_peered":"2024-04-25T01:52:43.077655+0000","last_unstale":"2024-04-25T01:52:47.061672+0000","last_undegraded":"2024-04-25T01:52:47.061672+0000","last_fullsized":"2024-04-25T01:52:47.061672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:24.820910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064151+0000","last_change":"2024-04-25T01:52:44.119560+0000","last_active":"2024-04-25T01:52:45.064151+0000","last_peered":"2024-04-25T01:52:45.064151+0000","last_clean":"2024-04-25T01:52:45.064151+0000","last_became_active":"2024-04-25T01:52:44.119106+0000","last_became_peered":"2024-04-25T01:52:44.119106+0000","last_unstale":"2024-04-25T01:52:45.064151+0000","last_undegraded":"2024-04-25T01:52:45.064151+0000","last_fullsized":"2024-04-25T01:52:45.064151+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:50.702393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.064882+0000","last_change":"2024-04-25T01:52:43.075783+0000","last_active":"2024-04-25T01:52:47.064882+0000","last_peered":"2024-04-25T01:52:47.064882+0000","last_clean":"2024-04-25T01:52:47.064882+0000","last_became_active":"2024-04-25T01:52:43.075462+0000","last_became_peered":"2024-04-25T01:52:43.075462+0000","last_unstale":"2024-04-25T01:52:47.064882+0000","last_undegraded":"2024-04-25T01:52:47.064882+0000","last_fullsized":"2024-04-25T01:52:47.064882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:07.077047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063269+0000","last_change":"2024-04-25T01:52:44.090695+0000","last_active":"2024-04-25T01:52:45.063269+0000","last_peered":"2024-04-25T01:52:45.063269+0000","last_clean":"2024-04-25T01:52:45.063269+0000","last_became_active":"2024-04-25T01:52:44.090203+0000","last_became_peered":"2024-04-25T01:52:44.090203+0000","last_unstale":"2024-04-25T01:52:45.063269+0000","last_undegraded":"2024-04-25T01:52:45.063269+0000","last_fullsized":"2024-04-25T01:52:45.063269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:58.104569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.068717+0000","last_change":"2024-04-25T01:52:43.086057+0000","last_active":"2024-04-25T01:52:47.068717+0000","last_peered":"2024-04-25T01:52:47.068717+0000","last_clean":"2024-04-25T01:52:47.068717+0000","last_became_active":"2024-04-25T01:52:43.085475+0000","last_became_peered":"2024-04-25T01:52:43.085475+0000","last_unstale":"2024-04-25T01:52:47.068717+0000","last_undegraded":"2024-04-25T01:52:47.068717+0000","last_fullsized":"2024-04-25T01:52:47.068717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:42.237414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092187+0000","last_change":"2024-04-25T01:52:44.106128+0000","last_active":"2024-04-25T01:52:46.092187+0000","last_peered":"2024-04-25T01:52:46.092187+0000","last_clean":"2024-04-25T01:52:46.092187+0000","last_became_active":"2024-04-25T01:52:44.105805+0000","last_became_peered":"2024-04-25T01:52:44.105805+0000","last_unstale":"2024-04-25T01:52:46.092187+0000","last_undegraded":"2024-04-25T01:52:46.092187+0000","last_fullsized":"2024-04-25T01:52:46.092187+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:41.901070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092497+0000","last_change":"2024-04-25T01:52:43.088347+0000","last_active":"2024-04-25T01:52:46.092497+0000","last_peered":"2024-04-25T01:52:46.092497+0000","last_clean":"2024-04-25T01:52:46.092497+0000","last_became_active":"2024-04-25T01:52:43.087932+0000","last_became_peered":"2024-04-25T01:52:43.087932+0000","last_unstale":"2024-04-25T01:52:46.092497+0000","last_undegraded":"2024-04-25T01:52:46.092497+0000","last_fullsized":"2024-04-25T01:52:46.092497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:26.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356784+0000","last_change":"2024-04-25T01:52:44.108686+0000","last_active":"2024-04-25T01:52:45.356784+0000","last_peered":"2024-04-25T01:52:45.356784+0000","last_clean":"2024-04-25T01:52:45.356784+0000","last_became_active":"2024-04-25T01:52:44.107732+0000","last_became_peered":"2024-04-25T01:52:44.107732+0000","last_unstale":"2024-04-25T01:52:45.356784+0000","last_undegraded":"2024-04-25T01:52:45.356784+0000","last_fullsized":"2024-04-25T01:52:45.356784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:14.820979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:47.061004+0000","last_change":"2024-04-25T01:52:43.069300+0000","last_active":"2024-04-25T01:52:47.061004+0000","last_peered":"2024-04-25T01:52:47.061004+0000","last_clean":"2024-04-25T01:52:47.061004+0000","last_became_active":"2024-04-25T01:52:43.068644+0000","last_became_peered":"2024-04-25T01:52:43.068644+0000","last_unstale":"2024-04-25T01:52:47.061004+0000","last_undegraded":"2024-04-25T01:52:47.061004+0000","last_fullsized":"2024-04-25T01:52:47.061004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:04.897638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064022+0000","last_change":"2024-04-25T01:52:44.100467+0000","last_active":"2024-04-25T01:52:45.064022+0000","last_peered":"2024-04-25T01:52:45.064022+0000","last_clean":"2024-04-25T01:52:45.064022+0000","last_became_active":"2024-04-25T01:52:44.099958+0000","last_became_peered":"2024-04-25T01:52:44.099958+0000","last_unstale":"2024-04-25T01:52:45.064022+0000","last_undegraded":"2024-04-25T01:52:45.064022+0000","last_fullsized":"2024-04-25T01:52:45.064022+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:29.919785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:48.095267+0000","last_change":"2024-04-25T01:52:43.090136+0000","last_active":"2024-04-25T01:52:48.095267+0000","last_peered":"2024-04-25T01:52:48.095267+0000","last_clean":"2024-04-25T01:52:48.095267+0000","last_became_active":"2024-04-25T01:52:43.089800+0000","last_became_peered":"2024-04-25T01:52:43.089800+0000","last_unstale":"2024-04-25T01:52:48.095267+0000","last_undegraded":"2024-04-25T01:52:48.095267+0000","last_fullsized":"2024-04-25T01:52:48.095267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:57:19.651527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063994+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:52:45.063994+0000","last_peered":"2024-04-25T01:52:45.063994+0000","last_clean":"2024-04-25T01:52:45.063994+0000","last_became_active":"2024-04-25T01:52:44.087911+0000","last_became_peered":"2024-04-25T01:52:44.087911+0000","last_unstale":"2024-04-25T01:52:45.063994+0000","last_undegraded":"2024-04-25T01:52:45.063994+0000","last_fullsized":"2024-04-25T01:52:45.063994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:29:38.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215585+0000","last_change":"2024-04-25T01:52:43.086083+0000","last_active":"2024-04-25T01:52:45.215585+0000","last_peered":"2024-04-25T01:52:45.215585+0000","last_clean":"2024-04-25T01:52:45.215585+0000","last_became_active":"2024-04-25T01:52:43.085613+0000","last_became_peered":"2024-04-25T01:52:43.085613+0000","last_unstale":"2024-04-25T01:52:45.215585+0000","last_undegraded":"2024-04-25T01:52:45.215585+0000","last_fullsized":"2024-04-25T01:52:45.215585+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:23.851812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.855271+0000","last_change":"2024-04-25T01:52:44.106435+0000","last_active":"2024-04-25T01:52:46.855271+0000","last_peered":"2024-04-25T01:52:46.855271+0000","last_clean":"2024-04-25T01:52:46.855271+0000","last_became_active":"2024-04-25T01:52:44.106074+0000","last_became_peered":"2024-04-25T01:52:44.106074+0000","last_unstale":"2024-04-25T01:52:46.855271+0000","last_undegraded":"2024-04-25T01:52:46.855271+0000","last_fullsized":"2024-04-25T01:52:46.855271+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:59.286266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092950+0000","last_change":"2024-04-25T01:52:43.088902+0000","last_active":"2024-04-25T01:52:46.092950+0000","last_peered":"2024-04-25T01:52:46.092950+0000","last_clean":"2024-04-25T01:52:46.092950+0000","last_became_active":"2024-04-25T01:52:43.088289+0000","last_became_peered":"2024-04-25T01:52:43.088289+0000","last_unstale":"2024-04-25T01:52:46.092950+0000","last_undegraded":"2024-04-25T01:52:46.092950+0000","last_fullsized":"2024-04-25T01:52:46.092950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:37:44.562369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.064218+0000","last_change":"2024-04-25T01:52:44.094185+0000","last_active":"2024-04-25T01:52:45.064218+0000","last_peered":"2024-04-25T01:52:45.064218+0000","last_clean":"2024-04-25T01:52:45.064218+0000","last_became_active":"2024-04-25T01:52:44.093753+0000","last_became_peered":"2024-04-25T01:52:44.093753+0000","last_unstale":"2024-04-25T01:52:45.064218+0000","last_undegraded":"2024-04-25T01:52:45.064218+0000","last_fullsized":"2024-04-25T01:52:45.064218+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:06.089670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092224+0000","last_change":"2024-04-25T01:52:44.093895+0000","last_active":"2024-04-25T01:52:46.092224+0000","last_peered":"2024-04-25T01:52:46.092224+0000","last_clean":"2024-04-25T01:52:46.092224+0000","last_became_active":"2024-04-25T01:52:44.093490+0000","last_became_peered":"2024-04-25T01:52:44.093490+0000","last_unstale":"2024-04-25T01:52:46.092224+0000","last_undegraded":"2024-04-25T01:52:46.092224+0000","last_fullsized":"2024-04-25T01:52:46.092224+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:53.848979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092345+0000","last_change":"2024-04-25T01:52:44.092500+0000","last_active":"2024-04-25T01:52:46.092345+0000","last_peered":"2024-04-25T01:52:46.092345+0000","last_clean":"2024-04-25T01:52:46.092345+0000","last_became_active":"2024-04-25T01:52:44.092044+0000","last_became_peered":"2024-04-25T01:52:44.092044+0000","last_unstale":"2024-04-25T01:52:46.092345+0000","last_undegraded":"2024-04-25T01:52:46.092345+0000","last_fullsized":"2024-04-25T01:52:46.092345+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:28.279676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092445+0000","last_change":"2024-04-25T01:52:44.116668+0000","last_active":"2024-04-25T01:52:46.092445+0000","last_peered":"2024-04-25T01:52:46.092445+0000","last_clean":"2024-04-25T01:52:46.092445+0000","last_became_active":"2024-04-25T01:52:44.116295+0000","last_became_peered":"2024-04-25T01:52:44.116295+0000","last_unstale":"2024-04-25T01:52:46.092445+0000","last_undegraded":"2024-04-25T01:52:46.092445+0000","last_fullsized":"2024-04-25T01:52:46.092445+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:49.109403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.063737+0000","last_change":"2024-04-25T01:52:44.109950+0000","last_active":"2024-04-25T01:52:45.063737+0000","last_peered":"2024-04-25T01:52:45.063737+0000","last_clean":"2024-04-25T01:52:45.063737+0000","last_became_active":"2024-04-25T01:52:44.109636+0000","last_became_peered":"2024-04-25T01:52:44.109636+0000","last_unstale":"2024-04-25T01:52:45.063737+0000","last_undegraded":"2024-04-25T01:52:45.063737+0000","last_fullsized":"2024-04-25T01:52:45.063737+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:02:57.610067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092976+0000","last_change":"2024-04-25T01:52:44.105237+0000","last_active":"2024-04-25T01:52:46.092976+0000","last_peered":"2024-04-25T01:52:46.092976+0000","last_clean":"2024-04-25T01:52:46.092976+0000","last_became_active":"2024-04-25T01:52:44.104531+0000","last_became_peered":"2024-04-25T01:52:44.104531+0000","last_unstale":"2024-04-25T01:52:46.092976+0000","last_undegraded":"2024-04-25T01:52:46.092976+0000","last_fullsized":"2024-04-25T01:52:46.092976+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.149119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092354+0000","last_change":"2024-04-25T01:52:44.095655+0000","last_active":"2024-04-25T01:52:46.092354+0000","last_peered":"2024-04-25T01:52:46.092354+0000","last_clean":"2024-04-25T01:52:46.092354+0000","last_became_active":"2024-04-25T01:52:44.095300+0000","last_became_peered":"2024-04-25T01:52:44.095300+0000","last_unstale":"2024-04-25T01:52:46.092354+0000","last_undegraded":"2024-04-25T01:52:46.092354+0000","last_fullsized":"2024-04-25T01:52:46.092354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:51.240170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.093105+0000","last_change":"2024-04-25T01:52:44.087382+0000","last_active":"2024-04-25T01:52:46.093105+0000","last_peered":"2024-04-25T01:52:46.093105+0000","last_clean":"2024-04-25T01:52:46.093105+0000","last_became_active":"2024-04-25T01:52:44.086815+0000","last_became_peered":"2024-04-25T01:52:44.086815+0000","last_unstale":"2024-04-25T01:52:46.093105+0000","last_undegraded":"2024-04-25T01:52:46.093105+0000","last_fullsized":"2024-04-25T01:52:46.093105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:52:13.682819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.357279+0000","last_change":"2024-04-25T01:52:44.091494+0000","last_active":"2024-04-25T01:52:45.357279+0000","last_peered":"2024-04-25T01:52:45.357279+0000","last_clean":"2024-04-25T01:52:45.357279+0000","last_became_active":"2024-04-25T01:52:44.091052+0000","last_became_peered":"2024-04-25T01:52:44.091052+0000","last_unstale":"2024-04-25T01:52:45.357279+0000","last_undegraded":"2024-04-25T01:52:45.357279+0000","last_fullsized":"2024-04-25T01:52:45.357279+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:37:43.696713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.215443+0000","last_change":"2024-04-25T01:52:44.114767+0000","last_active":"2024-04-25T01:52:45.215443+0000","last_peered":"2024-04-25T01:52:45.215443+0000","last_clean":"2024-04-25T01:52:45.215443+0000","last_became_active":"2024-04-25T01:52:44.114483+0000","last_became_peered":"2024-04-25T01:52:44.114483+0000","last_unstale":"2024-04-25T01:52:45.215443+0000","last_undegraded":"2024-04-25T01:52:45.215443+0000","last_fullsized":"2024-04-25T01:52:45.215443+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:14.439287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.356227+0000","last_change":"2024-04-25T01:52:44.094424+0000","last_active":"2024-04-25T01:52:45.356227+0000","last_peered":"2024-04-25T01:52:45.356227+0000","last_clean":"2024-04-25T01:52:45.356227+0000","last_became_active":"2024-04-25T01:52:44.093074+0000","last_became_peered":"2024-04-25T01:52:44.093074+0000","last_unstale":"2024-04-25T01:52:45.356227+0000","last_undegraded":"2024-04-25T01:52:45.356227+0000","last_fullsized":"2024-04-25T01:52:45.356227+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:20.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:45.469062+0000","last_change":"2024-04-25T01:52:44.113465+0000","last_active":"2024-04-25T01:52:45.469062+0000","last_peered":"2024-04-25T01:52:45.469062+0000","last_clean":"2024-04-25T01:52:45.469062+0000","last_became_active":"2024-04-25T01:52:44.113168+0000","last_became_peered":"2024-04-25T01:52:44.113168+0000","last_unstale":"2024-04-25T01:52:45.469062+0000","last_undegraded":"2024-04-25T01:52:45.469062+0000","last_fullsized":"2024-04-25T01:52:45.469062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:42:47.758996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.093097+0000","last_change":"2024-04-25T01:52:44.104763+0000","last_active":"2024-04-25T01:52:46.093097+0000","last_peered":"2024-04-25T01:52:46.093097+0000","last_clean":"2024-04-25T01:52:46.093097+0000","last_became_active":"2024-04-25T01:52:44.104377+0000","last_became_peered":"2024-04-25T01:52:44.104377+0000","last_unstale":"2024-04-25T01:52:46.093097+0000","last_undegraded":"2024-04-25T01:52:46.093097+0000","last_fullsized":"2024-04-25T01:52:46.093097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:13.647020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-25T01:52:46.092191+0000","last_change":"2024-04-25T01:52:44.088003+0000","last_active":"2024-04-25T01:52:46.092191+0000","last_peered":"2024-04-25T01:52:46.092191+0000","last_clean":"2024-04-25T01:52:46.092191+0000","last_became_active":"2024-04-25T01:52:44.087675+0000","last_became_peered":"2024-04-25T01:52:44.087675+0000","last_unstale":"2024-04-25T01:52:46.092191+0000","last_undegraded":"2024-04-25T01:52:46.092191+0000","last_fullsized":"2024-04-25T01:52:46.092191+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:55:58.066240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":41572,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":33,"num_read_kb":60,"num_write":136,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":78681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118,"ondisk_log_size":118,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318289,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":445784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579924,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41048,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704104,"statfs":{"total":95995035648,"available":95953002496,"internally_reserved":0,"allocated":1589248,"data_stored":1058160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743451,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":446020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005089,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":445940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266727,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1544192,"data_stored":1037870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430251,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":445409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":453414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":446680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":448164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117052,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":954368,"data_stored":445720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855414,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":445224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691889,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1037554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":20895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":9953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":39771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:52:53.433 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T01:52:53.433 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T01:52:53.433 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T01:52:53.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:53 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3503039888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:53.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:53 smithi110 ceph-mon[32848]: pgmap v322: 97 pgs: 97 active+clean; 617 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 9.0 KiB/s wr, 19 op/s 2024-04-25T01:52:53.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:53 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3503039888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:52:53.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:53 smithi193 ceph-mon[32981]: pgmap v322: 97 pgs: 97 active+clean; 617 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 9.0 KiB/s wr, 19 op/s 2024-04-25T01:52:53.901 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:54.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:54 smithi102 ceph-mon[27796]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:54.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:54 smithi102 ceph-mon[27796]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:54.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:54 smithi110 ceph-mon[32848]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:54.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:54 smithi110 ceph-mon[32848]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:54.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:54 smithi193 ceph-mon[32981]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:54.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:54 smithi193 ceph-mon[32981]: from='client.24725 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:52:55.486 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:55 smithi110 ceph-mon[32848]: pgmap v323: 97 pgs: 97 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-04-25T01:52:55.504 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:55 smithi193 ceph-mon[32981]: pgmap v323: 97 pgs: 97 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-04-25T01:52:55.581 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:55 smithi102 ceph-mon[27796]: pgmap v323: 97 pgs: 97 active+clean; 618 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-04-25T01:52:56.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.276+0000 7f0fb9400700 1 -- 172.21.15.102:0/2670470737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41022b0 msgr2=0x7f0fb41026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:56.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.276+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2670470737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41022b0 0x7f0fb41026d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0fa40099c0 tx=0x7f0fa4009cd0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.276+0000 7f0fb9400700 1 -- 172.21.15.102:0/2670470737 shutdown_connections 2024-04-25T01:52:56.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.276+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2670470737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fb4103e50 0x7f0fb410a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.276+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2670470737 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41022b0 0x7f0fb41026d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.276+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2670470737 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fb41034b0 0x7f0fb4103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.276+0000 7f0fb9400700 1 -- 172.21.15.102:0/2670470737 >> 172.21.15.102:0/2670470737 conn(0x7f0fb40fd850 msgr2=0x7f0fb40ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:56.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.277+0000 7f0fb9400700 1 -- 172.21.15.102:0/2670470737 shutdown_connections 2024-04-25T01:52:56.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.277+0000 7f0fb9400700 1 -- 172.21.15.102:0/2670470737 wait complete. 2024-04-25T01:52:56.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.278+0000 7f0fb9400700 1 Processor -- start 2024-04-25T01:52:56.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.278+0000 7f0fb9400700 1 -- start start 2024-04-25T01:52:56.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fb9400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fb41022b0 0x7f0fb4075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:56.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fb9400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41034b0 0x7f0fb4074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:56.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fb9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fb4103e50 0x7f0fb4074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:56.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fb9400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb410dee0 con 0x7f0fb41034b0 2024-04-25T01:52:56.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fb9400700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb410dd60 con 0x7f0fb41022b0 2024-04-25T01:52:56.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fb9400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb410e060 con 0x7f0fb4103e50 2024-04-25T01:52:56.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fa9e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41034b0 0x7f0fb4074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:56.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fa9e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41034b0 0x7f0fb4074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57936/0 (socket says 172.21.15.102:57936) 2024-04-25T01:52:56.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fa9e00700 1 -- 172.21.15.102:0/2955961578 learned_addr learned my addr 172.21.15.102:0/2955961578 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:52:56.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fb2800700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fb41022b0 0x7f0fb4075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:56.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fb3200700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fb4103e50 0x7f0fb4074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:56.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fa9e00700 1 -- 172.21.15.102:0/2955961578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fb4103e50 msgr2=0x7f0fb4074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:56.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fa9e00700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fb4103e50 0x7f0fb4074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fa9e00700 1 -- 172.21.15.102:0/2955961578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fb41022b0 msgr2=0x7f0fb4075fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.279+0000 7f0fa9e00700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fb41022b0 0x7f0fb4075fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.280+0000 7f0fa9e00700 1 -- 172.21.15.102:0/2955961578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb4075410 con 0x7f0fb41034b0 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.280+0000 7f0fb2800700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fb41022b0 0x7f0fb4075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.280+0000 7f0fa9e00700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41034b0 0x7f0fb4074620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00e9a0 tx=0x7f0f9c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.280+0000 7f0fabe00700 1 -- 172.21.15.102:0/2955961578 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c00ca10 con 0x7f0fb41034b0 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.280+0000 7f0fabe00700 1 -- 172.21.15.102:0/2955961578 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f9c00cbb0 con 0x7f0fb41034b0 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.280+0000 7f0fabe00700 1 -- 172.21.15.102:0/2955961578 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c018680 con 0x7f0fb41034b0 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.280+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb41a25f0 con 0x7f0fb41034b0 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.280+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb41a2ab0 con 0x7f0fb41034b0 2024-04-25T01:52:56.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.282+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb40507a0 con 0x7f0fb41034b0 2024-04-25T01:52:56.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.283+0000 7f0fabe00700 1 -- 172.21.15.102:0/2955961578 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0f9c010430 con 0x7f0fb41034b0 2024-04-25T01:52:56.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.285+0000 7f0fabe00700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f980a2de0 0x7f0f980a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:52:56.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.285+0000 7f0fabe00700 1 -- 172.21.15.102:0/2955961578 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f0f9c0cf2c0 con 0x7f0fb41034b0 2024-04-25T01:52:56.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.285+0000 7f0fb2800700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f980a2de0 0x7f0f980a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:52:56.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.286+0000 7f0fb2800700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f980a2de0 0x7f0f980a52a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b580 tx=0x7f0fa4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:52:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.289+0000 7f0fabe00700 1 -- 172.21.15.102:0/2955961578 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0f9c09c170 con 0x7f0fb41034b0 2024-04-25T01:52:56.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.603+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f0fb4062e40 con 0x7f0fb41034b0 2024-04-25T01:52:56.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.608+0000 7f0fabe00700 1 -- 172.21.15.102:0/2955961578 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f0f9c09ba70 con 0x7f0fb41034b0 2024-04-25T01:52:56.609 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-04-25T01:52:56.610 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQB4tylmmicjJBAAnkxlBu7GFmVVW37Jenxnqg== 2024-04-25T01:52:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f980a2de0 msgr2=0x7f0f980a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f980a2de0 0x7f0f980a52a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400b580 tx=0x7f0fa4005fb0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41034b0 msgr2=0x7f0fb4074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:52:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41034b0 0x7f0fb4074620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00e9a0 tx=0x7f0f9c00ecb0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 shutdown_connections 2024-04-25T01:52:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f980a2de0 0x7f0f980a52a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fb41022b0 0x7f0fb4075fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fb4103e50 0x7f0fb4074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 --2- 172.21.15.102:0/2955961578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0fb41034b0 0x7f0fb4074620 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:52:56.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.612+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 >> 172.21.15.102:0/2955961578 conn(0x7f0fb40fd850 msgr2=0x7f0fb41067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:52:56.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.613+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 shutdown_connections 2024-04-25T01:52:56.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:52:56.613+0000 7f0fb9400700 1 -- 172.21.15.102:0/2955961578 wait complete. 2024-04-25T01:52:56.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:56 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2955961578' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:52:56.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:56 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2955961578' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:52:56.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:56 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2955961578' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:52:56.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:56 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2955961578' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:52:57.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:56 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2955961578' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T01:52:57.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:56 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2955961578' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T01:52:57.608 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:52:57.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T01:52:57.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T01:52:57.738 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T01:52:57.738 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T01:52:57.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T01:52:57.848 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:57 smithi102 ceph-mon[27796]: pgmap v324: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.9 KiB/s wr, 17 op/s 2024-04-25T01:52:57.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:57 smithi110 ceph-mon[32848]: pgmap v324: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.9 KiB/s wr, 17 op/s 2024-04-25T01:52:58.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:57 smithi193 ceph-mon[32981]: pgmap v324: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.9 KiB/s wr, 17 op/s 2024-04-25T01:52:58.216 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:52:59.983 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:52:59 smithi102 ceph-mon[27796]: pgmap v325: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 16 op/s 2024-04-25T01:53:00.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:52:59 smithi110 ceph-mon[32848]: pgmap v325: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 16 op/s 2024-04-25T01:53:00.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:52:59 smithi193 ceph-mon[32981]: pgmap v325: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 KiB/s wr, 16 op/s 2024-04-25T01:53:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.556+0000 7fbaee200700 1 -- 172.21.15.102:0/2091983119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8107160 msgr2=0x7fbae8069f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.556+0000 7fbaee200700 1 --2- 172.21.15.102:0/2091983119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8107160 0x7fbae8069f40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009990 tx=0x7fbad8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.556+0000 7fbaee200700 1 -- 172.21.15.102:0/2091983119 shutdown_connections 2024-04-25T01:53:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.556+0000 7fbaee200700 1 --2- 172.21.15.102:0/2091983119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8107160 0x7fbae8069f40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.556+0000 7fbaee200700 1 --2- 172.21.15.102:0/2091983119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbae8104c70 0x7fbae8105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.556+0000 7fbaee200700 1 --2- 172.21.15.102:0/2091983119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbae806a480 0x7fbae806a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.556+0000 7fbaee200700 1 -- 172.21.15.102:0/2091983119 >> 172.21.15.102:0/2091983119 conn(0x7fbae80780e0 msgr2=0x7fbae80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.556+0000 7fbaee200700 1 -- 172.21.15.102:0/2091983119 shutdown_connections 2024-04-25T01:53:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.557+0000 7fbaee200700 1 -- 172.21.15.102:0/2091983119 wait complete. 2024-04-25T01:53:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.558+0000 7fbaee200700 1 Processor -- start 2024-04-25T01:53:00.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbaee200700 1 -- start start 2024-04-25T01:53:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbaee200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbae806a480 0x7fbae819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbaee200700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8104c70 0x7fbae819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbaee200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbae8107160 0x7fbae81a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbaee200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbae8118b40 con 0x7fbae806a480 2024-04-25T01:53:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbaee200700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbae81189c0 con 0x7fbae8104c70 2024-04-25T01:53:00.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbaee200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbae8118cc0 con 0x7fbae8107160 2024-04-25T01:53:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbae7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbae806a480 0x7fbae819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbadfe00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8104c70 0x7fbae819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbae7400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbae806a480 0x7fbae819f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57962/0 (socket says 172.21.15.102:57962) 2024-04-25T01:53:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.559+0000 7fbae7400700 1 -- 172.21.15.102:0/220143533 learned_addr learned my addr 172.21.15.102:0/220143533 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.560+0000 7fbae7e00700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbae8107160 0x7fbae81a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.560+0000 7fbadfe00700 1 -- 172.21.15.102:0/220143533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbae8107160 msgr2=0x7fbae81a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.560+0000 7fbadfe00700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbae8107160 0x7fbae81a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.560+0000 7fbadfe00700 1 -- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbae806a480 msgr2=0x7fbae819f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.560+0000 7fbadfe00700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbae806a480 0x7fbae819f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.560+0000 7fbadfe00700 1 -- 172.21.15.102:0/220143533 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae81a3ff0 con 0x7fbae8104c70 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.560+0000 7fbae7400700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbae806a480 0x7fbae819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.560+0000 7fbadfe00700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8104c70 0x7fbae819f570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbad000e9f0 tx=0x7fbad000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.561+0000 7fbae4c00700 1 -- 172.21.15.102:0/220143533 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad000c9c0 con 0x7fbae8104c70 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.561+0000 7fbae4c00700 1 -- 172.21.15.102:0/220143533 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbad000cb60 con 0x7fbae8104c70 2024-04-25T01:53:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.561+0000 7fbae4c00700 1 -- 172.21.15.102:0/220143533 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0018680 con 0x7fbae8104c70 2024-04-25T01:53:00.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.561+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae81a42e0 con 0x7fbae8104c70 2024-04-25T01:53:00.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.561+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae81abb60 con 0x7fbae8104c70 2024-04-25T01:53:00.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.562+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae8068f20 con 0x7fbae8104c70 2024-04-25T01:53:00.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.564+0000 7fbae4c00700 1 -- 172.21.15.102:0/220143533 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fbad0010430 con 0x7fbae8104c70 2024-04-25T01:53:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.565+0000 7fbae4c00700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbacc0a2d30 0x7fbacc0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.566+0000 7fbae4c00700 1 -- 172.21.15.102:0/220143533 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fbad00cf1a0 con 0x7fbae8104c70 2024-04-25T01:53:00.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.566+0000 7fbae7400700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbacc0a2d30 0x7fbacc0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:00.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.566+0000 7fbae7400700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbacc0a2d30 0x7fbacc0a51f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbad8006010 tx=0x7fbad800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:00.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.570+0000 7fbae4c00700 1 -- 172.21.15.102:0/220143533 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbad009c050 con 0x7fbae8104c70 2024-04-25T01:53:00.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.861+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fbae81abd60 con 0x7fbae8104c70 2024-04-25T01:53:00.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.867+0000 7fbae4c00700 1 -- 172.21.15.102:0/220143533 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263202 (secure 0 0 0) 0x7fbad009b950 con 0x7fbae8104c70 2024-04-25T01:53:00.872 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:53:00.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.872+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbacc0a2d30 msgr2=0x7fbacc0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:00.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.872+0000 7fbaee200700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbacc0a2d30 0x7fbacc0a51f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbad8006010 tx=0x7fbad800b260 comp rx=0 tx=0).stop 2024-04-25T01:53:00.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.872+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8104c70 msgr2=0x7fbae819f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:00.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.872+0000 7fbaee200700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8104c70 0x7fbae819f570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbad000e9f0 tx=0x7fbad000ed00 comp rx=0 tx=0).stop 2024-04-25T01:53:00.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.873+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 shutdown_connections 2024-04-25T01:53:00.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.873+0000 7fbaee200700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbacc0a2d30 0x7fbacc0a51f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.873+0000 7fbaee200700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbae8107160 0x7fbae81a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.873+0000 7fbaee200700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbae806a480 0x7fbae819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.873+0000 7fbaee200700 1 --2- 172.21.15.102:0/220143533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8104c70 0x7fbae819f570 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:00.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.873+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 >> 172.21.15.102:0/220143533 conn(0x7fbae80780e0 msgr2=0x7fbae8100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:00.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.873+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 shutdown_connections 2024-04-25T01:53:00.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:00.873+0000 7fbaee200700 1 -- 172.21.15.102:0/220143533 wait complete. 2024-04-25T01:53:01.709 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":83904525},{"type":"v1","addr":"172.21.15.102:6801","nonce":83904525}]},"active_addr":"172.21.15.102:6801/83904525","active_change":"2024-04-25T01:44:42.581452+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.102:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":511834001}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":459072323}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":2875896366}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":1835499087}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":4056663447}]}]} 2024-04-25T01:53:01.714 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T01:53:01.714 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T01:53:01.715 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:53:01.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:01 smithi102 ceph-mon[27796]: pgmap v326: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-04-25T01:53:01.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:01 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/220143533' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:53:02.189 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:02.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:01 smithi110 ceph-mon[32848]: pgmap v326: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-04-25T01:53:02.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:01 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/220143533' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:53:02.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:01 smithi193 ceph-mon[32981]: pgmap v326: 97 pgs: 97 active+clean; 623 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-04-25T01:53:02.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:01 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/220143533' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T01:53:04.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:03 smithi102 ceph-mon[27796]: pgmap v327: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-04-25T01:53:04.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:03 smithi110 ceph-mon[32848]: pgmap v327: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-04-25T01:53:04.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:03 smithi193 ceph-mon[32981]: pgmap v327: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-04-25T01:53:04.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.462+0000 7f61f7000700 1 -- 172.21.15.102:0/1430613346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f010ece0 msgr2=0x7f61f01110d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.462+0000 7f61f7000700 1 --2- 172.21.15.102:0/1430613346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f010ece0 0x7f61f01110d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009f90 tx=0x7f61e800b330 comp rx=0 tx=0).stop 2024-04-25T01:53:04.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.463+0000 7f61f7000700 1 -- 172.21.15.102:0/1430613346 shutdown_connections 2024-04-25T01:53:04.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.463+0000 7f61f7000700 1 --2- 172.21.15.102:0/1430613346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f010ece0 0x7f61f01110d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.463+0000 7f61f7000700 1 --2- 172.21.15.102:0/1430613346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61f0069f40 0x7f61f006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.463+0000 7f61f7000700 1 --2- 172.21.15.102:0/1430613346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f006a860 0x7f61f010e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.463+0000 7f61f7000700 1 -- 172.21.15.102:0/1430613346 >> 172.21.15.102:0/1430613346 conn(0x7f61f00780e0 msgr2=0x7f61f00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:04.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.463+0000 7f61f7000700 1 -- 172.21.15.102:0/1430613346 shutdown_connections 2024-04-25T01:53:04.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.463+0000 7f61f7000700 1 -- 172.21.15.102:0/1430613346 wait complete. 2024-04-25T01:53:04.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.464+0000 7f61f7000700 1 Processor -- start 2024-04-25T01:53:04.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.464+0000 7f61f7000700 1 -- start start 2024-04-25T01:53:04.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.464+0000 7f61f7000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f0069f40 0x7f61f01a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:04.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.464+0000 7f61f7000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61f006a860 0x7f61f01a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.464+0000 7f61f7000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f010ece0 0x7f61f019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61f7000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0114780 con 0x7f61f010ece0 2024-04-25T01:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61f7000700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0114600 con 0x7f61f0069f40 2024-04-25T01:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61f7000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0114900 con 0x7f61f006a860 2024-04-25T01:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61f5000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f010ece0 0x7f61f019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61e7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61f006a860 0x7f61f01a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61e7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61f006a860 0x7f61f01a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:59064/0 (socket says 172.21.15.102:59064) 2024-04-25T01:53:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61e7e00700 1 -- 172.21.15.102:0/1566876894 learned_addr learned my addr 172.21.15.102:0/1566876894 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61f5000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f010ece0 0x7f61f019aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52802/0 (socket says 172.21.15.102:52802) 2024-04-25T01:53:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61efe00700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f0069f40 0x7f61f01a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61e7e00700 1 -- 172.21.15.102:0/1566876894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f0069f40 msgr2=0x7f61f01a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61e7e00700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f0069f40 0x7f61f01a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61e7e00700 1 -- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f010ece0 msgr2=0x7f61f019aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61e7e00700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f010ece0 0x7f61f019aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61e7e00700 1 -- 172.21.15.102:0/1566876894 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f019b180 con 0x7f61f006a860 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61efe00700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f0069f40 0x7f61f01a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.465+0000 7f61f5000700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f010ece0 0x7f61f019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.466+0000 7f61e7e00700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61f006a860 0x7f61f01a0d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f61d800ec10 tx=0x7f61d800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.466+0000 7f61ed600700 1 -- 172.21.15.102:0/1566876894 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d800cc90 con 0x7f61f006a860 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.466+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f019b470 con 0x7f61f006a860 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.466+0000 7f61ed600700 1 -- 172.21.15.102:0/1566876894 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61d800ce30 con 0x7f61f006a860 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.466+0000 7f61ed600700 1 -- 172.21.15.102:0/1566876894 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d8018630 con 0x7f61f006a860 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.466+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f01a76e0 con 0x7f61f006a860 2024-04-25T01:53:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.467+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f0068f20 con 0x7f61f006a860 2024-04-25T01:53:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.469+0000 7f61ed600700 1 -- 172.21.15.102:0/1566876894 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f61d8003680 con 0x7f61f006a860 2024-04-25T01:53:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.470+0000 7f61ed600700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61d40a2cb0 0x7f61d40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.471+0000 7f61ed600700 1 -- 172.21.15.102:0/1566876894 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f61d80cf360 con 0x7f61f006a860 2024-04-25T01:53:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.471+0000 7f61efe00700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61d40a2cb0 0x7f61d40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.471+0000 7f61efe00700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61d40a2cb0 0x7f61d40a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f61f0078770 tx=0x7f61e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:04.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.475+0000 7f61ed600700 1 -- 172.21.15.102:0/1566876894 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f61d809c210 con 0x7f61f006a860 2024-04-25T01:53:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.701+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61f002db30 con 0x7f61f006a860 2024-04-25T01:53:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.703+0000 7f61ed600700 1 -- 172.21.15.102:0/1566876894 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22253 (secure 0 0 0) 0x7f61d809bb10 con 0x7f61f006a860 2024-04-25T01:53:04.704 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:53:04.705 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":87,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:52:45.058627+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:53:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61d40a2cb0 msgr2=0x7f61d40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61d40a2cb0 0x7f61d40a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f61f0078770 tx=0x7f61e0009260 comp rx=0 tx=0).stop 2024-04-25T01:53:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61f006a860 msgr2=0x7f61f01a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61f006a860 0x7f61f01a0d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f61d800ec10 tx=0x7f61d800ef20 comp rx=0 tx=0).stop 2024-04-25T01:53:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 shutdown_connections 2024-04-25T01:53:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f61d40a2cb0 0x7f61d40a5170 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61f0069f40 0x7f61f01a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61f010ece0 0x7f61f019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 --2- 172.21.15.102:0/1566876894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61f006a860 0x7f61f01a0d50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 >> 172.21.15.102:0/1566876894 conn(0x7f61f00780e0 msgr2=0x7f61f0110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 shutdown_connections 2024-04-25T01:53:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:04.707+0000 7f61f7000700 1 -- 172.21.15.102:0/1566876894 wait complete. 2024-04-25T01:53:05.086 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:04 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1566876894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:05.137 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:04 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1566876894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:05.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:04 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1566876894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:05.635 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T01:53:05.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T01:53:05.922 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:05 smithi102 ceph-mon[27796]: pgmap v328: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-25T01:53:06.098 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:06.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:05 smithi110 ceph-mon[32848]: pgmap v328: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-25T01:53:06.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:05 smithi193 ceph-mon[32981]: pgmap v328: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-04-25T01:53:08.062 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:07 smithi102 ceph-mon[27796]: pgmap v329: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-25T01:53:08.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:07 smithi110 ceph-mon[32848]: pgmap v329: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-25T01:53:08.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:07 smithi193 ceph-mon[32981]: pgmap v329: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-04-25T01:53:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.346+0000 7f5e72a00700 1 -- 172.21.15.102:0/3682984109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c10a4b0 msgr2=0x7f5e6c10a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.346+0000 7f5e72a00700 1 --2- 172.21.15.102:0/3682984109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c10a4b0 0x7f5e6c10a890 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c009990 tx=0x7f5e5c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.346+0000 7f5e72a00700 1 -- 172.21.15.102:0/3682984109 shutdown_connections 2024-04-25T01:53:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.346+0000 7f5e72a00700 1 --2- 172.21.15.102:0/3682984109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c10a4b0 0x7f5e6c10a890 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.346+0000 7f5e72a00700 1 --2- 172.21.15.102:0/3682984109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e6c104de0 0x7f5e6c1112f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.346+0000 7f5e72a00700 1 --2- 172.21.15.102:0/3682984109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e6c104420 0x7f5e6c1048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.346+0000 7f5e72a00700 1 -- 172.21.15.102:0/3682984109 >> 172.21.15.102:0/3682984109 conn(0x7f5e6c0fff10 msgr2=0x7f5e6c102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.346+0000 7f5e72a00700 1 -- 172.21.15.102:0/3682984109 shutdown_connections 2024-04-25T01:53:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.347+0000 7f5e72a00700 1 -- 172.21.15.102:0/3682984109 wait complete. 2024-04-25T01:53:08.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.348+0000 7f5e72a00700 1 Processor -- start 2024-04-25T01:53:08.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.348+0000 7f5e72a00700 1 -- start start 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e72a00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c104420 0x7f5e6c19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e72a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e6c104de0 0x7f5e6c19b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e72a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e6c10a4b0 0x7f5e6c19f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e72a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c114780 con 0x7f5e6c104de0 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e72a00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c114600 con 0x7f5e6c104420 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e72a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c114900 con 0x7f5e6c10a4b0 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e6be00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c104420 0x7f5e6c19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e6be00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c104420 0x7f5e6c19ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36110/0 (socket says 172.21.15.102:36110) 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e6be00700 1 -- 172.21.15.102:0/2738663991 learned_addr learned my addr 172.21.15.102:0/2738663991 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e6b400700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e6c104de0 0x7f5e6c19b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:08.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e70a00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e6c10a4b0 0x7f5e6c19f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:08.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e6be00700 1 -- 172.21.15.102:0/2738663991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e6c10a4b0 msgr2=0x7f5e6c19f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:08.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e6be00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e6c10a4b0 0x7f5e6c19f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e6be00700 1 -- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e6c104de0 msgr2=0x7f5e6c19b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e6be00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e6c104de0 0x7f5e6c19b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.349+0000 7f5e6be00700 1 -- 172.21.15.102:0/2738663991 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6c19fc50 con 0x7f5e6c104420 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.350+0000 7f5e6be00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c104420 0x7f5e6c19ac50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c000c00 tx=0x7f5e5c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.350+0000 7f5e68c00700 1 -- 172.21.15.102:0/2738663991 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c027070 con 0x7f5e6c104420 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.350+0000 7f5e68c00700 1 -- 172.21.15.102:0/2738663991 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e5c0056d0 con 0x7f5e6c104420 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.350+0000 7f5e68c00700 1 -- 172.21.15.102:0/2738663991 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c022e10 con 0x7f5e6c104420 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.350+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e6c19fee0 con 0x7f5e6c104420 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.350+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e6c1a0370 con 0x7f5e6c104420 2024-04-25T01:53:08.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.351+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e6c10c9b0 con 0x7f5e6c104420 2024-04-25T01:53:08.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.354+0000 7f5e68c00700 1 -- 172.21.15.102:0/2738663991 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5e5c009d10 con 0x7f5e6c104420 2024-04-25T01:53:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.355+0000 7f5e68c00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e500a2cb0 0x7f5e500a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.355+0000 7f5e68c00700 1 -- 172.21.15.102:0/2738663991 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f5e5c0d02e0 con 0x7f5e6c104420 2024-04-25T01:53:08.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.355+0000 7f5e6b400700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e500a2cb0 0x7f5e500a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:08.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.356+0000 7f5e6b400700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e500a2cb0 0x7f5e500a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c109880 tx=0x7f5e58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.357+0000 7f5e68c00700 1 -- 172.21.15.102:0/2738663991 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5e5c013070 con 0x7f5e6c104420 2024-04-25T01:53:08.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.581+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e6c19bfb0 con 0x7f5e6c104420 2024-04-25T01:53:08.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.583+0000 7f5e68c00700 1 -- 172.21.15.102:0/2738663991 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22253 (secure 0 0 0) 0x7f5e5c09d220 con 0x7f5e6c104420 2024-04-25T01:53:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:53:08.585 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":87,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:52:45.058627+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:53:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.587+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e500a2cb0 msgr2=0x7f5e500a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.587+0000 7f5e72a00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e500a2cb0 0x7f5e500a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c109880 tx=0x7f5e58009260 comp rx=0 tx=0).stop 2024-04-25T01:53:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.587+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c104420 msgr2=0x7f5e6c19ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.587+0000 7f5e72a00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c104420 0x7f5e6c19ac50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c000c00 tx=0x7f5e5c00bab0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.588+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 shutdown_connections 2024-04-25T01:53:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.588+0000 7f5e72a00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e500a2cb0 0x7f5e500a5170 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.588+0000 7f5e72a00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e6c104420 0x7f5e6c19ac50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.588+0000 7f5e72a00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e6c10a4b0 0x7f5e6c19f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.588+0000 7f5e72a00700 1 --2- 172.21.15.102:0/2738663991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e6c104de0 0x7f5e6c19b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.588+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 >> 172.21.15.102:0/2738663991 conn(0x7f5e6c0fff10 msgr2=0x7f5e6c1004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.588+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 shutdown_connections 2024-04-25T01:53:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:08.588+0000 7f5e72a00700 1 -- 172.21.15.102:0/2738663991 wait complete. 2024-04-25T01:53:08.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:08 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2738663991' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:09.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:08 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2738663991' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:09.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:08 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2738663991' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:53:09.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T01:53:09.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T01:53:09.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T01:53:09.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T01:53:09.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T01:53:09.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T01:53:09.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T01:53:09.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T01:53:09.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T01:53:09.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T01:53:09.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T01:53:09.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T01:53:09.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:09 smithi102 ceph-mon[27796]: pgmap v330: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T01:53:10.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:09 smithi110 ceph-mon[32848]: pgmap v330: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T01:53:10.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:09 smithi193 ceph-mon[32981]: pgmap v330: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-25T01:53:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.582 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.583 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.591 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.592 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.592 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.593 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.596 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.596 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.617 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.634 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:10.745 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:12.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:11 smithi102 ceph-mon[27796]: pgmap v331: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T01:53:12.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:11 smithi110 ceph-mon[32848]: pgmap v331: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T01:53:12.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:11 smithi193 ceph-mon[32981]: pgmap v331: 97 pgs: 97 active+clean; 635 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T01:53:13.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:13 smithi102 ceph-mon[27796]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-25T01:53:14.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:13 smithi110 ceph-mon[32848]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-25T01:53:14.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:13 smithi193 ceph-mon[32981]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-25T01:53:15.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:15 smithi102 ceph-mon[27796]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:53:16.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:15 smithi110 ceph-mon[32848]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:53:16.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:15 smithi193 ceph-mon[32981]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:53:17.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:17 smithi102 ceph-mon[27796]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:53:18.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:17 smithi110 ceph-mon[32848]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:53:18.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:17 smithi193 ceph-mon[32981]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:53:19.172 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:18 smithi102 ceph-mon[27796]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:53:19.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:18 smithi110 ceph-mon[32848]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:53:19.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:18 smithi193 ceph-mon[32981]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:53:19.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.431+0000 7f89ca200700 1 -- 172.21.15.102:0/2596212222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89c4074660 msgr2=0x7f89c4074a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.431+0000 7f89ca200700 1 --2- 172.21.15.102:0/2596212222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89c4074660 0x7f89c4074a40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f89b80098f0 tx=0x7f89b8009c00 comp rx=0 tx=0).stop 2024-04-25T01:53:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.431+0000 7f89ca200700 1 -- 172.21.15.102:0/2596212222 shutdown_connections 2024-04-25T01:53:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.431+0000 7f89ca200700 1 --2- 172.21.15.102:0/2596212222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c410e9a0 0x7f89c4110e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.431+0000 7f89ca200700 1 --2- 172.21.15.102:0/2596212222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89c4074660 0x7f89c4074a40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.431+0000 7f89ca200700 1 --2- 172.21.15.102:0/2596212222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c4074f80 0x7f89c410e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.431+0000 7f89ca200700 1 -- 172.21.15.102:0/2596212222 >> 172.21.15.102:0/2596212222 conn(0x7f89c40fdcb0 msgr2=0x7f89c41000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.433+0000 7f89ca200700 1 -- 172.21.15.102:0/2596212222 shutdown_connections 2024-04-25T01:53:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.433+0000 7f89ca200700 1 -- 172.21.15.102:0/2596212222 wait complete. 2024-04-25T01:53:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.434+0000 7f89ca200700 1 Processor -- start 2024-04-25T01:53:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.434+0000 7f89ca200700 1 -- start start 2024-04-25T01:53:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.434+0000 7f89ca200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c4074660 0x7f89c41a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89ca200700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c4074f80 0x7f89c41a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89ca200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89c410e9a0 0x7f89c419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:19.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89ca200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f89c41146c0 con 0x7f89c4074660 2024-04-25T01:53:19.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89ca200700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f89c4114540 con 0x7f89c4074f80 2024-04-25T01:53:19.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89ca200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f89c4114840 con 0x7f89c410e9a0 2024-04-25T01:53:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c3e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c4074f80 0x7f89c41a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c8e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c4074660 0x7f89c41a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c3e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c4074f80 0x7f89c41a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34190/0 (socket says 172.21.15.102:34190) 2024-04-25T01:53:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c3e00700 1 -- 172.21.15.102:0/2007768170 learned_addr learned my addr 172.21.15.102:0/2007768170 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c9800700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89c410e9a0 0x7f89c419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c3e00700 1 -- 172.21.15.102:0/2007768170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89c410e9a0 msgr2=0x7f89c419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c3e00700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89c410e9a0 0x7f89c419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:19.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c3e00700 1 -- 172.21.15.102:0/2007768170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c4074660 msgr2=0x7f89c41a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c3e00700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c4074660 0x7f89c41a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c3e00700 1 -- 172.21.15.102:0/2007768170 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c419b140 con 0x7f89c4074f80 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.435+0000 7f89c8e00700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89c4074660 0x7f89c41a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.436+0000 7f89c3e00700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c4074f80 0x7f89c41a0d10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f89b000d950 tx=0x7f89b000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.436+0000 7f89c1600700 1 -- 172.21.15.102:0/2007768170 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0016070 con 0x7f89c4074f80 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.436+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c419b430 con 0x7f89c4074f80 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.436+0000 7f89c1600700 1 -- 172.21.15.102:0/2007768170 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89b000fd70 con 0x7f89c4074f80 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.436+0000 7f89c1600700 1 -- 172.21.15.102:0/2007768170 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b000e610 con 0x7f89c4074f80 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.436+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c41a76a0 con 0x7f89c4074f80 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.437+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f89c40629c0 con 0x7f89c4074f80 2024-04-25T01:53:19.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.441+0000 7f89c1600700 1 -- 172.21.15.102:0/2007768170 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f89b0019070 con 0x7f89c4074f80 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.442+0000 7f89c1600700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f89ac0a2d30 0x7f89ac0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.443+0000 7f89c1600700 1 -- 172.21.15.102:0/2007768170 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f89b00ceff0 con 0x7f89c4074f80 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.443+0000 7f89c8e00700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f89ac0a2d30 0x7f89ac0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.443+0000 7f89c1600700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] conn(0x7f89ac0ac510 0x7f89ac0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.443+0000 7f89c1600700 1 -- 172.21.15.102:0/2007768170 --> [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89ac0aefe0 con 0x7f89ac0ac510 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.443+0000 7f89c1600700 1 -- 172.21.15.102:0/2007768170 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f89b00cf400 con 0x7f89c4074f80 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.443+0000 7f89c9800700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] conn(0x7f89ac0ac510 0x7f89ac0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.443+0000 7f89c8e00700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f89ac0a2d30 0x7f89ac0a51f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009810 tx=0x7f89b800bfb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.444+0000 7f89c9800700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] conn(0x7f89ac0ac510 0x7f89ac0ae930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.445+0000 7f89c1600700 1 -- 172.21.15.102:0/2007768170 <== osd.4 v2:172.21.15.110:6808/4200899007 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f89ac0aefe0 con 0x7f89ac0ac510 2024-04-25T01:53:19.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.463+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 --> [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f89c4062dc0 con 0x7f89ac0ac510 2024-04-25T01:53:19.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.464+0000 7f89c1600700 1 -- 172.21.15.102:0/2007768170 <== osd.4 v2:172.21.15.110:6808/4200899007 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f89c4062dc0 con 0x7f89ac0ac510 2024-04-25T01:53:19.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.464+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 >> [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] conn(0x7f89ac0ac510 msgr2=0x7f89ac0ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:19.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.464+0000 7f89ca200700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] conn(0x7f89ac0ac510 0x7f89ac0ae930 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:19.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.464+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f89ac0a2d30 msgr2=0x7f89ac0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:19.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.464+0000 7f89ca200700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f89ac0a2d30 0x7f89ac0a51f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009810 tx=0x7f89b800bfb0 comp rx=0 tx=0).stop 2024-04-25T01:53:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.464+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c4074f80 msgr2=0x7f89c41a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.464+0000 7f89ca200700 1 --2- 172.21.15.102:0/2007768170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c4074f80 0x7f89c41a0d10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f89b000d950 tx=0x7f89b000dc60 comp rx=0 tx=0).stop 2024-04-25T01:53:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.465+0000 7f89c9800700 1 -- 172.21.15.102:0/2007768170 reap_dead start 2024-04-25T01:53:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.465+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 shutdown_connections 2024-04-25T01:53:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.465+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 >> 172.21.15.102:0/2007768170 conn(0x7f89c40fdcb0 msgr2=0x7f89c40fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.465+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 shutdown_connections 2024-04-25T01:53:19.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.465+0000 7f89ca200700 1 -- 172.21.15.102:0/2007768170 wait complete. 2024-04-25T01:53:20.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.566+0000 7ff245400700 1 -- 172.21.15.102:0/3260186207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff240104f50 msgr2=0x7ff240108fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.566+0000 7ff245400700 1 --2- 172.21.15.102:0/3260186207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff240104f50 0x7ff240108fa0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff22c0099c0 tx=0x7ff22c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.566+0000 7ff245400700 1 -- 172.21.15.102:0/3260186207 shutdown_connections 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.566+0000 7ff245400700 1 --2- 172.21.15.102:0/3260186207 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2401045a0 0x7ff240104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.566+0000 7ff245400700 1 --2- 172.21.15.102:0/3260186207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff240109680 0x7ff240111250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.566+0000 7ff245400700 1 --2- 172.21.15.102:0/3260186207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff240104f50 0x7ff240108fa0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.566+0000 7ff245400700 1 -- 172.21.15.102:0/3260186207 >> 172.21.15.102:0/3260186207 conn(0x7ff2400ffe10 msgr2=0x7ff240102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.566+0000 7ff245400700 1 -- 172.21.15.102:0/3260186207 shutdown_connections 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.566+0000 7ff245400700 1 -- 172.21.15.102:0/3260186207 wait complete. 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.567+0000 7ff245400700 1 Processor -- start 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.567+0000 7ff245400700 1 -- start start 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff245400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2401045a0 0x7ff2401a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:20.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff245400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff240104f50 0x7ff2401a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff245400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff240109680 0x7ff24019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff245400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff240114640 con 0x7ff2401045a0 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff245400700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401144c0 con 0x7ff240109680 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff245400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401147c0 con 0x7ff240104f50 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff23fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2401045a0 0x7ff2401a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff23fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2401045a0 0x7ff2401a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54842/0 (socket says 172.21.15.102:54842) 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff23fe00700 1 -- 172.21.15.102:0/2626969086 learned_addr learned my addr 172.21.15.102:0/2626969086 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff244a00700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff240109680 0x7ff24019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff23f400700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff240104f50 0x7ff2401a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff244a00700 1 -- 172.21.15.102:0/2626969086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff240104f50 msgr2=0x7ff2401a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff244a00700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff240104f50 0x7ff2401a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff244a00700 1 -- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2401045a0 msgr2=0x7ff2401a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff244a00700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2401045a0 0x7ff2401a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff244a00700 1 -- 172.21.15.102:0/2626969086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24019b0a0 con 0x7ff240109680 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.568+0000 7ff23fe00700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2401045a0 0x7ff2401a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.569+0000 7ff244a00700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff240109680 0x7ff24019a940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff23800efb0 tx=0x7ff23800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.569+0000 7ff23cc00700 1 -- 172.21.15.102:0/2626969086 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238003a00 con 0x7ff240109680 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.569+0000 7ff23cc00700 1 -- 172.21.15.102:0/2626969086 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff238003ba0 con 0x7ff240109680 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.569+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24019b390 con 0x7ff240109680 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.569+0000 7ff23cc00700 1 -- 172.21.15.102:0/2626969086 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238004850 con 0x7ff240109680 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.569+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2401a7650 con 0x7ff240109680 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.570+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff2400629c0 con 0x7ff240109680 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.572+0000 7ff23cc00700 1 -- 172.21.15.102:0/2626969086 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff238012050 con 0x7ff240109680 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.573+0000 7ff23cc00700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff2240a2cb0 0x7ff2240a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.574+0000 7ff23cc00700 1 -- 172.21.15.102:0/2626969086 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff2380cee80 con 0x7ff240109680 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.574+0000 7ff23fe00700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff2240a2cb0 0x7ff2240a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:20.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.574+0000 7ff23cc00700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] conn(0x7ff2240ac490 0x7ff2240ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:20.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.574+0000 7ff23cc00700 1 -- 172.21.15.102:0/2626969086 --> [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2240aef60 con 0x7ff2240ac490 2024-04-25T01:53:20.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.574+0000 7ff23cc00700 1 -- 172.21.15.102:0/2626969086 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7ff2380cf230 con 0x7ff240109680 2024-04-25T01:53:20.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.574+0000 7ff23f400700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] conn(0x7ff2240ac490 0x7ff2240ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:20.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.574+0000 7ff23fe00700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff2240a2cb0 0x7ff2240a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff24019ba80 tx=0x7ff230009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:20.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.575+0000 7ff23f400700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] conn(0x7ff2240ac490 0x7ff2240ae8b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:20.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.576+0000 7ff23cc00700 1 -- 172.21.15.102:0/2626969086 <== osd.3 v2:172.21.15.102:6810/3993205287 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2240aef60 con 0x7ff2240ac490 2024-04-25T01:53:20.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.594+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 --> [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff240062e40 con 0x7ff2240ac490 2024-04-25T01:53:20.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.594+0000 7ff23cc00700 1 -- 172.21.15.102:0/2626969086 <== osd.3 v2:172.21.15.102:6810/3993205287 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff240062e40 con 0x7ff2240ac490 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] conn(0x7ff2240ac490 msgr2=0x7ff2240ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] conn(0x7ff2240ac490 0x7ff2240ae8b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff2240a2cb0 msgr2=0x7ff2240a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff2240a2cb0 0x7ff2240a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff24019ba80 tx=0x7ff230009260 comp rx=0 tx=0).stop 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff240109680 msgr2=0x7ff24019a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 --2- 172.21.15.102:0/2626969086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff240109680 0x7ff24019a940 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff23800efb0 tx=0x7ff23800c3a0 comp rx=0 tx=0).stop 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff244a00700 1 -- 172.21.15.102:0/2626969086 reap_dead start 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 shutdown_connections 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 >> 172.21.15.102:0/2626969086 conn(0x7ff2400ffe10 msgr2=0x7ff2401020f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 shutdown_connections 2024-04-25T01:53:20.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:19.595+0000 7ff245400700 1 -- 172.21.15.102:0/2626969086 wait complete. 2024-04-25T01:53:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.380+0000 7fd433e00700 1 -- 172.21.15.102:0/3119444548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4341018c0 msgr2=0x7fd43410e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.380+0000 7fd433e00700 1 --2- 172.21.15.102:0/3119444548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4341018c0 0x7fd43410e580 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd420009990 tx=0x7fd420009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.380+0000 7fd433e00700 1 -- 172.21.15.102:0/3119444548 shutdown_connections 2024-04-25T01:53:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.380+0000 7fd433e00700 1 --2- 172.21.15.102:0/3119444548 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd43410eac0 0x7fd434110eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.380+0000 7fd433e00700 1 --2- 172.21.15.102:0/3119444548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd434100f10 0x7fd4341012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.380+0000 7fd433e00700 1 --2- 172.21.15.102:0/3119444548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4341018c0 0x7fd43410e580 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.380+0000 7fd433e00700 1 -- 172.21.15.102:0/3119444548 >> 172.21.15.102:0/3119444548 conn(0x7fd4340fcb50 msgr2=0x7fd4340fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:21.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.380+0000 7fd433e00700 1 -- 172.21.15.102:0/3119444548 shutdown_connections 2024-04-25T01:53:21.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.380+0000 7fd433e00700 1 -- 172.21.15.102:0/3119444548 wait complete. 2024-04-25T01:53:21.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.381+0000 7fd433e00700 1 Processor -- start 2024-04-25T01:53:21.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.382+0000 7fd433e00700 1 -- start start 2024-04-25T01:53:21.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.382+0000 7fd433e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd434100f10 0x7fd4341a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.382+0000 7fd433e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4341018c0 0x7fd4341a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.382+0000 7fd433e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43410eac0 0x7fd43419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.382+0000 7fd433e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341146e0 con 0x7fd434100f10 2024-04-25T01:53:21.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.382+0000 7fd433e00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd434114560 con 0x7fd4341018c0 2024-04-25T01:53:21.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.382+0000 7fd433e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd434114860 con 0x7fd43410eac0 2024-04-25T01:53:21.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.382+0000 7fd432a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd434100f10 0x7fd4341a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd434100f10 0x7fd4341a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54856/0 (socket says 172.21.15.102:54856) 2024-04-25T01:53:21.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432a00700 1 -- 172.21.15.102:0/2658527416 learned_addr learned my addr 172.21.15.102:0/2658527416 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:21.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432000700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4341018c0 0x7fd4341a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd433400700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43410eac0 0x7fd43419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432a00700 1 -- 172.21.15.102:0/2658527416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43410eac0 msgr2=0x7fd43419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432a00700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43410eac0 0x7fd43419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432a00700 1 -- 172.21.15.102:0/2658527416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4341018c0 msgr2=0x7fd4341a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432a00700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4341018c0 0x7fd4341a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432a00700 1 -- 172.21.15.102:0/2658527416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43419b180 con 0x7fd434100f10 2024-04-25T01:53:21.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432000700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4341018c0 0x7fd4341a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:21.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.383+0000 7fd432a00700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd434100f10 0x7fd4341a0810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd42400e9a0 tx=0x7fd42400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:21.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.384+0000 7fd42b400700 1 -- 172.21.15.102:0/2658527416 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42400ca10 con 0x7fd434100f10 2024-04-25T01:53:21.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.384+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd43419b470 con 0x7fd434100f10 2024-04-25T01:53:21.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.384+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4341a76e0 con 0x7fd434100f10 2024-04-25T01:53:21.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.384+0000 7fd42b400700 1 -- 172.21.15.102:0/2658527416 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd42400cbb0 con 0x7fd434100f10 2024-04-25T01:53:21.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.384+0000 7fd42b400700 1 -- 172.21.15.102:0/2658527416 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424026060 con 0x7fd434100f10 2024-04-25T01:53:21.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.385+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd414001030 con 0x7fd434100f10 2024-04-25T01:53:21.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.387+0000 7fd42b400700 1 -- 172.21.15.102:0/2658527416 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd424010430 con 0x7fd434100f10 2024-04-25T01:53:21.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.388+0000 7fd42b400700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd4180a2d30 0x7fd4180a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.388+0000 7fd42b400700 1 -- 172.21.15.102:0/2658527416 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fd4240cf230 con 0x7fd434100f10 2024-04-25T01:53:21.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.389+0000 7fd432000700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd4180a2d30 0x7fd4180a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.389+0000 7fd42b400700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] conn(0x7fd4180ac510 0x7fd4180ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.389+0000 7fd42b400700 1 -- 172.21.15.102:0/2658527416 --> [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd4180aefe0 con 0x7fd4180ac510 2024-04-25T01:53:21.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.389+0000 7fd42b400700 1 -- 172.21.15.102:0/2658527416 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fd4240cf620 con 0x7fd434100f10 2024-04-25T01:53:21.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.389+0000 7fd433400700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] conn(0x7fd4180ac510 0x7fd4180ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.389+0000 7fd432000700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd4180a2d30 0x7fd4180a51f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd4340ff0d0 tx=0x7fd42000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:21.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.390+0000 7fd433400700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] conn(0x7fd4180ac510 0x7fd4180ae930 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:21.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.390+0000 7fd42b400700 1 -- 172.21.15.102:0/2658527416 <== osd.11 v2:172.21.15.193:6824/40469709 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd4180aefe0 con 0x7fd4180ac510 2024-04-25T01:53:21.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.408+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 --> [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd414002e60 con 0x7fd4180ac510 2024-04-25T01:53:21.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd42b400700 1 -- 172.21.15.102:0/2658527416 <== osd.11 v2:172.21.15.193:6824/40469709 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd414002e60 con 0x7fd4180ac510 2024-04-25T01:53:21.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 >> [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] conn(0x7fd4180ac510 msgr2=0x7fd4180ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd433e00700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] conn(0x7fd4180ac510 0x7fd4180ae930 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd4180a2d30 msgr2=0x7fd4180a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd433e00700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd4180a2d30 0x7fd4180a51f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd4340ff0d0 tx=0x7fd42000b260 comp rx=0 tx=0).stop 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd434100f10 msgr2=0x7fd4341a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd433e00700 1 --2- 172.21.15.102:0/2658527416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd434100f10 0x7fd4341a0810 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd42400e9a0 tx=0x7fd42400ecb0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd433400700 1 -- 172.21.15.102:0/2658527416 reap_dead start 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 shutdown_connections 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.409+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 >> 172.21.15.102:0/2658527416 conn(0x7fd4340fcb50 msgr2=0x7fd43410fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.410+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 shutdown_connections 2024-04-25T01:53:21.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.410+0000 7fd433e00700 1 -- 172.21.15.102:0/2658527416 wait complete. 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.874+0000 7fab9a400700 1 -- 172.21.15.102:0/2329666237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9410ec30 msgr2=0x7fab94111020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.874+0000 7fab9a400700 1 --2- 172.21.15.102:0/2329666237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9410ec30 0x7fab94111020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fab8c00b0a0 tx=0x7fab8c00b3b0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.874+0000 7fab9a400700 1 -- 172.21.15.102:0/2329666237 shutdown_connections 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.874+0000 7fab9a400700 1 --2- 172.21.15.102:0/2329666237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab94074460 0x7fab94074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.874+0000 7fab9a400700 1 --2- 172.21.15.102:0/2329666237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab9410ec30 0x7fab94111020 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.874+0000 7fab9a400700 1 --2- 172.21.15.102:0/2329666237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab94074d80 0x7fab9410e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.874+0000 7fab9a400700 1 -- 172.21.15.102:0/2329666237 >> 172.21.15.102:0/2329666237 conn(0x7fab94079390 msgr2=0x7fab940797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.874+0000 7fab9a400700 1 -- 172.21.15.102:0/2329666237 shutdown_connections 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.875+0000 7fab9a400700 1 -- 172.21.15.102:0/2329666237 wait complete. 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.875+0000 7fab9a400700 1 Processor -- start 2024-04-25T01:53:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab9a400700 1 -- start start 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab9a400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab94074460 0x7fab941a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab9a400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab94074d80 0x7fab941a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab9a400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9410ec30 0x7fab9419a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab9a400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fab94114750 con 0x7fab94074460 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab9a400700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fab941145d0 con 0x7fab9410ec30 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab9a400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab941148d0 con 0x7fab94074d80 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab93400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab94074460 0x7fab941a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab93400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab94074460 0x7fab941a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54878/0 (socket says 172.21.15.102:54878) 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab93400700 1 -- 172.21.15.102:0/2751347043 learned_addr learned my addr 172.21.15.102:0/2751347043 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab8be00700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab94074d80 0x7fab941a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.876+0000 7fab93e00700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9410ec30 0x7fab9419a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab93400700 1 -- 172.21.15.102:0/2751347043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab94074d80 msgr2=0x7fab941a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab93400700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab94074d80 0x7fab941a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab93400700 1 -- 172.21.15.102:0/2751347043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9410ec30 msgr2=0x7fab9419a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab93400700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab9410ec30 0x7fab9419a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab93400700 1 -- 172.21.15.102:0/2751347043 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9419b150 con 0x7fab94074460 2024-04-25T01:53:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab93400700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab94074460 0x7fab941a07e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab8400e9a0 tx=0x7fab8400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab90c00700 1 -- 172.21.15.102:0/2751347043 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8400ca10 con 0x7fab94074460 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab90c00700 1 -- 172.21.15.102:0/2751347043 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab8400cbb0 con 0x7fab94074460 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9419b440 con 0x7fab94074460 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.877+0000 7fab90c00700 1 -- 172.21.15.102:0/2751347043 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab84018680 con 0x7fab94074460 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.878+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab941a76b0 con 0x7fab94074460 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.878+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fab940ffe00 con 0x7fab94074460 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.881+0000 7fab90c00700 1 -- 172.21.15.102:0/2751347043 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fab84010430 con 0x7fab94074460 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.882+0000 7fab90c00700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab780a2d30 0x7fab780a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.882+0000 7fab90c00700 1 -- 172.21.15.102:0/2751347043 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fab840cfaf0 con 0x7fab94074460 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.882+0000 7fab8be00700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab780a2d30 0x7fab780a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.882+0000 7fab90c00700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] conn(0x7fab780ac510 0x7fab780ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.882+0000 7fab90c00700 1 -- 172.21.15.102:0/2751347043 --> [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab8401bec0 con 0x7fab780ac510 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.882+0000 7fab90c00700 1 -- 172.21.15.102:0/2751347043 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fab840cfea0 con 0x7fab94074460 2024-04-25T01:53:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.883+0000 7fab93e00700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] conn(0x7fab780ac510 0x7fab780ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.883+0000 7fab8be00700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab780a2d30 0x7fab780a51f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fab941108c0 tx=0x7fab7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.883+0000 7fab93e00700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] conn(0x7fab780ac510 0x7fab780ae930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.884+0000 7fab90c00700 1 -- 172.21.15.102:0/2751347043 <== osd.1 v2:172.21.15.110:6800/2162810099 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fab8401bec0 con 0x7fab780ac510 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.902+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 --> [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fab94075b50 con 0x7fab780ac510 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.903+0000 7fab90c00700 1 -- 172.21.15.102:0/2751347043 <== osd.1 v2:172.21.15.110:6800/2162810099 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fab94075b50 con 0x7fab780ac510 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.903+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 >> [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] conn(0x7fab780ac510 msgr2=0x7fab780ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.903+0000 7fab9a400700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] conn(0x7fab780ac510 0x7fab780ae930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.903+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab780a2d30 msgr2=0x7fab780a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.903+0000 7fab9a400700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab780a2d30 0x7fab780a51f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fab941108c0 tx=0x7fab7c009260 comp rx=0 tx=0).stop 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.903+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab94074460 msgr2=0x7fab941a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.903+0000 7fab9a400700 1 --2- 172.21.15.102:0/2751347043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab94074460 0x7fab941a07e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab8400e9a0 tx=0x7fab8400ecb0 comp rx=0 tx=0).stop 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.903+0000 7fab93e00700 1 -- 172.21.15.102:0/2751347043 reap_dead start 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.904+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 shutdown_connections 2024-04-25T01:53:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.904+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 >> 172.21.15.102:0/2751347043 conn(0x7fab94079390 msgr2=0x7fab940ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:21.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.904+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 shutdown_connections 2024-04-25T01:53:21.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:21.904+0000 7fab9a400700 1 -- 172.21.15.102:0/2751347043 wait complete. 2024-04-25T01:53:21.984 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:21 smithi102 ceph-mon[27796]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:53:22.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:21 smithi110 ceph-mon[32848]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:53:22.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:21 smithi193 ceph-mon[32981]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:53:23.372 INFO:teuthology.orchestra.run.smithi102.stdout:55834574926 2024-04-25T01:53:23.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T01:53:23.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:23 smithi102 ceph-mon[27796]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-25T01:53:24.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:23 smithi110 ceph-mon[32848]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-25T01:53:24.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:23 smithi193 ceph-mon[32981]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-25T01:53:25.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.538+0000 7fdb4ec00700 1 -- 172.21.15.102:0/3661692650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb48074460 msgr2=0x7fdb48074840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.538+0000 7fdb4ec00700 1 --2- 172.21.15.102:0/3661692650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb48074460 0x7fdb48074840 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdb38009990 tx=0x7fdb38009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.539+0000 7fdb4ec00700 1 -- 172.21.15.102:0/3661692650 shutdown_connections 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.539+0000 7fdb4ec00700 1 --2- 172.21.15.102:0/3661692650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb4810ec30 0x7fdb48111020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.539+0000 7fdb4ec00700 1 --2- 172.21.15.102:0/3661692650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb48074460 0x7fdb48074840 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.539+0000 7fdb4ec00700 1 --2- 172.21.15.102:0/3661692650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb48074d80 0x7fdb4810e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.539+0000 7fdb4ec00700 1 -- 172.21.15.102:0/3661692650 >> 172.21.15.102:0/3661692650 conn(0x7fdb48079390 msgr2=0x7fdb480797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.539+0000 7fdb4ec00700 1 -- 172.21.15.102:0/3661692650 shutdown_connections 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.539+0000 7fdb4ec00700 1 -- 172.21.15.102:0/3661692650 wait complete. 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.540+0000 7fdb4ec00700 1 Processor -- start 2024-04-25T01:53:25.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.540+0000 7fdb4ec00700 1 -- start start 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.540+0000 7fdb4ec00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb48074460 0x7fdb481a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.540+0000 7fdb4ec00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb48074d80 0x7fdb481a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.540+0000 7fdb4ec00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb4810ec30 0x7fdb4819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.540+0000 7fdb4ec00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdb48114780 con 0x7fdb4810ec30 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.540+0000 7fdb4ec00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdb48114600 con 0x7fdb48074460 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.540+0000 7fdb4ec00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdb48114900 con 0x7fdb48074d80 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb48074460 0x7fdb481a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb48074460 0x7fdb481a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51750/0 (socket says 172.21.15.102:51750) 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb4cc00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb4810ec30 0x7fdb4819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47e00700 1 -- 172.21.15.102:0/1425015162 learned_addr learned my addr 172.21.15.102:0/1425015162 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47400700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb48074d80 0x7fdb481a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47e00700 1 -- 172.21.15.102:0/1425015162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb48074d80 msgr2=0x7fdb481a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47e00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb48074d80 0x7fdb481a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47e00700 1 -- 172.21.15.102:0/1425015162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb4810ec30 msgr2=0x7fdb4819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47e00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb4810ec30 0x7fdb4819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47e00700 1 -- 172.21.15.102:0/1425015162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4819b1a0 con 0x7fdb48074460 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.541+0000 7fdb47e00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb48074460 0x7fdb481a0830 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdb38000c00 tx=0x7fdb3800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.542+0000 7fdb44c00700 1 -- 172.21.15.102:0/1425015162 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb38027070 con 0x7fdb48074460 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.542+0000 7fdb44c00700 1 -- 172.21.15.102:0/1425015162 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb38003d70 con 0x7fdb48074460 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.542+0000 7fdb44c00700 1 -- 172.21.15.102:0/1425015162 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3800f460 con 0x7fdb48074460 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.542+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4819b430 con 0x7fdb48074460 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.542+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb481a7700 con 0x7fdb48074460 2024-04-25T01:53:25.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.543+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdb480ffea0 con 0x7fdb48074460 2024-04-25T01:53:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.545+0000 7fdb44c00700 1 -- 172.21.15.102:0/1425015162 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fdb3802a070 con 0x7fdb48074460 2024-04-25T01:53:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.546+0000 7fdb44c00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdb2c0a2cb0 0x7fdb2c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.546+0000 7fdb44c00700 1 -- 172.21.15.102:0/1425015162 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7fdb380d0a80 con 0x7fdb48074460 2024-04-25T01:53:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.547+0000 7fdb47400700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdb2c0a2cb0 0x7fdb2c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.547+0000 7fdb44c00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] conn(0x7fdb2c0ac490 0x7fdb2c0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.547+0000 7fdb44c00700 1 -- 172.21.15.102:0/1425015162 --> [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb2c0aef60 con 0x7fdb2c0ac490 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.547+0000 7fdb44c00700 1 -- 172.21.15.102:0/1425015162 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fdb380d0e30 con 0x7fdb48074460 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.547+0000 7fdb4cc00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] conn(0x7fdb2c0ac490 0x7fdb2c0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.547+0000 7fdb47400700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdb2c0a2cb0 0x7fdb2c0a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009680 tx=0x7fdb34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.548+0000 7fdb4cc00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] conn(0x7fdb2c0ac490 0x7fdb2c0ae8b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.549+0000 7fdb44c00700 1 -- 172.21.15.102:0/1425015162 <== osd.10 v2:172.21.15.110:6824/851854931 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb2c0aef60 con 0x7fdb2c0ac490 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.566+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 --> [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdb48075b50 con 0x7fdb2c0ac490 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.567+0000 7fdb44c00700 1 -- 172.21.15.102:0/1425015162 <== osd.10 v2:172.21.15.110:6824/851854931 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdb48075b50 con 0x7fdb2c0ac490 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.568+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 >> [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] conn(0x7fdb2c0ac490 msgr2=0x7fdb2c0ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.568+0000 7fdb4ec00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] conn(0x7fdb2c0ac490 0x7fdb2c0ae8b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.568+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdb2c0a2cb0 msgr2=0x7fdb2c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.568+0000 7fdb4ec00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdb2c0a2cb0 0x7fdb2c0a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009680 tx=0x7fdb34009260 comp rx=0 tx=0).stop 2024-04-25T01:53:25.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.568+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb48074460 msgr2=0x7fdb481a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.568+0000 7fdb4ec00700 1 --2- 172.21.15.102:0/1425015162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb48074460 0x7fdb481a0830 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdb38000c00 tx=0x7fdb3800bab0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.568+0000 7fdb4cc00700 1 -- 172.21.15.102:0/1425015162 reap_dead start 2024-04-25T01:53:25.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.568+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 shutdown_connections 2024-04-25T01:53:25.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.568+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 >> 172.21.15.102:0/1425015162 conn(0x7fdb48079390 msgr2=0x7fdb480ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:25.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.569+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 shutdown_connections 2024-04-25T01:53:25.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.569+0000 7fdb4ec00700 1 -- 172.21.15.102:0/1425015162 wait complete. 2024-04-25T01:53:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.635+0000 7f50af200700 1 -- 172.21.15.102:0/88341872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50a8102d70 msgr2=0x7f50a8106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.635+0000 7f50af200700 1 --2- 172.21.15.102:0/88341872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50a8102d70 0x7f50a8106dc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5098009990 tx=0x7f5098009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.635+0000 7f50af200700 1 -- 172.21.15.102:0/88341872 shutdown_connections 2024-04-25T01:53:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.635+0000 7f50af200700 1 --2- 172.21.15.102:0/88341872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50a81074a0 0x7f50a810f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.635+0000 7f50af200700 1 --2- 172.21.15.102:0/88341872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50a81023c0 0x7f50a81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.635+0000 7f50af200700 1 --2- 172.21.15.102:0/88341872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50a8102d70 0x7f50a8106dc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.635+0000 7f50af200700 1 -- 172.21.15.102:0/88341872 >> 172.21.15.102:0/88341872 conn(0x7f50a80fdcb0 msgr2=0x7f50a81000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.635+0000 7f50af200700 1 -- 172.21.15.102:0/88341872 shutdown_connections 2024-04-25T01:53:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.635+0000 7f50af200700 1 -- 172.21.15.102:0/88341872 wait complete. 2024-04-25T01:53:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.636+0000 7f50af200700 1 Processor -- start 2024-04-25T01:53:25.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.636+0000 7f50af200700 1 -- start start 2024-04-25T01:53:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.636+0000 7f50af200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50a81023c0 0x7f50a81a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.636+0000 7f50af200700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50a8102d70 0x7f50a81a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.636+0000 7f50af200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50a81074a0 0x7f50a819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.636+0000 7f50af200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f50a8112460 con 0x7f50a81074a0 2024-04-25T01:53:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.636+0000 7f50af200700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f50a81122e0 con 0x7f50a8102d70 2024-04-25T01:53:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.636+0000 7f50af200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50a81125e0 con 0x7f50a81023c0 2024-04-25T01:53:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ad400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50a8102d70 0x7f50a81a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ade00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50a81023c0 0x7f50a81a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ae800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50a81074a0 0x7f50a819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ad400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50a8102d70 0x7f50a81a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51768/0 (socket says 172.21.15.102:51768) 2024-04-25T01:53:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ad400700 1 -- 172.21.15.102:0/4080792447 learned_addr learned my addr 172.21.15.102:0/4080792447 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ad400700 1 -- 172.21.15.102:0/4080792447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50a81023c0 msgr2=0x7f50a81a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ad400700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50a81023c0 0x7f50a81a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ad400700 1 -- 172.21.15.102:0/4080792447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50a81074a0 msgr2=0x7f50a819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ad400700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50a81074a0 0x7f50a819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ad400700 1 -- 172.21.15.102:0/4080792447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a819b040 con 0x7f50a8102d70 2024-04-25T01:53:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50ad400700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50a8102d70 0x7f50a81a0c60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5098000c00 tx=0x7f509800b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.637+0000 7f50a2a00700 1 -- 172.21.15.102:0/4080792447 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098027070 con 0x7f50a8102d70 2024-04-25T01:53:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.638+0000 7f50a2a00700 1 -- 172.21.15.102:0/4080792447 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5098022470 con 0x7f50a8102d70 2024-04-25T01:53:25.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.638+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a819b2d0 con 0x7f50a8102d70 2024-04-25T01:53:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.638+0000 7f50a2a00700 1 -- 172.21.15.102:0/4080792447 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098017650 con 0x7f50a8102d70 2024-04-25T01:53:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.638+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a81a75f0 con 0x7f50a8102d70 2024-04-25T01:53:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.639+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f50a80757c0 con 0x7f50a8102d70 2024-04-25T01:53:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.641+0000 7f50a2a00700 1 -- 172.21.15.102:0/4080792447 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f509802a070 con 0x7f50a8102d70 2024-04-25T01:53:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.642+0000 7f50a2a00700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f50900a2cb0 0x7f50900a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.642+0000 7f50a2a00700 1 -- 172.21.15.102:0/4080792447 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f50980d0260 con 0x7f50a8102d70 2024-04-25T01:53:25.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.642+0000 7f50ade00700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f50900a2cb0 0x7f50900a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.642+0000 7f50a2a00700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] conn(0x7f50900ac490 0x7f50900ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.642+0000 7f50a2a00700 1 -- 172.21.15.102:0/4080792447 --> [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50900aef60 con 0x7f50900ac490 2024-04-25T01:53:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.643+0000 7f50a2a00700 1 -- 172.21.15.102:0/4080792447 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f50980d0610 con 0x7f50a8102d70 2024-04-25T01:53:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.643+0000 7f50ae800700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] conn(0x7f50900ac490 0x7f50900ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:25.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.643+0000 7f50ade00700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f50900a2cb0 0x7f50900a5170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f50a819ba20 tx=0x7f509c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.643+0000 7f50ae800700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] conn(0x7f50900ac490 0x7f50900ae8b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.644+0000 7f50a2a00700 1 -- 172.21.15.102:0/4080792447 <== osd.2 v2:172.21.15.193:6800/602865964 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f50900aef60 con 0x7f50900ac490 2024-04-25T01:53:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.662+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 --> [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f50a81038a0 con 0x7f50900ac490 2024-04-25T01:53:25.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50a2a00700 1 -- 172.21.15.102:0/4080792447 <== osd.2 v2:172.21.15.193:6800/602865964 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f50a81038a0 con 0x7f50900ac490 2024-04-25T01:53:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 >> [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] conn(0x7f50900ac490 msgr2=0x7f50900ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50af200700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] conn(0x7f50900ac490 0x7f50900ae8b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f50900a2cb0 msgr2=0x7f50900a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50af200700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f50900a2cb0 0x7f50900a5170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f50a819ba20 tx=0x7f509c009260 comp rx=0 tx=0).stop 2024-04-25T01:53:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50a8102d70 msgr2=0x7f50a81a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:25.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50af200700 1 --2- 172.21.15.102:0/4080792447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50a8102d70 0x7f50a81a0c60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5098000c00 tx=0x7f509800b610 comp rx=0 tx=0).stop 2024-04-25T01:53:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50ae800700 1 -- 172.21.15.102:0/4080792447 reap_dead start 2024-04-25T01:53:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 shutdown_connections 2024-04-25T01:53:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 >> 172.21.15.102:0/4080792447 conn(0x7f50a80fdcb0 msgr2=0x7f50a80fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.663+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 shutdown_connections 2024-04-25T01:53:25.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:25.664+0000 7f50af200700 1 -- 172.21.15.102:0/4080792447 wait complete. 2024-04-25T01:53:25.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:25 smithi102 ceph-mon[27796]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T01:53:26.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:25 smithi110 ceph-mon[32848]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T01:53:26.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:25 smithi193 ceph-mon[32981]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T01:53:26.859 INFO:teuthology.orchestra.run.smithi102.stdout:150323855421 2024-04-25T01:53:26.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T01:53:28.028 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:27 smithi102 ceph-mon[27796]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:53:28.057 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:27 smithi193 ceph-mon[32981]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:53:28.174 INFO:teuthology.orchestra.run.smithi102.stdout:352187318295 2024-04-25T01:53:28.174 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T01:53:28.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:27 smithi110 ceph-mon[32848]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:53:30.124 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:29 smithi102 ceph-mon[27796]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T01:53:30.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:29 smithi110 ceph-mon[32848]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T01:53:30.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:29 smithi193 ceph-mon[32981]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T01:53:30.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.260+0000 7f1516200700 1 -- 172.21.15.102:0/3588252398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1510074460 msgr2=0x7f1510074840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:30.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.260+0000 7f1516200700 1 --2- 172.21.15.102:0/3588252398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1510074460 0x7f1510074840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1500009990 tx=0x7f1500009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:30.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.260+0000 7f1516200700 1 -- 172.21.15.102:0/3588252398 shutdown_connections 2024-04-25T01:53:30.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.260+0000 7f1516200700 1 --2- 172.21.15.102:0/3588252398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1510113400 0x7f15101157f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:30.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.260+0000 7f1516200700 1 --2- 172.21.15.102:0/3588252398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1510074460 0x7f1510074840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:30.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.260+0000 7f1516200700 1 --2- 172.21.15.102:0/3588252398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1510074d80 0x7f1510112ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:30.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.260+0000 7f1516200700 1 -- 172.21.15.102:0/3588252398 >> 172.21.15.102:0/3588252398 conn(0x7f1510079390 msgr2=0x7f15100797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:30.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.261+0000 7f1516200700 1 -- 172.21.15.102:0/3588252398 shutdown_connections 2024-04-25T01:53:30.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.261+0000 7f1516200700 1 -- 172.21.15.102:0/3588252398 wait complete. 2024-04-25T01:53:30.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.262+0000 7f1516200700 1 Processor -- start 2024-04-25T01:53:30.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.262+0000 7f1516200700 1 -- start start 2024-04-25T01:53:30.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.262+0000 7f1516200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1510074460 0x7f151019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:30.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.262+0000 7f1516200700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1510074d80 0x7f151019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.262+0000 7f1516200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1510113400 0x7f15101a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.262+0000 7f1516200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1510118b30 con 0x7f1510074460 2024-04-25T01:53:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.262+0000 7f1516200700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f15101189b0 con 0x7f1510074d80 2024-04-25T01:53:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.262+0000 7f1516200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1510118cb0 con 0x7f1510113400 2024-04-25T01:53:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.262+0000 7f150f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1510074460 0x7f151019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150f400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1510074460 0x7f151019f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37018/0 (socket says 172.21.15.102:37018) 2024-04-25T01:53:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150f400700 1 -- 172.21.15.102:0/2616076277 learned_addr learned my addr 172.21.15.102:0/2616076277 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150ea00700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1510074d80 0x7f151019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150fe00700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1510113400 0x7f15101a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150f400700 1 -- 172.21.15.102:0/2616076277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1510113400 msgr2=0x7f15101a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150f400700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1510113400 0x7f15101a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:30.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150f400700 1 -- 172.21.15.102:0/2616076277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1510074d80 msgr2=0x7f151019f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:30.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150f400700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1510074d80 0x7f151019f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:30.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150f400700 1 -- 172.21.15.102:0/2616076277 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15101a4030 con 0x7f1510074460 2024-04-25T01:53:30.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.263+0000 7f150f400700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1510074460 0x7f151019f020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1500000c00 tx=0x7f150000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:30.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.264+0000 7f1507e00700 1 -- 172.21.15.102:0/2616076277 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1500027070 con 0x7f1510074460 2024-04-25T01:53:30.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.264+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15101a42c0 con 0x7f1510074460 2024-04-25T01:53:30.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.264+0000 7f1507e00700 1 -- 172.21.15.102:0/2616076277 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1500003d70 con 0x7f1510074460 2024-04-25T01:53:30.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.264+0000 7f1507e00700 1 -- 172.21.15.102:0/2616076277 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150000f460 con 0x7f1510074460 2024-04-25T01:53:30.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.264+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15101abba0 con 0x7f1510074460 2024-04-25T01:53:30.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.265+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f151010f900 con 0x7f1510074460 2024-04-25T01:53:30.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.267+0000 7f1507e00700 1 -- 172.21.15.102:0/2616076277 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f150002a070 con 0x7f1510074460 2024-04-25T01:53:30.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.268+0000 7f1507e00700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14f40a2d30 0x7f14f40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.269+0000 7f1507e00700 1 -- 172.21.15.102:0/2616076277 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f15000d0690 con 0x7f1510074460 2024-04-25T01:53:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.269+0000 7f150ea00700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14f40a2d30 0x7f14f40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.269+0000 7f1507e00700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] conn(0x7f14f40ac510 0x7f14f40ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.269+0000 7f1507e00700 1 -- 172.21.15.102:0/2616076277 --> [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14f40aefe0 con 0x7f14f40ac510 2024-04-25T01:53:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.269+0000 7f1507e00700 1 -- 172.21.15.102:0/2616076277 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f15000d0a40 con 0x7f1510074460 2024-04-25T01:53:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.269+0000 7f150fe00700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] conn(0x7f14f40ac510 0x7f14f40ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:30.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.269+0000 7f150ea00700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14f40a2d30 0x7f14f40a51f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1510112980 tx=0x7f14fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:30.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.270+0000 7f150fe00700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] conn(0x7f14f40ac510 0x7f14f40ae930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:30.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.271+0000 7f1507e00700 1 -- 172.21.15.102:0/2616076277 <== osd.9 v2:172.21.15.102:6826/1305468681 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14f40aefe0 con 0x7f14f40ac510 2024-04-25T01:53:30.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.290+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 --> [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1510075b50 con 0x7f14f40ac510 2024-04-25T01:53:30.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.290+0000 7f1507e00700 1 -- 172.21.15.102:0/2616076277 <== osd.9 v2:172.21.15.102:6826/1305468681 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1510075b50 con 0x7f14f40ac510 2024-04-25T01:53:30.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] conn(0x7f14f40ac510 msgr2=0x7f14f40ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:30.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] conn(0x7f14f40ac510 0x7f14f40ae930 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14f40a2d30 msgr2=0x7f14f40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14f40a2d30 0x7f14f40a51f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1510112980 tx=0x7f14fc009260 comp rx=0 tx=0).stop 2024-04-25T01:53:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1510074460 msgr2=0x7f151019f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 --2- 172.21.15.102:0/2616076277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1510074460 0x7f151019f020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1500000c00 tx=0x7f150000bab0 comp rx=0 tx=0).stop 2024-04-25T01:53:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f150fe00700 1 -- 172.21.15.102:0/2616076277 reap_dead start 2024-04-25T01:53:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 shutdown_connections 2024-04-25T01:53:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 >> 172.21.15.102:0/2616076277 conn(0x7f1510079390 msgr2=0x7f151010eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 shutdown_connections 2024-04-25T01:53:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:29.291+0000 7f1516200700 1 -- 172.21.15.102:0/2616076277 wait complete. 2024-04-25T01:53:31.072 INFO:teuthology.orchestra.run.smithi102.stdout:317827579931 2024-04-25T01:53:31.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T01:53:32.094 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:31 smithi102 ceph-mon[27796]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-25T01:53:32.144 INFO:teuthology.orchestra.run.smithi102.stdout:296352743459 2024-04-25T01:53:32.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T01:53:32.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:31 smithi110 ceph-mon[32848]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-25T01:53:32.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:31 smithi193 ceph-mon[32981]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 4 op/s 2024-04-25T01:53:32.795 INFO:teuthology.orchestra.run.smithi102.stdout:115964117058 2024-04-25T01:53:32.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T01:53:34.145 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:33 smithi102 ceph-mon[27796]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:53:34.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:33 smithi110 ceph-mon[32848]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:53:34.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:33 smithi193 ceph-mon[32981]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-25T01:53:34.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.512+0000 7f0f5d400700 1 -- 172.21.15.102:0/1182575986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f581024e0 msgr2=0x7f0f581028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.512+0000 7f0f5d400700 1 --2- 172.21.15.102:0/1182575986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f581024e0 0x7f0f581028c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0f48009990 tx=0x7f0f48009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.512+0000 7f0f5d400700 1 -- 172.21.15.102:0/1182575986 shutdown_connections 2024-04-25T01:53:34.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.512+0000 7f0f5d400700 1 --2- 172.21.15.102:0/1182575986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f58107570 0x7f0f5810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.512+0000 7f0f5d400700 1 --2- 172.21.15.102:0/1182575986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f581024e0 0x7f0f581028c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.512+0000 7f0f5d400700 1 --2- 172.21.15.102:0/1182575986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f58102e90 0x7f0f58106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.512+0000 7f0f5d400700 1 -- 172.21.15.102:0/1182575986 >> 172.21.15.102:0/1182575986 conn(0x7f0f58079390 msgr2=0x7f0f580797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:34.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.512+0000 7f0f5d400700 1 -- 172.21.15.102:0/1182575986 shutdown_connections 2024-04-25T01:53:34.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.512+0000 7f0f5d400700 1 -- 172.21.15.102:0/1182575986 wait complete. 2024-04-25T01:53:34.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.513+0000 7f0f5d400700 1 Processor -- start 2024-04-25T01:53:34.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f5d400700 1 -- start start 2024-04-25T01:53:34.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f5d400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f581024e0 0x7f0f5819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:34.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f5d400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f58102e90 0x7f0f5819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:34.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f5d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f58107570 0x7f0f58198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f5d400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0f58112550 con 0x7f0f58102e90 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f5d400700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0f581123d0 con 0x7f0f581024e0 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f5d400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0f581126d0 con 0x7f0f58107570 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f55e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f58102e90 0x7f0f5819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f55e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f58102e90 0x7f0f5819ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58002/0 (socket says 172.21.15.102:58002) 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f57200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f58107570 0x7f0f58198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f55e00700 1 -- 172.21.15.102:0/1639646145 learned_addr learned my addr 172.21.15.102:0/1639646145 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.514+0000 7f0f56800700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f581024e0 0x7f0f5819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f55e00700 1 -- 172.21.15.102:0/1639646145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f58107570 msgr2=0x7f0f58198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f55e00700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f58107570 0x7f0f58198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f55e00700 1 -- 172.21.15.102:0/1639646145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f581024e0 msgr2=0x7f0f5819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f55e00700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f581024e0 0x7f0f5819e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f55e00700 1 -- 172.21.15.102:0/1639646145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f58199010 con 0x7f0f58102e90 2024-04-25T01:53:34.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f56800700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f581024e0 0x7f0f5819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:34.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f55e00700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f58102e90 0x7f0f5819ebf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400e9f0 tx=0x7f0f4400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:34.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f4f400700 1 -- 172.21.15.102:0/1639646145 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44004a30 con 0x7f0f58102e90 2024-04-25T01:53:34.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f4f400700 1 -- 172.21.15.102:0/1639646145 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f4400cca0 con 0x7f0f58102e90 2024-04-25T01:53:34.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.515+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f58199300 con 0x7f0f58102e90 2024-04-25T01:53:34.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.516+0000 7f0f4f400700 1 -- 172.21.15.102:0/1639646145 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f440056c0 con 0x7f0f58102e90 2024-04-25T01:53:34.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.516+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f581a5510 con 0x7f0f58102e90 2024-04-25T01:53:34.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.517+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f580629c0 con 0x7f0f58102e90 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.519+0000 7f0f4f400700 1 -- 172.21.15.102:0/1639646145 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0f4400c7c0 con 0x7f0f58102e90 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.521+0000 7f0f4f400700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f3c0a2d30 0x7f0f3c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.521+0000 7f0f4f400700 1 -- 172.21.15.102:0/1639646145 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f0f440cf610 con 0x7f0f58102e90 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.521+0000 7f0f56800700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f3c0a2d30 0x7f0f3c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.521+0000 7f0f4f400700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] conn(0x7f0f3c0ac510 0x7f0f3c0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.521+0000 7f0f4f400700 1 -- 172.21.15.102:0/1639646145 --> [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f4400e1a0 con 0x7f0f3c0ac510 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.521+0000 7f0f4f400700 1 -- 172.21.15.102:0/1639646145 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f0f440cf9c0 con 0x7f0f58102e90 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.522+0000 7f0f57200700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] conn(0x7f0f3c0ac510 0x7f0f3c0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.522+0000 7f0f56800700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f3c0a2d30 0x7f0f3c0a51f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0f480098e0 tx=0x7f0f48005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:34.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.522+0000 7f0f57200700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] conn(0x7f0f3c0ac510 0x7f0f3c0ae930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:34.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.523+0000 7f0f4f400700 1 -- 172.21.15.102:0/1639646145 <== osd.6 v2:172.21.15.102:6818/2856826689 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f4400e1a0 con 0x7f0f3c0ac510 2024-04-25T01:53:34.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.541+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 --> [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0f58062e40 con 0x7f0f3c0ac510 2024-04-25T01:53:34.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.541+0000 7f0f4f400700 1 -- 172.21.15.102:0/1639646145 <== osd.6 v2:172.21.15.102:6818/2856826689 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0f58062e40 con 0x7f0f3c0ac510 2024-04-25T01:53:34.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] conn(0x7f0f3c0ac510 msgr2=0x7f0f3c0ae930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] conn(0x7f0f3c0ac510 0x7f0f3c0ae930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f3c0a2d30 msgr2=0x7f0f3c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0f3c0a2d30 0x7f0f3c0a51f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0f480098e0 tx=0x7f0f48005960 comp rx=0 tx=0).stop 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f58102e90 msgr2=0x7f0f5819ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 --2- 172.21.15.102:0/1639646145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f58102e90 0x7f0f5819ebf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400e9f0 tx=0x7f0f4400ed00 comp rx=0 tx=0).stop 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f57200700 1 -- 172.21.15.102:0/1639646145 reap_dead start 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 shutdown_connections 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 >> 172.21.15.102:0/1639646145 conn(0x7f0f58079390 msgr2=0x7f0f58101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 shutdown_connections 2024-04-25T01:53:34.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.542+0000 7f0f5d400700 1 -- 172.21.15.102:0/1639646145 wait complete. 2024-04-25T01:53:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.765+0000 7ff16ea00700 1 -- 172.21.15.102:0/565924178 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff168103cc0 msgr2=0x7ff16810c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.765+0000 7ff16ea00700 1 --2- 172.21.15.102:0/565924178 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff168103cc0 0x7ff16810c1b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff154009990 tx=0x7ff154009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.765+0000 7ff16ea00700 1 -- 172.21.15.102:0/565924178 shutdown_connections 2024-04-25T01:53:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.765+0000 7ff16ea00700 1 --2- 172.21.15.102:0/565924178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff16810c780 0x7ff16810ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.765+0000 7ff16ea00700 1 --2- 172.21.15.102:0/565924178 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1681033a0 0x7ff168103780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.765+0000 7ff16ea00700 1 --2- 172.21.15.102:0/565924178 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff168103cc0 0x7ff16810c1b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.765+0000 7ff16ea00700 1 -- 172.21.15.102:0/565924178 >> 172.21.15.102:0/565924178 conn(0x7ff1680fcb50 msgr2=0x7ff1680fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.765+0000 7ff16ea00700 1 -- 172.21.15.102:0/565924178 shutdown_connections 2024-04-25T01:53:35.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.765+0000 7ff16ea00700 1 -- 172.21.15.102:0/565924178 wait complete. 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.766+0000 7ff16ea00700 1 Processor -- start 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.766+0000 7ff16ea00700 1 -- start start 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16ea00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1681033a0 0x7ff1681a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff168103cc0 0x7ff1681a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff16810c780 0x7ff16819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16ea00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff168112490 con 0x7ff16810c780 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16ea00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff168112310 con 0x7ff1681033a0 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16ea00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff168112610 con 0x7ff168103cc0 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16d600700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1681033a0 0x7ff1681a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16cc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff168103cc0 0x7ff1681a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16d600700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1681033a0 0x7ff1681a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51654/0 (socket says 172.21.15.102:51654) 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16d600700 1 -- 172.21.15.102:0/1216360693 learned_addr learned my addr 172.21.15.102:0/1216360693 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:35.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.767+0000 7ff16e000700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff16810c780 0x7ff16819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff16cc00700 1 -- 172.21.15.102:0/1216360693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1681033a0 msgr2=0x7ff1681a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff16cc00700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1681033a0 0x7ff1681a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff16cc00700 1 -- 172.21.15.102:0/1216360693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff16810c780 msgr2=0x7ff16819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff16cc00700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff16810c780 0x7ff16819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff16cc00700 1 -- 172.21.15.102:0/1216360693 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16819b170 con 0x7ff168103cc0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff16cc00700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff168103cc0 0x7ff1681a0d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff1540098e0 tx=0x7ff15400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff166000700 1 -- 172.21.15.102:0/1216360693 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154027070 con 0x7ff168103cc0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff16819b400 con 0x7ff168103cc0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff166000700 1 -- 172.21.15.102:0/1216360693 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff154003d70 con 0x7ff168103cc0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.768+0000 7ff166000700 1 -- 172.21.15.102:0/1216360693 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff15400f460 con 0x7ff168103cc0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.769+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1681a7720 con 0x7ff168103cc0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.769+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff168102aa0 con 0x7ff168103cc0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.772+0000 7ff166000700 1 -- 172.21.15.102:0/1216360693 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff15402a070 con 0x7ff168103cc0 2024-04-25T01:53:35.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.774+0000 7ff166000700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff1500a2db0 0x7ff1500a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.774+0000 7ff166000700 1 -- 172.21.15.102:0/1216360693 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7ff1540d0a30 con 0x7ff168103cc0 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.774+0000 7ff16d600700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff1500a2db0 0x7ff1500a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.774+0000 7ff166000700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] conn(0x7ff1500ac590 0x7ff1500ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.774+0000 7ff166000700 1 -- 172.21.15.102:0/1216360693 --> [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff154010ea0 con 0x7ff1500ac590 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.774+0000 7ff166000700 1 -- 172.21.15.102:0/1216360693 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7ff1540d0e20 con 0x7ff168103cc0 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.775+0000 7ff16e000700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] conn(0x7ff1500ac590 0x7ff1500ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.775+0000 7ff16d600700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff1500a2db0 0x7ff1500a5270 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff15c009fd0 tx=0x7ff15c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.776+0000 7ff16e000700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] conn(0x7ff1500ac590 0x7ff1500ae9b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.776+0000 7ff166000700 1 -- 172.21.15.102:0/1216360693 <== osd.7 v2:172.21.15.110:6816/2471551369 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff154010ea0 con 0x7ff1500ac590 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.798+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 --> [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff16806a770 con 0x7ff1500ac590 2024-04-25T01:53:35.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.799+0000 7ff166000700 1 -- 172.21.15.102:0/1216360693 <== osd.7 v2:172.21.15.110:6816/2471551369 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff16806a770 con 0x7ff1500ac590 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 >> [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] conn(0x7ff1500ac590 msgr2=0x7ff1500ae9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] conn(0x7ff1500ac590 0x7ff1500ae9b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff1500a2db0 msgr2=0x7ff1500a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff1500a2db0 0x7ff1500a5270 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff15c009fd0 tx=0x7ff15c00b040 comp rx=0 tx=0).stop 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff168103cc0 msgr2=0x7ff1681a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 --2- 172.21.15.102:0/1216360693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff168103cc0 0x7ff1681a0d90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff1540098e0 tx=0x7ff15400bab0 comp rx=0 tx=0).stop 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16e000700 1 -- 172.21.15.102:0/1216360693 reap_dead start 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 shutdown_connections 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 >> 172.21.15.102:0/1216360693 conn(0x7ff1680fcb50 msgr2=0x7ff1680fef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:35.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 shutdown_connections 2024-04-25T01:53:35.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:34.800+0000 7ff16ea00700 1 -- 172.21.15.102:0/1216360693 wait complete. 2024-04-25T01:53:36.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:35 smithi102 ceph-mon[27796]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T01:53:36.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:35 smithi110 ceph-mon[32848]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T01:53:36.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:35 smithi193 ceph-mon[32981]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-25T01:53:36.310 INFO:teuthology.orchestra.run.smithi102.stdout:77309411402 2024-04-25T01:53:36.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T01:53:36.851 INFO:teuthology.orchestra.run.smithi102.stdout:227633266736 2024-04-25T01:53:36.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T01:53:37.468 INFO:teuthology.orchestra.run.smithi102.stdout:206158430260 2024-04-25T01:53:37.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T01:53:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.752+0000 7f6d34800700 1 -- 172.21.15.102:0/4291624529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d2c107570 msgr2=0x7f6d2c10f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.752+0000 7f6d34800700 1 --2- 172.21.15.102:0/4291624529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d2c107570 0x7f6d2c10f140 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800b0a0 tx=0x7f6d2800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.752+0000 7f6d34800700 1 -- 172.21.15.102:0/4291624529 shutdown_connections 2024-04-25T01:53:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.752+0000 7f6d34800700 1 --2- 172.21.15.102:0/4291624529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d2c1024e0 0x7f6d2c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.752+0000 7f6d34800700 1 --2- 172.21.15.102:0/4291624529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d2c107570 0x7f6d2c10f140 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.752+0000 7f6d34800700 1 --2- 172.21.15.102:0/4291624529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d2c102e90 0x7f6d2c106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.752+0000 7f6d34800700 1 -- 172.21.15.102:0/4291624529 >> 172.21.15.102:0/4291624529 conn(0x7f6d2c079390 msgr2=0x7f6d2c0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:37.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.752+0000 7f6d34800700 1 -- 172.21.15.102:0/4291624529 shutdown_connections 2024-04-25T01:53:37.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.752+0000 7f6d34800700 1 -- 172.21.15.102:0/4291624529 wait complete. 2024-04-25T01:53:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.753+0000 7f6d34800700 1 Processor -- start 2024-04-25T01:53:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.753+0000 7f6d34800700 1 -- start start 2024-04-25T01:53:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d34800700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d2c1024e0 0x7f6d2c1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d34800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d2c102e90 0x7f6d2c1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d34800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d2c107570 0x7f6d2c19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d34800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c1125a0 con 0x7f6d2c107570 2024-04-25T01:53:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d34800700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c112420 con 0x7f6d2c1024e0 2024-04-25T01:53:37.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d34800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c112720 con 0x7f6d2c102e90 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d32800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d2c107570 0x7f6d2c19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d31e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d2c1024e0 0x7f6d2c1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d32800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d2c107570 0x7f6d2c19aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58042/0 (socket says 172.21.15.102:58042) 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d32800700 1 -- 172.21.15.102:0/1561929713 learned_addr learned my addr 172.21.15.102:0/1561929713 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d31e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d2c1024e0 0x7f6d2c1a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51676/0 (socket says 172.21.15.102:51676) 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.754+0000 7f6d31400700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d2c102e90 0x7f6d2c1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d31e00700 1 -- 172.21.15.102:0/1561929713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d2c102e90 msgr2=0x7f6d2c1a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d31e00700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d2c102e90 0x7f6d2c1a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d31e00700 1 -- 172.21.15.102:0/1561929713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d2c107570 msgr2=0x7f6d2c19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d31e00700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d2c107570 0x7f6d2c19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d31e00700 1 -- 172.21.15.102:0/1561929713 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2c19b170 con 0x7f6d2c1024e0 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d32800700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d2c107570 0x7f6d2c19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:37.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d31e00700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d2c1024e0 0x7f6d2c1a0850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000e9f0 tx=0x7f6d2000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d26a00700 1 -- 172.21.15.102:0/1561929713 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2000c9c0 con 0x7f6d2c1024e0 2024-04-25T01:53:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2c19b460 con 0x7f6d2c1024e0 2024-04-25T01:53:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d26a00700 1 -- 172.21.15.102:0/1561929713 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d2000cb60 con 0x7f6d2c1024e0 2024-04-25T01:53:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.755+0000 7f6d26a00700 1 -- 172.21.15.102:0/1561929713 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d20018680 con 0x7f6d2c1024e0 2024-04-25T01:53:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.756+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2c1a7720 con 0x7f6d2c1024e0 2024-04-25T01:53:37.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.756+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d2c0757c0 con 0x7f6d2c1024e0 2024-04-25T01:53:37.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.759+0000 7f6d26a00700 1 -- 172.21.15.102:0/1561929713 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f6d20010430 con 0x7f6d2c1024e0 2024-04-25T01:53:37.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.760+0000 7f6d26a00700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d140a2cb0 0x7f6d140a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:37.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.760+0000 7f6d26a00700 1 -- 172.21.15.102:0/1561929713 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f6d200cf660 con 0x7f6d2c1024e0 2024-04-25T01:53:37.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.760+0000 7f6d31400700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d140a2cb0 0x7f6d140a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.761+0000 7f6d26a00700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] conn(0x7f6d140ac490 0x7f6d140ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.761+0000 7f6d26a00700 1 -- 172.21.15.102:0/1561929713 --> [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d140aef60 con 0x7f6d140ac490 2024-04-25T01:53:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.761+0000 7f6d26a00700 1 -- 172.21.15.102:0/1561929713 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f6d200cfa10 con 0x7f6d2c1024e0 2024-04-25T01:53:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.761+0000 7f6d32800700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] conn(0x7f6d140ac490 0x7f6d140ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:37.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.761+0000 7f6d31400700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d140a2cb0 0x7f6d140a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c10f4f0 tx=0x7f6d1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:37.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.762+0000 7f6d32800700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] conn(0x7f6d140ac490 0x7f6d140ae8b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:37.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.762+0000 7f6d26a00700 1 -- 172.21.15.102:0/1561929713 <== osd.5 v2:172.21.15.193:6808/3543273137 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d140aef60 con 0x7f6d140ac490 2024-04-25T01:53:37.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.783+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 --> [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6d2c103970 con 0x7f6d140ac490 2024-04-25T01:53:37.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.783+0000 7f6d26a00700 1 -- 172.21.15.102:0/1561929713 <== osd.5 v2:172.21.15.193:6808/3543273137 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6d2c103970 con 0x7f6d140ac490 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.784+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 >> [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] conn(0x7f6d140ac490 msgr2=0x7f6d140ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.784+0000 7f6d34800700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] conn(0x7f6d140ac490 0x7f6d140ae8b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.784+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d140a2cb0 msgr2=0x7f6d140a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.784+0000 7f6d34800700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f6d140a2cb0 0x7f6d140a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c10f4f0 tx=0x7f6d1c009260 comp rx=0 tx=0).stop 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.784+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d2c1024e0 msgr2=0x7f6d2c1a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.784+0000 7f6d34800700 1 --2- 172.21.15.102:0/1561929713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d2c1024e0 0x7f6d2c1a0850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000e9f0 tx=0x7f6d2000ed00 comp rx=0 tx=0).stop 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.784+0000 7f6d32800700 1 -- 172.21.15.102:0/1561929713 reap_dead start 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.784+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 shutdown_connections 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.784+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 >> 172.21.15.102:0/1561929713 conn(0x7f6d2c079390 msgr2=0x7f6d2c100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.785+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 shutdown_connections 2024-04-25T01:53:37.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:37.785+0000 7f6d34800700 1 -- 172.21.15.102:0/1561929713 wait complete. 2024-04-25T01:53:38.044 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:37 smithi102 ceph-mon[27796]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-25T01:53:38.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:37 smithi110 ceph-mon[32848]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-25T01:53:38.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:37 smithi193 ceph-mon[32981]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-25T01:53:38.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.528+0000 7f5e5fe00700 1 -- 172.21.15.102:0/3947836085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e58109220 msgr2=0x7f5e58115730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.528+0000 7f5e5fe00700 1 --2- 172.21.15.102:0/3947836085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e58109220 0x7f5e58115730 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000b0a0 tx=0x7f5e5000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.528+0000 7f5e5fe00700 1 -- 172.21.15.102:0/3947836085 shutdown_connections 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.528+0000 7f5e5fe00700 1 --2- 172.21.15.102:0/3947836085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e5810e860 0x7f5e5810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.528+0000 7f5e5fe00700 1 --2- 172.21.15.102:0/3947836085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e58109220 0x7f5e58115730 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.528+0000 7f5e5fe00700 1 --2- 172.21.15.102:0/3947836085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e58108860 0x7f5e58108ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.528+0000 7f5e5fe00700 1 -- 172.21.15.102:0/3947836085 >> 172.21.15.102:0/3947836085 conn(0x7f5e58075a00 msgr2=0x7f5e58107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.528+0000 7f5e5fe00700 1 -- 172.21.15.102:0/3947836085 shutdown_connections 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.528+0000 7f5e5fe00700 1 -- 172.21.15.102:0/3947836085 wait complete. 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.529+0000 7f5e5fe00700 1 Processor -- start 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5fe00700 1 -- start start 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e58108860 0x7f5e5819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e58109220 0x7f5e5819f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5fe00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e5810e860 0x7f5e581a3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5fe00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5e58118c40 con 0x7f5e58109220 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5fe00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5e58118ac0 con 0x7f5e5810e860 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5fe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e58118dc0 con 0x7f5e58108860 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5de00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e5810e860 0x7f5e581a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5de00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e5810e860 0x7f5e581a3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51700/0 (socket says 172.21.15.102:51700) 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5de00700 1 -- 172.21.15.102:0/2055533333 learned_addr learned my addr 172.21.15.102:0/2055533333 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5ca00700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e58109220 0x7f5e5819f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.530+0000 7f5e5d400700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e58108860 0x7f5e5819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e5de00700 1 -- 172.21.15.102:0/2055533333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e58108860 msgr2=0x7f5e5819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e5de00700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e58108860 0x7f5e5819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e5de00700 1 -- 172.21.15.102:0/2055533333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e58109220 msgr2=0x7f5e5819f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e5de00700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e58109220 0x7f5e5819f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e5de00700 1 -- 172.21.15.102:0/2055533333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e581a4060 con 0x7f5e5810e860 2024-04-25T01:53:38.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e5d400700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e58108860 0x7f5e5819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:53:38.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e5de00700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e5810e860 0x7f5e581a3920 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000b070 tx=0x7f5e50004d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:38.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e56000700 1 -- 172.21.15.102:0/2055533333 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50021070 con 0x7f5e5810e860 2024-04-25T01:53:38.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e581a42f0 con 0x7f5e5810e860 2024-04-25T01:53:38.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e56000700 1 -- 172.21.15.102:0/2055533333 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e50007c00 con 0x7f5e5810e860 2024-04-25T01:53:38.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e56000700 1 -- 172.21.15.102:0/2055533333 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e500048b0 con 0x7f5e5810e860 2024-04-25T01:53:38.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.531+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e581abbd0 con 0x7f5e5810e860 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.532+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5e580768c0 con 0x7f5e5810e860 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.534+0000 7f5e56000700 1 -- 172.21.15.102:0/2055533333 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5e50004000 con 0x7f5e5810e860 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.536+0000 7f5e56000700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e400a2cb0 0x7f5e400a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.536+0000 7f5e56000700 1 -- 172.21.15.102:0/2055533333 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f5e5000f020 con 0x7f5e5810e860 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.536+0000 7f5e5d400700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e400a2cb0 0x7f5e400a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.536+0000 7f5e56000700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] conn(0x7f5e400ac490 0x7f5e400ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.536+0000 7f5e56000700 1 -- 172.21.15.102:0/2055533333 --> [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e500020b0 con 0x7f5e400ac490 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.536+0000 7f5e56000700 1 -- 172.21.15.102:0/2055533333 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f5e500d0b30 con 0x7f5e5810e860 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.537+0000 7f5e5ca00700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] conn(0x7f5e400ac490 0x7f5e400ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.537+0000 7f5e5d400700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e400a2cb0 0x7f5e400a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009c90 tx=0x7f5e4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.537+0000 7f5e5ca00700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] conn(0x7f5e400ac490 0x7f5e400ae8b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.538+0000 7f5e56000700 1 -- 172.21.15.102:0/2055533333 <== osd.8 v2:172.21.15.193:6816/3634064328 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5e500020b0 con 0x7f5e400ac490 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.556+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 --> [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5e58109fa0 con 0x7f5e400ac490 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.557+0000 7f5e56000700 1 -- 172.21.15.102:0/2055533333 <== osd.8 v2:172.21.15.193:6816/3634064328 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5e58109fa0 con 0x7f5e400ac490 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.557+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 >> [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] conn(0x7f5e400ac490 msgr2=0x7f5e400ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.557+0000 7f5e5fe00700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] conn(0x7f5e400ac490 0x7f5e400ae8b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.558+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e400a2cb0 msgr2=0x7f5e400a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.558+0000 7f5e5fe00700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5e400a2cb0 0x7f5e400a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009c90 tx=0x7f5e4c009260 comp rx=0 tx=0).stop 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.558+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e5810e860 msgr2=0x7f5e581a3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.558+0000 7f5e5fe00700 1 --2- 172.21.15.102:0/2055533333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e5810e860 0x7f5e581a3920 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000b070 tx=0x7f5e50004d10 comp rx=0 tx=0).stop 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.558+0000 7f5e5de00700 1 -- 172.21.15.102:0/2055533333 reap_dead start 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.558+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 shutdown_connections 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.558+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 >> 172.21.15.102:0/2055533333 conn(0x7f5e58075a00 msgr2=0x7f5e58075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.558+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 shutdown_connections 2024-04-25T01:53:38.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:38.558+0000 7f5e5fe00700 1 -- 172.21.15.102:0/2055533333 wait complete. 2024-04-25T01:53:39.565 INFO:teuthology.orchestra.run.smithi102.stdout:261993005099 2024-04-25T01:53:39.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T01:53:39.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.333+0000 7f4acce00700 1 -- 172.21.15.102:0/1294227646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac8069f40 msgr2=0x7f4ac806a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:39.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.333+0000 7f4acce00700 1 --2- 172.21.15.102:0/1294227646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac8069f40 0x7f4ac806a320 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009990 tx=0x7f4ab8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:39.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.333+0000 7f4acce00700 1 -- 172.21.15.102:0/1294227646 shutdown_connections 2024-04-25T01:53:39.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.333+0000 7f4acce00700 1 --2- 172.21.15.102:0/1294227646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac8069f40 0x7f4ac806a320 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.333+0000 7f4acce00700 1 --2- 172.21.15.102:0/1294227646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac810ece0 0x7f4ac81110d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.333+0000 7f4acce00700 1 --2- 172.21.15.102:0/1294227646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ac806a860 0x7f4ac810e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.333+0000 7f4acce00700 1 -- 172.21.15.102:0/1294227646 >> 172.21.15.102:0/1294227646 conn(0x7f4ac80780e0 msgr2=0x7f4ac80784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.333+0000 7f4acce00700 1 -- 172.21.15.102:0/1294227646 shutdown_connections 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.334+0000 7f4acce00700 1 -- 172.21.15.102:0/1294227646 wait complete. 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.335+0000 7f4acce00700 1 Processor -- start 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.335+0000 7f4acce00700 1 -- start start 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4acce00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ac8069f40 0x7f4ac81a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4acce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac806a860 0x7f4ac81a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4acce00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac810ece0 0x7f4ac819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4acce00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac8114760 con 0x7f4ac8069f40 2024-04-25T01:53:39.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4acce00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac81145e0 con 0x7f4ac810ece0 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4acce00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac81148e0 con 0x7f4ac806a860 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ac8069f40 0x7f4ac81a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ac8069f40 0x7f4ac81a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58088/0 (socket says 172.21.15.102:58088) 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac806a860 0x7f4ac81a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5e00700 1 -- 172.21.15.102:0/1966917806 learned_addr learned my addr 172.21.15.102:0/1966917806 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac6800700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac810ece0 0x7f4ac819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5e00700 1 -- 172.21.15.102:0/1966917806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac806a860 msgr2=0x7f4ac81a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5e00700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac806a860 0x7f4ac81a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5e00700 1 -- 172.21.15.102:0/1966917806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac810ece0 msgr2=0x7f4ac819aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5e00700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac810ece0 0x7f4ac819aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5e00700 1 -- 172.21.15.102:0/1966917806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac819b1a0 con 0x7f4ac8069f40 2024-04-25T01:53:39.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac6800700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ac810ece0 0x7f4ac819aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.336+0000 7f4ac5400700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ac806a860 0x7f4ac81a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.337+0000 7f4ac5e00700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ac8069f40 0x7f4ac81a0880 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8000c00 tx=0x7f4ab800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.337+0000 7f4abea00700 1 -- 172.21.15.102:0/1966917806 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8027070 con 0x7f4ac8069f40 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.337+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac819b430 con 0x7f4ac8069f40 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.337+0000 7f4abea00700 1 -- 172.21.15.102:0/1966917806 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ab8003d70 con 0x7f4ac8069f40 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.337+0000 7f4abea00700 1 -- 172.21.15.102:0/1966917806 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab800f460 con 0x7f4ac8069f40 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.337+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac81a7750 con 0x7f4ac8069f40 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.338+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4ac8062900 con 0x7f4ac8069f40 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.340+0000 7f4abea00700 1 -- 172.21.15.102:0/1966917806 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4ab802a070 con 0x7f4ac8069f40 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.342+0000 7f4abea00700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4aac0a2cb0 0x7f4aac0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:39.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.342+0000 7f4abea00700 1 -- 172.21.15.102:0/1966917806 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 7979+0+0 (secure 0 0 0) 0x7f4ab80d0500 con 0x7f4ac8069f40 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.342+0000 7f4ac5400700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4aac0a2cb0 0x7f4aac0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.342+0000 7f4abea00700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] conn(0x7f4aac0ac490 0x7f4aac0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.342+0000 7f4abea00700 1 -- 172.21.15.102:0/1966917806 --> [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4aac0aef60 con 0x7f4aac0ac490 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.342+0000 7f4abea00700 1 -- 172.21.15.102:0/1966917806 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7f4ab80d08b0 con 0x7f4ac8069f40 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.343+0000 7f4ac6800700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] conn(0x7f4aac0ac490 0x7f4aac0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.343+0000 7f4ac5400700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4aac0a2cb0 0x7f4aac0a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009680 tx=0x7f4ab4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.343+0000 7f4ac6800700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] conn(0x7f4aac0ac490 0x7f4aac0ae8b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.344+0000 7f4abea00700 1 -- 172.21.15.102:0/1966917806 <== osd.0 v2:172.21.15.102:6802/417549417 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4aac0aef60 con 0x7f4aac0ac490 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.363+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 --> [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4ac8062d00 con 0x7f4aac0ac490 2024-04-25T01:53:39.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.364+0000 7f4abea00700 1 -- 172.21.15.102:0/1966917806 <== osd.0 v2:172.21.15.102:6802/417549417 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4ac8062d00 con 0x7f4aac0ac490 2024-04-25T01:53:39.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.364+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] conn(0x7f4aac0ac490 msgr2=0x7f4aac0ae8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:39.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.364+0000 7f4acce00700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] conn(0x7f4aac0ac490 0x7f4aac0ae8b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:39.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.364+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4aac0a2cb0 msgr2=0x7f4aac0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:39.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.364+0000 7f4acce00700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4aac0a2cb0 0x7f4aac0a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009680 tx=0x7f4ab4009260 comp rx=0 tx=0).stop 2024-04-25T01:53:39.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.364+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ac8069f40 msgr2=0x7f4ac81a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:39.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.364+0000 7f4acce00700 1 --2- 172.21.15.102:0/1966917806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ac8069f40 0x7f4ac81a0880 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8000c00 tx=0x7f4ab800bab0 comp rx=0 tx=0).stop 2024-04-25T01:53:39.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.364+0000 7f4ac6800700 1 -- 172.21.15.102:0/1966917806 reap_dead start 2024-04-25T01:53:39.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.365+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 shutdown_connections 2024-04-25T01:53:39.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.365+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 >> 172.21.15.102:0/1966917806 conn(0x7f4ac80780e0 msgr2=0x7f4ac8110130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:39.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.365+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 shutdown_connections 2024-04-25T01:53:39.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:39.365+0000 7f4acce00700 1 -- 172.21.15.102:0/1966917806 wait complete. 2024-04-25T01:53:39.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:39 smithi102 ceph-mon[27796]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:53:40.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:39 smithi110 ceph-mon[32848]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:53:40.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:39 smithi193 ceph-mon[32981]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:53:40.412 INFO:teuthology.orchestra.run.smithi102.stdout:34359738454 2024-04-25T01:53:40.413 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T01:53:40.772 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:40.777 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:40.797 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:40.797 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:40.811 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:40.828 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:40.868 INFO:teuthology.orchestra.run.smithi102.stdout:171798691899 2024-04-25T01:53:40.869 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T01:53:42.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:41 smithi102 ceph-mon[27796]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:53:42.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:41 smithi110 ceph-mon[32848]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:53:42.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:41 smithi193 ceph-mon[32981]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:53:42.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-25T01:53:42.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-25T01:53:42.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-25T01:53:42.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:53:42.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T01:53:42.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-25T01:53:42.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-04-25T01:53:42.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:42 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-25T01:53:43.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-25T01:53:43.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-25T01:53:43.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-25T01:53:43.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:53:43.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T01:53:43.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-25T01:53:43.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-04-25T01:53:43.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:42 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-25T01:53:43.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-25T01:53:43.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-25T01:53:43.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-04-25T01:53:43.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-04-25T01:53:43.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-04-25T01:53:43.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-04-25T01:53:43.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]: dispatch 2024-04-25T01:53:43.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:42 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-25T01:53:44.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:53:44.138 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:43 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:53:44.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:53:44.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-25T01:53:44.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-25T01:53:44.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-25T01:53:44.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:53:44.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T01:53:44.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-25T01:53:44.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-04-25T01:53:44.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-25T01:53:44.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:53:44.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:43 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:53:44.154 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:53:44.154 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-25T01:53:44.154 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-25T01:53:44.154 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-04-25T01:53:44.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-04-25T01:53:44.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-04-25T01:53:44.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-04-25T01:53:44.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 8]}]': finished 2024-04-25T01:53:44.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-25T01:53:44.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T01:53:44.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:43 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:53:44.796 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:45.077 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:44 smithi102 ceph-mon[27796]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:53:45.077 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:44 smithi102 ceph-mon[27796]: pgmap v350: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T01:53:45.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:44 smithi110 ceph-mon[32848]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:53:45.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:44 smithi110 ceph-mon[32848]: pgmap v350: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T01:53:45.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:44 smithi193 ceph-mon[32981]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T01:53:45.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:44 smithi193 ceph-mon[32981]: pgmap v350: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s 2024-04-25T01:53:46.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:46 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:46 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:46 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:46 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:46 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:46 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:46 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:46 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:46 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:53:46.690 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:46.692 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:47.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.195+0000 7fef7d000700 1 -- 172.21.15.102:0/1362009504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 msgr2=0x7fef7810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:47.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.195+0000 7fef7d000700 1 --2- 172.21.15.102:0/1362009504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 0x7fef7810dfe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fef70009f90 tx=0x7fef7000b330 comp rx=0 tx=0).stop 2024-04-25T01:53:47.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.195+0000 7fef7d000700 1 -- 172.21.15.102:0/1362009504 shutdown_connections 2024-04-25T01:53:47.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.195+0000 7fef7d000700 1 --2- 172.21.15.102:0/1362009504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 0x7fef7810dfe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.195+0000 7fef7d000700 1 --2- 172.21.15.102:0/1362009504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef781071a0 0x7fef78107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.195+0000 7fef7d000700 1 --2- 172.21.15.102:0/1362009504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef78101130 0x7fef781015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.195+0000 7fef7d000700 1 -- 172.21.15.102:0/1362009504 >> 172.21.15.102:0/1362009504 conn(0x7fef780780e0 msgr2=0x7fef780784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:47.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.196+0000 7fef7d000700 1 -- 172.21.15.102:0/1362009504 shutdown_connections 2024-04-25T01:53:47.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.196+0000 7fef7d000700 1 -- 172.21.15.102:0/1362009504 wait complete. 2024-04-25T01:53:47.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.197+0000 7fef7d000700 1 Processor -- start 2024-04-25T01:53:47.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.197+0000 7fef7d000700 1 -- start start 2024-04-25T01:53:47.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.197+0000 7fef7d000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef78101130 0x7fef7819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:47.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.197+0000 7fef7d000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 0x7fef7819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:47.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.197+0000 7fef7d000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef781071a0 0x7fef781987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:47.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef7d000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fef78111460 con 0x7fef78101130 2024-04-25T01:53:47.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef7d000700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fef781112e0 con 0x7fef781071a0 2024-04-25T01:53:47.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef7d000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fef781115e0 con 0x7fef78101af0 2024-04-25T01:53:47.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef75400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 0x7fef7819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:47.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef75400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 0x7fef7819eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52516/0 (socket says 172.21.15.102:52516) 2024-04-25T01:53:47.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef75400700 1 -- 172.21.15.102:0/1746720559 learned_addr learned my addr 172.21.15.102:0/1746720559 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:47.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef75e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef78101130 0x7fef7819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:47.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef76800700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef781071a0 0x7fef781987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef75400700 1 -- 172.21.15.102:0/1746720559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef781071a0 msgr2=0x7fef781987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef75400700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef781071a0 0x7fef781987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef75400700 1 -- 172.21.15.102:0/1746720559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef78101130 msgr2=0x7fef7819e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef75400700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef78101130 0x7fef7819e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.198+0000 7fef75400700 1 -- 172.21.15.102:0/1746720559 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef78198f80 con 0x7fef78101af0 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.199+0000 7fef75400700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 0x7fef7819eb60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fef6400ec10 tx=0x7fef6400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.199+0000 7fef6ea00700 1 -- 172.21.15.102:0/1746720559 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef6400cc90 con 0x7fef78101af0 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.199+0000 7fef6ea00700 1 -- 172.21.15.102:0/1746720559 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef6400ce30 con 0x7fef78101af0 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.199+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef78199270 con 0x7fef78101af0 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.199+0000 7fef6ea00700 1 -- 172.21.15.102:0/1746720559 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef64010690 con 0x7fef78101af0 2024-04-25T01:53:47.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.199+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef781a5480 con 0x7fef78101af0 2024-04-25T01:53:47.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.200+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef78102850 con 0x7fef78101af0 2024-04-25T01:53:47.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.203+0000 7fef6ea00700 1 -- 172.21.15.102:0/1746720559 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fef64003680 con 0x7fef78101af0 2024-04-25T01:53:47.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.204+0000 7fef6ea00700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fef5c0a2cb0 0x7fef5c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:47.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.204+0000 7fef6ea00700 1 -- 172.21.15.102:0/1746720559 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fef640cf550 con 0x7fef78101af0 2024-04-25T01:53:47.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.204+0000 7fef75e00700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fef5c0a2cb0 0x7fef5c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:47.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.205+0000 7fef75e00700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fef5c0a2cb0 0x7fef5c0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef68009be0 tx=0x7fef68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:47.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.206+0000 7fef6ea00700 1 -- 172.21.15.102:0/1746720559 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fef6409c300 con 0x7fef78101af0 2024-04-25T01:53:47.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:47 smithi102 ceph-mon[27796]: pgmap v351: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T01:53:47.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.434+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fef781a5710 con 0x7fef78101af0 2024-04-25T01:53:47.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.435+0000 7fef6ea00700 1 -- 172.21.15.102:0/1746720559 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fef6409bc00 con 0x7fef78101af0 2024-04-25T01:53:47.438 INFO:teuthology.orchestra.run.smithi102.stdout:150323855426 2024-04-25T01:53:47.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.438+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fef5c0a2cb0 msgr2=0x7fef5c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:47.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.438+0000 7fef7d000700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fef5c0a2cb0 0x7fef5c0a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef68009be0 tx=0x7fef68009260 comp rx=0 tx=0).stop 2024-04-25T01:53:47.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.438+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 msgr2=0x7fef7819eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:47.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.438+0000 7fef7d000700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 0x7fef7819eb60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fef6400ec10 tx=0x7fef6400ef20 comp rx=0 tx=0).stop 2024-04-25T01:53:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.439+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 shutdown_connections 2024-04-25T01:53:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.439+0000 7fef7d000700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fef5c0a2cb0 0x7fef5c0a5170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.439+0000 7fef7d000700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef781071a0 0x7fef781987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.439+0000 7fef7d000700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef78101130 0x7fef7819e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.439+0000 7fef7d000700 1 --2- 172.21.15.102:0/1746720559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef78101af0 0x7fef7819eb60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.439+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 >> 172.21.15.102:0/1746720559 conn(0x7fef780780e0 msgr2=0x7fef781003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.439+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 shutdown_connections 2024-04-25T01:53:47.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:47.439+0000 7fef7d000700 1 -- 172.21.15.102:0/1746720559 wait complete. 2024-04-25T01:53:47.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:47 smithi110 ceph-mon[32848]: pgmap v351: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T01:53:47.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:47 smithi193 ceph-mon[32981]: pgmap v351: 97 pgs: 1 peering, 96 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 7.4 KiB/s, 0 objects/s recovering 2024-04-25T01:53:48.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:48 smithi102 ceph-mon[27796]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-25T01:53:48.358 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:48 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1746720559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:53:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:48 smithi110 ceph-mon[32848]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-25T01:53:48.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:48 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1746720559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:53:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:48 smithi193 ceph-mon[32981]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-25T01:53:48.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:48 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1746720559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T01:53:48.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.526+0000 7f4dafe00700 1 -- 172.21.15.102:0/4107479253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db0102d70 msgr2=0x7f4db0106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:48.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.526+0000 7f4dafe00700 1 --2- 172.21.15.102:0/4107479253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db0102d70 0x7f4db0106dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0099c0 tx=0x7f4d9c009cd0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.526+0000 7f4dafe00700 1 -- 172.21.15.102:0/4107479253 shutdown_connections 2024-04-25T01:53:48.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.526+0000 7f4dafe00700 1 --2- 172.21.15.102:0/4107479253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4db01023c0 0x7f4db01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.526+0000 7f4dafe00700 1 --2- 172.21.15.102:0/4107479253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4db01074a0 0x7f4db010f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.526+0000 7f4dafe00700 1 --2- 172.21.15.102:0/4107479253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db0102d70 0x7f4db0106dc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.526+0000 7f4dafe00700 1 -- 172.21.15.102:0/4107479253 >> 172.21.15.102:0/4107479253 conn(0x7f4db00fdcb0 msgr2=0x7f4db01000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:48.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.527+0000 7f4dafe00700 1 -- 172.21.15.102:0/4107479253 shutdown_connections 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.527+0000 7f4dafe00700 1 -- 172.21.15.102:0/4107479253 wait complete. 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.527+0000 7f4dafe00700 1 Processor -- start 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4dafe00700 1 -- start start 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4dafe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4db01023c0 0x7f4db0074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4dafe00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4db0102d70 0x7f4db0074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4dafe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db01074a0 0x7f4db0074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4dafe00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01124b0 con 0x7f4db01074a0 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4dafe00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4db0112330 con 0x7f4db0102d70 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4dafe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4db0112630 con 0x7f4db01023c0 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4daf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db01074a0 0x7f4db0074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4daf400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db01074a0 0x7f4db0074fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43644/0 (socket says 172.21.15.102:43644) 2024-04-25T01:53:48.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4daea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4db01023c0 0x7f4db0074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4daf400700 1 -- 172.21.15.102:0/3758759475 learned_addr learned my addr 172.21.15.102:0/3758759475 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.528+0000 7f4dae000700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4db0102d70 0x7f4db0074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.529+0000 7f4daf400700 1 -- 172.21.15.102:0/3758759475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4db01023c0 msgr2=0x7f4db0074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.529+0000 7f4daf400700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4db01023c0 0x7f4db0074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.529+0000 7f4daf400700 1 -- 172.21.15.102:0/3758759475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4db0102d70 msgr2=0x7f4db0074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.529+0000 7f4daf400700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4db0102d70 0x7f4db0074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.529+0000 7f4daf400700 1 -- 172.21.15.102:0/3758759475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db01a6e90 con 0x7f4db01074a0 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.529+0000 7f4daf400700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db01074a0 0x7f4db0074fb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4da800ef30 tx=0x7f4da800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.529+0000 7f4da7400700 1 -- 172.21.15.102:0/3758759475 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da80092e0 con 0x7f4db01074a0 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.530+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db01a7120 con 0x7f4db01074a0 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.530+0000 7f4da7400700 1 -- 172.21.15.102:0/3758759475 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4da8004bb0 con 0x7f4db01074a0 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.530+0000 7f4da7400700 1 -- 172.21.15.102:0/3758759475 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da8004280 con 0x7f4db01074a0 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.530+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db01a7630 con 0x7f4db01074a0 2024-04-25T01:53:48.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.531+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4db0068f20 con 0x7f4db01074a0 2024-04-25T01:53:48.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.533+0000 7f4da7400700 1 -- 172.21.15.102:0/3758759475 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4da801b030 con 0x7f4db01074a0 2024-04-25T01:53:48.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.535+0000 7f4da7400700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4d940a2cb0 0x7f4d940a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:48.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.535+0000 7f4da7400700 1 -- 172.21.15.102:0/3758759475 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f4da80cec80 con 0x7f4db01074a0 2024-04-25T01:53:48.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.535+0000 7f4daea00700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4d940a2cb0 0x7f4d940a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:48.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.536+0000 7f4daea00700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4d940a2cb0 0x7f4d940a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009c60 tx=0x7f4da0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:48.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.540+0000 7f4da7400700 1 -- 172.21.15.102:0/3758759475 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4da8022020 con 0x7f4db01074a0 2024-04-25T01:53:48.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.766+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f4db002db30 con 0x7f4db01074a0 2024-04-25T01:53:48.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.766+0000 7f4da7400700 1 -- 172.21.15.102:0/3758759475 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4da8010040 con 0x7f4db01074a0 2024-04-25T01:53:48.768 INFO:teuthology.orchestra.run.smithi102.stdout:296352743463 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4d940a2cb0 msgr2=0x7f4d940a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4d940a2cb0 0x7f4d940a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009c60 tx=0x7f4da0009260 comp rx=0 tx=0).stop 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db01074a0 msgr2=0x7f4db0074fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db01074a0 0x7f4db0074fb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4da800ef30 tx=0x7f4da800c3a0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 shutdown_connections 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4d940a2cb0 0x7f4d940a5170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4db01074a0 0x7f4db0074fb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4db01023c0 0x7f4db0074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 --2- 172.21.15.102:0/3758759475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4db0102d70 0x7f4db0074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 >> 172.21.15.102:0/3758759475 conn(0x7f4db00fdcb0 msgr2=0x7f4db00fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 shutdown_connections 2024-04-25T01:53:48.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:48.769+0000 7f4dafe00700 1 -- 172.21.15.102:0/3758759475 wait complete. 2024-04-25T01:53:49.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:49 smithi102 ceph-mon[27796]: pgmap v352: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:49.419 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:49 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3758759475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:53:49.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:49 smithi110 ceph-mon[32848]: pgmap v352: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:49.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:49 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3758759475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:53:49.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:49 smithi193 ceph-mon[32981]: pgmap v352: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:49.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:49 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3758759475' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T01:53:50.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:50 smithi102 ceph-mon[27796]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-25T01:53:50.417 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:50 smithi102 ceph-mon[27796]: Cluster is now healthy 2024-04-25T01:53:50.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:50 smithi110 ceph-mon[32848]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-25T01:53:50.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:50 smithi110 ceph-mon[32848]: Cluster is now healthy 2024-04-25T01:53:50.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:50 smithi193 ceph-mon[32981]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-25T01:53:50.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:50 smithi193 ceph-mon[32981]: Cluster is now healthy 2024-04-25T01:53:51.312 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:51 smithi102 ceph-mon[27796]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:51.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:51 smithi110 ceph-mon[32848]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:51.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:51 smithi193 ceph-mon[32981]: pgmap v353: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2 op/s; 7.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:51.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.643+0000 7fab42e00700 1 -- 172.21.15.102:0/3871854936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 msgr2=0x7fab3c10f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.643+0000 7fab42e00700 1 --2- 172.21.15.102:0/3871854936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 0x7fab3c10f170 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fab34009990 tx=0x7fab34009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.643+0000 7fab42e00700 1 -- 172.21.15.102:0/3871854936 shutdown_connections 2024-04-25T01:53:51.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.643+0000 7fab42e00700 1 --2- 172.21.15.102:0/3871854936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 0x7fab3c10f170 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.643+0000 7fab42e00700 1 --2- 172.21.15.102:0/3871854936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab3c1024e0 0x7fab3c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.643+0000 7fab42e00700 1 --2- 172.21.15.102:0/3871854936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab3c102e90 0x7fab3c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.643+0000 7fab42e00700 1 -- 172.21.15.102:0/3871854936 >> 172.21.15.102:0/3871854936 conn(0x7fab3c0793e0 msgr2=0x7fab3c0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.644+0000 7fab42e00700 1 -- 172.21.15.102:0/3871854936 shutdown_connections 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.644+0000 7fab42e00700 1 -- 172.21.15.102:0/3871854936 wait complete. 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.645+0000 7fab42e00700 1 Processor -- start 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.645+0000 7fab42e00700 1 -- start start 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.645+0000 7fab42e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab3c1024e0 0x7fab3c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.645+0000 7fab42e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab3c102e90 0x7fab3c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.645+0000 7fab42e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 0x7fab3c1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.645+0000 7fab42e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c1125d0 con 0x7fab3c1075a0 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.645+0000 7fab42e00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c112450 con 0x7fab3c102e90 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.645+0000 7fab42e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab3c112750 con 0x7fab3c1024e0 2024-04-25T01:53:51.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab40e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 0x7fab3c1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab40e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 0x7fab3c1988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43654/0 (socket says 172.21.15.102:43654) 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab40e00700 1 -- 172.21.15.102:0/2880733114 learned_addr learned my addr 172.21.15.102:0/2880733114 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab3b400700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab3c102e90 0x7fab3c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab3be00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab3c1024e0 0x7fab3c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab40e00700 1 -- 172.21.15.102:0/2880733114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab3c1024e0 msgr2=0x7fab3c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab40e00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab3c1024e0 0x7fab3c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab40e00700 1 -- 172.21.15.102:0/2880733114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab3c102e90 msgr2=0x7fab3c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab40e00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab3c102e90 0x7fab3c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab40e00700 1 -- 172.21.15.102:0/2880733114 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3c199010 con 0x7fab3c1075a0 2024-04-25T01:53:51.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.646+0000 7fab3b400700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab3c102e90 0x7fab3c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.647+0000 7fab40e00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 0x7fab3c1988b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab340095b0 tx=0x7fab3400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.647+0000 7fab38c00700 1 -- 172.21.15.102:0/2880733114 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab34004830 con 0x7fab3c1075a0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.647+0000 7fab38c00700 1 -- 172.21.15.102:0/2880733114 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab34005020 con 0x7fab3c1075a0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.647+0000 7fab38c00700 1 -- 172.21.15.102:0/2880733114 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab34003a60 con 0x7fab3c1075a0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.647+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3c1992a0 con 0x7fab3c1075a0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.647+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3c1a5570 con 0x7fab3c1075a0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.648+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab3c1039a0 con 0x7fab3c1075a0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.650+0000 7fab38c00700 1 -- 172.21.15.102:0/2880733114 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fab34009d10 con 0x7fab3c1075a0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.652+0000 7fab38c00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab200a2d30 0x7fab200a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.652+0000 7fab38c00700 1 -- 172.21.15.102:0/2880733114 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fab340cfdb0 con 0x7fab3c1075a0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.653+0000 7fab3be00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab200a2d30 0x7fab200a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.653+0000 7fab3be00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab200a2d30 0x7fab200a51f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab3c079c60 tx=0x7fab2c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.657+0000 7fab38c00700 1 -- 172.21.15.102:0/2880733114 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fab340d4070 con 0x7fab3c1075a0 2024-04-25T01:53:51.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.884+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fab3c02db30 con 0x7fab3c1075a0 2024-04-25T01:53:51.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.884+0000 7fab38c00700 1 -- 172.21.15.102:0/2880733114 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fab34013070 con 0x7fab3c1075a0 2024-04-25T01:53:51.886 INFO:teuthology.orchestra.run.smithi102.stdout:115964117064 2024-04-25T01:53:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.887+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab200a2d30 msgr2=0x7fab200a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.887+0000 7fab42e00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab200a2d30 0x7fab200a51f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab3c079c60 tx=0x7fab2c00b040 comp rx=0 tx=0).stop 2024-04-25T01:53:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.887+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 msgr2=0x7fab3c1988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.887+0000 7fab42e00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 0x7fab3c1988b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fab340095b0 tx=0x7fab3400b880 comp rx=0 tx=0).stop 2024-04-25T01:53:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.888+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 shutdown_connections 2024-04-25T01:53:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.888+0000 7fab42e00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fab200a2d30 0x7fab200a51f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.888+0000 7fab42e00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1075a0 0x7fab3c1988b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.888+0000 7fab42e00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab3c1024e0 0x7fab3c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.888+0000 7fab42e00700 1 --2- 172.21.15.102:0/2880733114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab3c102e90 0x7fab3c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.888+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 >> 172.21.15.102:0/2880733114 conn(0x7fab3c0793e0 msgr2=0x7fab3c100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.888+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 shutdown_connections 2024-04-25T01:53:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:51.888+0000 7fab42e00700 1 -- 172.21.15.102:0/2880733114 wait complete. 2024-04-25T01:53:52.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:52 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2880733114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:53:52.220 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:52.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:52 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2880733114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:53:52.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:52 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2880733114' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T01:53:53.324 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:53 smithi102 ceph-mon[27796]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 6.1 KiB/s, 1 objects/s recovering 2024-04-25T01:53:53.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:53 smithi110 ceph-mon[32848]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 6.1 KiB/s, 1 objects/s recovering 2024-04-25T01:53:53.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:53 smithi193 ceph-mon[32981]: pgmap v354: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s; 6.1 KiB/s, 1 objects/s recovering 2024-04-25T01:53:53.576 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743459 got 296352743463 for osd.9 2024-04-25T01:53:53.576 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:54.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:54 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3384734852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:53:54.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:54 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3384734852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:53:54.499 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117058 got 115964117064 for osd.3 2024-04-25T01:53:54.499 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:54.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:54 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3384734852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T01:53:55.423 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:55 smithi102 ceph-mon[27796]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 5.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:55.487 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:55 smithi110 ceph-mon[32848]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 5.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:55.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:55 smithi193 ceph-mon[32981]: pgmap v355: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 5.5 KiB/s, 1 objects/s recovering 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.822+0000 7f2928000700 1 -- 172.21.15.102:0/1769819212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29201024e0 msgr2=0x7f29201028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.822+0000 7f2928000700 1 --2- 172.21.15.102:0/1769819212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29201024e0 0x7f29201028c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2914009990 tx=0x7f2914009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.822+0000 7f2928000700 1 -- 172.21.15.102:0/1769819212 shutdown_connections 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.822+0000 7f2928000700 1 --2- 172.21.15.102:0/1769819212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2920107570 0x7f292010f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.822+0000 7f2928000700 1 --2- 172.21.15.102:0/1769819212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f29201024e0 0x7f29201028c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.822+0000 7f2928000700 1 --2- 172.21.15.102:0/1769819212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2920102e90 0x7f2920106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.822+0000 7f2928000700 1 -- 172.21.15.102:0/1769819212 >> 172.21.15.102:0/1769819212 conn(0x7f2920079390 msgr2=0x7f29200797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.822+0000 7f2928000700 1 -- 172.21.15.102:0/1769819212 shutdown_connections 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.822+0000 7f2928000700 1 -- 172.21.15.102:0/1769819212 wait complete. 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.823+0000 7f2928000700 1 Processor -- start 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.823+0000 7f2928000700 1 -- start start 2024-04-25T01:53:55.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2928000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29201024e0 0x7f292019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2928000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2920102e90 0x7f292019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2928000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2920107570 0x7f2920198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2928000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2920112550 con 0x7f2920107570 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2928000700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f29201123d0 con 0x7f29201024e0 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2928000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f29201126d0 con 0x7f2920102e90 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2926000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2920107570 0x7f2920198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2925600700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29201024e0 0x7f292019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2925600700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29201024e0 0x7f292019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:52764/0 (socket says 172.21.15.102:52764) 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2925600700 1 -- 172.21.15.102:0/3384734852 learned_addr learned my addr 172.21.15.102:0/3384734852 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.824+0000 7f2924c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2920102e90 0x7f292019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.825+0000 7f2925600700 1 -- 172.21.15.102:0/3384734852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2920102e90 msgr2=0x7f292019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.825+0000 7f2925600700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2920102e90 0x7f292019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.825+0000 7f2925600700 1 -- 172.21.15.102:0/3384734852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2920107570 msgr2=0x7f2920198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.825+0000 7f2925600700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2920107570 0x7f2920198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.825+0000 7f2925600700 1 -- 172.21.15.102:0/3384734852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2920199050 con 0x7f29201024e0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.825+0000 7f2924c00700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2920102e90 0x7f292019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.825+0000 7f2925600700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29201024e0 0x7f292019e6a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2914000c00 tx=0x7f291400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.825+0000 7f291e000700 1 -- 172.21.15.102:0/3384734852 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2914027070 con 0x7f29201024e0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.826+0000 7f291e000700 1 -- 172.21.15.102:0/3384734852 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f291400be70 con 0x7f29201024e0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.826+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29201992e0 con 0x7f29201024e0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.826+0000 7f291e000700 1 -- 172.21.15.102:0/3384734852 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f291400f460 con 0x7f29201024e0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.826+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2920074460 con 0x7f29201024e0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.827+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f292010a790 con 0x7f29201024e0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.829+0000 7f291e000700 1 -- 172.21.15.102:0/3384734852 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f291402a070 con 0x7f29201024e0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.830+0000 7f291e000700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f29080a2cb0 0x7f29080a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.831+0000 7f291e000700 1 -- 172.21.15.102:0/3384734852 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f29140d0510 con 0x7f29201024e0 2024-04-25T01:53:55.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.831+0000 7f2924c00700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f29080a2cb0 0x7f29080a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:55.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.831+0000 7f2924c00700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f29080a2cb0 0x7f29080a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2920101e00 tx=0x7f290c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:55.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:53.833+0000 7f291e000700 1 -- 172.21.15.102:0/3384734852 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2914031070 con 0x7f29201024e0 2024-04-25T01:53:55.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.075+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f2920199d50 con 0x7f29201024e0 2024-04-25T01:53:55.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.075+0000 7f291e000700 1 -- 172.21.15.102:0/3384734852 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f2914013070 con 0x7f29201024e0 2024-04-25T01:53:55.559 INFO:teuthology.orchestra.run.smithi102.stdout:352187318302 2024-04-25T01:53:55.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f29080a2cb0 msgr2=0x7f29080a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f29080a2cb0 0x7f29080a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2920101e00 tx=0x7f290c009260 comp rx=0 tx=0).stop 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29201024e0 msgr2=0x7f292019e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29201024e0 0x7f292019e6a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2914000c00 tx=0x7f291400bab0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 shutdown_connections 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f29080a2cb0 0x7f29080a5170 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29201024e0 0x7f292019e6a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2920107570 0x7f2920198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 --2- 172.21.15.102:0/3384734852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2920102e90 0x7f292019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 >> 172.21.15.102:0/3384734852 conn(0x7f2920079390 msgr2=0x7f29201086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 shutdown_connections 2024-04-25T01:53:55.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:54.078+0000 7f2928000700 1 -- 172.21.15.102:0/3384734852 wait complete. 2024-04-25T01:53:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.091+0000 7fdf41e00700 1 -- 172.21.15.102:0/772192528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c1045c0 msgr2=0x7fdf3c1049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.091+0000 7fdf41e00700 1 --2- 172.21.15.102:0/772192528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c1045c0 0x7fdf3c1049a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdf300099c0 tx=0x7fdf30009cd0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.091+0000 7fdf41e00700 1 -- 172.21.15.102:0/772192528 shutdown_connections 2024-04-25T01:53:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.091+0000 7fdf41e00700 1 --2- 172.21.15.102:0/772192528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf3c1096a0 0x7fdf3c111270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.091+0000 7fdf41e00700 1 --2- 172.21.15.102:0/772192528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c1045c0 0x7fdf3c1049a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.091+0000 7fdf41e00700 1 --2- 172.21.15.102:0/772192528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c104f70 0x7fdf3c108fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.091+0000 7fdf41e00700 1 -- 172.21.15.102:0/772192528 >> 172.21.15.102:0/772192528 conn(0x7fdf3c0ffe50 msgr2=0x7fdf3c102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.091+0000 7fdf41e00700 1 -- 172.21.15.102:0/772192528 shutdown_connections 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.091+0000 7fdf41e00700 1 -- 172.21.15.102:0/772192528 wait complete. 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.092+0000 7fdf41e00700 1 Processor -- start 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.092+0000 7fdf41e00700 1 -- start start 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf41e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf3c1045c0 0x7fdf3c1a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf41e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c104f70 0x7fdf3c1a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf41e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c1096a0 0x7fdf3c19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf41e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c114680 con 0x7fdf3c1045c0 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf41e00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c114500 con 0x7fdf3c1096a0 2024-04-25T01:53:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf41e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c114800 con 0x7fdf3c104f70 2024-04-25T01:53:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf40a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf3c1045c0 0x7fdf3c1a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf3be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c104f70 0x7fdf3c1a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf3be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c104f70 0x7fdf3c1a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52866/0 (socket says 172.21.15.102:52866) 2024-04-25T01:53:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf40a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf3c1045c0 0x7fdf3c1a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48528/0 (socket says 172.21.15.102:48528) 2024-04-25T01:53:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf3be00700 1 -- 172.21.15.102:0/2872394045 learned_addr learned my addr 172.21.15.102:0/2872394045 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf41400700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c1096a0 0x7fdf3c19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf3be00700 1 -- 172.21.15.102:0/2872394045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c1096a0 msgr2=0x7fdf3c19a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf3be00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c1096a0 0x7fdf3c19a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf3be00700 1 -- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf3c1045c0 msgr2=0x7fdf3c1a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf3be00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf3c1045c0 0x7fdf3c1a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf3be00700 1 -- 172.21.15.102:0/2872394045 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3c19afe0 con 0x7fdf3c104f70 2024-04-25T01:53:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.093+0000 7fdf40a00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf3c1045c0 0x7fdf3c1a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.094+0000 7fdf3be00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c104f70 0x7fdf3c1a0c00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800e970 tx=0x7fdf2800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.094+0000 7fdf39600700 1 -- 172.21.15.102:0/2872394045 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28004a30 con 0x7fdf3c104f70 2024-04-25T01:53:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.094+0000 7fdf39600700 1 -- 172.21.15.102:0/2872394045 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf2800cca0 con 0x7fdf3c104f70 2024-04-25T01:53:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.094+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf3c19b2d0 con 0x7fdf3c104f70 2024-04-25T01:53:56.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.094+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf3c1a7590 con 0x7fdf3c104f70 2024-04-25T01:53:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.094+0000 7fdf39600700 1 -- 172.21.15.102:0/2872394045 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf28005680 con 0x7fdf3c104f70 2024-04-25T01:53:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.096+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf1c005360 con 0x7fdf3c104f70 2024-04-25T01:53:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.098+0000 7fdf39600700 1 -- 172.21.15.102:0/2872394045 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fdf2800c7c0 con 0x7fdf3c104f70 2024-04-25T01:53:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.099+0000 7fdf39600700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdf240a2cb0 0x7fdf240a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.100+0000 7fdf39600700 1 -- 172.21.15.102:0/2872394045 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fdf280cf490 con 0x7fdf3c104f70 2024-04-25T01:53:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.100+0000 7fdf40a00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdf240a2cb0 0x7fdf240a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.100+0000 7fdf40a00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdf240a2cb0 0x7fdf240a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdf30000c00 tx=0x7fdf30005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:56.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.101+0000 7fdf39600700 1 -- 172.21.15.102:0/2872394045 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdf28053740 con 0x7fdf3c104f70 2024-04-25T01:53:56.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.332+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fdf1c005920 con 0x7fdf3c104f70 2024-04-25T01:53:56.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.333+0000 7fdf39600700 1 -- 172.21.15.102:0/2872394045 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdf2809c240 con 0x7fdf3c104f70 2024-04-25T01:53:56.335 INFO:teuthology.orchestra.run.smithi102.stdout:77309411408 2024-04-25T01:53:56.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdf240a2cb0 msgr2=0x7fdf240a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:56.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdf240a2cb0 0x7fdf240a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdf30000c00 tx=0x7fdf30005fb0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c104f70 msgr2=0x7fdf3c1a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:56.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c104f70 0x7fdf3c1a0c00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800e970 tx=0x7fdf2800ec80 comp rx=0 tx=0).stop 2024-04-25T01:53:56.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 shutdown_connections 2024-04-25T01:53:56.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fdf240a2cb0 0x7fdf240a5170 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c1096a0 0x7fdf3c19a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf3c1045c0 0x7fdf3c1a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 --2- 172.21.15.102:0/2872394045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf3c104f70 0x7fdf3c1a0c00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:56.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 >> 172.21.15.102:0/2872394045 conn(0x7fdf3c0ffe50 msgr2=0x7fdf3c101b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:56.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 shutdown_connections 2024-04-25T01:53:56.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:56.336+0000 7fdf41e00700 1 -- 172.21.15.102:0/2872394045 wait complete. 2024-04-25T01:53:56.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:56 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2872394045' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:53:56.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:56 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2872394045' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:53:56.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:56 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2872394045' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T01:53:57.115 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:53:57.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:57 smithi102 ceph-mon[27796]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 5.0 KiB/s, 1 objects/s recovering 2024-04-25T01:53:57.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:57 smithi110 ceph-mon[32848]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 5.0 KiB/s, 1 objects/s recovering 2024-04-25T01:53:57.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:57 smithi193 ceph-mon[32981]: pgmap v356: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 5.0 KiB/s, 1 objects/s recovering 2024-04-25T01:53:57.977 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855421 got 150323855426 for osd.4 2024-04-25T01:53:57.978 DEBUG:teuthology.parallel:result is None 2024-04-25T01:53:58.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.251+0000 7f2c0c200700 1 -- 172.21.15.102:0/3302332388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 msgr2=0x7f2c04108ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.251+0000 7f2c0c200700 1 --2- 172.21.15.102:0/3302332388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 0x7f2c04108ce0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0009990 tx=0x7f2bf0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.251+0000 7f2c0c200700 1 -- 172.21.15.102:0/3302332388 shutdown_connections 2024-04-25T01:53:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.251+0000 7f2c0c200700 1 --2- 172.21.15.102:0/3302332388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c0410e860 0x7f2c0410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.251+0000 7f2c0c200700 1 --2- 172.21.15.102:0/3302332388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c04109220 0x7f2c04115730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.251+0000 7f2c0c200700 1 --2- 172.21.15.102:0/3302332388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 0x7f2c04108ce0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.251+0000 7f2c0c200700 1 -- 172.21.15.102:0/3302332388 >> 172.21.15.102:0/3302332388 conn(0x7f2c04075a00 msgr2=0x7f2c04107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.251+0000 7f2c0c200700 1 -- 172.21.15.102:0/3302332388 shutdown_connections 2024-04-25T01:53:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.251+0000 7f2c0c200700 1 -- 172.21.15.102:0/3302332388 wait complete. 2024-04-25T01:53:58.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.252+0000 7f2c0c200700 1 Processor -- start 2024-04-25T01:53:58.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.252+0000 7f2c0c200700 1 -- start start 2024-04-25T01:53:58.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c0c200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 0x7f2c0419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:58.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c0c200700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c04109220 0x7f2c0419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:58.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c0c200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c0410e860 0x7f2c041a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:58.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c0c200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2c04118c00 con 0x7f2c04108860 2024-04-25T01:53:58.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c0c200700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2c04118a80 con 0x7f2c04109220 2024-04-25T01:53:58.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c0c200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2c04118d80 con 0x7f2c0410e860 2024-04-25T01:53:58.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c09800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 0x7f2c0419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:58.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c09800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 0x7f2c0419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48544/0 (socket says 172.21.15.102:48544) 2024-04-25T01:53:58.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c09800700 1 -- 172.21.15.102:0/4166436810 learned_addr learned my addr 172.21.15.102:0/4166436810 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:53:58.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c08e00700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c04109220 0x7f2c0419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:58.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.253+0000 7f2c0a200700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c0410e860 0x7f2c041a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:58.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c09800700 1 -- 172.21.15.102:0/4166436810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c0410e860 msgr2=0x7f2c041a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:58.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c09800700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c0410e860 0x7f2c041a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c09800700 1 -- 172.21.15.102:0/4166436810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c04109220 msgr2=0x7f2c0419f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:58.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c09800700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c04109220 0x7f2c0419f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c09800700 1 -- 172.21.15.102:0/4166436810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c041a4010 con 0x7f2c04108860 2024-04-25T01:53:58.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c08e00700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c04109220 0x7f2c0419f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:53:58.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c09800700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 0x7f2c0419f050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800e9a0 tx=0x7f2bf800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:58.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c02000700 1 -- 172.21.15.102:0/4166436810 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf800ca10 con 0x7f2c04108860 2024-04-25T01:53:58.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c02000700 1 -- 172.21.15.102:0/4166436810 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bf800cbb0 con 0x7f2c04108860 2024-04-25T01:53:58.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.254+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c041a4300 con 0x7f2c04108860 2024-04-25T01:53:58.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.255+0000 7f2c02000700 1 -- 172.21.15.102:0/4166436810 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8018680 con 0x7f2c04108860 2024-04-25T01:53:58.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.255+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c041abb80 con 0x7f2c04108860 2024-04-25T01:53:58.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.256+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c04068f20 con 0x7f2c04108860 2024-04-25T01:53:58.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.258+0000 7f2c02000700 1 -- 172.21.15.102:0/4166436810 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2bf8010430 con 0x7f2c04108860 2024-04-25T01:53:58.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.259+0000 7f2c02000700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2bec0a2cb0 0x7f2bec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:53:58.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.259+0000 7f2c02000700 1 -- 172.21.15.102:0/4166436810 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f2bf80cf830 con 0x7f2c04108860 2024-04-25T01:53:58.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.260+0000 7f2c08e00700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2bec0a2cb0 0x7f2bec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:53:58.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.260+0000 7f2c08e00700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2bec0a2cb0 0x7f2bec0a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2c040778e0 tx=0x7f2bf000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:53:58.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.262+0000 7f2c02000700 1 -- 172.21.15.102:0/4166436810 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2bf809c5e0 con 0x7f2c04108860 2024-04-25T01:53:58.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.486+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f2c0402db30 con 0x7f2c04108860 2024-04-25T01:53:58.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.487+0000 7f2c02000700 1 -- 172.21.15.102:0/4166436810 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2bf809bee0 con 0x7f2c04108860 2024-04-25T01:53:58.487 INFO:teuthology.orchestra.run.smithi102.stdout:55834574933 2024-04-25T01:53:58.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.489+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2bec0a2cb0 msgr2=0x7f2bec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:58.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.489+0000 7f2c0c200700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2bec0a2cb0 0x7f2bec0a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2c040778e0 tx=0x7f2bf000b260 comp rx=0 tx=0).stop 2024-04-25T01:53:58.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.489+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 msgr2=0x7f2c0419f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:53:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.489+0000 7f2c0c200700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 0x7f2c0419f050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800e9a0 tx=0x7f2bf800ecb0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.490+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 shutdown_connections 2024-04-25T01:53:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.490+0000 7f2c0c200700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f2bec0a2cb0 0x7f2bec0a5170 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.490+0000 7f2c0c200700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c0410e860 0x7f2c041a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.490+0000 7f2c0c200700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c04108860 0x7f2c0419f050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.490+0000 7f2c0c200700 1 --2- 172.21.15.102:0/4166436810 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c04109220 0x7f2c0419f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:53:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.490+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 >> 172.21.15.102:0/4166436810 conn(0x7f2c04075a00 msgr2=0x7f2c04075fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:53:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.490+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 shutdown_connections 2024-04-25T01:53:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:53:58.490+0000 7f2c0c200700 1 -- 172.21.15.102:0/4166436810 wait complete. 2024-04-25T01:53:58.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:58 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/4166436810' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:53:58.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:58 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/4166436810' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:53:59.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:58 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/4166436810' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T01:53:59.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:53:59 smithi102 ceph-mon[27796]: pgmap v357: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 59 B/s, 0 objects/s recovering 2024-04-25T01:53:59.934 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:53:59 smithi193 ceph-mon[32981]: pgmap v357: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 59 B/s, 0 objects/s recovering 2024-04-25T01:53:59.987 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:53:59 smithi110 ceph-mon[32848]: pgmap v357: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s; 59 B/s, 0 objects/s recovering 2024-04-25T01:54:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.802+0000 7f2888a00700 1 -- 172.21.15.102:0/194139977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2884102e90 msgr2=0x7f2884106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.802+0000 7f2888a00700 1 --2- 172.21.15.102:0/194139977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2884102e90 0x7f2884106e90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f28700098f0 tx=0x7f2870009c00 comp rx=0 tx=0).stop 2024-04-25T01:54:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.802+0000 7f2888a00700 1 -- 172.21.15.102:0/194139977 shutdown_connections 2024-04-25T01:54:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.802+0000 7f2888a00700 1 --2- 172.21.15.102:0/194139977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2884107570 0x7f288410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.802+0000 7f2888a00700 1 --2- 172.21.15.102:0/194139977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28841024e0 0x7f28841028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.802+0000 7f2888a00700 1 --2- 172.21.15.102:0/194139977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2884102e90 0x7f2884106e90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.802+0000 7f2888a00700 1 -- 172.21.15.102:0/194139977 >> 172.21.15.102:0/194139977 conn(0x7f2884079390 msgr2=0x7f28840797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.802+0000 7f2888a00700 1 -- 172.21.15.102:0/194139977 shutdown_connections 2024-04-25T01:54:00.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.802+0000 7f2888a00700 1 -- 172.21.15.102:0/194139977 wait complete. 2024-04-25T01:54:00.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.804+0000 7f2888a00700 1 Processor -- start 2024-04-25T01:54:00.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.804+0000 7f2888a00700 1 -- start start 2024-04-25T01:54:00.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.804+0000 7f2888a00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28841024e0 0x7f288419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:00.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.804+0000 7f2888a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2884102e90 0x7f288419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:00.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2888a00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2884107570 0x7f28841988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:00.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2888a00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2884112560 con 0x7f28841024e0 2024-04-25T01:54:00.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2888a00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f28841123e0 con 0x7f2884107570 2024-04-25T01:54:00.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2888a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28841126e0 con 0x7f2884102e90 2024-04-25T01:54:00.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28841024e0 0x7f288419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:00.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2884102e90 0x7f288419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:00.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28841024e0 0x7f288419e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48560/0 (socket says 172.21.15.102:48560) 2024-04-25T01:54:00.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881e00700 1 -- 172.21.15.102:0/774835079 learned_addr learned my addr 172.21.15.102:0/774835079 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:00.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2882800700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2884107570 0x7f28841988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:00.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881400700 1 -- 172.21.15.102:0/774835079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2884107570 msgr2=0x7f28841988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:00.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881400700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2884107570 0x7f28841988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881400700 1 -- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28841024e0 msgr2=0x7f288419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881400700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28841024e0 0x7f288419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881400700 1 -- 172.21.15.102:0/774835079 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2884199020 con 0x7f2884102e90 2024-04-25T01:54:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2881e00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28841024e0 0x7f288419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.805+0000 7f2882800700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2884107570 0x7f28841988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:00.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.806+0000 7f2881400700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2884102e90 0x7f288419ec40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2870000c00 tx=0x7f287000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:00.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.807+0000 7f287aa00700 1 -- 172.21.15.102:0/774835079 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870005040 con 0x7f2884102e90 2024-04-25T01:54:00.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.807+0000 7f287aa00700 1 -- 172.21.15.102:0/774835079 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28700056c0 con 0x7f2884102e90 2024-04-25T01:54:00.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.807+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28841992b0 con 0x7f2884102e90 2024-04-25T01:54:00.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.808+0000 7f287aa00700 1 -- 172.21.15.102:0/774835079 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870003a00 con 0x7f2884102e90 2024-04-25T01:54:00.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.808+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28841a55d0 con 0x7f2884102e90 2024-04-25T01:54:00.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.809+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2884068f20 con 0x7f2884102e90 2024-04-25T01:54:00.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.811+0000 7f287aa00700 1 -- 172.21.15.102:0/774835079 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f287000b560 con 0x7f2884102e90 2024-04-25T01:54:00.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.812+0000 7f287aa00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f28680a2d30 0x7f28680a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:00.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.812+0000 7f287aa00700 1 -- 172.21.15.102:0/774835079 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f28700cffd0 con 0x7f2884102e90 2024-04-25T01:54:00.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.812+0000 7f2881e00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f28680a2d30 0x7f28680a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:00.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.813+0000 7f2881e00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f28680a2d30 0x7f28680a51f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2874009750 tx=0x7f2874009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:00.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:00.816+0000 7f287aa00700 1 -- 172.21.15.102:0/774835079 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2870013030 con 0x7f2884102e90 2024-04-25T01:54:01.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.044+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f288402db30 con 0x7f2884102e90 2024-04-25T01:54:01.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.044+0000 7f287aa00700 1 -- 172.21.15.102:0/774835079 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2870021380 con 0x7f2884102e90 2024-04-25T01:54:01.045 INFO:teuthology.orchestra.run.smithi102.stdout:261993005103 2024-04-25T01:54:01.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.047+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f28680a2d30 msgr2=0x7f28680a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:01.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.047+0000 7f2888a00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f28680a2d30 0x7f28680a51f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2874009750 tx=0x7f2874009310 comp rx=0 tx=0).stop 2024-04-25T01:54:01.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.047+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2884102e90 msgr2=0x7f288419ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:01.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.047+0000 7f2888a00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2884102e90 0x7f288419ec40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2870000c00 tx=0x7f287000fa00 comp rx=0 tx=0).stop 2024-04-25T01:54:01.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.048+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 shutdown_connections 2024-04-25T01:54:01.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.048+0000 7f2888a00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f28680a2d30 0x7f28680a51f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.048+0000 7f2888a00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2884107570 0x7f28841988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.048+0000 7f2888a00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28841024e0 0x7f288419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.048+0000 7f2888a00700 1 --2- 172.21.15.102:0/774835079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2884102e90 0x7f288419ec40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.048+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 >> 172.21.15.102:0/774835079 conn(0x7f2884079390 msgr2=0x7f2884100a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:01.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.048+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 shutdown_connections 2024-04-25T01:54:01.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.048+0000 7f2888a00700 1 -- 172.21.15.102:0/774835079 wait complete. 2024-04-25T01:54:01.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.601+0000 7fd767000700 1 -- 172.21.15.102:0/2233175057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd760104e00 msgr2=0x7fd760111310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:01.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.601+0000 7fd767000700 1 --2- 172.21.15.102:0/2233175057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd760104e00 0x7fd760111310 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd75800b0a0 tx=0x7fd75800b3b0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.601+0000 7fd767000700 1 -- 172.21.15.102:0/2233175057 shutdown_connections 2024-04-25T01:54:01.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.601+0000 7fd767000700 1 --2- 172.21.15.102:0/2233175057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76010a4d0 0x7fd76010a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.601+0000 7fd767000700 1 --2- 172.21.15.102:0/2233175057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd760104e00 0x7fd760111310 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.601+0000 7fd767000700 1 --2- 172.21.15.102:0/2233175057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd760104440 0x7fd7601048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.601+0000 7fd767000700 1 -- 172.21.15.102:0/2233175057 >> 172.21.15.102:0/2233175057 conn(0x7fd7600fff70 msgr2=0x7fd760102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.602+0000 7fd767000700 1 -- 172.21.15.102:0/2233175057 shutdown_connections 2024-04-25T01:54:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.602+0000 7fd767000700 1 -- 172.21.15.102:0/2233175057 wait complete. 2024-04-25T01:54:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.603+0000 7fd767000700 1 Processor -- start 2024-04-25T01:54:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.603+0000 7fd767000700 1 -- start start 2024-04-25T01:54:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.603+0000 7fd767000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd760104440 0x7fd7601a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.603+0000 7fd767000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd760104e00 0x7fd7601a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.603+0000 7fd767000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76010a4d0 0x7fd76019aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.603+0000 7fd767000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd760114780 con 0x7fd760104e00 2024-04-25T01:54:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.603+0000 7fd767000700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd760114600 con 0x7fd76010a4d0 2024-04-25T01:54:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.603+0000 7fd767000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd760114900 con 0x7fd760104440 2024-04-25T01:54:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd75fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd760104440 0x7fd7601a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd75fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd760104440 0x7fd7601a0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52928/0 (socket says 172.21.15.102:52928) 2024-04-25T01:54:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd75fe00700 1 -- 172.21.15.102:0/1946357788 learned_addr learned my addr 172.21.15.102:0/1946357788 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd765000700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76010a4d0 0x7fd76019aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd75f400700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd760104e00 0x7fd7601a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd75fe00700 1 -- 172.21.15.102:0/1946357788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76010a4d0 msgr2=0x7fd76019aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd75fe00700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76010a4d0 0x7fd76019aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd75fe00700 1 -- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd760104e00 msgr2=0x7fd7601a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd75fe00700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd760104e00 0x7fd7601a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd75fe00700 1 -- 172.21.15.102:0/1946357788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76019b1a0 con 0x7fd760104440 2024-04-25T01:54:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.604+0000 7fd765000700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76010a4d0 0x7fd76019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.605+0000 7fd75fe00700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd760104440 0x7fd7601a0830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd75000e9a0 tx=0x7fd75000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.605+0000 7fd75cc00700 1 -- 172.21.15.102:0/1946357788 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75000ca10 con 0x7fd760104440 2024-04-25T01:54:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.605+0000 7fd75cc00700 1 -- 172.21.15.102:0/1946357788 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75000cbb0 con 0x7fd760104440 2024-04-25T01:54:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.605+0000 7fd75cc00700 1 -- 172.21.15.102:0/1946357788 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750018680 con 0x7fd760104440 2024-04-25T01:54:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.605+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76019b490 con 0x7fd760104440 2024-04-25T01:54:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.605+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7601a7700 con 0x7fd760104440 2024-04-25T01:54:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.606+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd760068f20 con 0x7fd760104440 2024-04-25T01:54:01.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.608+0000 7fd75cc00700 1 -- 172.21.15.102:0/1946357788 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd750010430 con 0x7fd760104440 2024-04-25T01:54:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.609+0000 7fd75cc00700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7440a2cb0 0x7fd7440a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.610+0000 7fd75cc00700 1 -- 172.21.15.102:0/1946357788 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fd7500cf720 con 0x7fd760104440 2024-04-25T01:54:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.610+0000 7fd75f400700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7440a2cb0 0x7fd7440a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.610+0000 7fd75f400700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7440a2cb0 0x7fd7440a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd760109a20 tx=0x7fd74c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:01.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.612+0000 7fd75cc00700 1 -- 172.21.15.102:0/1946357788 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd75009c4d0 con 0x7fd760104440 2024-04-25T01:54:01.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.842+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fd76002db30 con 0x7fd760104440 2024-04-25T01:54:01.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.843+0000 7fd75cc00700 1 -- 172.21.15.102:0/1946357788 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd75009bdd0 con 0x7fd760104440 2024-04-25T01:54:01.844 INFO:teuthology.orchestra.run.smithi102.stdout:317827579938 2024-04-25T01:54:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.847+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7440a2cb0 msgr2=0x7fd7440a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.847+0000 7fd767000700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7440a2cb0 0x7fd7440a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd760109a20 tx=0x7fd74c009260 comp rx=0 tx=0).stop 2024-04-25T01:54:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.847+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd760104440 msgr2=0x7fd7601a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.847+0000 7fd767000700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd760104440 0x7fd7601a0830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd75000e9a0 tx=0x7fd75000ecb0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.848+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 shutdown_connections 2024-04-25T01:54:01.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.848+0000 7fd767000700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7440a2cb0 0x7fd7440a5170 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.848+0000 7fd767000700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd76010a4d0 0x7fd76019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.848+0000 7fd767000700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd760104440 0x7fd7601a0830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.848+0000 7fd767000700 1 --2- 172.21.15.102:0/1946357788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd760104e00 0x7fd7601a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.848+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 >> 172.21.15.102:0/1946357788 conn(0x7fd7600fff70 msgr2=0x7fd760100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.848+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 shutdown_connections 2024-04-25T01:54:01.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:01.848+0000 7fd767000700 1 -- 172.21.15.102:0/1946357788 wait complete. 2024-04-25T01:54:02.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:01 smithi102 ceph-mon[27796]: pgmap v358: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:02.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:01 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/774835079' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:54:02.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:01 smithi110 ceph-mon[32848]: pgmap v358: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:02.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:01 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/774835079' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:54:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:01 smithi193 ceph-mon[32981]: pgmap v358: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:02.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:01 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/774835079' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T01:54:02.363 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005099 got 261993005103 for osd.8 2024-04-25T01:54:02.363 DEBUG:teuthology.parallel:result is None 2024-04-25T01:54:02.760 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318295 got 352187318302 for osd.11 2024-04-25T01:54:02.760 DEBUG:teuthology.parallel:result is None 2024-04-25T01:54:03.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:02 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1946357788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:54:03.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:02 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1946357788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:54:03.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:02 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1946357788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T01:54:03.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:03 smithi102 ceph-mon[27796]: pgmap v359: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:04.150 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411402 got 77309411408 for osd.2 2024-04-25T01:54:04.150 DEBUG:teuthology.parallel:result is None 2024-04-25T01:54:04.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:03 smithi110 ceph-mon[32848]: pgmap v359: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:04.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:03 smithi193 ceph-mon[32981]: pgmap v359: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:04.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.754+0000 7fb62ac00700 1 -- 172.21.15.102:0/1996653312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 msgr2=0x7fb624074840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:04.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.754+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1996653312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 0x7fb624074840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb614009990 tx=0x7fb614009ca0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.754+0000 7fb62ac00700 1 -- 172.21.15.102:0/1996653312 shutdown_connections 2024-04-25T01:54:04.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.754+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1996653312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 0x7fb624074840 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.754+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1996653312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb62410ec30 0x7fb624111020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.754+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1996653312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb624074d80 0x7fb62410e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.754+0000 7fb62ac00700 1 -- 172.21.15.102:0/1996653312 >> 172.21.15.102:0/1996653312 conn(0x7fb624079390 msgr2=0x7fb6240797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:04.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.754+0000 7fb62ac00700 1 -- 172.21.15.102:0/1996653312 shutdown_connections 2024-04-25T01:54:04.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.754+0000 7fb62ac00700 1 -- 172.21.15.102:0/1996653312 wait complete. 2024-04-25T01:54:04.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.755+0000 7fb62ac00700 1 Processor -- start 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.755+0000 7fb62ac00700 1 -- start start 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb62ac00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 0x7fb6241a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb62ac00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb624074d80 0x7fb6241a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb62ac00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb62410ec30 0x7fb62419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb62ac00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb624114730 con 0x7fb62410ec30 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb62ac00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb6241145b0 con 0x7fb624074460 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb62ac00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb6241148b0 con 0x7fb624074d80 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb623e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 0x7fb6241a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb623e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 0x7fb6241a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:57552/0 (socket says 172.21.15.102:57552) 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb623e00700 1 -- 172.21.15.102:0/1699648209 learned_addr learned my addr 172.21.15.102:0/1699648209 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb628c00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb62410ec30 0x7fb62419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.756+0000 7fb623400700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb624074d80 0x7fb6241a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb623e00700 1 -- 172.21.15.102:0/1699648209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb624074d80 msgr2=0x7fb6241a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb623e00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb624074d80 0x7fb6241a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb623e00700 1 -- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb62410ec30 msgr2=0x7fb62419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb623e00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb62410ec30 0x7fb62419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb623e00700 1 -- 172.21.15.102:0/1699648209 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62419b1f0 con 0x7fb624074460 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb623e00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 0x7fb6241a0880 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb61400bae0 tx=0x7fb61400bb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb620c00700 1 -- 172.21.15.102:0/1699648209 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614026070 con 0x7fb624074460 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb620c00700 1 -- 172.21.15.102:0/1699648209 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb614003d70 con 0x7fb624074460 2024-04-25T01:54:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62419b480 con 0x7fb624074460 2024-04-25T01:54:04.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb620c00700 1 -- 172.21.15.102:0/1699648209 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614021410 con 0x7fb624074460 2024-04-25T01:54:04.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.757+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6241a7750 con 0x7fb624074460 2024-04-25T01:54:04.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.759+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb624068f20 con 0x7fb624074460 2024-04-25T01:54:04.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.761+0000 7fb620c00700 1 -- 172.21.15.102:0/1699648209 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb614029070 con 0x7fb624074460 2024-04-25T01:54:04.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.762+0000 7fb620c00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6080a2cb0 0x7fb6080a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:04.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.762+0000 7fb620c00700 1 -- 172.21.15.102:0/1699648209 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fb6140d04d0 con 0x7fb624074460 2024-04-25T01:54:04.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.762+0000 7fb623400700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6080a2cb0 0x7fb6080a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:04.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.763+0000 7fb623400700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6080a2cb0 0x7fb6080a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb624110930 tx=0x7fb610009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:04.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.764+0000 7fb620c00700 1 -- 172.21.15.102:0/1699648209 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb614030080 con 0x7fb624074460 2024-04-25T01:54:04.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.993+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fb62402db30 con 0x7fb624074460 2024-04-25T01:54:04.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.994+0000 7fb620c00700 1 -- 172.21.15.102:0/1699648209 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb614013070 con 0x7fb624074460 2024-04-25T01:54:04.995 INFO:teuthology.orchestra.run.smithi102.stdout:171798691904 2024-04-25T01:54:04.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6080a2cb0 msgr2=0x7fb6080a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:04.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6080a2cb0 0x7fb6080a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb624110930 tx=0x7fb610009260 comp rx=0 tx=0).stop 2024-04-25T01:54:04.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 msgr2=0x7fb6241a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:04.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 0x7fb6241a0880 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb61400bae0 tx=0x7fb61400bb10 comp rx=0 tx=0).stop 2024-04-25T01:54:04.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 shutdown_connections 2024-04-25T01:54:04.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb6080a2cb0 0x7fb6080a5170 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb624074460 0x7fb6241a0880 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb62410ec30 0x7fb62419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 --2- 172.21.15.102:0/1699648209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb624074d80 0x7fb6241a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:04.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 >> 172.21.15.102:0/1699648209 conn(0x7fb624079390 msgr2=0x7fb6240ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:04.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 shutdown_connections 2024-04-25T01:54:04.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:04.997+0000 7fb62ac00700 1 -- 172.21.15.102:0/1699648209 wait complete. 2024-04-25T01:54:06.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:05 smithi102 ceph-mon[27796]: pgmap v360: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:06.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:05 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1699648209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:54:06.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.158+0000 7f259a800700 1 -- 172.21.15.102:0/34713614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2594074d80 msgr2=0x7f2594112ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.158+0000 7f259a800700 1 --2- 172.21.15.102:0/34713614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2594074d80 0x7f2594112ec0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2580009990 tx=0x7f2580009ca0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.159+0000 7f259a800700 1 -- 172.21.15.102:0/34713614 shutdown_connections 2024-04-25T01:54:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.159+0000 7f259a800700 1 --2- 172.21.15.102:0/34713614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2594074460 0x7f2594074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.159+0000 7f259a800700 1 --2- 172.21.15.102:0/34713614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2594113400 0x7f25941157f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.159+0000 7f259a800700 1 --2- 172.21.15.102:0/34713614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2594074d80 0x7f2594112ec0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.159+0000 7f259a800700 1 -- 172.21.15.102:0/34713614 >> 172.21.15.102:0/34713614 conn(0x7f2594079390 msgr2=0x7f25940797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.159+0000 7f259a800700 1 -- 172.21.15.102:0/34713614 shutdown_connections 2024-04-25T01:54:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.159+0000 7f259a800700 1 -- 172.21.15.102:0/34713614 wait complete. 2024-04-25T01:54:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.160+0000 7f259a800700 1 Processor -- start 2024-04-25T01:54:06.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.160+0000 7f259a800700 1 -- start start 2024-04-25T01:54:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.160+0000 7f259a800700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2594074460 0x7f259419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.160+0000 7f259a800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2594074d80 0x7f259419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.160+0000 7f259a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2594113400 0x7f25941a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.160+0000 7f259a800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2594118b90 con 0x7f2594074d80 2024-04-25T01:54:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.160+0000 7f259a800700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2594118a10 con 0x7f2594074460 2024-04-25T01:54:06.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.160+0000 7f259a800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2594118d10 con 0x7f2594113400 2024-04-25T01:54:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.161+0000 7f2593400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2594074460 0x7f259419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.161+0000 7f2593400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2594074460 0x7f259419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:57562/0 (socket says 172.21.15.102:57562) 2024-04-25T01:54:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.161+0000 7f2593400700 1 -- 172.21.15.102:0/2104133196 learned_addr learned my addr 172.21.15.102:0/2104133196 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.161+0000 7f2592a00700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2594074d80 0x7f259419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:06.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f2593e00700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2594113400 0x7f25941a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f2593400700 1 -- 172.21.15.102:0/2104133196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2594113400 msgr2=0x7f25941a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f2593400700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2594113400 0x7f25941a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f2593400700 1 -- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2594074d80 msgr2=0x7f259419f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f2593400700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2594074d80 0x7f259419f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f2593400700 1 -- 172.21.15.102:0/2104133196 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25941a4010 con 0x7f2594074460 2024-04-25T01:54:06.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f2593e00700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2594113400 0x7f25941a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:54:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f2593400700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2594074460 0x7f259419f000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f258400eb30 tx=0x7f258400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f258be00700 1 -- 172.21.15.102:0/2104133196 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258400cc90 con 0x7f2594074460 2024-04-25T01:54:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f258be00700 1 -- 172.21.15.102:0/2104133196 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2584004500 con 0x7f2594074460 2024-04-25T01:54:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f258be00700 1 -- 172.21.15.102:0/2104133196 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584010430 con 0x7f2594074460 2024-04-25T01:54:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.162+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25941a4300 con 0x7f2594074460 2024-04-25T01:54:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.163+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25941abb80 con 0x7f2594074460 2024-04-25T01:54:06.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.164+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2594068f20 con 0x7f2594074460 2024-04-25T01:54:06.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.166+0000 7f258be00700 1 -- 172.21.15.102:0/2104133196 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2584003f80 con 0x7f2594074460 2024-04-25T01:54:06.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.167+0000 7f258be00700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f25780a2d30 0x7f25780a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:06.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.167+0000 7f258be00700 1 -- 172.21.15.102:0/2104133196 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f25840cf480 con 0x7f2594074460 2024-04-25T01:54:06.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.167+0000 7f2592a00700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f25780a2d30 0x7f25780a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:06.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.168+0000 7f2592a00700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f25780a2d30 0x7f25780a51f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2580009750 tx=0x7f258000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:06.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.171+0000 7f258be00700 1 -- 172.21.15.102:0/2104133196 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f258409c230 con 0x7f2594074460 2024-04-25T01:54:06.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:05 smithi110 ceph-mon[32848]: pgmap v360: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:06.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:05 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1699648209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:54:06.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:05 smithi193 ceph-mon[32981]: pgmap v360: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:06.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:05 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1699648209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T01:54:06.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.399+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f25941a0360 con 0x7f2594074460 2024-04-25T01:54:06.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.399+0000 7f258be00700 1 -- 172.21.15.102:0/2104133196 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f258409bb30 con 0x7f2594074460 2024-04-25T01:54:06.400 INFO:teuthology.orchestra.run.smithi102.stdout:206158430266 2024-04-25T01:54:06.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.402+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f25780a2d30 msgr2=0x7f25780a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.402+0000 7f259a800700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f25780a2d30 0x7f25780a51f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2580009750 tx=0x7f258000b260 comp rx=0 tx=0).stop 2024-04-25T01:54:06.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.402+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2594074460 msgr2=0x7f259419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:06.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.402+0000 7f259a800700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2594074460 0x7f259419f000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f258400eb30 tx=0x7f258400ee40 comp rx=0 tx=0).stop 2024-04-25T01:54:06.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.403+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 shutdown_connections 2024-04-25T01:54:06.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.403+0000 7f259a800700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f25780a2d30 0x7f25780a51f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.403+0000 7f259a800700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2594074460 0x7f259419f000 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.403+0000 7f259a800700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2594113400 0x7f25941a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.403+0000 7f259a800700 1 --2- 172.21.15.102:0/2104133196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2594074d80 0x7f259419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:06.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.403+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 >> 172.21.15.102:0/2104133196 conn(0x7f2594079390 msgr2=0x7f25941121c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:06.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.403+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 shutdown_connections 2024-04-25T01:54:06.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:06.403+0000 7f259a800700 1 -- 172.21.15.102:0/2104133196 wait complete. 2024-04-25T01:54:06.671 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579931 got 317827579938 for osd.10 2024-04-25T01:54:06.671 DEBUG:teuthology.parallel:result is None 2024-04-25T01:54:06.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:06 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2104133196' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:54:07.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:06 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2104133196' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:54:07.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:06 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2104133196' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T01:54:07.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:07 smithi102 ceph-mon[27796]: pgmap v361: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:08.025 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430260 got 206158430266 for osd.6 2024-04-25T01:54:08.025 DEBUG:teuthology.parallel:result is None 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.001+0000 7fa6e3800700 1 -- 172.21.15.102:0/3323316412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6dc102e90 msgr2=0x7fa6dc106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.001+0000 7fa6e3800700 1 --2- 172.21.15.102:0/3323316412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6dc102e90 0x7fa6dc106e90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009990 tx=0x7fa6c8009ca0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.001+0000 7fa6e3800700 1 -- 172.21.15.102:0/3323316412 shutdown_connections 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.001+0000 7fa6e3800700 1 --2- 172.21.15.102:0/3323316412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6dc107570 0x7fa6dc10f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.001+0000 7fa6e3800700 1 --2- 172.21.15.102:0/3323316412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6dc1024e0 0x7fa6dc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.001+0000 7fa6e3800700 1 --2- 172.21.15.102:0/3323316412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6dc102e90 0x7fa6dc106e90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.001+0000 7fa6e3800700 1 -- 172.21.15.102:0/3323316412 >> 172.21.15.102:0/3323316412 conn(0x7fa6dc079390 msgr2=0x7fa6dc0797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.001+0000 7fa6e3800700 1 -- 172.21.15.102:0/3323316412 shutdown_connections 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.001+0000 7fa6e3800700 1 -- 172.21.15.102:0/3323316412 wait complete. 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.002+0000 7fa6e3800700 1 Processor -- start 2024-04-25T01:54:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e3800700 1 -- start start 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e3800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6dc1024e0 0x7fa6dc19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e3800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6dc102e90 0x7fa6dc19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e3800700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6dc107570 0x7fa6dc198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e3800700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc112580 con 0x7fa6dc102e90 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e3800700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc112400 con 0x7fa6dc107570 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e3800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa6dc112700 con 0x7fa6dc1024e0 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6dbe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6dc102e90 0x7fa6dc19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e0e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6dc1024e0 0x7fa6dc19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e1800700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6dc107570 0x7fa6dc198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:08.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.003+0000 7fa6e0e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6dc1024e0 0x7fa6dc19e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40324/0 (socket says 172.21.15.102:40324) 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.004+0000 7fa6e0e00700 1 -- 172.21.15.102:0/737631825 learned_addr learned my addr 172.21.15.102:0/737631825 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.004+0000 7fa6e0e00700 1 -- 172.21.15.102:0/737631825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6dc107570 msgr2=0x7fa6dc198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.004+0000 7fa6e0e00700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6dc107570 0x7fa6dc198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.004+0000 7fa6e0e00700 1 -- 172.21.15.102:0/737631825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6dc102e90 msgr2=0x7fa6dc19ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.004+0000 7fa6e0e00700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6dc102e90 0x7fa6dc19ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.004+0000 7fa6e0e00700 1 -- 172.21.15.102:0/737631825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6dc199060 con 0x7fa6dc1024e0 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.004+0000 7fa6e0e00700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6dc1024e0 0x7fa6dc19e740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000ef10 tx=0x7fa6d000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.004+0000 7fa6d9600700 1 -- 172.21.15.102:0/737631825 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d000c9c0 con 0x7fa6dc1024e0 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.005+0000 7fa6d9600700 1 -- 172.21.15.102:0/737631825 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6d0019070 con 0x7fa6dc1024e0 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.005+0000 7fa6d9600700 1 -- 172.21.15.102:0/737631825 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d0007e60 con 0x7fa6dc1024e0 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.005+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6dc1992f0 con 0x7fa6dc1024e0 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.005+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6dc1a5610 con 0x7fa6dc1024e0 2024-04-25T01:54:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.006+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6dc068f20 con 0x7fa6dc1024e0 2024-04-25T01:54:08.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.008+0000 7fa6d9600700 1 -- 172.21.15.102:0/737631825 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fa6d0012050 con 0x7fa6dc1024e0 2024-04-25T01:54:08.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.009+0000 7fa6d9600700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa6c40a2cb0 0x7fa6c40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:08.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.010+0000 7fa6d9600700 1 -- 172.21.15.102:0/737631825 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fa6d00cf3e0 con 0x7fa6dc1024e0 2024-04-25T01:54:08.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.010+0000 7fa6dbe00700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa6c40a2cb0 0x7fa6c40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:08.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.011+0000 7fa6dbe00700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa6c40a2cb0 0x7fa6c40a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc100de0 tx=0x7fa6c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:08.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.012+0000 7fa6d9600700 1 -- 172.21.15.102:0/737631825 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa6d0010040 con 0x7fa6dc1024e0 2024-04-25T01:54:08.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:07 smithi110 ceph-mon[32848]: pgmap v361: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:08.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.238+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fa6dc02db30 con 0x7fa6dc1024e0 2024-04-25T01:54:08.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.239+0000 7fa6d9600700 1 -- 172.21.15.102:0/737631825 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa6d009c190 con 0x7fa6dc1024e0 2024-04-25T01:54:08.239 INFO:teuthology.orchestra.run.smithi102.stdout:227633266742 2024-04-25T01:54:08.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa6c40a2cb0 msgr2=0x7fa6c40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:08.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa6c40a2cb0 0x7fa6c40a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc100de0 tx=0x7fa6c8005fd0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6dc1024e0 msgr2=0x7fa6dc19e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:08.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6dc1024e0 0x7fa6dc19e740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000ef10 tx=0x7fa6d000c3a0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 shutdown_connections 2024-04-25T01:54:08.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa6c40a2cb0 0x7fa6c40a5170 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6dc107570 0x7fa6dc198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6dc1024e0 0x7fa6dc19e740 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 --2- 172.21.15.102:0/737631825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6dc102e90 0x7fa6dc19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:08.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 >> 172.21.15.102:0/737631825 conn(0x7fa6dc079390 msgr2=0x7fa6dc101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:08.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 shutdown_connections 2024-04-25T01:54:08.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:08.242+0000 7fa6e3800700 1 -- 172.21.15.102:0/737631825 wait complete. 2024-04-25T01:54:08.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:07 smithi193 ceph-mon[32981]: pgmap v361: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:08.612 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574926 got 55834574933 for osd.1 2024-04-25T01:54:08.612 DEBUG:teuthology.parallel:result is None 2024-04-25T01:54:08.994 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:09.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:08 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/737631825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:54:09.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:08 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/737631825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:54:09.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:08 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/737631825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T01:54:10.128 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:09 smithi102 ceph-mon[27796]: pgmap v362: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:10.132 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266736 got 227633266742 for osd.7 2024-04-25T01:54:10.132 DEBUG:teuthology.parallel:result is None 2024-04-25T01:54:10.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:09 smithi110 ceph-mon[32848]: pgmap v362: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:10.243 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691899 got 171798691904 for osd.5 2024-04-25T01:54:10.243 DEBUG:teuthology.parallel:result is None 2024-04-25T01:54:10.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:09 smithi193 ceph-mon[32981]: pgmap v362: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:11.142 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:10 smithi102 ceph-mon[27796]: pgmap v363: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:11.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:10 smithi110 ceph-mon[32848]: pgmap v363: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:11.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:10 smithi193 ceph-mon[32981]: pgmap v363: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:12.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.299+0000 7f9843200700 1 -- 172.21.15.102:0/4168357125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 msgr2=0x7f983c10e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.299+0000 7f9843200700 1 --2- 172.21.15.102:0/4168357125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 0x7f983c10e460 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f982c009990 tx=0x7f982c009ca0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.300+0000 7f9843200700 1 -- 172.21.15.102:0/4168357125 shutdown_connections 2024-04-25T01:54:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.300+0000 7f9843200700 1 --2- 172.21.15.102:0/4168357125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f983c074660 0x7f983c074a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.300+0000 7f9843200700 1 --2- 172.21.15.102:0/4168357125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f983c10e9a0 0x7f983c110e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.300+0000 7f9843200700 1 --2- 172.21.15.102:0/4168357125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 0x7f983c10e460 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.300+0000 7f9843200700 1 -- 172.21.15.102:0/4168357125 >> 172.21.15.102:0/4168357125 conn(0x7f983c0fdcb0 msgr2=0x7f983c1000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.300+0000 7f9843200700 1 -- 172.21.15.102:0/4168357125 shutdown_connections 2024-04-25T01:54:12.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.300+0000 7f9843200700 1 -- 172.21.15.102:0/4168357125 wait complete. 2024-04-25T01:54:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.301+0000 7f9843200700 1 Processor -- start 2024-04-25T01:54:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.301+0000 7f9843200700 1 -- start start 2024-04-25T01:54:12.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9843200700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f983c074660 0x7f983c1a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9843200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 0x7f983c1a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9843200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f983c10e9a0 0x7f983c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9843200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f983c114680 con 0x7f983c10e9a0 2024-04-25T01:54:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9843200700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f983c114500 con 0x7f983c074660 2024-04-25T01:54:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9843200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f983c114800 con 0x7f983c074f80 2024-04-25T01:54:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9841400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 0x7f983c1a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9841e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f983c074660 0x7f983c1a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:12.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9841400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 0x7f983c1a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40336/0 (socket says 172.21.15.102:40336) 2024-04-25T01:54:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9841400700 1 -- 172.21.15.102:0/842508362 learned_addr learned my addr 172.21.15.102:0/842508362 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.302+0000 7f9842800700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f983c10e9a0 0x7f983c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:12.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9841400700 1 -- 172.21.15.102:0/842508362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f983c074660 msgr2=0x7f983c1a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9841400700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f983c074660 0x7f983c1a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9841400700 1 -- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f983c10e9a0 msgr2=0x7f983c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9841400700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f983c10e9a0 0x7f983c19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9841400700 1 -- 172.21.15.102:0/842508362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983c19b0a0 con 0x7f983c074f80 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9841400700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 0x7f983c1a0cc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f982c00bae0 tx=0x7f982c00bb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9836a00700 1 -- 172.21.15.102:0/842508362 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c026070 con 0x7f983c074f80 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f983c19b330 con 0x7f983c074f80 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9836a00700 1 -- 172.21.15.102:0/842508362 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f982c003d70 con 0x7f983c074f80 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9836a00700 1 -- 172.21.15.102:0/842508362 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c021410 con 0x7f983c074f80 2024-04-25T01:54:12.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.303+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f983c1a7650 con 0x7f983c074f80 2024-04-25T01:54:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.305+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f983c068f20 con 0x7f983c074f80 2024-04-25T01:54:12.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.306+0000 7f9836a00700 1 -- 172.21.15.102:0/842508362 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f982c029070 con 0x7f983c074f80 2024-04-25T01:54:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.308+0000 7f9836a00700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f98240a2cb0 0x7f98240a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:12.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.308+0000 7f9836a00700 1 -- 172.21.15.102:0/842508362 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f982c0cffb0 con 0x7f983c074f80 2024-04-25T01:54:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.308+0000 7f9841e00700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f98240a2cb0 0x7f98240a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.309+0000 7f9841e00700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f98240a2cb0 0x7f98240a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f983c19ba80 tx=0x7f9830009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.312+0000 7f9836a00700 1 -- 172.21.15.102:0/842508362 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f982c05d740 con 0x7f983c074f80 2024-04-25T01:54:12.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.547+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f983c19be70 con 0x7f983c074f80 2024-04-25T01:54:12.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.548+0000 7f9836a00700 1 -- 172.21.15.102:0/842508362 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f982c013070 con 0x7f983c074f80 2024-04-25T01:54:12.548 INFO:teuthology.orchestra.run.smithi102.stdout:34359738460 2024-04-25T01:54:12.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.550+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f98240a2cb0 msgr2=0x7f98240a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:12.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.550+0000 7f9843200700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f98240a2cb0 0x7f98240a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f983c19ba80 tx=0x7f9830009260 comp rx=0 tx=0).stop 2024-04-25T01:54:12.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 msgr2=0x7f983c1a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:12.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 0x7f983c1a0cc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f982c00bae0 tx=0x7f982c00bb10 comp rx=0 tx=0).stop 2024-04-25T01:54:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 shutdown_connections 2024-04-25T01:54:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f98240a2cb0 0x7f98240a5170 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f983c074660 0x7f983c1a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f983c10e9a0 0x7f983c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 --2- 172.21.15.102:0/842508362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f983c074f80 0x7f983c1a0cc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 >> 172.21.15.102:0/842508362 conn(0x7f983c0fdcb0 msgr2=0x7f983c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 shutdown_connections 2024-04-25T01:54:12.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:12.551+0000 7f9843200700 1 -- 172.21.15.102:0/842508362 wait complete. 2024-04-25T01:54:13.563 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738454 got 34359738460 for osd.0 2024-04-25T01:54:13.564 DEBUG:teuthology.parallel:result is None 2024-04-25T01:54:13.564 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T01:54:13.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:54:13.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:13 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:54:13.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:13 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:54:13.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:13 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/842508362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:54:13.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:13 smithi102 ceph-mon[27796]: pgmap v364: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:13.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:13 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:54:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:13 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:54:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:13 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/842508362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:54:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:13 smithi110 ceph-mon[32848]: pgmap v364: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:13.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:13 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:54:13.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:13 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:54:13.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:13 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/842508362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T01:54:13.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:13 smithi193 ceph-mon[32981]: pgmap v364: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:14.010 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:14 smithi110 ceph-mon[32848]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T01:54:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:54:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:54:14.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:14 smithi110 ceph-mon[32848]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:54:14.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:14 smithi193 ceph-mon[32981]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T01:54:14.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:54:14.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:54:14.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:14 smithi193 ceph-mon[32981]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:54:14.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:14 smithi102 ceph-mon[27796]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-25T01:54:14.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:54:14.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:54:14.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:14 smithi102 ceph-mon[27796]: from='mgr.14152 172.21.15.102:0/2137057640' entity='mgr.x' 2024-04-25T01:54:15.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:15 smithi102 ceph-mon[27796]: pgmap v365: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:15.737 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:15 smithi110 ceph-mon[32848]: pgmap v365: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:15.755 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:15 smithi193 ceph-mon[32981]: pgmap v365: 97 pgs: 97 active+clean; 640 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:16.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.227+0000 7fbbcee00700 1 -- 172.21.15.102:0/1281504776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104e10 msgr2=0x7fbbc8111320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.227+0000 7fbbcee00700 1 --2- 172.21.15.102:0/1281504776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104e10 0x7fbbc8111320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009f90 tx=0x7fbbc000b330 comp rx=0 tx=0).stop 2024-04-25T01:54:16.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.227+0000 7fbbcee00700 1 -- 172.21.15.102:0/1281504776 shutdown_connections 2024-04-25T01:54:16.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.227+0000 7fbbcee00700 1 --2- 172.21.15.102:0/1281504776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104e10 0x7fbbc8111320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.227+0000 7fbbcee00700 1 --2- 172.21.15.102:0/1281504776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbc810a4e0 0x7fbbc810a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.227+0000 7fbbcee00700 1 --2- 172.21.15.102:0/1281504776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc8104450 0x7fbbc81048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.227+0000 7fbbcee00700 1 -- 172.21.15.102:0/1281504776 >> 172.21.15.102:0/1281504776 conn(0x7fbbc80fff40 msgr2=0x7fbbc8102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:16.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.227+0000 7fbbcee00700 1 -- 172.21.15.102:0/1281504776 shutdown_connections 2024-04-25T01:54:16.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.227+0000 7fbbcee00700 1 -- 172.21.15.102:0/1281504776 wait complete. 2024-04-25T01:54:16.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.228+0000 7fbbcee00700 1 Processor -- start 2024-04-25T01:54:16.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.228+0000 7fbbcee00700 1 -- start start 2024-04-25T01:54:16.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbcee00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104450 0x7fbbc81a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbcee00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbc8104e10 0x7fbbc81a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbcee00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc810a4e0 0x7fbbc819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbcee00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc8114790 con 0x7fbbc810a4e0 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbcee00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc8114610 con 0x7fbbc8104450 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbcee00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc8114910 con 0x7fbbc8104e10 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbc7e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104450 0x7fbbc81a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbc7e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104450 0x7fbbc81a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56482/0 (socket says 172.21.15.102:56482) 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbc7e00700 1 -- 172.21.15.102:0/4074072665 learned_addr learned my addr 172.21.15.102:0/4074072665 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbcce00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc810a4e0 0x7fbbc819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbc7400700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbc8104e10 0x7fbbc81a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbc7e00700 1 -- 172.21.15.102:0/4074072665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbc8104e10 msgr2=0x7fbbc81a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbc7e00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbc8104e10 0x7fbbc81a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbc7e00700 1 -- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc810a4e0 msgr2=0x7fbbc819aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.229+0000 7fbbc7e00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc810a4e0 0x7fbbc819aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.230+0000 7fbbc7e00700 1 -- 172.21.15.102:0/4074072665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc819b1b0 con 0x7fbbc8104450 2024-04-25T01:54:16.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.230+0000 7fbbc7e00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104450 0x7fbbc81a0840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800e9c0 tx=0x7fbbb800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:16.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.230+0000 7fbbc4c00700 1 -- 172.21.15.102:0/4074072665 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb800cbd0 con 0x7fbbc8104450 2024-04-25T01:54:16.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.230+0000 7fbbc4c00700 1 -- 172.21.15.102:0/4074072665 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb800cd70 con 0x7fbbc8104450 2024-04-25T01:54:16.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.230+0000 7fbbc4c00700 1 -- 172.21.15.102:0/4074072665 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8005660 con 0x7fbbc8104450 2024-04-25T01:54:16.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.230+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc819b4a0 con 0x7fbbc8104450 2024-04-25T01:54:16.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.230+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc81a7710 con 0x7fbbc8104450 2024-04-25T01:54:16.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.232+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc8068f20 con 0x7fbbc8104450 2024-04-25T01:54:16.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.233+0000 7fbbc4c00700 1 -- 172.21.15.102:0/4074072665 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbbb8004750 con 0x7fbbc8104450 2024-04-25T01:54:16.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.235+0000 7fbbc4c00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbac0a2ca0 0x7fbbac0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:16.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.235+0000 7fbbc4c00700 1 -- 172.21.15.102:0/4074072665 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fbbb80cf100 con 0x7fbbc8104450 2024-04-25T01:54:16.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.235+0000 7fbbc7400700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbac0a2ca0 0x7fbbac0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:16.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.236+0000 7fbbc7400700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbac0a2ca0 0x7fbbac0a5160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8109970 tx=0x7fbbb4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:16.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.239+0000 7fbbc4c00700 1 -- 172.21.15.102:0/4074072665 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbbb809c010 con 0x7fbbc8104450 2024-04-25T01:54:16.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.457+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbbc8062670 con 0x7fbbac0a2ca0 2024-04-25T01:54:16.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.481+0000 7fbbc4c00700 1 -- 172.21.15.102:0/4074072665 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258771 (secure 0 0 0) 0x7fbbc8062670 con 0x7fbbac0a2ca0 2024-04-25T01:54:16.490 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.488+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbac0a2ca0 msgr2=0x7fbbac0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.488+0000 7fbbcee00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbac0a2ca0 0x7fbbac0a5160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8109970 tx=0x7fbbb4009260 comp rx=0 tx=0).stop 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.488+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104450 msgr2=0x7fbbc81a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.488+0000 7fbbcee00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104450 0x7fbbc81a0840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800e9c0 tx=0x7fbbb800ecd0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.489+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 shutdown_connections 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.489+0000 7fbbcee00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbbac0a2ca0 0x7fbbac0a5160 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.489+0000 7fbbcee00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbbc8104450 0x7fbbc81a0840 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.489+0000 7fbbcee00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc810a4e0 0x7fbbc819aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.489+0000 7fbbcee00700 1 --2- 172.21.15.102:0/4074072665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbbc8104e10 0x7fbbc81a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.489+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 >> 172.21.15.102:0/4074072665 conn(0x7fbbc80fff40 msgr2=0x7fbbc8100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.489+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 shutdown_connections 2024-04-25T01:54:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:16.489+0000 7fbbcee00700 1 -- 172.21.15.102:0/4074072665 wait complete. 2024-04-25T01:54:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-25T01:54:17.464 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-04-25T01:54:14.758303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655649,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":649,"num_read_kb":441,"num_write":339,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1242498,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379224,"kb_used_data":13356,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562600,"statfs":{"total":1151940427776,"available":1151552102400,"internally_reserved":0,"allocated":13676544,"data_stored":7514736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":374453563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005437"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.857461+0000","last_change":"2024-04-25T01:52:44.102143+0000","last_active":"2024-04-25T01:53:43.857461+0000","last_peered":"2024-04-25T01:53:43.857461+0000","last_clean":"2024-04-25T01:53:43.857461+0000","last_became_active":"2024-04-25T01:52:44.101810+0000","last_became_peered":"2024-04-25T01:52:44.101810+0000","last_unstale":"2024-04-25T01:53:43.857461+0000","last_undegraded":"2024-04-25T01:53:43.857461+0000","last_fullsized":"2024-04-25T01:53:43.857461+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:02.356294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.870183+0000","last_change":"2024-04-25T01:52:44.103074+0000","last_active":"2024-04-25T01:53:43.870183+0000","last_peered":"2024-04-25T01:53:43.870183+0000","last_clean":"2024-04-25T01:53:43.870183+0000","last_became_active":"2024-04-25T01:52:44.102103+0000","last_became_peered":"2024-04-25T01:52:44.102103+0000","last_unstale":"2024-04-25T01:53:43.870183+0000","last_undegraded":"2024-04-25T01:53:43.870183+0000","last_fullsized":"2024-04-25T01:53:43.870183+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:36:43.016756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.868468+0000","last_change":"2024-04-25T01:52:44.101467+0000","last_active":"2024-04-25T01:53:43.868468+0000","last_peered":"2024-04-25T01:53:43.868468+0000","last_clean":"2024-04-25T01:53:43.868468+0000","last_became_active":"2024-04-25T01:52:44.100911+0000","last_became_peered":"2024-04-25T01:52:44.100911+0000","last_unstale":"2024-04-25T01:53:43.868468+0000","last_undegraded":"2024-04-25T01:53:43.868468+0000","last_fullsized":"2024-04-25T01:53:43.868468+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:58.533213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.734344+0000","last_change":"2024-04-25T01:52:44.088592+0000","last_active":"2024-04-25T01:54:12.734344+0000","last_peered":"2024-04-25T01:54:12.734344+0000","last_clean":"2024-04-25T01:54:12.734344+0000","last_became_active":"2024-04-25T01:52:44.087651+0000","last_became_peered":"2024-04-25T01:52:44.087651+0000","last_unstale":"2024-04-25T01:54:12.734344+0000","last_undegraded":"2024-04-25T01:54:12.734344+0000","last_fullsized":"2024-04-25T01:54:12.734344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:52.240818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859099+0000","last_change":"2024-04-25T01:52:44.095033+0000","last_active":"2024-04-25T01:53:43.859099+0000","last_peered":"2024-04-25T01:53:43.859099+0000","last_clean":"2024-04-25T01:53:43.859099+0000","last_became_active":"2024-04-25T01:52:44.094719+0000","last_became_peered":"2024-04-25T01:52:44.094719+0000","last_unstale":"2024-04-25T01:53:43.859099+0000","last_undegraded":"2024-04-25T01:53:43.859099+0000","last_fullsized":"2024-04-25T01:53:43.859099+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:52.422544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858096+0000","last_change":"2024-04-25T01:52:44.102574+0000","last_active":"2024-04-25T01:53:43.858096+0000","last_peered":"2024-04-25T01:53:43.858096+0000","last_clean":"2024-04-25T01:53:43.858096+0000","last_became_active":"2024-04-25T01:52:44.102158+0000","last_became_peered":"2024-04-25T01:52:44.102158+0000","last_unstale":"2024-04-25T01:53:43.858096+0000","last_undegraded":"2024-04-25T01:53:43.858096+0000","last_fullsized":"2024-04-25T01:53:43.858096+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:20.931392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856003+0000","last_change":"2024-04-25T01:52:44.107662+0000","last_active":"2024-04-25T01:53:43.856003+0000","last_peered":"2024-04-25T01:53:43.856003+0000","last_clean":"2024-04-25T01:53:43.856003+0000","last_became_active":"2024-04-25T01:52:44.107304+0000","last_became_peered":"2024-04-25T01:52:44.107304+0000","last_unstale":"2024-04-25T01:53:43.856003+0000","last_undegraded":"2024-04-25T01:53:43.856003+0000","last_fullsized":"2024-04-25T01:53:43.856003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:42:54.636950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.866459+0000","last_change":"2024-04-25T01:52:44.100273+0000","last_active":"2024-04-25T01:53:43.866459+0000","last_peered":"2024-04-25T01:53:43.866459+0000","last_clean":"2024-04-25T01:53:43.866459+0000","last_became_active":"2024-04-25T01:52:44.098223+0000","last_became_peered":"2024-04-25T01:52:44.098223+0000","last_unstale":"2024-04-25T01:53:43.866459+0000","last_undegraded":"2024-04-25T01:53:43.866459+0000","last_fullsized":"2024-04-25T01:53:43.866459+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:05.003265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.861355+0000","last_change":"2024-04-25T01:52:44.089658+0000","last_active":"2024-04-25T01:53:43.861355+0000","last_peered":"2024-04-25T01:53:43.861355+0000","last_clean":"2024-04-25T01:53:43.861355+0000","last_became_active":"2024-04-25T01:52:44.089232+0000","last_became_peered":"2024-04-25T01:52:44.089232+0000","last_unstale":"2024-04-25T01:53:43.861355+0000","last_undegraded":"2024-04-25T01:53:43.861355+0000","last_fullsized":"2024-04-25T01:53:43.861355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:54.385782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858557+0000","last_change":"2024-04-25T01:52:44.086660+0000","last_active":"2024-04-25T01:53:43.858557+0000","last_peered":"2024-04-25T01:53:43.858557+0000","last_clean":"2024-04-25T01:53:43.858557+0000","last_became_active":"2024-04-25T01:52:44.086306+0000","last_became_peered":"2024-04-25T01:52:44.086306+0000","last_unstale":"2024-04-25T01:53:43.858557+0000","last_undegraded":"2024-04-25T01:53:43.858557+0000","last_fullsized":"2024-04-25T01:53:43.858557+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:33:20.726616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856231+0000","last_change":"2024-04-25T01:52:44.090239+0000","last_active":"2024-04-25T01:53:43.856231+0000","last_peered":"2024-04-25T01:53:43.856231+0000","last_clean":"2024-04-25T01:53:43.856231+0000","last_became_active":"2024-04-25T01:52:44.089736+0000","last_became_peered":"2024-04-25T01:52:44.089736+0000","last_unstale":"2024-04-25T01:53:43.856231+0000","last_undegraded":"2024-04-25T01:53:43.856231+0000","last_fullsized":"2024-04-25T01:53:43.856231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:22.954011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.730383+0000","last_change":"2024-04-25T01:52:44.088878+0000","last_active":"2024-04-25T01:54:12.730383+0000","last_peered":"2024-04-25T01:54:12.730383+0000","last_clean":"2024-04-25T01:54:12.730383+0000","last_became_active":"2024-04-25T01:52:44.088250+0000","last_became_peered":"2024-04-25T01:52:44.088250+0000","last_unstale":"2024-04-25T01:54:12.730383+0000","last_undegraded":"2024-04-25T01:54:12.730383+0000","last_fullsized":"2024-04-25T01:54:12.730383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:15.490131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.865396+0000","last_change":"2024-04-25T01:52:44.094675+0000","last_active":"2024-04-25T01:53:43.865396+0000","last_peered":"2024-04-25T01:53:43.865396+0000","last_clean":"2024-04-25T01:53:43.865396+0000","last_became_active":"2024-04-25T01:52:44.094219+0000","last_became_peered":"2024-04-25T01:52:44.094219+0000","last_unstale":"2024-04-25T01:53:43.865396+0000","last_undegraded":"2024-04-25T01:53:43.865396+0000","last_fullsized":"2024-04-25T01:53:43.865396+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:26.160156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.868293+0000","last_change":"2024-04-25T01:52:44.111131+0000","last_active":"2024-04-25T01:53:43.868293+0000","last_peered":"2024-04-25T01:53:43.868293+0000","last_clean":"2024-04-25T01:53:43.868293+0000","last_became_active":"2024-04-25T01:52:44.105670+0000","last_became_peered":"2024-04-25T01:52:44.105670+0000","last_unstale":"2024-04-25T01:53:43.868293+0000","last_undegraded":"2024-04-25T01:53:43.868293+0000","last_fullsized":"2024-04-25T01:53:43.868293+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:00.377724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.868284+0000","last_change":"2024-04-25T01:52:44.109707+0000","last_active":"2024-04-25T01:53:43.868284+0000","last_peered":"2024-04-25T01:53:43.868284+0000","last_clean":"2024-04-25T01:53:43.868284+0000","last_became_active":"2024-04-25T01:52:44.108550+0000","last_became_peered":"2024-04-25T01:52:44.108550+0000","last_unstale":"2024-04-25T01:53:43.868284+0000","last_undegraded":"2024-04-25T01:53:43.868284+0000","last_fullsized":"2024-04-25T01:53:43.868284+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:46.296789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.860523+0000","last_change":"2024-04-25T01:52:44.104504+0000","last_active":"2024-04-25T01:53:43.860523+0000","last_peered":"2024-04-25T01:53:43.860523+0000","last_clean":"2024-04-25T01:53:43.860523+0000","last_became_active":"2024-04-25T01:52:44.104180+0000","last_became_peered":"2024-04-25T01:52:44.104180+0000","last_unstale":"2024-04-25T01:53:43.860523+0000","last_undegraded":"2024-04-25T01:53:43.860523+0000","last_fullsized":"2024-04-25T01:53:43.860523+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:17.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.857632+0000","last_change":"2024-04-25T01:52:44.107550+0000","last_active":"2024-04-25T01:53:43.857632+0000","last_peered":"2024-04-25T01:53:43.857632+0000","last_clean":"2024-04-25T01:53:43.857632+0000","last_became_active":"2024-04-25T01:52:44.107195+0000","last_became_peered":"2024-04-25T01:52:44.107195+0000","last_unstale":"2024-04-25T01:53:43.857632+0000","last_undegraded":"2024-04-25T01:53:43.857632+0000","last_fullsized":"2024-04-25T01:53:43.857632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:59:42.220575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858299+0000","last_change":"2024-04-25T01:52:44.101165+0000","last_active":"2024-04-25T01:53:43.858299+0000","last_peered":"2024-04-25T01:53:43.858299+0000","last_clean":"2024-04-25T01:53:43.858299+0000","last_became_active":"2024-04-25T01:52:44.100386+0000","last_became_peered":"2024-04-25T01:52:44.100386+0000","last_unstale":"2024-04-25T01:53:43.858299+0000","last_undegraded":"2024-04-25T01:53:43.858299+0000","last_fullsized":"2024-04-25T01:53:43.858299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:30.922707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.754083+0000","last_change":"2024-04-25T01:52:44.116694+0000","last_active":"2024-04-25T01:54:12.754083+0000","last_peered":"2024-04-25T01:54:12.754083+0000","last_clean":"2024-04-25T01:54:12.754083+0000","last_became_active":"2024-04-25T01:52:44.116326+0000","last_became_peered":"2024-04-25T01:52:44.116326+0000","last_unstale":"2024-04-25T01:54:12.754083+0000","last_undegraded":"2024-04-25T01:54:12.754083+0000","last_fullsized":"2024-04-25T01:54:12.754083+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:16.329899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858757+0000","last_change":"2024-04-25T01:52:43.072358+0000","last_active":"2024-04-25T01:53:43.858757+0000","last_peered":"2024-04-25T01:53:43.858757+0000","last_clean":"2024-04-25T01:53:43.858757+0000","last_became_active":"2024-04-25T01:52:43.071815+0000","last_became_peered":"2024-04-25T01:52:43.071815+0000","last_unstale":"2024-04-25T01:53:43.858757+0000","last_undegraded":"2024-04-25T01:53:43.858757+0000","last_fullsized":"2024-04-25T01:53:43.858757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:15.295744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.867488+0000","last_change":"2024-04-25T01:52:44.094509+0000","last_active":"2024-04-25T01:53:43.867488+0000","last_peered":"2024-04-25T01:53:43.867488+0000","last_clean":"2024-04-25T01:53:43.867488+0000","last_became_active":"2024-04-25T01:52:44.093955+0000","last_became_peered":"2024-04-25T01:52:44.093955+0000","last_unstale":"2024-04-25T01:53:43.867488+0000","last_undegraded":"2024-04-25T01:53:43.867488+0000","last_fullsized":"2024-04-25T01:53:43.867488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:37.701835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.867618+0000","last_change":"2024-04-25T01:52:43.094435+0000","last_active":"2024-04-25T01:53:43.867618+0000","last_peered":"2024-04-25T01:53:43.867618+0000","last_clean":"2024-04-25T01:53:43.867618+0000","last_became_active":"2024-04-25T01:52:43.094018+0000","last_became_peered":"2024-04-25T01:52:43.094018+0000","last_unstale":"2024-04-25T01:53:43.867618+0000","last_undegraded":"2024-04-25T01:53:43.867618+0000","last_fullsized":"2024-04-25T01:53:43.867618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:36.781014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.870332+0000","last_change":"2024-04-25T01:52:44.104347+0000","last_active":"2024-04-25T01:53:43.870332+0000","last_peered":"2024-04-25T01:53:43.870332+0000","last_clean":"2024-04-25T01:53:43.870332+0000","last_became_active":"2024-04-25T01:52:44.103803+0000","last_became_peered":"2024-04-25T01:52:44.103803+0000","last_unstale":"2024-04-25T01:53:43.870332+0000","last_undegraded":"2024-04-25T01:53:43.870332+0000","last_fullsized":"2024-04-25T01:53:43.870332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:01.784893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859364+0000","last_change":"2024-04-25T01:52:43.070902+0000","last_active":"2024-04-25T01:53:43.859364+0000","last_peered":"2024-04-25T01:53:43.859364+0000","last_clean":"2024-04-25T01:53:43.859364+0000","last_became_active":"2024-04-25T01:52:43.070313+0000","last_became_peered":"2024-04-25T01:52:43.070313+0000","last_unstale":"2024-04-25T01:53:43.859364+0000","last_undegraded":"2024-04-25T01:53:43.859364+0000","last_fullsized":"2024-04-25T01:53:43.859364+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:10.677320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.861005+0000","last_change":"2024-04-25T01:52:44.093293+0000","last_active":"2024-04-25T01:53:43.861005+0000","last_peered":"2024-04-25T01:53:43.861005+0000","last_clean":"2024-04-25T01:53:43.861005+0000","last_became_active":"2024-04-25T01:52:44.092932+0000","last_became_peered":"2024-04-25T01:52:44.092932+0000","last_unstale":"2024-04-25T01:53:43.861005+0000","last_undegraded":"2024-04-25T01:53:43.861005+0000","last_fullsized":"2024-04-25T01:53:43.861005+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:58:37.705594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.860028+0000","last_change":"2024-04-25T01:52:43.069743+0000","last_active":"2024-04-25T01:53:43.860028+0000","last_peered":"2024-04-25T01:53:43.860028+0000","last_clean":"2024-04-25T01:53:43.860028+0000","last_became_active":"2024-04-25T01:52:43.068947+0000","last_became_peered":"2024-04-25T01:52:43.068947+0000","last_unstale":"2024-04-25T01:53:43.860028+0000","last_undegraded":"2024-04-25T01:53:43.860028+0000","last_fullsized":"2024-04-25T01:53:43.860028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:02:52.582176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.871177+0000","last_change":"2024-04-25T01:53:43.865168+0000","last_active":"2024-04-25T01:53:43.871177+0000","last_peered":"2024-04-25T01:53:43.871177+0000","last_clean":"2024-04-25T01:53:43.871177+0000","last_became_active":"2024-04-25T01:53:43.864055+0000","last_became_peered":"2024-04-25T01:53:43.864055+0000","last_unstale":"2024-04-25T01:53:43.871177+0000","last_undegraded":"2024-04-25T01:53:43.871177+0000","last_fullsized":"2024-04-25T01:53:43.871177+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:01.707773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"87'6","reported_seq":112,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:11.231281+0000","last_change":"2024-04-25T01:52:43.068450+0000","last_active":"2024-04-25T01:54:11.231281+0000","last_peered":"2024-04-25T01:54:11.231281+0000","last_clean":"2024-04-25T01:54:11.231281+0000","last_became_active":"2024-04-25T01:52:43.067881+0000","last_became_peered":"2024-04-25T01:52:43.067881+0000","last_unstale":"2024-04-25T01:54:11.231281+0000","last_undegraded":"2024-04-25T01:54:11.231281+0000","last_fullsized":"2024-04-25T01:54:11.231281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:12.739321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859372+0000","last_change":"2024-04-25T01:52:44.101743+0000","last_active":"2024-04-25T01:53:43.859372+0000","last_peered":"2024-04-25T01:53:43.859372+0000","last_clean":"2024-04-25T01:53:43.859372+0000","last_became_active":"2024-04-25T01:52:44.101351+0000","last_became_peered":"2024-04-25T01:52:44.101351+0000","last_unstale":"2024-04-25T01:53:43.859372+0000","last_undegraded":"2024-04-25T01:53:43.859372+0000","last_fullsized":"2024-04-25T01:53:43.859372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:54:24.104819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.857985+0000","last_change":"2024-04-25T01:52:43.075880+0000","last_active":"2024-04-25T01:53:43.857985+0000","last_peered":"2024-04-25T01:53:43.857985+0000","last_clean":"2024-04-25T01:53:43.857985+0000","last_became_active":"2024-04-25T01:52:43.075522+0000","last_became_peered":"2024-04-25T01:52:43.075522+0000","last_unstale":"2024-04-25T01:53:43.857985+0000","last_undegraded":"2024-04-25T01:53:43.857985+0000","last_fullsized":"2024-04-25T01:53:43.857985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:26.218774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864379+0000","last_change":"2024-04-25T01:52:44.116772+0000","last_active":"2024-04-25T01:53:43.864379+0000","last_peered":"2024-04-25T01:53:43.864379+0000","last_clean":"2024-04-25T01:53:43.864379+0000","last_became_active":"2024-04-25T01:52:44.116408+0000","last_became_peered":"2024-04-25T01:52:44.116408+0000","last_unstale":"2024-04-25T01:53:43.864379+0000","last_undegraded":"2024-04-25T01:53:43.864379+0000","last_fullsized":"2024-04-25T01:53:43.864379+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:21.988425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.869110+0000","last_change":"2024-04-25T01:52:43.076964+0000","last_active":"2024-04-25T01:53:43.869110+0000","last_peered":"2024-04-25T01:53:43.869110+0000","last_clean":"2024-04-25T01:53:43.869110+0000","last_became_active":"2024-04-25T01:52:43.073915+0000","last_became_peered":"2024-04-25T01:52:43.073915+0000","last_unstale":"2024-04-25T01:53:43.869110+0000","last_undegraded":"2024-04-25T01:53:43.869110+0000","last_fullsized":"2024-04-25T01:53:43.869110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:03.321920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.866351+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:53:43.866351+0000","last_peered":"2024-04-25T01:53:43.866351+0000","last_clean":"2024-04-25T01:53:43.866351+0000","last_became_active":"2024-04-25T01:52:44.088198+0000","last_became_peered":"2024-04-25T01:52:44.088198+0000","last_unstale":"2024-04-25T01:53:43.866351+0000","last_undegraded":"2024-04-25T01:53:43.866351+0000","last_fullsized":"2024-04-25T01:53:43.866351+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:30.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864096+0000","last_change":"2024-04-25T01:52:43.069853+0000","last_active":"2024-04-25T01:53:43.864096+0000","last_peered":"2024-04-25T01:53:43.864096+0000","last_clean":"2024-04-25T01:53:43.864096+0000","last_became_active":"2024-04-25T01:52:43.069526+0000","last_became_peered":"2024-04-25T01:52:43.069526+0000","last_unstale":"2024-04-25T01:53:43.864096+0000","last_undegraded":"2024-04-25T01:53:43.864096+0000","last_fullsized":"2024-04-25T01:53:43.864096+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:35:03.272610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.857359+0000","last_change":"2024-04-25T01:52:44.114296+0000","last_active":"2024-04-25T01:53:43.857359+0000","last_peered":"2024-04-25T01:53:43.857359+0000","last_clean":"2024-04-25T01:53:43.857359+0000","last_became_active":"2024-04-25T01:52:44.113958+0000","last_became_peered":"2024-04-25T01:52:44.113958+0000","last_unstale":"2024-04-25T01:53:43.857359+0000","last_undegraded":"2024-04-25T01:53:43.857359+0000","last_fullsized":"2024-04-25T01:53:43.857359+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:45.068080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":104,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:09.227374+0000","last_change":"2024-04-25T01:52:43.071388+0000","last_active":"2024-04-25T01:54:09.227374+0000","last_peered":"2024-04-25T01:54:09.227374+0000","last_clean":"2024-04-25T01:54:09.227374+0000","last_became_active":"2024-04-25T01:52:43.071005+0000","last_became_peered":"2024-04-25T01:52:43.071005+0000","last_unstale":"2024-04-25T01:54:09.227374+0000","last_undegraded":"2024-04-25T01:54:09.227374+0000","last_fullsized":"2024-04-25T01:54:09.227374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:56:59.580458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.861254+0000","last_change":"2024-04-25T01:52:44.106106+0000","last_active":"2024-04-25T01:53:43.861254+0000","last_peered":"2024-04-25T01:53:43.861254+0000","last_clean":"2024-04-25T01:53:43.861254+0000","last_became_active":"2024-04-25T01:52:44.105760+0000","last_became_peered":"2024-04-25T01:52:44.105760+0000","last_unstale":"2024-04-25T01:53:43.861254+0000","last_undegraded":"2024-04-25T01:53:43.861254+0000","last_fullsized":"2024-04-25T01:53:43.861254+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:24.626753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856325+0000","last_change":"2024-04-25T01:52:43.068926+0000","last_active":"2024-04-25T01:53:43.856325+0000","last_peered":"2024-04-25T01:53:43.856325+0000","last_clean":"2024-04-25T01:53:43.856325+0000","last_became_active":"2024-04-25T01:52:43.067356+0000","last_became_peered":"2024-04-25T01:52:43.067356+0000","last_unstale":"2024-04-25T01:53:43.856325+0000","last_undegraded":"2024-04-25T01:53:43.856325+0000","last_fullsized":"2024-04-25T01:53:43.856325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:19.134633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.870393+0000","last_change":"2024-04-25T01:52:44.108299+0000","last_active":"2024-04-25T01:53:43.870393+0000","last_peered":"2024-04-25T01:53:43.870393+0000","last_clean":"2024-04-25T01:53:43.870393+0000","last_became_active":"2024-04-25T01:52:44.107573+0000","last_became_peered":"2024-04-25T01:52:44.107573+0000","last_unstale":"2024-04-25T01:53:43.870393+0000","last_undegraded":"2024-04-25T01:53:43.870393+0000","last_fullsized":"2024-04-25T01:53:43.870393+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.804579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864522+0000","last_change":"2024-04-25T01:52:43.079263+0000","last_active":"2024-04-25T01:53:43.864522+0000","last_peered":"2024-04-25T01:53:43.864522+0000","last_clean":"2024-04-25T01:53:43.864522+0000","last_became_active":"2024-04-25T01:52:43.078459+0000","last_became_peered":"2024-04-25T01:52:43.078459+0000","last_unstale":"2024-04-25T01:53:43.864522+0000","last_undegraded":"2024-04-25T01:53:43.864522+0000","last_fullsized":"2024-04-25T01:53:43.864522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:59.349897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.861171+0000","last_change":"2024-04-25T01:52:44.088044+0000","last_active":"2024-04-25T01:53:43.861171+0000","last_peered":"2024-04-25T01:53:43.861171+0000","last_clean":"2024-04-25T01:53:43.861171+0000","last_became_active":"2024-04-25T01:52:44.087682+0000","last_became_peered":"2024-04-25T01:52:44.087682+0000","last_unstale":"2024-04-25T01:53:43.861171+0000","last_undegraded":"2024-04-25T01:53:43.861171+0000","last_fullsized":"2024-04-25T01:53:43.861171+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:08.933923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"87'28","reported_seq":237,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:10.904374+0000","last_change":"2024-04-25T01:53:43.869516+0000","last_active":"2024-04-25T01:54:10.904374+0000","last_peered":"2024-04-25T01:54:10.904374+0000","last_clean":"2024-04-25T01:54:10.904374+0000","last_became_active":"2024-04-25T01:53:43.861718+0000","last_became_peered":"2024-04-25T01:53:43.861718+0000","last_unstale":"2024-04-25T01:54:10.904374+0000","last_undegraded":"2024-04-25T01:54:10.904374+0000","last_fullsized":"2024-04-25T01:54:10.904374+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:13:01.820264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":164,"num_read_kb":141,"num_write":29,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859241+0000","last_change":"2024-04-25T01:52:44.091547+0000","last_active":"2024-04-25T01:53:43.859241+0000","last_peered":"2024-04-25T01:53:43.859241+0000","last_clean":"2024-04-25T01:53:43.859241+0000","last_became_active":"2024-04-25T01:52:44.091167+0000","last_became_peered":"2024-04-25T01:52:44.091167+0000","last_unstale":"2024-04-25T01:53:43.859241+0000","last_undegraded":"2024-04-25T01:53:43.859241+0000","last_fullsized":"2024-04-25T01:53:43.859241+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:24.559694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856061+0000","last_change":"2024-04-25T01:52:43.088385+0000","last_active":"2024-04-25T01:53:43.856061+0000","last_peered":"2024-04-25T01:53:43.856061+0000","last_clean":"2024-04-25T01:53:43.856061+0000","last_became_active":"2024-04-25T01:52:43.087982+0000","last_became_peered":"2024-04-25T01:52:43.087982+0000","last_unstale":"2024-04-25T01:53:43.856061+0000","last_undegraded":"2024-04-25T01:53:43.856061+0000","last_fullsized":"2024-04-25T01:53:43.856061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:28.944427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.863070+0000","last_change":"2024-04-25T01:52:44.099420+0000","last_active":"2024-04-25T01:53:43.863070+0000","last_peered":"2024-04-25T01:53:43.863070+0000","last_clean":"2024-04-25T01:53:43.863070+0000","last_became_active":"2024-04-25T01:52:44.099045+0000","last_became_peered":"2024-04-25T01:52:44.099045+0000","last_unstale":"2024-04-25T01:53:43.863070+0000","last_undegraded":"2024-04-25T01:53:43.863070+0000","last_fullsized":"2024-04-25T01:53:43.863070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:28.377091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864197+0000","last_change":"2024-04-25T01:52:43.086252+0000","last_active":"2024-04-25T01:53:43.864197+0000","last_peered":"2024-04-25T01:53:43.864197+0000","last_clean":"2024-04-25T01:53:43.864197+0000","last_became_active":"2024-04-25T01:52:43.085892+0000","last_became_peered":"2024-04-25T01:52:43.085892+0000","last_unstale":"2024-04-25T01:53:43.864197+0000","last_undegraded":"2024-04-25T01:53:43.864197+0000","last_fullsized":"2024-04-25T01:53:43.864197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:25:26.395705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856293+0000","last_change":"2024-04-25T01:52:44.097163+0000","last_active":"2024-04-25T01:53:43.856293+0000","last_peered":"2024-04-25T01:53:43.856293+0000","last_clean":"2024-04-25T01:53:43.856293+0000","last_became_active":"2024-04-25T01:52:44.096758+0000","last_became_peered":"2024-04-25T01:52:44.096758+0000","last_unstale":"2024-04-25T01:53:43.856293+0000","last_undegraded":"2024-04-25T01:53:43.856293+0000","last_fullsized":"2024-04-25T01:53:43.856293+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:22:47.394855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":108,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:11.243007+0000","last_change":"2024-04-25T01:52:43.069024+0000","last_active":"2024-04-25T01:54:11.243007+0000","last_peered":"2024-04-25T01:54:11.243007+0000","last_clean":"2024-04-25T01:54:11.243007+0000","last_became_active":"2024-04-25T01:52:43.068058+0000","last_became_peered":"2024-04-25T01:52:43.068058+0000","last_unstale":"2024-04-25T01:54:11.243007+0000","last_undegraded":"2024-04-25T01:54:11.243007+0000","last_fullsized":"2024-04-25T01:54:11.243007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:43.126131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.855921+0000","last_change":"2024-04-25T01:52:44.091829+0000","last_active":"2024-04-25T01:53:43.855921+0000","last_peered":"2024-04-25T01:53:43.855921+0000","last_clean":"2024-04-25T01:53:43.855921+0000","last_became_active":"2024-04-25T01:52:44.091528+0000","last_became_peered":"2024-04-25T01:52:44.091528+0000","last_unstale":"2024-04-25T01:53:43.855921+0000","last_undegraded":"2024-04-25T01:53:43.855921+0000","last_fullsized":"2024-04-25T01:53:43.855921+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:39.585296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.889407+0000","last_change":"2024-04-25T01:53:43.888956+0000","last_active":"2024-04-25T01:53:43.889407+0000","last_peered":"2024-04-25T01:53:43.889407+0000","last_clean":"2024-04-25T01:53:43.889407+0000","last_became_active":"2024-04-25T01:53:43.870789+0000","last_became_peered":"2024-04-25T01:53:43.870789+0000","last_unstale":"2024-04-25T01:53:43.889407+0000","last_undegraded":"2024-04-25T01:53:43.889407+0000","last_fullsized":"2024-04-25T01:53:43.889407+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:08:48.790006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864143+0000","last_change":"2024-04-25T01:52:44.094696+0000","last_active":"2024-04-25T01:53:43.864143+0000","last_peered":"2024-04-25T01:53:43.864143+0000","last_clean":"2024-04-25T01:53:43.864143+0000","last_became_active":"2024-04-25T01:52:44.094371+0000","last_became_peered":"2024-04-25T01:52:44.094371+0000","last_unstale":"2024-04-25T01:53:43.864143+0000","last_undegraded":"2024-04-25T01:53:43.864143+0000","last_fullsized":"2024-04-25T01:53:43.864143+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:20.035674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859323+0000","last_change":"2024-04-25T01:52:43.071275+0000","last_active":"2024-04-25T01:53:43.859323+0000","last_peered":"2024-04-25T01:53:43.859323+0000","last_clean":"2024-04-25T01:53:43.859323+0000","last_became_active":"2024-04-25T01:52:43.070958+0000","last_became_peered":"2024-04-25T01:52:43.070958+0000","last_unstale":"2024-04-25T01:53:43.859323+0000","last_undegraded":"2024-04-25T01:53:43.859323+0000","last_fullsized":"2024-04-25T01:53:43.859323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:23:55.284943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.857381+0000","last_change":"2024-04-25T01:52:44.092742+0000","last_active":"2024-04-25T01:53:43.857381+0000","last_peered":"2024-04-25T01:53:43.857381+0000","last_clean":"2024-04-25T01:53:43.857381+0000","last_became_active":"2024-04-25T01:52:44.092345+0000","last_became_peered":"2024-04-25T01:52:44.092345+0000","last_unstale":"2024-04-25T01:53:43.857381+0000","last_undegraded":"2024-04-25T01:53:43.857381+0000","last_fullsized":"2024-04-25T01:53:43.857381+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:14.325814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.857260+0000","last_change":"2024-04-25T01:52:43.090057+0000","last_active":"2024-04-25T01:53:43.857260+0000","last_peered":"2024-04-25T01:53:43.857260+0000","last_clean":"2024-04-25T01:53:43.857260+0000","last_became_active":"2024-04-25T01:52:43.089280+0000","last_became_peered":"2024-04-25T01:52:43.089280+0000","last_unstale":"2024-04-25T01:53:43.857260+0000","last_undegraded":"2024-04-25T01:53:43.857260+0000","last_fullsized":"2024-04-25T01:53:43.857260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:41.048908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858328+0000","last_change":"2024-04-25T01:52:44.097651+0000","last_active":"2024-04-25T01:53:43.858328+0000","last_peered":"2024-04-25T01:53:43.858328+0000","last_clean":"2024-04-25T01:53:43.858328+0000","last_became_active":"2024-04-25T01:52:44.097254+0000","last_became_peered":"2024-04-25T01:52:44.097254+0000","last_unstale":"2024-04-25T01:53:43.858328+0000","last_undegraded":"2024-04-25T01:53:43.858328+0000","last_fullsized":"2024-04-25T01:53:43.858328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:15.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856041+0000","last_change":"2024-04-25T01:52:43.076925+0000","last_active":"2024-04-25T01:53:43.856041+0000","last_peered":"2024-04-25T01:53:43.856041+0000","last_clean":"2024-04-25T01:53:43.856041+0000","last_became_active":"2024-04-25T01:52:43.076514+0000","last_became_peered":"2024-04-25T01:52:43.076514+0000","last_unstale":"2024-04-25T01:53:43.856041+0000","last_undegraded":"2024-04-25T01:53:43.856041+0000","last_fullsized":"2024-04-25T01:53:43.856041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:31:17.305379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858132+0000","last_change":"2024-04-25T01:52:44.103731+0000","last_active":"2024-04-25T01:53:43.858132+0000","last_peered":"2024-04-25T01:53:43.858132+0000","last_clean":"2024-04-25T01:53:43.858132+0000","last_became_active":"2024-04-25T01:52:44.103173+0000","last_became_peered":"2024-04-25T01:52:44.103173+0000","last_unstale":"2024-04-25T01:53:43.858132+0000","last_undegraded":"2024-04-25T01:53:43.858132+0000","last_fullsized":"2024-04-25T01:53:43.858132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:43.617579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864212+0000","last_change":"2024-04-25T01:52:44.090847+0000","last_active":"2024-04-25T01:53:43.864212+0000","last_peered":"2024-04-25T01:53:43.864212+0000","last_clean":"2024-04-25T01:53:43.864212+0000","last_became_active":"2024-04-25T01:52:44.090506+0000","last_became_peered":"2024-04-25T01:52:44.090506+0000","last_unstale":"2024-04-25T01:53:43.864212+0000","last_undegraded":"2024-04-25T01:53:43.864212+0000","last_fullsized":"2024-04-25T01:53:43.864212+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:24.539726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":106,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:10.240137+0000","last_change":"2024-04-25T01:52:43.090469+0000","last_active":"2024-04-25T01:54:10.240137+0000","last_peered":"2024-04-25T01:54:10.240137+0000","last_clean":"2024-04-25T01:54:10.240137+0000","last_became_active":"2024-04-25T01:52:43.090159+0000","last_became_peered":"2024-04-25T01:52:43.090159+0000","last_unstale":"2024-04-25T01:54:10.240137+0000","last_undegraded":"2024-04-25T01:54:10.240137+0000","last_fullsized":"2024-04-25T01:54:10.240137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:42:45.532017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.869798+0000","last_change":"2024-04-25T01:52:44.093527+0000","last_active":"2024-04-25T01:53:43.869798+0000","last_peered":"2024-04-25T01:53:43.869798+0000","last_clean":"2024-04-25T01:53:43.869798+0000","last_became_active":"2024-04-25T01:52:44.092729+0000","last_became_peered":"2024-04-25T01:52:44.092729+0000","last_unstale":"2024-04-25T01:53:43.869798+0000","last_undegraded":"2024-04-25T01:53:43.869798+0000","last_fullsized":"2024-04-25T01:53:43.869798+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:32:50.668352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859076+0000","last_change":"2024-04-25T01:52:43.075323+0000","last_active":"2024-04-25T01:53:43.859076+0000","last_peered":"2024-04-25T01:53:43.859076+0000","last_clean":"2024-04-25T01:53:43.859076+0000","last_became_active":"2024-04-25T01:52:43.074876+0000","last_became_peered":"2024-04-25T01:52:43.074876+0000","last_unstale":"2024-04-25T01:53:43.859076+0000","last_undegraded":"2024-04-25T01:53:43.859076+0000","last_fullsized":"2024-04-25T01:53:43.859076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:47.472920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856031+0000","last_change":"2024-04-25T01:51:20.173285+0000","last_active":"2024-04-25T01:53:43.856031+0000","last_peered":"2024-04-25T01:53:43.856031+0000","last_clean":"2024-04-25T01:53:43.856031+0000","last_became_active":"2024-04-25T01:51:20.172811+0000","last_became_peered":"2024-04-25T01:51:20.172811+0000","last_unstale":"2024-04-25T01:53:43.856031+0000","last_undegraded":"2024-04-25T01:53:43.856031+0000","last_fullsized":"2024-04-25T01:53:43.856031+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_clean_scrub_stamp":"2024-04-25T01:47:31.712321+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:33.708082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.874632+0000","last_change":"2024-04-25T01:53:43.874632+0000","last_active":"2024-04-25T01:53:43.874632+0000","last_peered":"2024-04-25T01:53:43.874632+0000","last_clean":"2024-04-25T01:53:43.874632+0000","last_became_active":"2024-04-25T01:53:43.873732+0000","last_became_peered":"2024-04-25T01:53:43.873732+0000","last_unstale":"2024-04-25T01:53:43.874632+0000","last_undegraded":"2024-04-25T01:53:43.874632+0000","last_fullsized":"2024-04-25T01:53:43.874632+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:27.874789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856171+0000","last_change":"2024-04-25T01:52:43.072565+0000","last_active":"2024-04-25T01:53:43.856171+0000","last_peered":"2024-04-25T01:53:43.856171+0000","last_clean":"2024-04-25T01:53:43.856171+0000","last_became_active":"2024-04-25T01:52:43.072193+0000","last_became_peered":"2024-04-25T01:52:43.072193+0000","last_unstale":"2024-04-25T01:53:43.856171+0000","last_undegraded":"2024-04-25T01:53:43.856171+0000","last_fullsized":"2024-04-25T01:53:43.856171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:07.261859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.878893+0000","last_change":"2024-04-25T01:53:43.878499+0000","last_active":"2024-04-25T01:53:43.878893+0000","last_peered":"2024-04-25T01:53:43.878893+0000","last_clean":"2024-04-25T01:53:43.878893+0000","last_became_active":"2024-04-25T01:53:43.870876+0000","last_became_peered":"2024-04-25T01:53:43.870876+0000","last_unstale":"2024-04-25T01:53:43.878893+0000","last_undegraded":"2024-04-25T01:53:43.878893+0000","last_fullsized":"2024-04-25T01:53:43.878893+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:08.809127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.863253+0000","last_change":"2024-04-25T01:52:44.104892+0000","last_active":"2024-04-25T01:53:43.863253+0000","last_peered":"2024-04-25T01:53:43.863253+0000","last_clean":"2024-04-25T01:53:43.863253+0000","last_became_active":"2024-04-25T01:52:44.103802+0000","last_became_peered":"2024-04-25T01:52:44.103802+0000","last_unstale":"2024-04-25T01:53:43.863253+0000","last_undegraded":"2024-04-25T01:53:43.863253+0000","last_fullsized":"2024-04-25T01:53:43.863253+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:16.445968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'8","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858613+0000","last_change":"2024-04-25T01:52:43.087143+0000","last_active":"2024-04-25T01:53:43.858613+0000","last_peered":"2024-04-25T01:53:43.858613+0000","last_clean":"2024-04-25T01:53:43.858613+0000","last_became_active":"2024-04-25T01:52:43.085987+0000","last_became_peered":"2024-04-25T01:52:43.085987+0000","last_unstale":"2024-04-25T01:53:43.858613+0000","last_undegraded":"2024-04-25T01:53:43.858613+0000","last_fullsized":"2024-04-25T01:53:43.858613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:17.936671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.867146+0000","last_change":"2024-04-25T01:52:44.092342+0000","last_active":"2024-04-25T01:53:43.867146+0000","last_peered":"2024-04-25T01:53:43.867146+0000","last_clean":"2024-04-25T01:53:43.867146+0000","last_became_active":"2024-04-25T01:52:44.091991+0000","last_became_peered":"2024-04-25T01:52:44.091991+0000","last_unstale":"2024-04-25T01:53:43.867146+0000","last_undegraded":"2024-04-25T01:53:43.867146+0000","last_fullsized":"2024-04-25T01:53:43.867146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:22.147284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859134+0000","last_change":"2024-04-25T01:52:43.077972+0000","last_active":"2024-04-25T01:53:43.859134+0000","last_peered":"2024-04-25T01:53:43.859134+0000","last_clean":"2024-04-25T01:53:43.859134+0000","last_became_active":"2024-04-25T01:52:43.077655+0000","last_became_peered":"2024-04-25T01:52:43.077655+0000","last_unstale":"2024-04-25T01:53:43.859134+0000","last_undegraded":"2024-04-25T01:53:43.859134+0000","last_fullsized":"2024-04-25T01:53:43.859134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:24.820910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.870086+0000","last_change":"2024-04-25T01:52:44.119560+0000","last_active":"2024-04-25T01:53:43.870086+0000","last_peered":"2024-04-25T01:53:43.870086+0000","last_clean":"2024-04-25T01:53:43.870086+0000","last_became_active":"2024-04-25T01:52:44.119106+0000","last_became_peered":"2024-04-25T01:52:44.119106+0000","last_unstale":"2024-04-25T01:53:43.870086+0000","last_undegraded":"2024-04-25T01:53:43.870086+0000","last_fullsized":"2024-04-25T01:53:43.870086+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:50.702393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.860595+0000","last_change":"2024-04-25T01:52:43.075783+0000","last_active":"2024-04-25T01:53:43.860595+0000","last_peered":"2024-04-25T01:53:43.860595+0000","last_clean":"2024-04-25T01:53:43.860595+0000","last_became_active":"2024-04-25T01:52:43.075462+0000","last_became_peered":"2024-04-25T01:52:43.075462+0000","last_unstale":"2024-04-25T01:53:43.860595+0000","last_undegraded":"2024-04-25T01:53:43.860595+0000","last_fullsized":"2024-04-25T01:53:43.860595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:07.077047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859164+0000","last_change":"2024-04-25T01:52:44.090695+0000","last_active":"2024-04-25T01:53:43.859164+0000","last_peered":"2024-04-25T01:53:43.859164+0000","last_clean":"2024-04-25T01:53:43.859164+0000","last_became_active":"2024-04-25T01:52:44.090203+0000","last_became_peered":"2024-04-25T01:52:44.090203+0000","last_unstale":"2024-04-25T01:53:43.859164+0000","last_undegraded":"2024-04-25T01:53:43.859164+0000","last_fullsized":"2024-04-25T01:53:43.859164+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:58.104569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.866657+0000","last_change":"2024-04-25T01:52:43.086057+0000","last_active":"2024-04-25T01:53:43.866657+0000","last_peered":"2024-04-25T01:53:43.866657+0000","last_clean":"2024-04-25T01:53:43.866657+0000","last_became_active":"2024-04-25T01:52:43.085475+0000","last_became_peered":"2024-04-25T01:52:43.085475+0000","last_unstale":"2024-04-25T01:53:43.866657+0000","last_undegraded":"2024-04-25T01:53:43.866657+0000","last_fullsized":"2024-04-25T01:53:43.866657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:42.237414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858153+0000","last_change":"2024-04-25T01:52:44.106128+0000","last_active":"2024-04-25T01:53:43.858153+0000","last_peered":"2024-04-25T01:53:43.858153+0000","last_clean":"2024-04-25T01:53:43.858153+0000","last_became_active":"2024-04-25T01:52:44.105805+0000","last_became_peered":"2024-04-25T01:52:44.105805+0000","last_unstale":"2024-04-25T01:53:43.858153+0000","last_undegraded":"2024-04-25T01:53:43.858153+0000","last_fullsized":"2024-04-25T01:53:43.858153+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:41.901070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856616+0000","last_change":"2024-04-25T01:52:43.088347+0000","last_active":"2024-04-25T01:53:43.856616+0000","last_peered":"2024-04-25T01:53:43.856616+0000","last_clean":"2024-04-25T01:53:43.856616+0000","last_became_active":"2024-04-25T01:52:43.087932+0000","last_became_peered":"2024-04-25T01:52:43.087932+0000","last_unstale":"2024-04-25T01:53:43.856616+0000","last_undegraded":"2024-04-25T01:53:43.856616+0000","last_fullsized":"2024-04-25T01:53:43.856616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:26.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.867219+0000","last_change":"2024-04-25T01:52:44.108686+0000","last_active":"2024-04-25T01:53:43.867219+0000","last_peered":"2024-04-25T01:53:43.867219+0000","last_clean":"2024-04-25T01:53:43.867219+0000","last_became_active":"2024-04-25T01:52:44.107732+0000","last_became_peered":"2024-04-25T01:52:44.107732+0000","last_unstale":"2024-04-25T01:53:43.867219+0000","last_undegraded":"2024-04-25T01:53:43.867219+0000","last_fullsized":"2024-04-25T01:53:43.867219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:14.820979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859230+0000","last_change":"2024-04-25T01:52:43.069300+0000","last_active":"2024-04-25T01:53:43.859230+0000","last_peered":"2024-04-25T01:53:43.859230+0000","last_clean":"2024-04-25T01:53:43.859230+0000","last_became_active":"2024-04-25T01:52:43.068644+0000","last_became_peered":"2024-04-25T01:52:43.068644+0000","last_unstale":"2024-04-25T01:53:43.859230+0000","last_undegraded":"2024-04-25T01:53:43.859230+0000","last_fullsized":"2024-04-25T01:53:43.859230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:04.897638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856156+0000","last_change":"2024-04-25T01:52:44.100467+0000","last_active":"2024-04-25T01:53:43.856156+0000","last_peered":"2024-04-25T01:53:43.856156+0000","last_clean":"2024-04-25T01:53:43.856156+0000","last_became_active":"2024-04-25T01:52:44.099958+0000","last_became_peered":"2024-04-25T01:52:44.099958+0000","last_unstale":"2024-04-25T01:53:43.856156+0000","last_undegraded":"2024-04-25T01:53:43.856156+0000","last_fullsized":"2024-04-25T01:53:43.856156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:29.919785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.875865+0000","last_change":"2024-04-25T01:53:43.875865+0000","last_active":"2024-04-25T01:53:43.875865+0000","last_peered":"2024-04-25T01:53:43.875865+0000","last_clean":"2024-04-25T01:53:43.875865+0000","last_became_active":"2024-04-25T01:53:43.871525+0000","last_became_peered":"2024-04-25T01:53:43.871525+0000","last_unstale":"2024-04-25T01:53:43.875865+0000","last_undegraded":"2024-04-25T01:53:43.875865+0000","last_fullsized":"2024-04-25T01:53:43.875865+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:25.179902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856322+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:53:43.856322+0000","last_peered":"2024-04-25T01:53:43.856322+0000","last_clean":"2024-04-25T01:53:43.856322+0000","last_became_active":"2024-04-25T01:52:44.087911+0000","last_became_peered":"2024-04-25T01:52:44.087911+0000","last_unstale":"2024-04-25T01:53:43.856322+0000","last_undegraded":"2024-04-25T01:53:43.856322+0000","last_fullsized":"2024-04-25T01:53:43.856322+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:29:38.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864407+0000","last_change":"2024-04-25T01:52:43.086083+0000","last_active":"2024-04-25T01:53:43.864407+0000","last_peered":"2024-04-25T01:53:43.864407+0000","last_clean":"2024-04-25T01:53:43.864407+0000","last_became_active":"2024-04-25T01:52:43.085613+0000","last_became_peered":"2024-04-25T01:52:43.085613+0000","last_unstale":"2024-04-25T01:53:43.864407+0000","last_undegraded":"2024-04-25T01:53:43.864407+0000","last_fullsized":"2024-04-25T01:53:43.864407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:23.851812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"87'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856062+0000","last_change":"2024-04-25T01:52:44.106435+0000","last_active":"2024-04-25T01:53:43.856062+0000","last_peered":"2024-04-25T01:53:43.856062+0000","last_clean":"2024-04-25T01:53:43.856062+0000","last_became_active":"2024-04-25T01:52:44.106074+0000","last_became_peered":"2024-04-25T01:52:44.106074+0000","last_unstale":"2024-04-25T01:53:43.856062+0000","last_undegraded":"2024-04-25T01:53:43.856062+0000","last_fullsized":"2024-04-25T01:53:43.856062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:59.286266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864297+0000","last_change":"2024-04-25T01:52:43.088902+0000","last_active":"2024-04-25T01:53:43.864297+0000","last_peered":"2024-04-25T01:53:43.864297+0000","last_clean":"2024-04-25T01:53:43.864297+0000","last_became_active":"2024-04-25T01:52:43.088289+0000","last_became_peered":"2024-04-25T01:52:43.088289+0000","last_unstale":"2024-04-25T01:53:43.864297+0000","last_undegraded":"2024-04-25T01:53:43.864297+0000","last_fullsized":"2024-04-25T01:53:43.864297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:37:44.562369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856402+0000","last_change":"2024-04-25T01:52:44.094185+0000","last_active":"2024-04-25T01:53:43.856402+0000","last_peered":"2024-04-25T01:53:43.856402+0000","last_clean":"2024-04-25T01:53:43.856402+0000","last_became_active":"2024-04-25T01:52:44.093753+0000","last_became_peered":"2024-04-25T01:52:44.093753+0000","last_unstale":"2024-04-25T01:53:43.856402+0000","last_undegraded":"2024-04-25T01:53:43.856402+0000","last_fullsized":"2024-04-25T01:53:43.856402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:06.089670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858163+0000","last_change":"2024-04-25T01:52:44.093895+0000","last_active":"2024-04-25T01:53:43.858163+0000","last_peered":"2024-04-25T01:53:43.858163+0000","last_clean":"2024-04-25T01:53:43.858163+0000","last_became_active":"2024-04-25T01:52:44.093490+0000","last_became_peered":"2024-04-25T01:52:44.093490+0000","last_unstale":"2024-04-25T01:53:43.858163+0000","last_undegraded":"2024-04-25T01:53:43.858163+0000","last_fullsized":"2024-04-25T01:53:43.858163+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:53.848979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.733855+0000","last_change":"2024-04-25T01:52:44.092500+0000","last_active":"2024-04-25T01:54:12.733855+0000","last_peered":"2024-04-25T01:54:12.733855+0000","last_clean":"2024-04-25T01:54:12.733855+0000","last_became_active":"2024-04-25T01:52:44.092044+0000","last_became_peered":"2024-04-25T01:52:44.092044+0000","last_unstale":"2024-04-25T01:54:12.733855+0000","last_undegraded":"2024-04-25T01:54:12.733855+0000","last_fullsized":"2024-04-25T01:54:12.733855+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:28.279676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.757034+0000","last_change":"2024-04-25T01:52:44.116668+0000","last_active":"2024-04-25T01:54:12.757034+0000","last_peered":"2024-04-25T01:54:12.757034+0000","last_clean":"2024-04-25T01:54:12.757034+0000","last_became_active":"2024-04-25T01:52:44.116295+0000","last_became_peered":"2024-04-25T01:52:44.116295+0000","last_unstale":"2024-04-25T01:54:12.757034+0000","last_undegraded":"2024-04-25T01:54:12.757034+0000","last_fullsized":"2024-04-25T01:54:12.757034+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:49.109403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.868993+0000","last_change":"2024-04-25T01:52:44.109950+0000","last_active":"2024-04-25T01:53:43.868993+0000","last_peered":"2024-04-25T01:53:43.868993+0000","last_clean":"2024-04-25T01:53:43.868993+0000","last_became_active":"2024-04-25T01:52:44.109636+0000","last_became_peered":"2024-04-25T01:52:44.109636+0000","last_unstale":"2024-04-25T01:53:43.868993+0000","last_undegraded":"2024-04-25T01:53:43.868993+0000","last_fullsized":"2024-04-25T01:53:43.868993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:02:57.610067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.863146+0000","last_change":"2024-04-25T01:52:44.105237+0000","last_active":"2024-04-25T01:53:43.863146+0000","last_peered":"2024-04-25T01:53:43.863146+0000","last_clean":"2024-04-25T01:53:43.863146+0000","last_became_active":"2024-04-25T01:52:44.104531+0000","last_became_peered":"2024-04-25T01:52:44.104531+0000","last_unstale":"2024-04-25T01:53:43.863146+0000","last_undegraded":"2024-04-25T01:53:43.863146+0000","last_fullsized":"2024-04-25T01:53:43.863146+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.149119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.857339+0000","last_change":"2024-04-25T01:52:44.095655+0000","last_active":"2024-04-25T01:53:43.857339+0000","last_peered":"2024-04-25T01:53:43.857339+0000","last_clean":"2024-04-25T01:53:43.857339+0000","last_became_active":"2024-04-25T01:52:44.095300+0000","last_became_peered":"2024-04-25T01:52:44.095300+0000","last_unstale":"2024-04-25T01:53:43.857339+0000","last_undegraded":"2024-04-25T01:53:43.857339+0000","last_fullsized":"2024-04-25T01:53:43.857339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:51.240170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.732535+0000","last_change":"2024-04-25T01:52:44.087382+0000","last_active":"2024-04-25T01:54:12.732535+0000","last_peered":"2024-04-25T01:54:12.732535+0000","last_clean":"2024-04-25T01:54:12.732535+0000","last_became_active":"2024-04-25T01:52:44.086815+0000","last_became_peered":"2024-04-25T01:52:44.086815+0000","last_unstale":"2024-04-25T01:54:12.732535+0000","last_undegraded":"2024-04-25T01:54:12.732535+0000","last_fullsized":"2024-04-25T01:54:12.732535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:52:13.682819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.871148+0000","last_change":"2024-04-25T01:53:43.871148+0000","last_active":"2024-04-25T01:53:43.871148+0000","last_peered":"2024-04-25T01:53:43.871148+0000","last_clean":"2024-04-25T01:53:43.871148+0000","last_became_active":"2024-04-25T01:53:43.870490+0000","last_became_peered":"2024-04-25T01:53:43.870490+0000","last_unstale":"2024-04-25T01:53:43.871148+0000","last_undegraded":"2024-04-25T01:53:43.871148+0000","last_fullsized":"2024-04-25T01:53:43.871148+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:41.790442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864152+0000","last_change":"2024-04-25T01:52:44.114767+0000","last_active":"2024-04-25T01:53:43.864152+0000","last_peered":"2024-04-25T01:53:43.864152+0000","last_clean":"2024-04-25T01:53:43.864152+0000","last_became_active":"2024-04-25T01:52:44.114483+0000","last_became_peered":"2024-04-25T01:52:44.114483+0000","last_unstale":"2024-04-25T01:53:43.864152+0000","last_undegraded":"2024-04-25T01:53:43.864152+0000","last_fullsized":"2024-04-25T01:53:43.864152+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:14.439287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.866501+0000","last_change":"2024-04-25T01:52:44.094424+0000","last_active":"2024-04-25T01:53:43.866501+0000","last_peered":"2024-04-25T01:53:43.866501+0000","last_clean":"2024-04-25T01:53:43.866501+0000","last_became_active":"2024-04-25T01:52:44.093074+0000","last_became_peered":"2024-04-25T01:52:44.093074+0000","last_unstale":"2024-04-25T01:53:43.866501+0000","last_undegraded":"2024-04-25T01:53:43.866501+0000","last_fullsized":"2024-04-25T01:53:43.866501+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:20.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.860882+0000","last_change":"2024-04-25T01:52:44.113465+0000","last_active":"2024-04-25T01:53:43.860882+0000","last_peered":"2024-04-25T01:53:43.860882+0000","last_clean":"2024-04-25T01:53:43.860882+0000","last_became_active":"2024-04-25T01:52:44.113168+0000","last_became_peered":"2024-04-25T01:52:44.113168+0000","last_unstale":"2024-04-25T01:53:43.860882+0000","last_undegraded":"2024-04-25T01:53:43.860882+0000","last_fullsized":"2024-04-25T01:53:43.860882+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:42:47.758996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.865681+0000","last_change":"2024-04-25T01:52:44.104763+0000","last_active":"2024-04-25T01:53:43.865681+0000","last_peered":"2024-04-25T01:53:43.865681+0000","last_clean":"2024-04-25T01:53:43.865681+0000","last_became_active":"2024-04-25T01:52:44.104377+0000","last_became_peered":"2024-04-25T01:52:44.104377+0000","last_unstale":"2024-04-25T01:53:43.865681+0000","last_undegraded":"2024-04-25T01:53:43.865681+0000","last_fullsized":"2024-04-25T01:53:43.865681+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:13.647020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864792+0000","last_change":"2024-04-25T01:53:43.864792+0000","last_active":"2024-04-25T01:53:43.864792+0000","last_peered":"2024-04-25T01:53:43.864792+0000","last_clean":"2024-04-25T01:53:43.864792+0000","last_became_active":"2024-04-25T01:53:43.864432+0000","last_became_peered":"2024-04-25T01:53:43.864432+0000","last_unstale":"2024-04-25T01:53:43.864792+0000","last_undegraded":"2024-04-25T01:53:43.864792+0000","last_fullsized":"2024-04-25T01:53:43.864792+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:28.149564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65047,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":513,"num_read_kb":322,"num_write":156,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":61762,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":195531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318306,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":463020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579941,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41120,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704032,"statfs":{"total":95995035648,"available":95952928768,"internally_reserved":0,"allocated":1654784,"data_stored":1115413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743468,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":463122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005106,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":462908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266744,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28552,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716600,"statfs":{"total":95995035648,"available":95965798400,"internally_reserved":0,"allocated":1568768,"data_stored":1054972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430268,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":462511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":523350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738461,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":983040,"data_stored":463782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":463604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117069,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":462822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855432,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":524576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691906,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1581056,"data_stored":1054656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:54:17.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T01:54:17.907 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:17 smithi102 ceph-mon[27796]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:18.087 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:17 smithi102 ceph-mon[27796]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-25T01:54:18.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:17 smithi110 ceph-mon[32848]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:18.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:17 smithi110 ceph-mon[32848]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-25T01:54:18.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:17 smithi193 ceph-mon[32981]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:18.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:17 smithi193 ceph-mon[32981]: pgmap v366: 97 pgs: 97 active+clean; 641 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-25T01:54:19.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:19 smithi102 ceph-mon[27796]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:54:20.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.214+0000 7f39b9600700 1 -- 172.21.15.102:0/4283943067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 msgr2=0x7f39b4108ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.214+0000 7f39b9600700 1 --2- 172.21.15.102:0/4283943067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 0x7f39b4108ce0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009990 tx=0x7f39a0009ca0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.214+0000 7f39b9600700 1 -- 172.21.15.102:0/4283943067 shutdown_connections 2024-04-25T01:54:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.214+0000 7f39b9600700 1 --2- 172.21.15.102:0/4283943067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39b410e860 0x7f39b410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.214+0000 7f39b9600700 1 --2- 172.21.15.102:0/4283943067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b4109220 0x7f39b4115730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.214+0000 7f39b9600700 1 --2- 172.21.15.102:0/4283943067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 0x7f39b4108ce0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.214+0000 7f39b9600700 1 -- 172.21.15.102:0/4283943067 >> 172.21.15.102:0/4283943067 conn(0x7f39b4075a00 msgr2=0x7f39b4107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.214+0000 7f39b9600700 1 -- 172.21.15.102:0/4283943067 shutdown_connections 2024-04-25T01:54:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.215+0000 7f39b9600700 1 -- 172.21.15.102:0/4283943067 wait complete. 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.216+0000 7f39b9600700 1 Processor -- start 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.216+0000 7f39b9600700 1 -- start start 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.216+0000 7f39b9600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 0x7f39b419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.216+0000 7f39b9600700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39b4109220 0x7f39b419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.216+0000 7f39b9600700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b410e860 0x7f39b41a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.216+0000 7f39b9600700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4118bf0 con 0x7f39b410e860 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.216+0000 7f39b9600700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4118a70 con 0x7f39b4109220 2024-04-25T01:54:20.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.216+0000 7f39b9600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4118d70 con 0x7f39b4108860 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b3400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b410e860 0x7f39b41a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 0x7f39b419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 0x7f39b419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:57544/0 (socket says 172.21.15.102:57544) 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2a00700 1 -- 172.21.15.102:0/2301316459 learned_addr learned my addr 172.21.15.102:0/2301316459 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2000700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39b4109220 0x7f39b419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2a00700 1 -- 172.21.15.102:0/2301316459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39b4109220 msgr2=0x7f39b419f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2a00700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39b4109220 0x7f39b419f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2a00700 1 -- 172.21.15.102:0/2301316459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b410e860 msgr2=0x7f39b41a3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2a00700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b410e860 0x7f39b41a3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2a00700 1 -- 172.21.15.102:0/2301316459 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b41a3fc0 con 0x7f39b4108860 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.217+0000 7f39b2000700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39b4109220 0x7f39b419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.218+0000 7f39b2a00700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 0x7f39b419f000 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f39a400e9a0 tx=0x7f39a400ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.218+0000 7f39ab400700 1 -- 172.21.15.102:0/2301316459 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a400cbd0 con 0x7f39b4108860 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.218+0000 7f39ab400700 1 -- 172.21.15.102:0/2301316459 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39a400cd70 con 0x7f39b4108860 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.218+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b41a42b0 con 0x7f39b4108860 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.218+0000 7f39ab400700 1 -- 172.21.15.102:0/2301316459 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4018710 con 0x7f39b4108860 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.218+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b41abb30 con 0x7f39b4108860 2024-04-25T01:54:20.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.219+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b4068f20 con 0x7f39b4108860 2024-04-25T01:54:20.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.221+0000 7f39ab400700 1 -- 172.21.15.102:0/2301316459 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f39a4010430 con 0x7f39b4108860 2024-04-25T01:54:20.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.222+0000 7f39ab400700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39980a2c20 0x7f39980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:20.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.223+0000 7f39ab400700 1 -- 172.21.15.102:0/2301316459 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f39a40cf640 con 0x7f39b4108860 2024-04-25T01:54:20.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.223+0000 7f39b2000700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39980a2c20 0x7f39980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:20.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.223+0000 7f39b2000700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39980a2c20 0x7f39980a50e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f39a000b2e0 tx=0x7f39a001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.225+0000 7f39ab400700 1 -- 172.21.15.102:0/2301316459 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f39a409c550 con 0x7f39b4108860 2024-04-25T01:54:20.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:19 smithi110 ceph-mon[32848]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:54:20.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:19 smithi193 ceph-mon[32981]: pgmap v367: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:54:20.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.443+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f39b4062670 con 0x7f39980a2c20 2024-04-25T01:54:20.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.457+0000 7f39ab400700 1 -- 172.21.15.102:0/2301316459 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258964 (secure 0 0 0) 0x7f39a00663b0 con 0x7f39980a2c20 2024-04-25T01:54:20.462 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:20.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.462+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39980a2c20 msgr2=0x7f39980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.462+0000 7f39b9600700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39980a2c20 0x7f39980a50e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f39a000b2e0 tx=0x7f39a001a040 comp rx=0 tx=0).stop 2024-04-25T01:54:20.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.462+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 msgr2=0x7f39b419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:20.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.462+0000 7f39b9600700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 0x7f39b419f000 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f39a400e9a0 tx=0x7f39a400ed60 comp rx=0 tx=0).stop 2024-04-25T01:54:20.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.463+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 shutdown_connections 2024-04-25T01:54:20.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.463+0000 7f39b9600700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f39980a2c20 0x7f39980a50e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.463+0000 7f39b9600700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b410e860 0x7f39b41a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.463+0000 7f39b9600700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108860 0x7f39b419f000 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.463+0000 7f39b9600700 1 --2- 172.21.15.102:0/2301316459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39b4109220 0x7f39b419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:20.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.463+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 >> 172.21.15.102:0/2301316459 conn(0x7f39b4075a00 msgr2=0x7f39b4075fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:20.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.463+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 shutdown_connections 2024-04-25T01:54:20.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:20.463+0000 7f39b9600700 1 -- 172.21.15.102:0/2301316459 wait complete. 2024-04-25T01:54:20.465 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-25T01:54:21.375 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-04-25T01:54:18.759958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656039,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":681,"num_read_kb":458,"num_write":346,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1242498,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":780,"ondisk_log_size":780,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383308,"kb_used_data":13356,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124558516,"statfs":{"total":1151940427776,"available":1151547920384,"internally_reserved":0,"allocated":13676544,"data_stored":7515765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":378713403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":72,"num_read_kb":32,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005265"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.748495+0000","last_change":"2024-04-25T01:52:44.102143+0000","last_active":"2024-04-25T01:54:12.748495+0000","last_peered":"2024-04-25T01:54:12.748495+0000","last_clean":"2024-04-25T01:54:12.748495+0000","last_became_active":"2024-04-25T01:52:44.101810+0000","last_became_peered":"2024-04-25T01:52:44.101810+0000","last_unstale":"2024-04-25T01:54:12.748495+0000","last_undegraded":"2024-04-25T01:54:12.748495+0000","last_fullsized":"2024-04-25T01:54:12.748495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:02.356294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.741072+0000","last_change":"2024-04-25T01:52:44.103074+0000","last_active":"2024-04-25T01:54:12.741072+0000","last_peered":"2024-04-25T01:54:12.741072+0000","last_clean":"2024-04-25T01:54:12.741072+0000","last_became_active":"2024-04-25T01:52:44.102103+0000","last_became_peered":"2024-04-25T01:52:44.102103+0000","last_unstale":"2024-04-25T01:54:12.741072+0000","last_undegraded":"2024-04-25T01:54:12.741072+0000","last_fullsized":"2024-04-25T01:54:12.741072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:36:43.016756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.750691+0000","last_change":"2024-04-25T01:52:44.101467+0000","last_active":"2024-04-25T01:54:12.750691+0000","last_peered":"2024-04-25T01:54:12.750691+0000","last_clean":"2024-04-25T01:54:12.750691+0000","last_became_active":"2024-04-25T01:52:44.100911+0000","last_became_peered":"2024-04-25T01:52:44.100911+0000","last_unstale":"2024-04-25T01:54:12.750691+0000","last_undegraded":"2024-04-25T01:54:12.750691+0000","last_fullsized":"2024-04-25T01:54:12.750691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:58.533213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.734344+0000","last_change":"2024-04-25T01:52:44.088592+0000","last_active":"2024-04-25T01:54:12.734344+0000","last_peered":"2024-04-25T01:54:12.734344+0000","last_clean":"2024-04-25T01:54:12.734344+0000","last_became_active":"2024-04-25T01:52:44.087651+0000","last_became_peered":"2024-04-25T01:52:44.087651+0000","last_unstale":"2024-04-25T01:54:12.734344+0000","last_undegraded":"2024-04-25T01:54:12.734344+0000","last_fullsized":"2024-04-25T01:54:12.734344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:52.240818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.752993+0000","last_change":"2024-04-25T01:52:44.095033+0000","last_active":"2024-04-25T01:54:12.752993+0000","last_peered":"2024-04-25T01:54:12.752993+0000","last_clean":"2024-04-25T01:54:12.752993+0000","last_became_active":"2024-04-25T01:52:44.094719+0000","last_became_peered":"2024-04-25T01:52:44.094719+0000","last_unstale":"2024-04-25T01:54:12.752993+0000","last_undegraded":"2024-04-25T01:54:12.752993+0000","last_fullsized":"2024-04-25T01:54:12.752993+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:52.422544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.741178+0000","last_change":"2024-04-25T01:52:44.102574+0000","last_active":"2024-04-25T01:54:12.741178+0000","last_peered":"2024-04-25T01:54:12.741178+0000","last_clean":"2024-04-25T01:54:12.741178+0000","last_became_active":"2024-04-25T01:52:44.102158+0000","last_became_peered":"2024-04-25T01:52:44.102158+0000","last_unstale":"2024-04-25T01:54:12.741178+0000","last_undegraded":"2024-04-25T01:54:12.741178+0000","last_fullsized":"2024-04-25T01:54:12.741178+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:20.931392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.742951+0000","last_change":"2024-04-25T01:52:44.107662+0000","last_active":"2024-04-25T01:54:12.742951+0000","last_peered":"2024-04-25T01:54:12.742951+0000","last_clean":"2024-04-25T01:54:12.742951+0000","last_became_active":"2024-04-25T01:52:44.107304+0000","last_became_peered":"2024-04-25T01:52:44.107304+0000","last_unstale":"2024-04-25T01:54:12.742951+0000","last_undegraded":"2024-04-25T01:54:12.742951+0000","last_fullsized":"2024-04-25T01:54:12.742951+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:42:54.636950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.756354+0000","last_change":"2024-04-25T01:52:44.100273+0000","last_active":"2024-04-25T01:54:12.756354+0000","last_peered":"2024-04-25T01:54:12.756354+0000","last_clean":"2024-04-25T01:54:12.756354+0000","last_became_active":"2024-04-25T01:52:44.098223+0000","last_became_peered":"2024-04-25T01:52:44.098223+0000","last_unstale":"2024-04-25T01:54:12.756354+0000","last_undegraded":"2024-04-25T01:54:12.756354+0000","last_fullsized":"2024-04-25T01:54:12.756354+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:05.003265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.747129+0000","last_change":"2024-04-25T01:52:44.089658+0000","last_active":"2024-04-25T01:54:12.747129+0000","last_peered":"2024-04-25T01:54:12.747129+0000","last_clean":"2024-04-25T01:54:12.747129+0000","last_became_active":"2024-04-25T01:52:44.089232+0000","last_became_peered":"2024-04-25T01:52:44.089232+0000","last_unstale":"2024-04-25T01:54:12.747129+0000","last_undegraded":"2024-04-25T01:54:12.747129+0000","last_fullsized":"2024-04-25T01:54:12.747129+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:54.385782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.738459+0000","last_change":"2024-04-25T01:52:44.086660+0000","last_active":"2024-04-25T01:54:12.738459+0000","last_peered":"2024-04-25T01:54:12.738459+0000","last_clean":"2024-04-25T01:54:12.738459+0000","last_became_active":"2024-04-25T01:52:44.086306+0000","last_became_peered":"2024-04-25T01:52:44.086306+0000","last_unstale":"2024-04-25T01:54:12.738459+0000","last_undegraded":"2024-04-25T01:54:12.738459+0000","last_fullsized":"2024-04-25T01:54:12.738459+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:33:20.726616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.746073+0000","last_change":"2024-04-25T01:52:44.090239+0000","last_active":"2024-04-25T01:54:12.746073+0000","last_peered":"2024-04-25T01:54:12.746073+0000","last_clean":"2024-04-25T01:54:12.746073+0000","last_became_active":"2024-04-25T01:52:44.089736+0000","last_became_peered":"2024-04-25T01:52:44.089736+0000","last_unstale":"2024-04-25T01:54:12.746073+0000","last_undegraded":"2024-04-25T01:54:12.746073+0000","last_fullsized":"2024-04-25T01:54:12.746073+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:22.954011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.730383+0000","last_change":"2024-04-25T01:52:44.088878+0000","last_active":"2024-04-25T01:54:12.730383+0000","last_peered":"2024-04-25T01:54:12.730383+0000","last_clean":"2024-04-25T01:54:12.730383+0000","last_became_active":"2024-04-25T01:52:44.088250+0000","last_became_peered":"2024-04-25T01:52:44.088250+0000","last_unstale":"2024-04-25T01:54:12.730383+0000","last_undegraded":"2024-04-25T01:54:12.730383+0000","last_fullsized":"2024-04-25T01:54:12.730383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:15.490131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.746477+0000","last_change":"2024-04-25T01:52:44.094675+0000","last_active":"2024-04-25T01:54:12.746477+0000","last_peered":"2024-04-25T01:54:12.746477+0000","last_clean":"2024-04-25T01:54:12.746477+0000","last_became_active":"2024-04-25T01:52:44.094219+0000","last_became_peered":"2024-04-25T01:52:44.094219+0000","last_unstale":"2024-04-25T01:54:12.746477+0000","last_undegraded":"2024-04-25T01:54:12.746477+0000","last_fullsized":"2024-04-25T01:54:12.746477+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:26.160156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.743240+0000","last_change":"2024-04-25T01:52:44.111131+0000","last_active":"2024-04-25T01:54:12.743240+0000","last_peered":"2024-04-25T01:54:12.743240+0000","last_clean":"2024-04-25T01:54:12.743240+0000","last_became_active":"2024-04-25T01:52:44.105670+0000","last_became_peered":"2024-04-25T01:52:44.105670+0000","last_unstale":"2024-04-25T01:54:12.743240+0000","last_undegraded":"2024-04-25T01:54:12.743240+0000","last_fullsized":"2024-04-25T01:54:12.743240+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:00.377724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.747283+0000","last_change":"2024-04-25T01:52:44.109707+0000","last_active":"2024-04-25T01:54:12.747283+0000","last_peered":"2024-04-25T01:54:12.747283+0000","last_clean":"2024-04-25T01:54:12.747283+0000","last_became_active":"2024-04-25T01:52:44.108550+0000","last_became_peered":"2024-04-25T01:52:44.108550+0000","last_unstale":"2024-04-25T01:54:12.747283+0000","last_undegraded":"2024-04-25T01:54:12.747283+0000","last_fullsized":"2024-04-25T01:54:12.747283+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:46.296789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.754672+0000","last_change":"2024-04-25T01:52:44.104504+0000","last_active":"2024-04-25T01:54:12.754672+0000","last_peered":"2024-04-25T01:54:12.754672+0000","last_clean":"2024-04-25T01:54:12.754672+0000","last_became_active":"2024-04-25T01:52:44.104180+0000","last_became_peered":"2024-04-25T01:52:44.104180+0000","last_unstale":"2024-04-25T01:54:12.754672+0000","last_undegraded":"2024-04-25T01:54:12.754672+0000","last_fullsized":"2024-04-25T01:54:12.754672+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:17.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.748671+0000","last_change":"2024-04-25T01:52:44.107550+0000","last_active":"2024-04-25T01:54:12.748671+0000","last_peered":"2024-04-25T01:54:12.748671+0000","last_clean":"2024-04-25T01:54:12.748671+0000","last_became_active":"2024-04-25T01:52:44.107195+0000","last_became_peered":"2024-04-25T01:52:44.107195+0000","last_unstale":"2024-04-25T01:54:12.748671+0000","last_undegraded":"2024-04-25T01:54:12.748671+0000","last_fullsized":"2024-04-25T01:54:12.748671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:59:42.220575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.754929+0000","last_change":"2024-04-25T01:52:44.101165+0000","last_active":"2024-04-25T01:54:12.754929+0000","last_peered":"2024-04-25T01:54:12.754929+0000","last_clean":"2024-04-25T01:54:12.754929+0000","last_became_active":"2024-04-25T01:52:44.100386+0000","last_became_peered":"2024-04-25T01:52:44.100386+0000","last_unstale":"2024-04-25T01:54:12.754929+0000","last_undegraded":"2024-04-25T01:54:12.754929+0000","last_fullsized":"2024-04-25T01:54:12.754929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:30.922707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.754083+0000","last_change":"2024-04-25T01:52:44.116694+0000","last_active":"2024-04-25T01:54:12.754083+0000","last_peered":"2024-04-25T01:54:12.754083+0000","last_clean":"2024-04-25T01:54:12.754083+0000","last_became_active":"2024-04-25T01:52:44.116326+0000","last_became_peered":"2024-04-25T01:52:44.116326+0000","last_unstale":"2024-04-25T01:54:12.754083+0000","last_undegraded":"2024-04-25T01:54:12.754083+0000","last_fullsized":"2024-04-25T01:54:12.754083+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:16.329899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858757+0000","last_change":"2024-04-25T01:52:43.072358+0000","last_active":"2024-04-25T01:53:43.858757+0000","last_peered":"2024-04-25T01:53:43.858757+0000","last_clean":"2024-04-25T01:53:43.858757+0000","last_became_active":"2024-04-25T01:52:43.071815+0000","last_became_peered":"2024-04-25T01:52:43.071815+0000","last_unstale":"2024-04-25T01:53:43.858757+0000","last_undegraded":"2024-04-25T01:53:43.858757+0000","last_fullsized":"2024-04-25T01:53:43.858757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:15.295744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.755137+0000","last_change":"2024-04-25T01:52:44.094509+0000","last_active":"2024-04-25T01:54:12.755137+0000","last_peered":"2024-04-25T01:54:12.755137+0000","last_clean":"2024-04-25T01:54:12.755137+0000","last_became_active":"2024-04-25T01:52:44.093955+0000","last_became_peered":"2024-04-25T01:52:44.093955+0000","last_unstale":"2024-04-25T01:54:12.755137+0000","last_undegraded":"2024-04-25T01:54:12.755137+0000","last_fullsized":"2024-04-25T01:54:12.755137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:37.701835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.867618+0000","last_change":"2024-04-25T01:52:43.094435+0000","last_active":"2024-04-25T01:53:43.867618+0000","last_peered":"2024-04-25T01:53:43.867618+0000","last_clean":"2024-04-25T01:53:43.867618+0000","last_became_active":"2024-04-25T01:52:43.094018+0000","last_became_peered":"2024-04-25T01:52:43.094018+0000","last_unstale":"2024-04-25T01:53:43.867618+0000","last_undegraded":"2024-04-25T01:53:43.867618+0000","last_fullsized":"2024-04-25T01:53:43.867618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:36.781014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.751637+0000","last_change":"2024-04-25T01:52:44.104347+0000","last_active":"2024-04-25T01:54:12.751637+0000","last_peered":"2024-04-25T01:54:12.751637+0000","last_clean":"2024-04-25T01:54:12.751637+0000","last_became_active":"2024-04-25T01:52:44.103803+0000","last_became_peered":"2024-04-25T01:52:44.103803+0000","last_unstale":"2024-04-25T01:54:12.751637+0000","last_undegraded":"2024-04-25T01:54:12.751637+0000","last_fullsized":"2024-04-25T01:54:12.751637+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:01.784893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859364+0000","last_change":"2024-04-25T01:52:43.070902+0000","last_active":"2024-04-25T01:53:43.859364+0000","last_peered":"2024-04-25T01:53:43.859364+0000","last_clean":"2024-04-25T01:53:43.859364+0000","last_became_active":"2024-04-25T01:52:43.070313+0000","last_became_peered":"2024-04-25T01:52:43.070313+0000","last_unstale":"2024-04-25T01:53:43.859364+0000","last_undegraded":"2024-04-25T01:53:43.859364+0000","last_fullsized":"2024-04-25T01:53:43.859364+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:10.677320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.749417+0000","last_change":"2024-04-25T01:52:44.093293+0000","last_active":"2024-04-25T01:54:12.749417+0000","last_peered":"2024-04-25T01:54:12.749417+0000","last_clean":"2024-04-25T01:54:12.749417+0000","last_became_active":"2024-04-25T01:52:44.092932+0000","last_became_peered":"2024-04-25T01:52:44.092932+0000","last_unstale":"2024-04-25T01:54:12.749417+0000","last_undegraded":"2024-04-25T01:54:12.749417+0000","last_fullsized":"2024-04-25T01:54:12.749417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:58:37.705594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.860028+0000","last_change":"2024-04-25T01:52:43.069743+0000","last_active":"2024-04-25T01:53:43.860028+0000","last_peered":"2024-04-25T01:53:43.860028+0000","last_clean":"2024-04-25T01:53:43.860028+0000","last_became_active":"2024-04-25T01:52:43.068947+0000","last_became_peered":"2024-04-25T01:52:43.068947+0000","last_unstale":"2024-04-25T01:53:43.860028+0000","last_undegraded":"2024-04-25T01:53:43.860028+0000","last_fullsized":"2024-04-25T01:53:43.860028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:02:52.582176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.754289+0000","last_change":"2024-04-25T01:53:43.865168+0000","last_active":"2024-04-25T01:54:12.754289+0000","last_peered":"2024-04-25T01:54:12.754289+0000","last_clean":"2024-04-25T01:54:12.754289+0000","last_became_active":"2024-04-25T01:53:43.864055+0000","last_became_peered":"2024-04-25T01:53:43.864055+0000","last_unstale":"2024-04-25T01:54:12.754289+0000","last_undegraded":"2024-04-25T01:54:12.754289+0000","last_fullsized":"2024-04-25T01:54:12.754289+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:01.707773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"89'7","reported_seq":119,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:16.239989+0000","last_change":"2024-04-25T01:52:43.068450+0000","last_active":"2024-04-25T01:54:16.239989+0000","last_peered":"2024-04-25T01:54:16.239989+0000","last_clean":"2024-04-25T01:54:16.239989+0000","last_became_active":"2024-04-25T01:52:43.067881+0000","last_became_peered":"2024-04-25T01:52:43.067881+0000","last_unstale":"2024-04-25T01:54:16.239989+0000","last_undegraded":"2024-04-25T01:54:16.239989+0000","last_fullsized":"2024-04-25T01:54:16.239989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:12.739321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.735228+0000","last_change":"2024-04-25T01:52:44.101743+0000","last_active":"2024-04-25T01:54:12.735228+0000","last_peered":"2024-04-25T01:54:12.735228+0000","last_clean":"2024-04-25T01:54:12.735228+0000","last_became_active":"2024-04-25T01:52:44.101351+0000","last_became_peered":"2024-04-25T01:52:44.101351+0000","last_unstale":"2024-04-25T01:54:12.735228+0000","last_undegraded":"2024-04-25T01:54:12.735228+0000","last_fullsized":"2024-04-25T01:54:12.735228+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:54:24.104819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.857985+0000","last_change":"2024-04-25T01:52:43.075880+0000","last_active":"2024-04-25T01:53:43.857985+0000","last_peered":"2024-04-25T01:53:43.857985+0000","last_clean":"2024-04-25T01:53:43.857985+0000","last_became_active":"2024-04-25T01:52:43.075522+0000","last_became_peered":"2024-04-25T01:52:43.075522+0000","last_unstale":"2024-04-25T01:53:43.857985+0000","last_undegraded":"2024-04-25T01:53:43.857985+0000","last_fullsized":"2024-04-25T01:53:43.857985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:26.218774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.755743+0000","last_change":"2024-04-25T01:52:44.116772+0000","last_active":"2024-04-25T01:54:12.755743+0000","last_peered":"2024-04-25T01:54:12.755743+0000","last_clean":"2024-04-25T01:54:12.755743+0000","last_became_active":"2024-04-25T01:52:44.116408+0000","last_became_peered":"2024-04-25T01:52:44.116408+0000","last_unstale":"2024-04-25T01:54:12.755743+0000","last_undegraded":"2024-04-25T01:54:12.755743+0000","last_fullsized":"2024-04-25T01:54:12.755743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:21.988425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.869110+0000","last_change":"2024-04-25T01:52:43.076964+0000","last_active":"2024-04-25T01:53:43.869110+0000","last_peered":"2024-04-25T01:53:43.869110+0000","last_clean":"2024-04-25T01:53:43.869110+0000","last_became_active":"2024-04-25T01:52:43.073915+0000","last_became_peered":"2024-04-25T01:52:43.073915+0000","last_unstale":"2024-04-25T01:53:43.869110+0000","last_undegraded":"2024-04-25T01:53:43.869110+0000","last_fullsized":"2024-04-25T01:53:43.869110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:03.321920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.753438+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:54:12.753438+0000","last_peered":"2024-04-25T01:54:12.753438+0000","last_clean":"2024-04-25T01:54:12.753438+0000","last_became_active":"2024-04-25T01:52:44.088198+0000","last_became_peered":"2024-04-25T01:52:44.088198+0000","last_unstale":"2024-04-25T01:54:12.753438+0000","last_undegraded":"2024-04-25T01:54:12.753438+0000","last_fullsized":"2024-04-25T01:54:12.753438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:30.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864096+0000","last_change":"2024-04-25T01:52:43.069853+0000","last_active":"2024-04-25T01:53:43.864096+0000","last_peered":"2024-04-25T01:53:43.864096+0000","last_clean":"2024-04-25T01:53:43.864096+0000","last_became_active":"2024-04-25T01:52:43.069526+0000","last_became_peered":"2024-04-25T01:52:43.069526+0000","last_unstale":"2024-04-25T01:53:43.864096+0000","last_undegraded":"2024-04-25T01:53:43.864096+0000","last_fullsized":"2024-04-25T01:53:43.864096+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:35:03.272610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.758034+0000","last_change":"2024-04-25T01:52:44.114296+0000","last_active":"2024-04-25T01:54:12.758034+0000","last_peered":"2024-04-25T01:54:12.758034+0000","last_clean":"2024-04-25T01:54:12.758034+0000","last_became_active":"2024-04-25T01:52:44.113958+0000","last_became_peered":"2024-04-25T01:52:44.113958+0000","last_unstale":"2024-04-25T01:54:12.758034+0000","last_undegraded":"2024-04-25T01:54:12.758034+0000","last_fullsized":"2024-04-25T01:54:12.758034+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:45.068080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":110,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:14.236006+0000","last_change":"2024-04-25T01:52:43.071388+0000","last_active":"2024-04-25T01:54:14.236006+0000","last_peered":"2024-04-25T01:54:14.236006+0000","last_clean":"2024-04-25T01:54:14.236006+0000","last_became_active":"2024-04-25T01:52:43.071005+0000","last_became_peered":"2024-04-25T01:52:43.071005+0000","last_unstale":"2024-04-25T01:54:14.236006+0000","last_undegraded":"2024-04-25T01:54:14.236006+0000","last_fullsized":"2024-04-25T01:54:14.236006+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:56:59.580458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.749767+0000","last_change":"2024-04-25T01:52:44.106106+0000","last_active":"2024-04-25T01:54:12.749767+0000","last_peered":"2024-04-25T01:54:12.749767+0000","last_clean":"2024-04-25T01:54:12.749767+0000","last_became_active":"2024-04-25T01:52:44.105760+0000","last_became_peered":"2024-04-25T01:52:44.105760+0000","last_unstale":"2024-04-25T01:54:12.749767+0000","last_undegraded":"2024-04-25T01:54:12.749767+0000","last_fullsized":"2024-04-25T01:54:12.749767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:24.626753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856325+0000","last_change":"2024-04-25T01:52:43.068926+0000","last_active":"2024-04-25T01:53:43.856325+0000","last_peered":"2024-04-25T01:53:43.856325+0000","last_clean":"2024-04-25T01:53:43.856325+0000","last_became_active":"2024-04-25T01:52:43.067356+0000","last_became_peered":"2024-04-25T01:52:43.067356+0000","last_unstale":"2024-04-25T01:53:43.856325+0000","last_undegraded":"2024-04-25T01:53:43.856325+0000","last_fullsized":"2024-04-25T01:53:43.856325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:19.134633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.751002+0000","last_change":"2024-04-25T01:52:44.108299+0000","last_active":"2024-04-25T01:54:12.751002+0000","last_peered":"2024-04-25T01:54:12.751002+0000","last_clean":"2024-04-25T01:54:12.751002+0000","last_became_active":"2024-04-25T01:52:44.107573+0000","last_became_peered":"2024-04-25T01:52:44.107573+0000","last_unstale":"2024-04-25T01:54:12.751002+0000","last_undegraded":"2024-04-25T01:54:12.751002+0000","last_fullsized":"2024-04-25T01:54:12.751002+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.804579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864522+0000","last_change":"2024-04-25T01:52:43.079263+0000","last_active":"2024-04-25T01:53:43.864522+0000","last_peered":"2024-04-25T01:53:43.864522+0000","last_clean":"2024-04-25T01:53:43.864522+0000","last_became_active":"2024-04-25T01:52:43.078459+0000","last_became_peered":"2024-04-25T01:52:43.078459+0000","last_unstale":"2024-04-25T01:53:43.864522+0000","last_undegraded":"2024-04-25T01:53:43.864522+0000","last_fullsized":"2024-04-25T01:53:43.864522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:59.349897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.747761+0000","last_change":"2024-04-25T01:52:44.088044+0000","last_active":"2024-04-25T01:54:12.747761+0000","last_peered":"2024-04-25T01:54:12.747761+0000","last_clean":"2024-04-25T01:54:12.747761+0000","last_became_active":"2024-04-25T01:52:44.087682+0000","last_became_peered":"2024-04-25T01:52:44.087682+0000","last_unstale":"2024-04-25T01:54:12.747761+0000","last_undegraded":"2024-04-25T01:54:12.747761+0000","last_fullsized":"2024-04-25T01:54:12.747761+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:08.933923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"89'31","reported_seq":251,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:15.912793+0000","last_change":"2024-04-25T01:53:43.869516+0000","last_active":"2024-04-25T01:54:15.912793+0000","last_peered":"2024-04-25T01:54:15.912793+0000","last_clean":"2024-04-25T01:54:15.912793+0000","last_became_active":"2024-04-25T01:53:43.861718+0000","last_became_peered":"2024-04-25T01:53:43.861718+0000","last_unstale":"2024-04-25T01:54:15.912793+0000","last_undegraded":"2024-04-25T01:54:15.912793+0000","last_fullsized":"2024-04-25T01:54:15.912793+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:13:01.820264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61295,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":175,"num_read_kb":147,"num_write":32,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.716269+0000","last_change":"2024-04-25T01:52:44.091547+0000","last_active":"2024-04-25T01:54:12.716269+0000","last_peered":"2024-04-25T01:54:12.716269+0000","last_clean":"2024-04-25T01:54:12.716269+0000","last_became_active":"2024-04-25T01:52:44.091167+0000","last_became_peered":"2024-04-25T01:52:44.091167+0000","last_unstale":"2024-04-25T01:54:12.716269+0000","last_undegraded":"2024-04-25T01:54:12.716269+0000","last_fullsized":"2024-04-25T01:54:12.716269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:24.559694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856061+0000","last_change":"2024-04-25T01:52:43.088385+0000","last_active":"2024-04-25T01:53:43.856061+0000","last_peered":"2024-04-25T01:53:43.856061+0000","last_clean":"2024-04-25T01:53:43.856061+0000","last_became_active":"2024-04-25T01:52:43.087982+0000","last_became_peered":"2024-04-25T01:52:43.087982+0000","last_unstale":"2024-04-25T01:53:43.856061+0000","last_undegraded":"2024-04-25T01:53:43.856061+0000","last_fullsized":"2024-04-25T01:53:43.856061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:28.944427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.758173+0000","last_change":"2024-04-25T01:52:44.099420+0000","last_active":"2024-04-25T01:54:12.758173+0000","last_peered":"2024-04-25T01:54:12.758173+0000","last_clean":"2024-04-25T01:54:12.758173+0000","last_became_active":"2024-04-25T01:52:44.099045+0000","last_became_peered":"2024-04-25T01:52:44.099045+0000","last_unstale":"2024-04-25T01:54:12.758173+0000","last_undegraded":"2024-04-25T01:54:12.758173+0000","last_fullsized":"2024-04-25T01:54:12.758173+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:28.377091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864197+0000","last_change":"2024-04-25T01:52:43.086252+0000","last_active":"2024-04-25T01:53:43.864197+0000","last_peered":"2024-04-25T01:53:43.864197+0000","last_clean":"2024-04-25T01:53:43.864197+0000","last_became_active":"2024-04-25T01:52:43.085892+0000","last_became_peered":"2024-04-25T01:52:43.085892+0000","last_unstale":"2024-04-25T01:53:43.864197+0000","last_undegraded":"2024-04-25T01:53:43.864197+0000","last_fullsized":"2024-04-25T01:53:43.864197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:25:26.395705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.740356+0000","last_change":"2024-04-25T01:52:44.097163+0000","last_active":"2024-04-25T01:54:12.740356+0000","last_peered":"2024-04-25T01:54:12.740356+0000","last_clean":"2024-04-25T01:54:12.740356+0000","last_became_active":"2024-04-25T01:52:44.096758+0000","last_became_peered":"2024-04-25T01:52:44.096758+0000","last_unstale":"2024-04-25T01:54:12.740356+0000","last_undegraded":"2024-04-25T01:54:12.740356+0000","last_fullsized":"2024-04-25T01:54:12.740356+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:22:47.394855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"87'3","reported_seq":113,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:16.252078+0000","last_change":"2024-04-25T01:52:43.069024+0000","last_active":"2024-04-25T01:54:16.252078+0000","last_peered":"2024-04-25T01:54:16.252078+0000","last_clean":"2024-04-25T01:54:16.252078+0000","last_became_active":"2024-04-25T01:52:43.068058+0000","last_became_peered":"2024-04-25T01:52:43.068058+0000","last_unstale":"2024-04-25T01:54:16.252078+0000","last_undegraded":"2024-04-25T01:54:16.252078+0000","last_fullsized":"2024-04-25T01:54:16.252078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:43.126131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.733350+0000","last_change":"2024-04-25T01:52:44.091829+0000","last_active":"2024-04-25T01:54:12.733350+0000","last_peered":"2024-04-25T01:54:12.733350+0000","last_clean":"2024-04-25T01:54:12.733350+0000","last_became_active":"2024-04-25T01:52:44.091528+0000","last_became_peered":"2024-04-25T01:52:44.091528+0000","last_unstale":"2024-04-25T01:54:12.733350+0000","last_undegraded":"2024-04-25T01:54:12.733350+0000","last_fullsized":"2024-04-25T01:54:12.733350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:39.585296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.889407+0000","last_change":"2024-04-25T01:53:43.888956+0000","last_active":"2024-04-25T01:53:43.889407+0000","last_peered":"2024-04-25T01:53:43.889407+0000","last_clean":"2024-04-25T01:53:43.889407+0000","last_became_active":"2024-04-25T01:53:43.870789+0000","last_became_peered":"2024-04-25T01:53:43.870789+0000","last_unstale":"2024-04-25T01:53:43.889407+0000","last_undegraded":"2024-04-25T01:53:43.889407+0000","last_fullsized":"2024-04-25T01:53:43.889407+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:08:48.790006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.735672+0000","last_change":"2024-04-25T01:52:44.094696+0000","last_active":"2024-04-25T01:54:12.735672+0000","last_peered":"2024-04-25T01:54:12.735672+0000","last_clean":"2024-04-25T01:54:12.735672+0000","last_became_active":"2024-04-25T01:52:44.094371+0000","last_became_peered":"2024-04-25T01:52:44.094371+0000","last_unstale":"2024-04-25T01:54:12.735672+0000","last_undegraded":"2024-04-25T01:54:12.735672+0000","last_fullsized":"2024-04-25T01:54:12.735672+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:20.035674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859323+0000","last_change":"2024-04-25T01:52:43.071275+0000","last_active":"2024-04-25T01:53:43.859323+0000","last_peered":"2024-04-25T01:53:43.859323+0000","last_clean":"2024-04-25T01:53:43.859323+0000","last_became_active":"2024-04-25T01:52:43.070958+0000","last_became_peered":"2024-04-25T01:52:43.070958+0000","last_unstale":"2024-04-25T01:53:43.859323+0000","last_undegraded":"2024-04-25T01:53:43.859323+0000","last_fullsized":"2024-04-25T01:53:43.859323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:23:55.284943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.739102+0000","last_change":"2024-04-25T01:52:44.092742+0000","last_active":"2024-04-25T01:54:12.739102+0000","last_peered":"2024-04-25T01:54:12.739102+0000","last_clean":"2024-04-25T01:54:12.739102+0000","last_became_active":"2024-04-25T01:52:44.092345+0000","last_became_peered":"2024-04-25T01:52:44.092345+0000","last_unstale":"2024-04-25T01:54:12.739102+0000","last_undegraded":"2024-04-25T01:54:12.739102+0000","last_fullsized":"2024-04-25T01:54:12.739102+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:14.325814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.857260+0000","last_change":"2024-04-25T01:52:43.090057+0000","last_active":"2024-04-25T01:53:43.857260+0000","last_peered":"2024-04-25T01:53:43.857260+0000","last_clean":"2024-04-25T01:53:43.857260+0000","last_became_active":"2024-04-25T01:52:43.089280+0000","last_became_peered":"2024-04-25T01:52:43.089280+0000","last_unstale":"2024-04-25T01:53:43.857260+0000","last_undegraded":"2024-04-25T01:53:43.857260+0000","last_fullsized":"2024-04-25T01:53:43.857260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:41.048908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.755029+0000","last_change":"2024-04-25T01:52:44.097651+0000","last_active":"2024-04-25T01:54:12.755029+0000","last_peered":"2024-04-25T01:54:12.755029+0000","last_clean":"2024-04-25T01:54:12.755029+0000","last_became_active":"2024-04-25T01:52:44.097254+0000","last_became_peered":"2024-04-25T01:52:44.097254+0000","last_unstale":"2024-04-25T01:54:12.755029+0000","last_undegraded":"2024-04-25T01:54:12.755029+0000","last_fullsized":"2024-04-25T01:54:12.755029+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:15.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856041+0000","last_change":"2024-04-25T01:52:43.076925+0000","last_active":"2024-04-25T01:53:43.856041+0000","last_peered":"2024-04-25T01:53:43.856041+0000","last_clean":"2024-04-25T01:53:43.856041+0000","last_became_active":"2024-04-25T01:52:43.076514+0000","last_became_peered":"2024-04-25T01:52:43.076514+0000","last_unstale":"2024-04-25T01:53:43.856041+0000","last_undegraded":"2024-04-25T01:53:43.856041+0000","last_fullsized":"2024-04-25T01:53:43.856041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:31:17.305379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.742658+0000","last_change":"2024-04-25T01:52:44.103731+0000","last_active":"2024-04-25T01:54:12.742658+0000","last_peered":"2024-04-25T01:54:12.742658+0000","last_clean":"2024-04-25T01:54:12.742658+0000","last_became_active":"2024-04-25T01:52:44.103173+0000","last_became_peered":"2024-04-25T01:52:44.103173+0000","last_unstale":"2024-04-25T01:54:12.742658+0000","last_undegraded":"2024-04-25T01:54:12.742658+0000","last_fullsized":"2024-04-25T01:54:12.742658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:43.617579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.735631+0000","last_change":"2024-04-25T01:52:44.090847+0000","last_active":"2024-04-25T01:54:12.735631+0000","last_peered":"2024-04-25T01:54:12.735631+0000","last_clean":"2024-04-25T01:54:12.735631+0000","last_became_active":"2024-04-25T01:52:44.090506+0000","last_became_peered":"2024-04-25T01:52:44.090506+0000","last_unstale":"2024-04-25T01:54:12.735631+0000","last_undegraded":"2024-04-25T01:54:12.735631+0000","last_fullsized":"2024-04-25T01:54:12.735631+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:24.539726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":111,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:15.249917+0000","last_change":"2024-04-25T01:52:43.090469+0000","last_active":"2024-04-25T01:54:15.249917+0000","last_peered":"2024-04-25T01:54:15.249917+0000","last_clean":"2024-04-25T01:54:15.249917+0000","last_became_active":"2024-04-25T01:52:43.090159+0000","last_became_peered":"2024-04-25T01:52:43.090159+0000","last_unstale":"2024-04-25T01:54:15.249917+0000","last_undegraded":"2024-04-25T01:54:15.249917+0000","last_fullsized":"2024-04-25T01:54:15.249917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:42:45.532017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.758050+0000","last_change":"2024-04-25T01:52:44.093527+0000","last_active":"2024-04-25T01:54:12.758050+0000","last_peered":"2024-04-25T01:54:12.758050+0000","last_clean":"2024-04-25T01:54:12.758050+0000","last_became_active":"2024-04-25T01:52:44.092729+0000","last_became_peered":"2024-04-25T01:52:44.092729+0000","last_unstale":"2024-04-25T01:54:12.758050+0000","last_undegraded":"2024-04-25T01:54:12.758050+0000","last_fullsized":"2024-04-25T01:54:12.758050+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:32:50.668352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859076+0000","last_change":"2024-04-25T01:52:43.075323+0000","last_active":"2024-04-25T01:53:43.859076+0000","last_peered":"2024-04-25T01:53:43.859076+0000","last_clean":"2024-04-25T01:53:43.859076+0000","last_became_active":"2024-04-25T01:52:43.074876+0000","last_became_peered":"2024-04-25T01:52:43.074876+0000","last_unstale":"2024-04-25T01:53:43.859076+0000","last_undegraded":"2024-04-25T01:53:43.859076+0000","last_fullsized":"2024-04-25T01:53:43.859076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:47.472920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856031+0000","last_change":"2024-04-25T01:51:20.173285+0000","last_active":"2024-04-25T01:53:43.856031+0000","last_peered":"2024-04-25T01:53:43.856031+0000","last_clean":"2024-04-25T01:53:43.856031+0000","last_became_active":"2024-04-25T01:51:20.172811+0000","last_became_peered":"2024-04-25T01:51:20.172811+0000","last_unstale":"2024-04-25T01:53:43.856031+0000","last_undegraded":"2024-04-25T01:53:43.856031+0000","last_fullsized":"2024-04-25T01:53:43.856031+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_clean_scrub_stamp":"2024-04-25T01:47:31.712321+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:33.708082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.740239+0000","last_change":"2024-04-25T01:53:43.874632+0000","last_active":"2024-04-25T01:54:12.740239+0000","last_peered":"2024-04-25T01:54:12.740239+0000","last_clean":"2024-04-25T01:54:12.740239+0000","last_became_active":"2024-04-25T01:53:43.873732+0000","last_became_peered":"2024-04-25T01:53:43.873732+0000","last_unstale":"2024-04-25T01:54:12.740239+0000","last_undegraded":"2024-04-25T01:54:12.740239+0000","last_fullsized":"2024-04-25T01:54:12.740239+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:27.874789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856171+0000","last_change":"2024-04-25T01:52:43.072565+0000","last_active":"2024-04-25T01:53:43.856171+0000","last_peered":"2024-04-25T01:53:43.856171+0000","last_clean":"2024-04-25T01:53:43.856171+0000","last_became_active":"2024-04-25T01:52:43.072193+0000","last_became_peered":"2024-04-25T01:52:43.072193+0000","last_unstale":"2024-04-25T01:53:43.856171+0000","last_undegraded":"2024-04-25T01:53:43.856171+0000","last_fullsized":"2024-04-25T01:53:43.856171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:07.261859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.878893+0000","last_change":"2024-04-25T01:53:43.878499+0000","last_active":"2024-04-25T01:53:43.878893+0000","last_peered":"2024-04-25T01:53:43.878893+0000","last_clean":"2024-04-25T01:53:43.878893+0000","last_became_active":"2024-04-25T01:53:43.870876+0000","last_became_peered":"2024-04-25T01:53:43.870876+0000","last_unstale":"2024-04-25T01:53:43.878893+0000","last_undegraded":"2024-04-25T01:53:43.878893+0000","last_fullsized":"2024-04-25T01:53:43.878893+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:08.809127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.741556+0000","last_change":"2024-04-25T01:52:44.104892+0000","last_active":"2024-04-25T01:54:12.741556+0000","last_peered":"2024-04-25T01:54:12.741556+0000","last_clean":"2024-04-25T01:54:12.741556+0000","last_became_active":"2024-04-25T01:52:44.103802+0000","last_became_peered":"2024-04-25T01:52:44.103802+0000","last_unstale":"2024-04-25T01:54:12.741556+0000","last_undegraded":"2024-04-25T01:54:12.741556+0000","last_fullsized":"2024-04-25T01:54:12.741556+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:16.445968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.858753+0000","last_change":"2024-04-25T01:52:43.087143+0000","last_active":"2024-04-25T01:53:43.858753+0000","last_peered":"2024-04-25T01:53:43.858753+0000","last_clean":"2024-04-25T01:53:43.858753+0000","last_became_active":"2024-04-25T01:52:43.085987+0000","last_became_peered":"2024-04-25T01:52:43.085987+0000","last_unstale":"2024-04-25T01:53:43.858753+0000","last_undegraded":"2024-04-25T01:53:43.858753+0000","last_fullsized":"2024-04-25T01:53:43.858753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:17.936671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.755394+0000","last_change":"2024-04-25T01:52:44.092342+0000","last_active":"2024-04-25T01:54:12.755394+0000","last_peered":"2024-04-25T01:54:12.755394+0000","last_clean":"2024-04-25T01:54:12.755394+0000","last_became_active":"2024-04-25T01:52:44.091991+0000","last_became_peered":"2024-04-25T01:52:44.091991+0000","last_unstale":"2024-04-25T01:54:12.755394+0000","last_undegraded":"2024-04-25T01:54:12.755394+0000","last_fullsized":"2024-04-25T01:54:12.755394+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:22.147284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859134+0000","last_change":"2024-04-25T01:52:43.077972+0000","last_active":"2024-04-25T01:53:43.859134+0000","last_peered":"2024-04-25T01:53:43.859134+0000","last_clean":"2024-04-25T01:53:43.859134+0000","last_became_active":"2024-04-25T01:52:43.077655+0000","last_became_peered":"2024-04-25T01:52:43.077655+0000","last_unstale":"2024-04-25T01:53:43.859134+0000","last_undegraded":"2024-04-25T01:53:43.859134+0000","last_fullsized":"2024-04-25T01:53:43.859134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:24.820910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.753889+0000","last_change":"2024-04-25T01:52:44.119560+0000","last_active":"2024-04-25T01:54:12.753889+0000","last_peered":"2024-04-25T01:54:12.753889+0000","last_clean":"2024-04-25T01:54:12.753889+0000","last_became_active":"2024-04-25T01:52:44.119106+0000","last_became_peered":"2024-04-25T01:52:44.119106+0000","last_unstale":"2024-04-25T01:54:12.753889+0000","last_undegraded":"2024-04-25T01:54:12.753889+0000","last_fullsized":"2024-04-25T01:54:12.753889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:50.702393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.860595+0000","last_change":"2024-04-25T01:52:43.075783+0000","last_active":"2024-04-25T01:53:43.860595+0000","last_peered":"2024-04-25T01:53:43.860595+0000","last_clean":"2024-04-25T01:53:43.860595+0000","last_became_active":"2024-04-25T01:52:43.075462+0000","last_became_peered":"2024-04-25T01:52:43.075462+0000","last_unstale":"2024-04-25T01:53:43.860595+0000","last_undegraded":"2024-04-25T01:53:43.860595+0000","last_fullsized":"2024-04-25T01:53:43.860595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:07.077047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.702317+0000","last_change":"2024-04-25T01:52:44.090695+0000","last_active":"2024-04-25T01:54:12.702317+0000","last_peered":"2024-04-25T01:54:12.702317+0000","last_clean":"2024-04-25T01:54:12.702317+0000","last_became_active":"2024-04-25T01:52:44.090203+0000","last_became_peered":"2024-04-25T01:52:44.090203+0000","last_unstale":"2024-04-25T01:54:12.702317+0000","last_undegraded":"2024-04-25T01:54:12.702317+0000","last_fullsized":"2024-04-25T01:54:12.702317+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:58.104569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.866657+0000","last_change":"2024-04-25T01:52:43.086057+0000","last_active":"2024-04-25T01:53:43.866657+0000","last_peered":"2024-04-25T01:53:43.866657+0000","last_clean":"2024-04-25T01:53:43.866657+0000","last_became_active":"2024-04-25T01:52:43.085475+0000","last_became_peered":"2024-04-25T01:52:43.085475+0000","last_unstale":"2024-04-25T01:53:43.866657+0000","last_undegraded":"2024-04-25T01:53:43.866657+0000","last_fullsized":"2024-04-25T01:53:43.866657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:42.237414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.743738+0000","last_change":"2024-04-25T01:52:44.106128+0000","last_active":"2024-04-25T01:54:12.743738+0000","last_peered":"2024-04-25T01:54:12.743738+0000","last_clean":"2024-04-25T01:54:12.743738+0000","last_became_active":"2024-04-25T01:52:44.105805+0000","last_became_peered":"2024-04-25T01:52:44.105805+0000","last_unstale":"2024-04-25T01:54:12.743738+0000","last_undegraded":"2024-04-25T01:54:12.743738+0000","last_fullsized":"2024-04-25T01:54:12.743738+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:41.901070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.856616+0000","last_change":"2024-04-25T01:52:43.088347+0000","last_active":"2024-04-25T01:53:43.856616+0000","last_peered":"2024-04-25T01:53:43.856616+0000","last_clean":"2024-04-25T01:53:43.856616+0000","last_became_active":"2024-04-25T01:52:43.087932+0000","last_became_peered":"2024-04-25T01:52:43.087932+0000","last_unstale":"2024-04-25T01:53:43.856616+0000","last_undegraded":"2024-04-25T01:53:43.856616+0000","last_fullsized":"2024-04-25T01:53:43.856616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:26.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.756066+0000","last_change":"2024-04-25T01:52:44.108686+0000","last_active":"2024-04-25T01:54:12.756066+0000","last_peered":"2024-04-25T01:54:12.756066+0000","last_clean":"2024-04-25T01:54:12.756066+0000","last_became_active":"2024-04-25T01:52:44.107732+0000","last_became_peered":"2024-04-25T01:52:44.107732+0000","last_unstale":"2024-04-25T01:54:12.756066+0000","last_undegraded":"2024-04-25T01:54:12.756066+0000","last_fullsized":"2024-04-25T01:54:12.756066+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:14.820979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.859230+0000","last_change":"2024-04-25T01:52:43.069300+0000","last_active":"2024-04-25T01:53:43.859230+0000","last_peered":"2024-04-25T01:53:43.859230+0000","last_clean":"2024-04-25T01:53:43.859230+0000","last_became_active":"2024-04-25T01:52:43.068644+0000","last_became_peered":"2024-04-25T01:52:43.068644+0000","last_unstale":"2024-04-25T01:53:43.859230+0000","last_undegraded":"2024-04-25T01:53:43.859230+0000","last_fullsized":"2024-04-25T01:53:43.859230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:04.897638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.746156+0000","last_change":"2024-04-25T01:52:44.100467+0000","last_active":"2024-04-25T01:54:12.746156+0000","last_peered":"2024-04-25T01:54:12.746156+0000","last_clean":"2024-04-25T01:54:12.746156+0000","last_became_active":"2024-04-25T01:52:44.099958+0000","last_became_peered":"2024-04-25T01:52:44.099958+0000","last_unstale":"2024-04-25T01:54:12.746156+0000","last_undegraded":"2024-04-25T01:54:12.746156+0000","last_fullsized":"2024-04-25T01:54:12.746156+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:29.919785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.875865+0000","last_change":"2024-04-25T01:53:43.875865+0000","last_active":"2024-04-25T01:53:43.875865+0000","last_peered":"2024-04-25T01:53:43.875865+0000","last_clean":"2024-04-25T01:53:43.875865+0000","last_became_active":"2024-04-25T01:53:43.871525+0000","last_became_peered":"2024-04-25T01:53:43.871525+0000","last_unstale":"2024-04-25T01:53:43.875865+0000","last_undegraded":"2024-04-25T01:53:43.875865+0000","last_fullsized":"2024-04-25T01:53:43.875865+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:25.179902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.746457+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:54:12.746457+0000","last_peered":"2024-04-25T01:54:12.746457+0000","last_clean":"2024-04-25T01:54:12.746457+0000","last_became_active":"2024-04-25T01:52:44.087911+0000","last_became_peered":"2024-04-25T01:52:44.087911+0000","last_unstale":"2024-04-25T01:54:12.746457+0000","last_undegraded":"2024-04-25T01:54:12.746457+0000","last_fullsized":"2024-04-25T01:54:12.746457+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:29:38.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864407+0000","last_change":"2024-04-25T01:52:43.086083+0000","last_active":"2024-04-25T01:53:43.864407+0000","last_peered":"2024-04-25T01:53:43.864407+0000","last_clean":"2024-04-25T01:53:43.864407+0000","last_became_active":"2024-04-25T01:52:43.085613+0000","last_became_peered":"2024-04-25T01:52:43.085613+0000","last_unstale":"2024-04-25T01:53:43.864407+0000","last_undegraded":"2024-04-25T01:53:43.864407+0000","last_fullsized":"2024-04-25T01:53:43.864407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:23.851812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.753430+0000","last_change":"2024-04-25T01:52:44.106435+0000","last_active":"2024-04-25T01:54:12.753430+0000","last_peered":"2024-04-25T01:54:12.753430+0000","last_clean":"2024-04-25T01:54:12.753430+0000","last_became_active":"2024-04-25T01:52:44.106074+0000","last_became_peered":"2024-04-25T01:52:44.106074+0000","last_unstale":"2024-04-25T01:54:12.753430+0000","last_undegraded":"2024-04-25T01:54:12.753430+0000","last_fullsized":"2024-04-25T01:54:12.753430+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:59.286266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:53:43.864297+0000","last_change":"2024-04-25T01:52:43.088902+0000","last_active":"2024-04-25T01:53:43.864297+0000","last_peered":"2024-04-25T01:53:43.864297+0000","last_clean":"2024-04-25T01:53:43.864297+0000","last_became_active":"2024-04-25T01:52:43.088289+0000","last_became_peered":"2024-04-25T01:52:43.088289+0000","last_unstale":"2024-04-25T01:53:43.864297+0000","last_undegraded":"2024-04-25T01:53:43.864297+0000","last_fullsized":"2024-04-25T01:53:43.864297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:37:44.562369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.732685+0000","last_change":"2024-04-25T01:52:44.094185+0000","last_active":"2024-04-25T01:54:12.732685+0000","last_peered":"2024-04-25T01:54:12.732685+0000","last_clean":"2024-04-25T01:54:12.732685+0000","last_became_active":"2024-04-25T01:52:44.093753+0000","last_became_peered":"2024-04-25T01:52:44.093753+0000","last_unstale":"2024-04-25T01:54:12.732685+0000","last_undegraded":"2024-04-25T01:54:12.732685+0000","last_fullsized":"2024-04-25T01:54:12.732685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:06.089670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.734661+0000","last_change":"2024-04-25T01:52:44.093895+0000","last_active":"2024-04-25T01:54:12.734661+0000","last_peered":"2024-04-25T01:54:12.734661+0000","last_clean":"2024-04-25T01:54:12.734661+0000","last_became_active":"2024-04-25T01:52:44.093490+0000","last_became_peered":"2024-04-25T01:52:44.093490+0000","last_unstale":"2024-04-25T01:54:12.734661+0000","last_undegraded":"2024-04-25T01:54:12.734661+0000","last_fullsized":"2024-04-25T01:54:12.734661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:53.848979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.733855+0000","last_change":"2024-04-25T01:52:44.092500+0000","last_active":"2024-04-25T01:54:12.733855+0000","last_peered":"2024-04-25T01:54:12.733855+0000","last_clean":"2024-04-25T01:54:12.733855+0000","last_became_active":"2024-04-25T01:52:44.092044+0000","last_became_peered":"2024-04-25T01:52:44.092044+0000","last_unstale":"2024-04-25T01:54:12.733855+0000","last_undegraded":"2024-04-25T01:54:12.733855+0000","last_fullsized":"2024-04-25T01:54:12.733855+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:28.279676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.757034+0000","last_change":"2024-04-25T01:52:44.116668+0000","last_active":"2024-04-25T01:54:12.757034+0000","last_peered":"2024-04-25T01:54:12.757034+0000","last_clean":"2024-04-25T01:54:12.757034+0000","last_became_active":"2024-04-25T01:52:44.116295+0000","last_became_peered":"2024-04-25T01:52:44.116295+0000","last_unstale":"2024-04-25T01:54:12.757034+0000","last_undegraded":"2024-04-25T01:54:12.757034+0000","last_fullsized":"2024-04-25T01:54:12.757034+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:49.109403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.754251+0000","last_change":"2024-04-25T01:52:44.109950+0000","last_active":"2024-04-25T01:54:12.754251+0000","last_peered":"2024-04-25T01:54:12.754251+0000","last_clean":"2024-04-25T01:54:12.754251+0000","last_became_active":"2024-04-25T01:52:44.109636+0000","last_became_peered":"2024-04-25T01:52:44.109636+0000","last_unstale":"2024-04-25T01:54:12.754251+0000","last_undegraded":"2024-04-25T01:54:12.754251+0000","last_fullsized":"2024-04-25T01:54:12.754251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:02:57.610067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.751132+0000","last_change":"2024-04-25T01:52:44.105237+0000","last_active":"2024-04-25T01:54:12.751132+0000","last_peered":"2024-04-25T01:54:12.751132+0000","last_clean":"2024-04-25T01:54:12.751132+0000","last_became_active":"2024-04-25T01:52:44.104531+0000","last_became_peered":"2024-04-25T01:52:44.104531+0000","last_unstale":"2024-04-25T01:54:12.751132+0000","last_undegraded":"2024-04-25T01:54:12.751132+0000","last_fullsized":"2024-04-25T01:54:12.751132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.149119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.732830+0000","last_change":"2024-04-25T01:52:44.095655+0000","last_active":"2024-04-25T01:54:12.732830+0000","last_peered":"2024-04-25T01:54:12.732830+0000","last_clean":"2024-04-25T01:54:12.732830+0000","last_became_active":"2024-04-25T01:52:44.095300+0000","last_became_peered":"2024-04-25T01:52:44.095300+0000","last_unstale":"2024-04-25T01:54:12.732830+0000","last_undegraded":"2024-04-25T01:54:12.732830+0000","last_fullsized":"2024-04-25T01:54:12.732830+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:51.240170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.732535+0000","last_change":"2024-04-25T01:52:44.087382+0000","last_active":"2024-04-25T01:54:12.732535+0000","last_peered":"2024-04-25T01:54:12.732535+0000","last_clean":"2024-04-25T01:54:12.732535+0000","last_became_active":"2024-04-25T01:52:44.086815+0000","last_became_peered":"2024-04-25T01:52:44.086815+0000","last_unstale":"2024-04-25T01:54:12.732535+0000","last_undegraded":"2024-04-25T01:54:12.732535+0000","last_fullsized":"2024-04-25T01:54:12.732535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:52:13.682819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.743220+0000","last_change":"2024-04-25T01:53:43.871148+0000","last_active":"2024-04-25T01:54:12.743220+0000","last_peered":"2024-04-25T01:54:12.743220+0000","last_clean":"2024-04-25T01:54:12.743220+0000","last_became_active":"2024-04-25T01:53:43.870490+0000","last_became_peered":"2024-04-25T01:53:43.870490+0000","last_unstale":"2024-04-25T01:54:12.743220+0000","last_undegraded":"2024-04-25T01:54:12.743220+0000","last_fullsized":"2024-04-25T01:54:12.743220+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:41.790442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.752637+0000","last_change":"2024-04-25T01:52:44.114767+0000","last_active":"2024-04-25T01:54:12.752637+0000","last_peered":"2024-04-25T01:54:12.752637+0000","last_clean":"2024-04-25T01:54:12.752637+0000","last_became_active":"2024-04-25T01:52:44.114483+0000","last_became_peered":"2024-04-25T01:52:44.114483+0000","last_unstale":"2024-04-25T01:54:12.752637+0000","last_undegraded":"2024-04-25T01:54:12.752637+0000","last_fullsized":"2024-04-25T01:54:12.752637+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:14.439287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.754512+0000","last_change":"2024-04-25T01:52:44.094424+0000","last_active":"2024-04-25T01:54:12.754512+0000","last_peered":"2024-04-25T01:54:12.754512+0000","last_clean":"2024-04-25T01:54:12.754512+0000","last_became_active":"2024-04-25T01:52:44.093074+0000","last_became_peered":"2024-04-25T01:52:44.093074+0000","last_unstale":"2024-04-25T01:54:12.754512+0000","last_undegraded":"2024-04-25T01:54:12.754512+0000","last_fullsized":"2024-04-25T01:54:12.754512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:20.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.758295+0000","last_change":"2024-04-25T01:52:44.113465+0000","last_active":"2024-04-25T01:54:12.758295+0000","last_peered":"2024-04-25T01:54:12.758295+0000","last_clean":"2024-04-25T01:54:12.758295+0000","last_became_active":"2024-04-25T01:52:44.113168+0000","last_became_peered":"2024-04-25T01:52:44.113168+0000","last_unstale":"2024-04-25T01:54:12.758295+0000","last_undegraded":"2024-04-25T01:54:12.758295+0000","last_fullsized":"2024-04-25T01:54:12.758295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:42:47.758996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.750933+0000","last_change":"2024-04-25T01:52:44.104763+0000","last_active":"2024-04-25T01:54:12.750933+0000","last_peered":"2024-04-25T01:54:12.750933+0000","last_clean":"2024-04-25T01:54:12.750933+0000","last_became_active":"2024-04-25T01:52:44.104377+0000","last_became_peered":"2024-04-25T01:52:44.104377+0000","last_unstale":"2024-04-25T01:54:12.750933+0000","last_undegraded":"2024-04-25T01:54:12.750933+0000","last_fullsized":"2024-04-25T01:54:12.750933+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:13.647020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-25T01:54:12.739093+0000","last_change":"2024-04-25T01:53:43.864792+0000","last_active":"2024-04-25T01:54:12.739093+0000","last_peered":"2024-04-25T01:54:12.739093+0000","last_clean":"2024-04-25T01:54:12.739093+0000","last_became_active":"2024-04-25T01:53:43.864432+0000","last_became_peered":"2024-04-25T01:53:43.864432+0000","last_unstale":"2024-04-25T01:54:12.739093+0000","last_undegraded":"2024-04-25T01:54:12.739093+0000","last_fullsized":"2024-04-25T01:54:12.739093+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:28.149564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65437,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":545,"num_read_kb":339,"num_write":163,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":61762,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":196311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318307,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":463020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579942,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41112,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704040,"statfs":{"total":95995035648,"available":95952936960,"internally_reserved":0,"allocated":1654784,"data_stored":1115662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743469,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":463122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005107,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":462908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266745,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1055113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430269,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":462511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":523599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":463782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36288,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708864,"statfs":{"total":95995035648,"available":95957876736,"internally_reserved":0,"allocated":974848,"data_stored":463604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117070,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":462822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855433,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":524825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691907,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1581056,"data_stored":1054797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":61295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:54:21.380 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T01:54:21.380 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T01:54:21.380 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T01:54:21.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T01:54:21.867 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:22.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:21 smithi102 ceph-mon[27796]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:22.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:21 smithi102 ceph-mon[27796]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:54:22.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:21 smithi110 ceph-mon[32848]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:22.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:21 smithi110 ceph-mon[32848]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:54:22.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:21 smithi193 ceph-mon[32981]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:54:22.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:21 smithi193 ceph-mon[32981]: pgmap v368: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T01:54:24.060 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:23 smithi102 ceph-mon[27796]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:54:24.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:23 smithi110 ceph-mon[32848]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:54:24.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:23 smithi193 ceph-mon[32981]: pgmap v369: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:54:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.380+0000 7f20ecc00700 1 -- 172.21.15.102:0/1707563950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e810ec30 msgr2=0x7f20e8111020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.380+0000 7f20ecc00700 1 --2- 172.21.15.102:0/1707563950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e810ec30 0x7f20e8111020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f20e000b0a0 tx=0x7f20e000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.380+0000 7f20ecc00700 1 -- 172.21.15.102:0/1707563950 shutdown_connections 2024-04-25T01:54:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.380+0000 7f20ecc00700 1 --2- 172.21.15.102:0/1707563950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e810ec30 0x7f20e8111020 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.380+0000 7f20ecc00700 1 --2- 172.21.15.102:0/1707563950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20e8074460 0x7f20e8074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.380+0000 7f20ecc00700 1 --2- 172.21.15.102:0/1707563950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20e8074d80 0x7f20e810e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.380+0000 7f20ecc00700 1 -- 172.21.15.102:0/1707563950 >> 172.21.15.102:0/1707563950 conn(0x7f20e8079390 msgr2=0x7f20e80797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.380+0000 7f20ecc00700 1 -- 172.21.15.102:0/1707563950 shutdown_connections 2024-04-25T01:54:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.380+0000 7f20ecc00700 1 -- 172.21.15.102:0/1707563950 wait complete. 2024-04-25T01:54:24.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.381+0000 7f20ecc00700 1 Processor -- start 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.381+0000 7f20ecc00700 1 -- start start 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.381+0000 7f20ecc00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20e8074460 0x7f20e81a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20ecc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20e8074d80 0x7f20e81a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20ecc00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e810ec30 0x7f20e819a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20ecc00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f20e8114710 con 0x7f20e810ec30 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20ecc00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f20e8114590 con 0x7f20e8074460 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20ecc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f20e8114890 con 0x7f20e8074d80 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e6800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e810ec30 0x7f20e819a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e6800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e810ec30 0x7f20e819a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55618/0 (socket says 172.21.15.102:55618) 2024-04-25T01:54:24.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e6800700 1 -- 172.21.15.102:0/3894127052 learned_addr learned my addr 172.21.15.102:0/3894127052 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e5400700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20e8074d80 0x7f20e81a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e5e00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20e8074460 0x7f20e81a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e5400700 1 -- 172.21.15.102:0/3894127052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20e8074460 msgr2=0x7f20e81a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e5400700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20e8074460 0x7f20e81a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e5400700 1 -- 172.21.15.102:0/3894127052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e810ec30 msgr2=0x7f20e819a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e5400700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e810ec30 0x7f20e819a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.382+0000 7f20e5400700 1 -- 172.21.15.102:0/3894127052 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e819b130 con 0x7f20e8074d80 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.383+0000 7f20e5400700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20e8074d80 0x7f20e81a0d00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f20d400e9c0 tx=0x7f20d400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.383+0000 7f20dea00700 1 -- 172.21.15.102:0/3894127052 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4004220 con 0x7f20e8074d80 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.383+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e819b420 con 0x7f20e8074d80 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.383+0000 7f20dea00700 1 -- 172.21.15.102:0/3894127052 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d40043c0 con 0x7f20e8074d80 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.383+0000 7f20dea00700 1 -- 172.21.15.102:0/3894127052 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d4003cc0 con 0x7f20e8074d80 2024-04-25T01:54:24.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.383+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e81a7690 con 0x7f20e8074d80 2024-04-25T01:54:24.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.384+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e8068f20 con 0x7f20e8074d80 2024-04-25T01:54:24.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.387+0000 7f20dea00700 1 -- 172.21.15.102:0/3894127052 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f20d400c7c0 con 0x7f20e8074d80 2024-04-25T01:54:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.388+0000 7f20dea00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f20cc0a2c20 0x7f20cc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.388+0000 7f20dea00700 1 -- 172.21.15.102:0/3894127052 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f20d40cf260 con 0x7f20e8074d80 2024-04-25T01:54:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.388+0000 7f20e5e00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f20cc0a2c20 0x7f20cc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.389+0000 7f20e5e00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f20cc0a2c20 0x7f20cc0a50e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20e8079b30 tx=0x7f20d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:24.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.392+0000 7f20dea00700 1 -- 172.21.15.102:0/3894127052 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f20d409c170 con 0x7f20e8074d80 2024-04-25T01:54:24.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.686+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f20e819beb0 con 0x7f20e8074d80 2024-04-25T01:54:24.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.687+0000 7f20dea00700 1 -- 172.21.15.102:0/3894127052 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f20d409ba70 con 0x7f20e8074d80 2024-04-25T01:54:24.689 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:24.689 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f20cc0a2c20 msgr2=0x7f20cc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f20cc0a2c20 0x7f20cc0a50e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f20e8079b30 tx=0x7f20d8009260 comp rx=0 tx=0).stop 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20e8074d80 msgr2=0x7f20e81a0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20e8074d80 0x7f20e81a0d00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f20d400e9c0 tx=0x7f20d400ecd0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 shutdown_connections 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f20cc0a2c20 0x7f20cc0a50e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20e8074460 0x7f20e81a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20e810ec30 0x7f20e819a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 --2- 172.21.15.102:0/3894127052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20e8074d80 0x7f20e81a0d00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 >> 172.21.15.102:0/3894127052 conn(0x7f20e8079390 msgr2=0x7f20e80ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:24.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 shutdown_connections 2024-04-25T01:54:24.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:24.690+0000 7f20ecc00700 1 -- 172.21.15.102:0/3894127052 wait complete. 2024-04-25T01:54:24.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:24 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3894127052' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:54:25.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:24 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3894127052' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:54:25.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:24 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3894127052' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T01:54:25.694 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T01:54:25.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.911+0000 7fd667200640 1 -- 172.21.15.102:0/3559342136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd668108db0 msgr2=0x7fd668109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.911+0000 7fd667200640 1 --2- 172.21.15.102:0/3559342136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd668108db0 0x7fd668109230 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd650009900 tx=0x7fd65002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:54:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.912+0000 7fd667200640 1 -- 172.21.15.102:0/3559342136 shutdown_connections 2024-04-25T01:54:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.912+0000 7fd667200640 1 --2- 172.21.15.102:0/3559342136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd668109770 0x7fd668110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.912+0000 7fd667200640 1 --2- 172.21.15.102:0/3559342136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd668108db0 0x7fd668109230 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.912+0000 7fd667200640 1 --2- 172.21.15.102:0/3559342136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd668107bc0 0x7fd668107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.912+0000 7fd667200640 1 -- 172.21.15.102:0/3559342136 >> 172.21.15.102:0/3559342136 conn(0x7fd668076b80 msgr2=0x7fd668078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.912+0000 7fd667200640 1 -- 172.21.15.102:0/3559342136 shutdown_connections 2024-04-25T01:54:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.912+0000 7fd667200640 1 -- 172.21.15.102:0/3559342136 wait complete. 2024-04-25T01:54:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd667200640 1 Processor -- start 2024-04-25T01:54:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd667200640 1 -- start start 2024-04-25T01:54:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd667200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd668107bc0 0x7fd66819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd667200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd668108db0 0x7fd66819e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd667200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd668109770 0x7fd6681a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd667200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd668113820 con 0x7fd668109770 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd667200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd6681136a0 con 0x7fd668108db0 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd667200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6681139a0 con 0x7fd668107bc0 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd665e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd668107bc0 0x7fd66819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd65d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd668108db0 0x7fd66819e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd665e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd668107bc0 0x7fd66819de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52996/0 (socket says 172.21.15.102:52996) 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd65d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd668108db0 0x7fd66819e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40778/0 (socket says 172.21.15.102:40778) 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.913+0000 7fd665e00640 1 -- 172.21.15.102:0/1972645318 learned_addr learned my addr 172.21.15.102:0/1972645318 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd666800640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd668109770 0x7fd6681a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd665e00640 1 -- 172.21.15.102:0/1972645318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd668108db0 msgr2=0x7fd66819e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd665e00640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd668108db0 0x7fd66819e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd665e00640 1 -- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd668109770 msgr2=0x7fd6681a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd665e00640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd668109770 0x7fd6681a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd665e00640 1 -- 172.21.15.102:0/1972645318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6681a5ab0 con 0x7fd668107bc0 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd666800640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd668109770 0x7fd6681a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd665e00640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd668107bc0 0x7fd66819de90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd65800e9c0 tx=0x7fd65800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd65f400640 1 -- 172.21.15.102:0/1972645318 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65800cde0 con 0x7fd668107bc0 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd65f400640 1 -- 172.21.15.102:0/1972645318 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd658010910 con 0x7fd668107bc0 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6681a5da0 con 0x7fd668107bc0 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd65f400640 1 -- 172.21.15.102:0/1972645318 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd658018a00 con 0x7fd668107bc0 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.914+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6681a6260 con 0x7fd668107bc0 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.916+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd66810eee0 con 0x7fd668107bc0 2024-04-25T01:54:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.917+0000 7fd65f400640 1 -- 172.21.15.102:0/1972645318 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd65801f020 con 0x7fd668107bc0 2024-04-25T01:54:25.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.919+0000 7fd65f400640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd64c0a2c10 0x7fd64c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:25.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.919+0000 7fd65f400640 1 -- 172.21.15.102:0/1972645318 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fd6580cf2c0 con 0x7fd668107bc0 2024-04-25T01:54:25.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.919+0000 7fd65d400640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd64c0a2c10 0x7fd64c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:25.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.919+0000 7fd65d400640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd64c0a2c10 0x7fd64c0a50d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd6500314e0 tx=0x7fd650000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:25.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:25 smithi102 ceph-mon[27796]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:54:25.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:25.923+0000 7fd65f400640 1 -- 172.21.15.102:0/1972645318 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd65809c1d0 con 0x7fd668107bc0 2024-04-25T01:54:26.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.126+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd66802d420 con 0x7fd668107bc0 2024-04-25T01:54:26.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.128+0000 7fd65f400640 1 -- 172.21.15.102:0/1972645318 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22686 (secure 0 0 0) 0x7fd65809bad0 con 0x7fd668107bc0 2024-04-25T01:54:26.130 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:26.130 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:53:43.851432+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd64c0a2c10 msgr2=0x7fd64c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd64c0a2c10 0x7fd64c0a50d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd6500314e0 tx=0x7fd650000bc0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd668107bc0 msgr2=0x7fd66819de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd668107bc0 0x7fd66819de90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd65800e9c0 tx=0x7fd65800ee80 comp rx=0 tx=0).stop 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 shutdown_connections 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd64c0a2c10 0x7fd64c0a50d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd668109770 0x7fd6681a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd668108db0 0x7fd66819e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 --2- 172.21.15.102:0/1972645318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd668107bc0 0x7fd66819de90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 >> 172.21.15.102:0/1972645318 conn(0x7fd668076b80 msgr2=0x7fd66810c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:26.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 shutdown_connections 2024-04-25T01:54:26.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.131+0000 7fd667200640 1 -- 172.21.15.102:0/1972645318 wait complete. 2024-04-25T01:54:26.153 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:41.585137+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:43.022666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:54:26.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:54:26.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:25 smithi110 ceph-mon[32848]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:54:26.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:25 smithi193 ceph-mon[32981]: pgmap v370: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T01:54:26.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.356+0000 7f8875a00640 1 -- 172.21.15.102:0/297469682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8870103880 msgr2=0x7f8870103ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.356+0000 7f8875a00640 1 --2- 172.21.15.102:0/297469682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8870103880 0x7f8870103ca0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8860009900 tx=0x7f886002a410 comp rx=0 tx=0).stop 2024-04-25T01:54:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.357+0000 7f8875a00640 1 -- 172.21.15.102:0/297469682 shutdown_connections 2024-04-25T01:54:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.357+0000 7f8875a00640 1 --2- 172.21.15.102:0/297469682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8870105430 0x7f887010bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.357+0000 7f8875a00640 1 --2- 172.21.15.102:0/297469682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8870104a70 0x7f8870104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.357+0000 7f8875a00640 1 --2- 172.21.15.102:0/297469682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8870103880 0x7f8870103ca0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.357+0000 7f8875a00640 1 -- 172.21.15.102:0/297469682 >> 172.21.15.102:0/297469682 conn(0x7f88700fedf0 msgr2=0x7f8870101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.357+0000 7f8875a00640 1 -- 172.21.15.102:0/297469682 shutdown_connections 2024-04-25T01:54:26.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.357+0000 7f8875a00640 1 -- 172.21.15.102:0/297469682 wait complete. 2024-04-25T01:54:26.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f8875a00640 1 Processor -- start 2024-04-25T01:54:26.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f8875a00640 1 -- start start 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f8875a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8870103880 0x7f8870199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f8875a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8870104a70 0x7f887019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f8875a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8870105430 0x7f88701a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f8875a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f887010f4f0 con 0x7f8870103880 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f8875a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f887010f370 con 0x7f8870104a70 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f8875a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f887010f670 con 0x7f8870105430 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f886fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8870103880 0x7f8870199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f886fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8870103880 0x7f8870199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55660/0 (socket says 172.21.15.102:55660) 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f886fe00640 1 -- 172.21.15.102:0/713801162 learned_addr learned my addr 172.21.15.102:0/713801162 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f8875000640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8870105430 0x7f88701a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.358+0000 7f886f400640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8870104a70 0x7f887019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:26.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.359+0000 7f886fe00640 1 -- 172.21.15.102:0/713801162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8870105430 msgr2=0x7f88701a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.359+0000 7f886fe00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8870105430 0x7f88701a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.359+0000 7f886fe00640 1 -- 172.21.15.102:0/713801162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8870104a70 msgr2=0x7f887019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.359+0000 7f886fe00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8870104a70 0x7f887019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.359+0000 7f886fe00640 1 -- 172.21.15.102:0/713801162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88701a17e0 con 0x7f8870103880 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.359+0000 7f8875000640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8870105430 0x7f88701a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.359+0000 7f886fe00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8870103880 0x7f8870199bc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8860031610 tx=0x7f8860004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.359+0000 7f886cc00640 1 -- 172.21.15.102:0/713801162 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886003f030 con 0x7f8870103880 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.359+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88701a1a70 con 0x7f8870103880 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.360+0000 7f886cc00640 1 -- 172.21.15.102:0/713801162 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8860003ad0 con 0x7f8870103880 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.360+0000 7f886cc00640 1 -- 172.21.15.102:0/713801162 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860035070 con 0x7f8870103880 2024-04-25T01:54:26.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.360+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88701a1fb0 con 0x7f8870103880 2024-04-25T01:54:26.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.361+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887010ac20 con 0x7f8870103880 2024-04-25T01:54:26.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.363+0000 7f886cc00640 1 -- 172.21.15.102:0/713801162 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8860042030 con 0x7f8870103880 2024-04-25T01:54:26.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.365+0000 7f886cc00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f88540a2b90 0x7f88540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:26.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.365+0000 7f886cc00640 1 -- 172.21.15.102:0/713801162 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f88600ef620 con 0x7f8870103880 2024-04-25T01:54:26.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.365+0000 7f886f400640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f88540a2b90 0x7f88540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:26.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.366+0000 7f886f400640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f88540a2b90 0x7f88540a5050 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f887019b0d0 tx=0x7f885c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:26.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.370+0000 7f886cc00640 1 -- 172.21.15.102:0/713801162 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f88600bbee0 con 0x7f8870103880 2024-04-25T01:54:26.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.570+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8870062340 con 0x7f8870103880 2024-04-25T01:54:26.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.571+0000 7f886cc00640 1 -- 172.21.15.102:0/713801162 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f88600c01e0 con 0x7f8870103880 2024-04-25T01:54:26.572 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:54:26.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f88540a2b90 msgr2=0x7f88540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f88540a2b90 0x7f88540a5050 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f887019b0d0 tx=0x7f885c005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:26.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8870103880 msgr2=0x7f8870199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8870103880 0x7f8870199bc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8860031610 tx=0x7f8860004270 comp rx=0 tx=0).stop 2024-04-25T01:54:26.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 shutdown_connections 2024-04-25T01:54:26.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f88540a2b90 0x7f88540a5050 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8870105430 0x7f88701a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8870104a70 0x7f887019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 --2- 172.21.15.102:0/713801162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8870103880 0x7f8870199bc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.573+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 >> 172.21.15.102:0/713801162 conn(0x7f88700fedf0 msgr2=0x7f8870107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:26.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.574+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 shutdown_connections 2024-04-25T01:54:26.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.574+0000 7f8875a00640 1 -- 172.21.15.102:0/713801162 wait complete. 2024-04-25T01:54:26.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:54:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.799+0000 7f279ec00640 1 -- 172.21.15.102:0/2667990770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2798109830 msgr2=0x7f27981100c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.799+0000 7f279ec00640 1 --2- 172.21.15.102:0/2667990770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2798109830 0x7f27981100c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2790009ca0 tx=0x7f279002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.799+0000 7f279ec00640 1 -- 172.21.15.102:0/2667990770 shutdown_connections 2024-04-25T01:54:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.799+0000 7f279ec00640 1 --2- 172.21.15.102:0/2667990770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2798109830 0x7f27981100c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.799+0000 7f279ec00640 1 --2- 172.21.15.102:0/2667990770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2798108e70 0x7f27981092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.799+0000 7f279ec00640 1 --2- 172.21.15.102:0/2667990770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798107c80 0x7f27981080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.799+0000 7f279ec00640 1 -- 172.21.15.102:0/2667990770 >> 172.21.15.102:0/2667990770 conn(0x7f2798076c10 msgr2=0x7f2798079050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.799+0000 7f279ec00640 1 -- 172.21.15.102:0/2667990770 shutdown_connections 2024-04-25T01:54:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.799+0000 7f279ec00640 1 -- 172.21.15.102:0/2667990770 wait complete. 2024-04-25T01:54:26.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.800+0000 7f279ec00640 1 Processor -- start 2024-04-25T01:54:26.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ec00640 1 -- start start 2024-04-25T01:54:26.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ec00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2798107c80 0x7f279819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:26.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ec00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798108e70 0x7f279819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:26.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ec00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2798109830 0x7f27981a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:26.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ec00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2798113930 con 0x7f2798108e70 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ec00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f27981137b0 con 0x7f2798109830 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ec00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2798113ab0 con 0x7f2798107c80 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798108e70 0x7f279819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798108e70 0x7f279819e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55690/0 (socket says 172.21.15.102:55690) 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279ce00640 1 -- 172.21.15.102:0/4011529834 learned_addr learned my addr 172.21.15.102:0/4011529834 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279d800640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2798107c80 0x7f279819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.801+0000 7f279e200640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2798109830 0x7f27981a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f279ce00640 1 -- 172.21.15.102:0/4011529834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2798107c80 msgr2=0x7f279819dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f279ce00640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2798107c80 0x7f279819dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f279ce00640 1 -- 172.21.15.102:0/4011529834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2798109830 msgr2=0x7f27981a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f279ce00640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2798109830 0x7f27981a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f279ce00640 1 -- 172.21.15.102:0/4011529834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27981a5ae0 con 0x7f2798108e70 2024-04-25T01:54:26.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f279ce00640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798108e70 0x7f279819e400 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f278800e9f0 tx=0x7f278800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:26.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f2796000640 1 -- 172.21.15.102:0/4011529834 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278800cd80 con 0x7f2798108e70 2024-04-25T01:54:26.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27981a5dd0 con 0x7f2798108e70 2024-04-25T01:54:26.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f2796000640 1 -- 172.21.15.102:0/4011529834 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2788010910 con 0x7f2798108e70 2024-04-25T01:54:26.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.802+0000 7f2796000640 1 -- 172.21.15.102:0/4011529834 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27880188f0 con 0x7f2798108e70 2024-04-25T01:54:26.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.803+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27981a6290 con 0x7f2798108e70 2024-04-25T01:54:26.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.804+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f279810eed0 con 0x7f2798108e70 2024-04-25T01:54:26.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.806+0000 7f2796000640 1 -- 172.21.15.102:0/4011529834 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f2788010ac0 con 0x7f2798108e70 2024-04-25T01:54:26.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.807+0000 7f2796000640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f27800a2b90 0x7f27800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:26.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.807+0000 7f2796000640 1 -- 172.21.15.102:0/4011529834 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f27880cf3b0 con 0x7f2798108e70 2024-04-25T01:54:26.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.807+0000 7f279d800640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f27800a2b90 0x7f27800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:26.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.808+0000 7f279d800640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f27800a2b90 0x7f27800a5050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f278c005fd0 tx=0x7f278c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:26.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:26.811+0000 7f2796000640 1 -- 172.21.15.102:0/4011529834 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f278809bc70 con 0x7f2798108e70 2024-04-25T01:54:27.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.014+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2798062340 con 0x7f2798108e70 2024-04-25T01:54:27.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.015+0000 7f2796000640 1 -- 172.21.15.102:0/4011529834 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f278809ff70 con 0x7f2798108e70 2024-04-25T01:54:27.017 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-25T01:54:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f27800a2b90 msgr2=0x7f27800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f27800a2b90 0x7f27800a5050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f278c005fd0 tx=0x7f278c00b410 comp rx=0 tx=0).stop 2024-04-25T01:54:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798108e70 msgr2=0x7f279819e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798108e70 0x7f279819e400 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f278800e9f0 tx=0x7f278800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 shutdown_connections 2024-04-25T01:54:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f27800a2b90 0x7f27800a5050 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2798109830 0x7f27981a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2798108e70 0x7f279819e400 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 --2- 172.21.15.102:0/4011529834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2798107c80 0x7f279819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 >> 172.21.15.102:0/4011529834 conn(0x7f2798076c10 msgr2=0x7f279810c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:27.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 shutdown_connections 2024-04-25T01:54:27.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.018+0000 7f279ec00640 1 -- 172.21.15.102:0/4011529834 wait complete. 2024-04-25T01:54:27.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:54:27.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:26 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1972645318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:27.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:26 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/713801162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:27.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:26 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1972645318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:27.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:26 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/713801162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.247+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3421018631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ea4103200 msgr2=0x7f8ea4109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.247+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3421018631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ea4103200 0x7f8ea4109a90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009ca0 tx=0x7f8e9c02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.247+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3421018631 shutdown_connections 2024-04-25T01:54:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.247+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3421018631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ea4103200 0x7f8ea4109a90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.247+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3421018631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea4102840 0x7f8ea4102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.247+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3421018631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ea4101650 0x7f8ea4101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.247+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3421018631 >> 172.21.15.102:0/3421018631 conn(0x7f8ea40fcbe0 msgr2=0x7f8ea40ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:27.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.247+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3421018631 shutdown_connections 2024-04-25T01:54:27.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.247+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3421018631 wait complete. 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.248+0000 7f8ea9a00640 1 Processor -- start 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.248+0000 7f8ea9a00640 1 -- start start 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ea4101650 0x7f8ea40754f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea9a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ea4102840 0x7f8ea4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea9a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea4103200 0x7f8ea4074070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea9a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea410d2b0 con 0x7f8ea4101650 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea9a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea410d130 con 0x7f8ea4102840 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea9a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea410d430 con 0x7f8ea4103200 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ea4101650 0x7f8ea40754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ea4101650 0x7f8ea40754f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55708/0 (socket says 172.21.15.102:55708) 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea3e00640 1 -- 172.21.15.102:0/3532933853 learned_addr learned my addr 172.21.15.102:0/3532933853 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea3400640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ea4102840 0x7f8ea4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:27.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea9000640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea4103200 0x7f8ea4074070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea3e00640 1 -- 172.21.15.102:0/3532933853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea4103200 msgr2=0x7f8ea4074070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea3e00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea4103200 0x7f8ea4074070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea3e00640 1 -- 172.21.15.102:0/3532933853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ea4102840 msgr2=0x7f8ea4073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea3e00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ea4102840 0x7f8ea4073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea3e00640 1 -- 172.21.15.102:0/3532933853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea4074900 con 0x7f8ea4101650 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.249+0000 7f8ea9000640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea4103200 0x7f8ea4074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.250+0000 7f8ea3e00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ea4101650 0x7f8ea40754f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400e9c0 tx=0x7f8e9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.250+0000 7f8ea0c00640 1 -- 172.21.15.102:0/3532933853 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9400cd80 con 0x7f8ea4101650 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.250+0000 7f8ea0c00640 1 -- 172.21.15.102:0/3532933853 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e940035f0 con 0x7f8ea4101650 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.250+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea41a19f0 con 0x7f8ea4101650 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.250+0000 7f8ea0c00640 1 -- 172.21.15.102:0/3532933853 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e94010690 con 0x7f8ea4101650 2024-04-25T01:54:27.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.250+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea41a1f40 con 0x7f8ea4101650 2024-04-25T01:54:27.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.251+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ea4108970 con 0x7f8ea4101650 2024-04-25T01:54:27.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.253+0000 7f8ea0c00640 1 -- 172.21.15.102:0/3532933853 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8e940037a0 con 0x7f8ea4101650 2024-04-25T01:54:27.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:26 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1972645318' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:27.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:26 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/713801162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:27.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.255+0000 7f8ea0c00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8e880a2b90 0x7f8e880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:27.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.256+0000 7f8ea3400640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8e880a2b90 0x7f8e880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:27.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.256+0000 7f8ea0c00640 1 -- 172.21.15.102:0/3532933853 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f8e940cf520 con 0x7f8ea4101650 2024-04-25T01:54:27.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.256+0000 7f8ea3400640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8e880a2b90 0x7f8e880a5050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8e900095f0 tx=0x7f8e90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:27.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.260+0000 7f8ea0c00640 1 -- 172.21.15.102:0/3532933853 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8e9409c490 con 0x7f8ea4101650 2024-04-25T01:54:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.460+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8ea402d170 con 0x7f8ea4101650 2024-04-25T01:54:27.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.460+0000 7f8ea0c00640 1 -- 172.21.15.102:0/3532933853 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f8ea402d170 con 0x7f8ea4101650 2024-04-25T01:54:27.462 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-25T01:54:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8e880a2b90 msgr2=0x7f8e880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8e880a2b90 0x7f8e880a5050 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8e900095f0 tx=0x7f8e90005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ea4101650 msgr2=0x7f8ea40754f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ea4101650 0x7f8ea40754f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8e9400e9c0 tx=0x7f8e9400ee80 comp rx=0 tx=0).stop 2024-04-25T01:54:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 shutdown_connections 2024-04-25T01:54:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8e880a2b90 0x7f8e880a5050 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ea4103200 0x7f8ea4074070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ea4102840 0x7f8ea4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 --2- 172.21.15.102:0/3532933853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ea4101650 0x7f8ea40754f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.463+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 >> 172.21.15.102:0/3532933853 conn(0x7f8ea40fcbe0 msgr2=0x7f8ea4105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:27.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.464+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 shutdown_connections 2024-04-25T01:54:27.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.464+0000 7f8ea9a00640 1 -- 172.21.15.102:0/3532933853 wait complete. 2024-04-25T01:54:27.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:54:27.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.684+0000 7f6682e00640 1 -- 172.21.15.102:0/2314305425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c074140 msgr2=0x7f667c0745a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.684+0000 7f6682e00640 1 --2- 172.21.15.102:0/2314305425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c074140 0x7f667c0745a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6668009900 tx=0x7f666802a480 comp rx=0 tx=0).stop 2024-04-25T01:54:27.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.684+0000 7f6682e00640 1 -- 172.21.15.102:0/2314305425 shutdown_connections 2024-04-25T01:54:27.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.684+0000 7f6682e00640 1 --2- 172.21.15.102:0/2314305425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f667c109440 0x7f667c10b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.684+0000 7f6682e00640 1 --2- 172.21.15.102:0/2314305425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c074140 0x7f667c0745a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.684+0000 7f6682e00640 1 --2- 172.21.15.102:0/2314305425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f667c075720 0x7f667c073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.684+0000 7f6682e00640 1 -- 172.21.15.102:0/2314305425 >> 172.21.15.102:0/2314305425 conn(0x7f667c0fcb20 msgr2=0x7f667c0fef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:27.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.684+0000 7f6682e00640 1 -- 172.21.15.102:0/2314305425 shutdown_connections 2024-04-25T01:54:27.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.684+0000 7f6682e00640 1 -- 172.21.15.102:0/2314305425 wait complete. 2024-04-25T01:54:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.685+0000 7f6682e00640 1 Processor -- start 2024-04-25T01:54:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.685+0000 7f6682e00640 1 -- start start 2024-04-25T01:54:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.685+0000 7f6682e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f667c074140 0x7f667c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.685+0000 7f6682e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f667c075720 0x7f667c19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.685+0000 7f6682e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c109440 0x7f667c1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:27.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.685+0000 7f6682e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f667c10f480 con 0x7f667c109440 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.685+0000 7f6682e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f667c10f300 con 0x7f667c075720 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.685+0000 7f6682e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f667c10f600 con 0x7f667c074140 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6682400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c109440 0x7f667c1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6682400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c109440 0x7f667c1a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55736/0 (socket says 172.21.15.102:55736) 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6681a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f667c074140 0x7f667c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6682400640 1 -- 172.21.15.102:0/3825111561 learned_addr learned my addr 172.21.15.102:0/3825111561 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6681000640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f667c075720 0x7f667c19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6682400640 1 -- 172.21.15.102:0/3825111561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f667c074140 msgr2=0x7f667c199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6682400640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f667c074140 0x7f667c199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6682400640 1 -- 172.21.15.102:0/3825111561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f667c075720 msgr2=0x7f667c19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6682400640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f667c075720 0x7f667c19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6682400640 1 -- 172.21.15.102:0/3825111561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c1a1720 con 0x7f667c109440 2024-04-25T01:54:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6681a00640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f667c074140 0x7f667c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6681000640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f667c075720 0x7f667c19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.686+0000 7f6682400640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c109440 0x7f667c1a10b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f667400eb10 tx=0x7f667400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.687+0000 7f667a000640 1 -- 172.21.15.102:0/3825111561 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674004b20 con 0x7f667c109440 2024-04-25T01:54:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.687+0000 7f667a000640 1 -- 172.21.15.102:0/3825111561 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6674004cc0 con 0x7f667c109440 2024-04-25T01:54:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.687+0000 7f667a000640 1 -- 172.21.15.102:0/3825111561 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674010760 con 0x7f667c109440 2024-04-25T01:54:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.687+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667c1a1a10 con 0x7f667c109440 2024-04-25T01:54:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.687+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f667c1a1ee0 con 0x7f667c109440 2024-04-25T01:54:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.688+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667c108200 con 0x7f667c109440 2024-04-25T01:54:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.690+0000 7f667a000640 1 -- 172.21.15.102:0/3825111561 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6674005020 con 0x7f667c109440 2024-04-25T01:54:27.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.692+0000 7f667a000640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f66640a2b90 0x7f66640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:27.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.693+0000 7f667a000640 1 -- 172.21.15.102:0/3825111561 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f66740cf640 con 0x7f667c109440 2024-04-25T01:54:27.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.693+0000 7f6681a00640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f66640a2b90 0x7f66640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:27.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.693+0000 7f6681a00640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f66640a2b90 0x7f66640a5050 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6670009fc0 tx=0x7f6670005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.697+0000 7f667a000640 1 -- 172.21.15.102:0/3825111561 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f667409c550 con 0x7f667c109440 2024-04-25T01:54:27.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.937+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f667c02d170 con 0x7f667c109440 2024-04-25T01:54:27.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.938+0000 7f667a000640 1 -- 172.21.15.102:0/3825111561 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11896 (secure 0 0 0) 0x7f667409be50 con 0x7f667c109440 2024-04-25T01:54:27.940 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 22 2024-04-25T01:54:27.940 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:27.941 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:52:49.089206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T01:54:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f66640a2b90 msgr2=0x7f66640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f66640a2b90 0x7f66640a5050 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6670009fc0 tx=0x7f6670005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c109440 msgr2=0x7f667c1a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c109440 0x7f667c1a10b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f667400eb10 tx=0x7f667400c2d0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 shutdown_connections 2024-04-25T01:54:27.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f66640a2b90 0x7f66640a5050 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f667c109440 0x7f667c1a10b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f667c075720 0x7f667c19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 --2- 172.21.15.102:0/3825111561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f667c074140 0x7f667c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 >> 172.21.15.102:0/3825111561 conn(0x7f667c0fcb20 msgr2=0x7f667c1053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.941+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 shutdown_connections 2024-04-25T01:54:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:27.942+0000 7f6682e00640 1 -- 172.21.15.102:0/3825111561 wait complete. 2024-04-25T01:54:27.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:28.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:27 smithi193 ceph-mon[32981]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:54:28.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:27 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/4011529834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:28.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:27 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3532933853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.170+0000 7f264da00640 1 -- 172.21.15.102:0/3895673836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26481039c0 msgr2=0x7f2648103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.170+0000 7f264da00640 1 --2- 172.21.15.102:0/3895673836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26481039c0 0x7f2648103de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2638009ca0 tx=0x7f263802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.170+0000 7f264da00640 1 -- 172.21.15.102:0/3895673836 shutdown_connections 2024-04-25T01:54:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.170+0000 7f264da00640 1 --2- 172.21.15.102:0/3895673836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2648105570 0x7f264810be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.170+0000 7f264da00640 1 --2- 172.21.15.102:0/3895673836 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2648104bb0 0x7f2648105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.170+0000 7f264da00640 1 --2- 172.21.15.102:0/3895673836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26481039c0 0x7f2648103de0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.170+0000 7f264da00640 1 -- 172.21.15.102:0/3895673836 >> 172.21.15.102:0/3895673836 conn(0x7f26480fef30 msgr2=0x7f2648101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.170+0000 7f264da00640 1 -- 172.21.15.102:0/3895673836 shutdown_connections 2024-04-25T01:54:28.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.170+0000 7f264da00640 1 -- 172.21.15.102:0/3895673836 wait complete. 2024-04-25T01:54:28.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.171+0000 7f264da00640 1 Processor -- start 2024-04-25T01:54:28.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.171+0000 7f264da00640 1 -- start start 2024-04-25T01:54:28.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:27 smithi102 ceph-mon[27796]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:54:28.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:27 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/4011529834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:28.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:27 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3532933853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:28.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f264da00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26481039c0 0x7f26480794f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f264da00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2648104bb0 0x7f2648079a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f264da00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2648105570 0x7f2648076060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f264da00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f264810f610 con 0x7f26481039c0 2024-04-25T01:54:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f264da00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f264810f490 con 0x7f2648104bb0 2024-04-25T01:54:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f264da00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f264810f790 con 0x7f2648105570 2024-04-25T01:54:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2646000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2648104bb0 0x7f2648079a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2646000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2648104bb0 0x7f2648079a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40868/0 (socket says 172.21.15.102:40868) 2024-04-25T01:54:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2646000640 1 -- 172.21.15.102:0/1564020457 learned_addr learned my addr 172.21.15.102:0/1564020457 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2646a00640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26481039c0 0x7f26480794f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2647400640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2648105570 0x7f2648076060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2646000640 1 -- 172.21.15.102:0/1564020457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2648105570 msgr2=0x7f2648076060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2646000640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2648105570 0x7f2648076060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2646000640 1 -- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26481039c0 msgr2=0x7f26480794f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2646000640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26481039c0 0x7f26480794f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.172+0000 7f2646000640 1 -- 172.21.15.102:0/1564020457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2648076920 con 0x7f2648104bb0 2024-04-25T01:54:28.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.173+0000 7f2646000640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2648104bb0 0x7f2648079a30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f263400e9f0 tx=0x7f263400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:28.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.173+0000 7f263f400640 1 -- 172.21.15.102:0/1564020457 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263400cd80 con 0x7f2648104bb0 2024-04-25T01:54:28.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.173+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26481a6160 con 0x7f2648104bb0 2024-04-25T01:54:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.173+0000 7f263f400640 1 -- 172.21.15.102:0/1564020457 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2634010910 con 0x7f2648104bb0 2024-04-25T01:54:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.173+0000 7f263f400640 1 -- 172.21.15.102:0/1564020457 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26340188f0 con 0x7f2648104bb0 2024-04-25T01:54:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.173+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26481a65c0 con 0x7f2648104bb0 2024-04-25T01:54:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.174+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f264810ad80 con 0x7f2648104bb0 2024-04-25T01:54:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.177+0000 7f263f400640 1 -- 172.21.15.102:0/1564020457 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f2634010ac0 con 0x7f2648104bb0 2024-04-25T01:54:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.179+0000 7f263f400640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f262c0a2b90 0x7f262c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.179+0000 7f263f400640 1 -- 172.21.15.102:0/1564020457 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f26340cf980 con 0x7f2648104bb0 2024-04-25T01:54:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.179+0000 7f2646a00640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f262c0a2b90 0x7f262c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.179+0000 7f2646a00640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f262c0a2b90 0x7f262c0a5050 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2638006010 tx=0x7f2638000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:28.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.180+0000 7f263f400640 1 -- 172.21.15.102:0/1564020457 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f263409c240 con 0x7f2648104bb0 2024-04-25T01:54:28.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:27 smithi110 ceph-mon[32848]: pgmap v371: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T01:54:28.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:27 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/4011529834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:28.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:27 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3532933853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:28.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.380+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2648062340 con 0x7f2648104bb0 2024-04-25T01:54:28.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.382+0000 7f263f400640 1 -- 172.21.15.102:0/1564020457 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22686 (secure 0 0 0) 0x7f26340a0540 con 0x7f2648104bb0 2024-04-25T01:54:28.384 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:28.384 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:53:43.851432+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f262c0a2b90 msgr2=0x7f262c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f262c0a2b90 0x7f262c0a5050 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2638006010 tx=0x7f2638000f40 comp rx=0 tx=0).stop 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2648104bb0 msgr2=0x7f2648079a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2648104bb0 0x7f2648079a30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f263400e9f0 tx=0x7f263400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 shutdown_connections 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f262c0a2b90 0x7f262c0a5050 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2648105570 0x7f2648076060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2648104bb0 0x7f2648079a30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 --2- 172.21.15.102:0/1564020457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26481039c0 0x7f26480794f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 >> 172.21.15.102:0/1564020457 conn(0x7f26480fef30 msgr2=0x7f2648107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 shutdown_connections 2024-04-25T01:54:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.385+0000 7f264da00640 1 -- 172.21.15.102:0/1564020457 wait complete. 2024-04-25T01:54:28.407 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:41.585137+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:43.022666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:54:28.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:54:28.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.613+0000 7f2001200640 1 -- 172.21.15.102:0/3084762625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc077ce0 msgr2=0x7f1ffc078580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.613+0000 7f2001200640 1 --2- 172.21.15.102:0/3084762625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc077ce0 0x7f1ffc078580 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009ca0 tx=0x7f1ff402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.614+0000 7f2001200640 1 -- 172.21.15.102:0/3084762625 shutdown_connections 2024-04-25T01:54:28.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.614+0000 7f2001200640 1 --2- 172.21.15.102:0/3084762625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc077ce0 0x7f1ffc078580 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.614+0000 7f2001200640 1 --2- 172.21.15.102:0/3084762625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ffc077340 0x7f1ffc0777a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.614+0000 7f2001200640 1 --2- 172.21.15.102:0/3084762625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ffc076100 0x7f1ffc076520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.614+0000 7f2001200640 1 -- 172.21.15.102:0/3084762625 >> 172.21.15.102:0/3084762625 conn(0x7f1ffc0fef30 msgr2=0x7f1ffc101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:28.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.614+0000 7f2001200640 1 -- 172.21.15.102:0/3084762625 shutdown_connections 2024-04-25T01:54:28.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.614+0000 7f2001200640 1 -- 172.21.15.102:0/3084762625 wait complete. 2024-04-25T01:54:28.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.615+0000 7f2001200640 1 Processor -- start 2024-04-25T01:54:28.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.615+0000 7f2001200640 1 -- start start 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.615+0000 7f2001200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc076100 0x7f1ffc19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.615+0000 7f2001200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ffc077340 0x7f1ffc19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.615+0000 7f2001200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ffc077ce0 0x7f1ffc1a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.615+0000 7f2001200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc113a30 con 0x7f1ffc077ce0 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.615+0000 7f2001200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc1138b0 con 0x7f1ffc076100 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.615+0000 7f2001200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc113bb0 con 0x7f1ffc077340 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ffa800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ffc077ce0 0x7f1ffc1a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff9e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc076100 0x7f1ffc19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ffc077340 0x7f1ffc19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff9e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc076100 0x7f1ffc19e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40888/0 (socket says 172.21.15.102:40888) 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ffa800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ffc077ce0 0x7f1ffc1a55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55788/0 (socket says 172.21.15.102:55788) 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ffc077340 0x7f1ffc19e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53104/0 (socket says 172.21.15.102:53104) 2024-04-25T01:54:28.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff9e00640 1 -- 172.21.15.102:0/1124525715 learned_addr learned my addr 172.21.15.102:0/1124525715 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff9e00640 1 -- 172.21.15.102:0/1124525715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ffc077340 msgr2=0x7f1ffc19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff9e00640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ffc077340 0x7f1ffc19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff9e00640 1 -- 172.21.15.102:0/1124525715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ffc077ce0 msgr2=0x7f1ffc1a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff9e00640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ffc077ce0 0x7f1ffc1a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff9e00640 1 -- 172.21.15.102:0/1124525715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc1a5c50 con 0x7f1ffc076100 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.616+0000 7f1ff9e00640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc076100 0x7f1ffc19e030 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00e9f0 tx=0x7f1fec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.617+0000 7f1ff2a00640 1 -- 172.21.15.102:0/1124525715 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00cd80 con 0x7f1ffc076100 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.617+0000 7f1ff2a00640 1 -- 172.21.15.102:0/1124525715 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fec010910 con 0x7f1ffc076100 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.617+0000 7f1ff2a00640 1 -- 172.21.15.102:0/1124525715 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fec018990 con 0x7f1ffc076100 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.617+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc1a5f40 con 0x7f1ffc076100 2024-04-25T01:54:28.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.617+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc1a6400 con 0x7f1ffc076100 2024-04-25T01:54:28.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.618+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ffc110310 con 0x7f1ffc076100 2024-04-25T01:54:28.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.620+0000 7f1ff2a00640 1 -- 172.21.15.102:0/1124525715 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1fec010ac0 con 0x7f1ffc076100 2024-04-25T01:54:28.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.621+0000 7f1ff2a00640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1fe00a2c10 0x7f1fe00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:28.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.621+0000 7f1ff2a00640 1 -- 172.21.15.102:0/1124525715 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f1fec0cf300 con 0x7f1ffc076100 2024-04-25T01:54:28.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.621+0000 7f1ff3e00640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1fe00a2c10 0x7f1fe00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:28.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.622+0000 7f1ff3e00640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1fe00a2c10 0x7f1fe00a50d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc19f540 tx=0x7f1fe4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:28.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.625+0000 7f1ff2a00640 1 -- 172.21.15.102:0/1124525715 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1fec09bbc0 con 0x7f1ffc076100 2024-04-25T01:54:28.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.826+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1ffc0623e0 con 0x7f1ffc076100 2024-04-25T01:54:28.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.827+0000 7f1ff2a00640 1 -- 172.21.15.102:0/1124525715 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f1fec09fec0 con 0x7f1ffc076100 2024-04-25T01:54:28.830 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:54:28.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1fe00a2c10 msgr2=0x7f1fe00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1fe00a2c10 0x7f1fe00a50d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc19f540 tx=0x7f1fe4005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:28.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc076100 msgr2=0x7f1ffc19e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:28.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc076100 0x7f1ffc19e030 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00e9f0 tx=0x7f1fec00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 shutdown_connections 2024-04-25T01:54:28.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1fe00a2c10 0x7f1fe00a50d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ffc077ce0 0x7f1ffc1a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ffc077340 0x7f1ffc19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 --2- 172.21.15.102:0/1124525715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ffc076100 0x7f1ffc19e030 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 >> 172.21.15.102:0/1124525715 conn(0x7f1ffc0fef30 msgr2=0x7f1ffc100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 shutdown_connections 2024-04-25T01:54:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:28.830+0000 7f2001200640 1 -- 172.21.15.102:0/1124525715 wait complete. 2024-04-25T01:54:28.851 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:54:29.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.058+0000 7f05afe00640 1 -- 172.21.15.102:0/2258848986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b0101670 msgr2=0x7f05b0101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.058+0000 7f05afe00640 1 --2- 172.21.15.102:0/2258848986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b0101670 0x7f05b0101a90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009ca0 tx=0x7f05a002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.058+0000 7f05afe00640 1 -- 172.21.15.102:0/2258848986 shutdown_connections 2024-04-25T01:54:29.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.058+0000 7f05afe00640 1 --2- 172.21.15.102:0/2258848986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05b0103220 0x7f05b0109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.058+0000 7f05afe00640 1 --2- 172.21.15.102:0/2258848986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05b0102860 0x7f05b0102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.058+0000 7f05afe00640 1 --2- 172.21.15.102:0/2258848986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b0101670 0x7f05b0101a90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.058+0000 7f05afe00640 1 -- 172.21.15.102:0/2258848986 >> 172.21.15.102:0/2258848986 conn(0x7f05b00fcbe0 msgr2=0x7f05b00ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:29.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.058+0000 7f05afe00640 1 -- 172.21.15.102:0/2258848986 shutdown_connections 2024-04-25T01:54:29.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.058+0000 7f05afe00640 1 -- 172.21.15.102:0/2258848986 wait complete. 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.059+0000 7f05afe00640 1 Processor -- start 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.059+0000 7f05afe00640 1 -- start start 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05afe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05b0101670 0x7f05b0197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05afe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b0102860 0x7f05b0197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05afe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05b0103220 0x7f05b019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05afe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f05b010d310 con 0x7f05b0102860 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05afe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f05b010d190 con 0x7f05b0101670 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05afe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f05b010d490 con 0x7f05b0103220 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05aea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05b0101670 0x7f05b0197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05ae000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b0102860 0x7f05b0197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05aea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05b0101670 0x7f05b0197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40894/0 (socket says 172.21.15.102:40894) 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05aea00640 1 -- 172.21.15.102:0/1167712252 learned_addr learned my addr 172.21.15.102:0/1167712252 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05af400640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05b0103220 0x7f05b019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05aea00640 1 -- 172.21.15.102:0/1167712252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05b0103220 msgr2=0x7f05b019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05aea00640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05b0103220 0x7f05b019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.060+0000 7f05aea00640 1 -- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b0102860 msgr2=0x7f05b0197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.061+0000 7f05aea00640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b0102860 0x7f05b0197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.061+0000 7f05aea00640 1 -- 172.21.15.102:0/1167712252 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b019f5a0 con 0x7f05b0101670 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.061+0000 7f05ae000640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b0102860 0x7f05b0197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.061+0000 7f05aea00640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05b0101670 0x7f05b0197980 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f05a0000f80 tx=0x7f05a00319f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.061+0000 7f05a7400640 1 -- 172.21.15.102:0/1167712252 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0003b70 con 0x7f05b0101670 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.061+0000 7f05a7400640 1 -- 172.21.15.102:0/1167712252 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05a0003d10 con 0x7f05b0101670 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.061+0000 7f05a7400640 1 -- 172.21.15.102:0/1167712252 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a002fc50 con 0x7f05b0101670 2024-04-25T01:54:29.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.061+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b019f830 con 0x7f05b0101670 2024-04-25T01:54:29.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.061+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b019fd80 con 0x7f05b0101670 2024-04-25T01:54:29.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.062+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05b0108a00 con 0x7f05b0101670 2024-04-25T01:54:29.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.064+0000 7f05a7400640 1 -- 172.21.15.102:0/1167712252 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f05a0004040 con 0x7f05b0101670 2024-04-25T01:54:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.066+0000 7f05a7400640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05940a2b90 0x7f05940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.066+0000 7f05a7400640 1 -- 172.21.15.102:0/1167712252 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f05a00f05f0 con 0x7f05b0101670 2024-04-25T01:54:29.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.066+0000 7f05ae000640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05940a2b90 0x7f05940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.067+0000 7f05ae000640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05940a2b90 0x7f05940a5050 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f05b0198e90 tx=0x7f0598005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:29.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.070+0000 7f05a7400640 1 -- 172.21.15.102:0/1167712252 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f05a00bd500 con 0x7f05b0101670 2024-04-25T01:54:29.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:28 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3825111561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:29.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:28 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1564020457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:29.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:28 smithi102 ceph-mon[27796]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T01:54:29.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:28 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1124525715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:29.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:28 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3825111561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:29.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:28 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1564020457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:29.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:28 smithi110 ceph-mon[32848]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T01:54:29.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:28 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1124525715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:29.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:28 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3825111561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:29.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:28 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1564020457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:29.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:28 smithi193 ceph-mon[32981]: pgmap v372: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T01:54:29.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:28 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1124525715' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:54:29.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.272+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f05b0062340 con 0x7f05b0101670 2024-04-25T01:54:29.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.273+0000 7f05a7400640 1 -- 172.21.15.102:0/1167712252 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f05a00bce00 con 0x7f05b0101670 2024-04-25T01:54:29.275 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05940a2b90 msgr2=0x7f05940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05940a2b90 0x7f05940a5050 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f05b0198e90 tx=0x7f0598005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05b0101670 msgr2=0x7f05b0197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05b0101670 0x7f05b0197980 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f05a0000f80 tx=0x7f05a00319f0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 shutdown_connections 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05940a2b90 0x7f05940a5050 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05b0103220 0x7f05b019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b0102860 0x7f05b0197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 --2- 172.21.15.102:0/1167712252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05b0101670 0x7f05b0197980 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 >> 172.21.15.102:0/1167712252 conn(0x7f05b00fcbe0 msgr2=0x7f05b0105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:29.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 shutdown_connections 2024-04-25T01:54:29.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.276+0000 7f05afe00640 1 -- 172.21.15.102:0/1167712252 wait complete. 2024-04-25T01:54:29.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:54:29.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.492+0000 7f682ba00640 1 -- 172.21.15.102:0/1427520706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68241039a0 msgr2=0x7f6824103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.492+0000 7f682ba00640 1 --2- 172.21.15.102:0/1427520706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68241039a0 0x7f6824103dc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6814009900 tx=0x7f681402a3f0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.493+0000 7f682ba00640 1 -- 172.21.15.102:0/1427520706 shutdown_connections 2024-04-25T01:54:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.493+0000 7f682ba00640 1 --2- 172.21.15.102:0/1427520706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6824105550 0x7f682410bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.493+0000 7f682ba00640 1 --2- 172.21.15.102:0/1427520706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6824104b90 0x7f6824105010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.493+0000 7f682ba00640 1 --2- 172.21.15.102:0/1427520706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68241039a0 0x7f6824103dc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.493+0000 7f682ba00640 1 -- 172.21.15.102:0/1427520706 >> 172.21.15.102:0/1427520706 conn(0x7f68240fef30 msgr2=0x7f6824101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.493+0000 7f682ba00640 1 -- 172.21.15.102:0/1427520706 shutdown_connections 2024-04-25T01:54:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.493+0000 7f682ba00640 1 -- 172.21.15.102:0/1427520706 wait complete. 2024-04-25T01:54:29.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.494+0000 7f682ba00640 1 Processor -- start 2024-04-25T01:54:29.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.494+0000 7f682ba00640 1 -- start start 2024-04-25T01:54:29.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.494+0000 7f682ba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68241039a0 0x7f6824199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.494+0000 7f682ba00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6824104b90 0x7f682419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.494+0000 7f682ba00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6824105550 0x7f68241a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.494+0000 7f682ba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f682410f620 con 0x7f68241039a0 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.494+0000 7f682ba00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f682410f4a0 con 0x7f6824104b90 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.494+0000 7f682ba00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f682410f7a0 con 0x7f6824105550 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68241039a0 0x7f6824199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68241039a0 0x7f6824199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55810/0 (socket says 172.21.15.102:55810) 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829000640 1 -- 172.21.15.102:0/1520362697 learned_addr learned my addr 172.21.15.102:0/1520362697 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f681be00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6824104b90 0x7f682419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829a00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6824105550 0x7f68241a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829000640 1 -- 172.21.15.102:0/1520362697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6824105550 msgr2=0x7f68241a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829000640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6824105550 0x7f68241a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829000640 1 -- 172.21.15.102:0/1520362697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6824104b90 msgr2=0x7f682419a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829000640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6824104b90 0x7f682419a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829000640 1 -- 172.21.15.102:0/1520362697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68241a1910 con 0x7f68241039a0 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f681be00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6824104b90 0x7f682419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.495+0000 7f6829000640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68241039a0 0x7f6824199d00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6814031610 tx=0x7f6814004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.496+0000 7f6822000640 1 -- 172.21.15.102:0/1520362697 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681403f030 con 0x7f68241039a0 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.496+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68241a1ba0 con 0x7f68241039a0 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.496+0000 7f6822000640 1 -- 172.21.15.102:0/1520362697 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6814003ad0 con 0x7f68241039a0 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.496+0000 7f6822000640 1 -- 172.21.15.102:0/1520362697 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6814035070 con 0x7f68241039a0 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.496+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68241a20e0 con 0x7f68241039a0 2024-04-25T01:54:29.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.497+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f682410ad90 con 0x7f68241039a0 2024-04-25T01:54:29.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.499+0000 7f6822000640 1 -- 172.21.15.102:0/1520362697 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6814042030 con 0x7f68241039a0 2024-04-25T01:54:29.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.500+0000 7f6822000640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68080a2c10 0x7f68080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.501+0000 7f6822000640 1 -- 172.21.15.102:0/1520362697 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f68140ef5b0 con 0x7f68241039a0 2024-04-25T01:54:29.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.501+0000 7f681be00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68080a2c10 0x7f68080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.501+0000 7f681be00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68080a2c10 0x7f68080a50d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f680c009fc0 tx=0x7f680c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:29.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.504+0000 7f6822000640 1 -- 172.21.15.102:0/1520362697 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68140bbed0 con 0x7f68241039a0 2024-04-25T01:54:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.701+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6824062370 con 0x7f68241039a0 2024-04-25T01:54:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.702+0000 7f6822000640 1 -- 172.21.15.102:0/1520362697 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f68140c01d0 con 0x7f68241039a0 2024-04-25T01:54:29.704 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68080a2c10 msgr2=0x7f68080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68080a2c10 0x7f68080a50d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f680c009fc0 tx=0x7f680c005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68241039a0 msgr2=0x7f6824199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68241039a0 0x7f6824199d00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6814031610 tx=0x7f6814004270 comp rx=0 tx=0).stop 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 shutdown_connections 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68080a2c10 0x7f68080a50d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6824105550 0x7f68241a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6824104b90 0x7f682419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 --2- 172.21.15.102:0/1520362697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68241039a0 0x7f6824199d00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 >> 172.21.15.102:0/1520362697 conn(0x7f68240fef30 msgr2=0x7f6824107f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 shutdown_connections 2024-04-25T01:54:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.705+0000 7f682ba00640 1 -- 172.21.15.102:0/1520362697 wait complete. 2024-04-25T01:54:29.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:54:29.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.922+0000 7ff62d000640 1 -- 172.21.15.102:0/4077838525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff628077340 msgr2=0x7ff6280777a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.922+0000 7ff62d000640 1 --2- 172.21.15.102:0/4077838525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff628077340 0x7ff6280777a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff610009860 tx=0x7ff61002a340 comp rx=0 tx=0).stop 2024-04-25T01:54:29.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.922+0000 7ff62d000640 1 -- 172.21.15.102:0/4077838525 shutdown_connections 2024-04-25T01:54:29.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.922+0000 7ff62d000640 1 --2- 172.21.15.102:0/4077838525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff628077ce0 0x7ff628078580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.922+0000 7ff62d000640 1 --2- 172.21.15.102:0/4077838525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff628077340 0x7ff6280777a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.922+0000 7ff62d000640 1 --2- 172.21.15.102:0/4077838525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff628076100 0x7ff628076520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.922+0000 7ff62d000640 1 -- 172.21.15.102:0/4077838525 >> 172.21.15.102:0/4077838525 conn(0x7ff6280fef30 msgr2=0x7ff628101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:29.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.922+0000 7ff62d000640 1 -- 172.21.15.102:0/4077838525 shutdown_connections 2024-04-25T01:54:29.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.922+0000 7ff62d000640 1 -- 172.21.15.102:0/4077838525 wait complete. 2024-04-25T01:54:29.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.923+0000 7ff62d000640 1 Processor -- start 2024-04-25T01:54:29.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.923+0000 7ff62d000640 1 -- start start 2024-04-25T01:54:29.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff62d000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff628076100 0x7ff62819e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff62d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff628077340 0x7ff62819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff62d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff628077ce0 0x7ff6281a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff62d000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff628113a10 con 0x7ff628077340 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff62d000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff628113890 con 0x7ff628076100 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff62d000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff628113b90 con 0x7ff628077ce0 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff61fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff628077340 0x7ff62819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff61fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff628077340 0x7ff62819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55832/0 (socket says 172.21.15.102:55832) 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff61fe00640 1 -- 172.21.15.102:0/2593868685 learned_addr learned my addr 172.21.15.102:0/2593868685 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff625e00640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff628076100 0x7ff62819e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff626800640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff628077ce0 0x7ff6281a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff61fe00640 1 -- 172.21.15.102:0/2593868685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff628077ce0 msgr2=0x7ff6281a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff61fe00640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff628077ce0 0x7ff6281a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff61fe00640 1 -- 172.21.15.102:0/2593868685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff628076100 msgr2=0x7ff62819e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff61fe00640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff628076100 0x7ff62819e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff61fe00640 1 -- 172.21.15.102:0/2593868685 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6281a5d00 con 0x7ff628077340 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.924+0000 7ff625e00640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff628076100 0x7ff62819e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:29.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.925+0000 7ff61fe00640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff628077340 0x7ff62819e620 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff610005940 tx=0x7ff610033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:29.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.925+0000 7ff61ea00640 1 -- 172.21.15.102:0/2593868685 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61003f020 con 0x7ff628077340 2024-04-25T01:54:29.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.925+0000 7ff61ea00640 1 -- 172.21.15.102:0/2593868685 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff610030850 con 0x7ff628077340 2024-04-25T01:54:29.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.925+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6281a5f90 con 0x7ff628077340 2024-04-25T01:54:29.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.925+0000 7ff61ea00640 1 -- 172.21.15.102:0/2593868685 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610037040 con 0x7ff628077340 2024-04-25T01:54:29.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.925+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6281a63f0 con 0x7ff628077340 2024-04-25T01:54:29.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.926+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff628110520 con 0x7ff628077340 2024-04-25T01:54:29.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.928+0000 7ff61ea00640 1 -- 172.21.15.102:0/2593868685 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff610042020 con 0x7ff628077340 2024-04-25T01:54:29.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.930+0000 7ff61ea00640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff60c0a2b90 0x7ff60c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:29.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.930+0000 7ff61ea00640 1 -- 172.21.15.102:0/2593868685 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7ff6100ef330 con 0x7ff628077340 2024-04-25T01:54:29.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.930+0000 7ff625e00640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff60c0a2b90 0x7ff60c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:29.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.930+0000 7ff625e00640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff60c0a2b90 0x7ff60c0a5050 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff618009fc0 tx=0x7ff618005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:29.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:29.934+0000 7ff61ea00640 1 -- 172.21.15.102:0/2593868685 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff610049020 con 0x7ff628077340 2024-04-25T01:54:30.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.172+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff62802d390 con 0x7ff628077340 2024-04-25T01:54:30.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:29 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1167712252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:30.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:29 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1520362697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:30.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.174+0000 7ff61ea00640 1 -- 172.21.15.102:0/2593868685 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11896 (secure 0 0 0) 0x7ff6100bc2a0 con 0x7ff628077340 2024-04-25T01:54:30.176 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:30.176 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:52:49.089206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T01:54:30.176 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 22 2024-04-25T01:54:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff60c0a2b90 msgr2=0x7ff60c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff60c0a2b90 0x7ff60c0a5050 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff618009fc0 tx=0x7ff618005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff628077340 msgr2=0x7ff62819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff628077340 0x7ff62819e620 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff610005940 tx=0x7ff610033d90 comp rx=0 tx=0).stop 2024-04-25T01:54:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 shutdown_connections 2024-04-25T01:54:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7ff60c0a2b90 0x7ff60c0a5050 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff628077ce0 0x7ff6281a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff628077340 0x7ff62819e620 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 --2- 172.21.15.102:0/2593868685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff628076100 0x7ff62819e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.176+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 >> 172.21.15.102:0/2593868685 conn(0x7ff6280fef30 msgr2=0x7ff628100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.177+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 shutdown_connections 2024-04-25T01:54:30.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.177+0000 7ff62d000640 1 -- 172.21.15.102:0/2593868685 wait complete. 2024-04-25T01:54:30.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:30.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:29 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1167712252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:30.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:29 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1520362697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:30.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:29 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1167712252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:54:30.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:29 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1520362697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:54:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.395+0000 7f543e200640 1 -- 172.21.15.102:0/1420949119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5438108db0 msgr2=0x7f5438109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.395+0000 7f543e200640 1 --2- 172.21.15.102:0/1420949119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5438108db0 0x7f5438109230 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5420009900 tx=0x7f542002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.395+0000 7f543e200640 1 -- 172.21.15.102:0/1420949119 shutdown_connections 2024-04-25T01:54:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.395+0000 7f543e200640 1 --2- 172.21.15.102:0/1420949119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5438109770 0x7f5438110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.395+0000 7f543e200640 1 --2- 172.21.15.102:0/1420949119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5438108db0 0x7f5438109230 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.395+0000 7f543e200640 1 --2- 172.21.15.102:0/1420949119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5438107bc0 0x7f5438107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.395+0000 7f543e200640 1 -- 172.21.15.102:0/1420949119 >> 172.21.15.102:0/1420949119 conn(0x7f5438076b80 msgr2=0x7f5438078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.395+0000 7f543e200640 1 -- 172.21.15.102:0/1420949119 shutdown_connections 2024-04-25T01:54:30.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.395+0000 7f543e200640 1 -- 172.21.15.102:0/1420949119 wait complete. 2024-04-25T01:54:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543e200640 1 Processor -- start 2024-04-25T01:54:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543e200640 1 -- start start 2024-04-25T01:54:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543e200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5438107bc0 0x7f543819df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5438108db0 0x7f543819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543e200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5438109770 0x7f54381a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5438113890 con 0x7f5438108db0 2024-04-25T01:54:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543e200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5438113710 con 0x7f5438107bc0 2024-04-25T01:54:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543e200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5438113a10 con 0x7f5438109770 2024-04-25T01:54:30.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f542fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5438108db0 0x7f543819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543d800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5438109770 0x7f54381a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543ce00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5438107bc0 0x7f543819df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f542fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5438108db0 0x7f543819e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55858/0 (socket says 172.21.15.102:55858) 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f542fe00640 1 -- 172.21.15.102:0/3378174459 learned_addr learned my addr 172.21.15.102:0/3378174459 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.397+0000 7f543d800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5438109770 0x7f54381a5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53172/0 (socket says 172.21.15.102:53172) 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f543ce00640 1 -- 172.21.15.102:0/3378174459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5438109770 msgr2=0x7f54381a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f543ce00640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5438109770 0x7f54381a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f543ce00640 1 -- 172.21.15.102:0/3378174459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5438108db0 msgr2=0x7f543819e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f543ce00640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5438108db0 0x7f543819e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f543ce00640 1 -- 172.21.15.102:0/3378174459 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54381a5b00 con 0x7f5438107bc0 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f543ce00640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5438107bc0 0x7f543819df30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f542800e9c0 tx=0x7f542800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f5436000640 1 -- 172.21.15.102:0/3378174459 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542800cd80 con 0x7f5438107bc0 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54381a5df0 con 0x7f5438107bc0 2024-04-25T01:54:30.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f5436000640 1 -- 172.21.15.102:0/3378174459 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5428010910 con 0x7f5438107bc0 2024-04-25T01:54:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f5436000640 1 -- 172.21.15.102:0/3378174459 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5428018980 con 0x7f5438107bc0 2024-04-25T01:54:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.398+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54381a6250 con 0x7f5438107bc0 2024-04-25T01:54:30.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.399+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543810ee80 con 0x7f5438107bc0 2024-04-25T01:54:30.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.402+0000 7f5436000640 1 -- 172.21.15.102:0/3378174459 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f542801f020 con 0x7f5438107bc0 2024-04-25T01:54:30.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.403+0000 7f5436000640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f541c0a2b90 0x7f541c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.403+0000 7f5436000640 1 -- 172.21.15.102:0/3378174459 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f54280cf190 con 0x7f5438107bc0 2024-04-25T01:54:30.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.403+0000 7f542fe00640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f541c0a2b90 0x7f541c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.404+0000 7f542fe00640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f541c0a2b90 0x7f541c0a5050 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f543819f3f0 tx=0x7f54200315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.405+0000 7f5436000640 1 -- 172.21.15.102:0/3378174459 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f542809c0a0 con 0x7f5438107bc0 2024-04-25T01:54:30.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.592+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f543802d390 con 0x7f5438107bc0 2024-04-25T01:54:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.594+0000 7f5436000640 1 -- 172.21.15.102:0/3378174459 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22686 (secure 0 0 0) 0x7f542809b9a0 con 0x7f5438107bc0 2024-04-25T01:54:30.596 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:30.597 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:53:43.851432+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.597+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f541c0a2b90 msgr2=0x7f541c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.597+0000 7f543e200640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f541c0a2b90 0x7f541c0a5050 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f543819f3f0 tx=0x7f54200315b0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5438107bc0 msgr2=0x7f543819df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5438107bc0 0x7f543819df30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f542800e9c0 tx=0x7f542800ee80 comp rx=0 tx=0).stop 2024-04-25T01:54:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 shutdown_connections 2024-04-25T01:54:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f541c0a2b90 0x7f541c0a5050 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5438109770 0x7f54381a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5438108db0 0x7f543819e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 --2- 172.21.15.102:0/3378174459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5438107bc0 0x7f543819df30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 >> 172.21.15.102:0/3378174459 conn(0x7f5438076b80 msgr2=0x7f543810c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 shutdown_connections 2024-04-25T01:54:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.598+0000 7f543e200640 1 -- 172.21.15.102:0/3378174459 wait complete. 2024-04-25T01:54:30.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:54:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.812+0000 7f4fc6e00640 1 -- 172.21.15.102:0/2035115904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc01039c0 msgr2=0x7f4fc0103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.812+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/2035115904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc01039c0 0x7f4fc0103de0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009900 tx=0x7f4fb002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.813+0000 7f4fc6e00640 1 -- 172.21.15.102:0/2035115904 shutdown_connections 2024-04-25T01:54:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.813+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/2035115904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc0105570 0x7f4fc010be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.813+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/2035115904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fc0104bb0 0x7f4fc0105030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.813+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/2035115904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc01039c0 0x7f4fc0103de0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.813+0000 7f4fc6e00640 1 -- 172.21.15.102:0/2035115904 >> 172.21.15.102:0/2035115904 conn(0x7f4fc00fef30 msgr2=0x7f4fc0101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.813+0000 7f4fc6e00640 1 -- 172.21.15.102:0/2035115904 shutdown_connections 2024-04-25T01:54:30.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.813+0000 7f4fc6e00640 1 -- 172.21.15.102:0/2035115904 wait complete. 2024-04-25T01:54:30.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fc6e00640 1 Processor -- start 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fc6e00640 1 -- start start 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fc6e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fc01039c0 0x7f4fc0199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fc6e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc0104bb0 0x7f4fc019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fc6e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc0105570 0x7f4fc01a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fc6e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc010f660 con 0x7f4fc0104bb0 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fc6e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc010f4e0 con 0x7f4fc01039c0 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fc6e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc010f7e0 con 0x7f4fc0105570 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fb7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc0104bb0 0x7f4fc019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fbfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fc01039c0 0x7f4fc0199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fc4e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc0105570 0x7f4fc01a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fb7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc0104bb0 0x7f4fc019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55864/0 (socket says 172.21.15.102:55864) 2024-04-25T01:54:30.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fb7e00640 1 -- 172.21.15.102:0/554649337 learned_addr learned my addr 172.21.15.102:0/554649337 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.814+0000 7f4fbfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fc01039c0 0x7f4fc0199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40986/0 (socket says 172.21.15.102:40986) 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fb7e00640 1 -- 172.21.15.102:0/554649337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc0105570 msgr2=0x7f4fc01a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fb7e00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc0105570 0x7f4fc01a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fb7e00640 1 -- 172.21.15.102:0/554649337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fc01039c0 msgr2=0x7f4fc0199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fb7e00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fc01039c0 0x7f4fc0199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fb7e00640 1 -- 172.21.15.102:0/554649337 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc01a1930 con 0x7f4fc0104bb0 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fbfe00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fc01039c0 0x7f4fc0199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fb7e00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc0104bb0 0x7f4fc019a260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800e9f0 tx=0x7f4fa800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fbd600640 1 -- 172.21.15.102:0/554649337 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa800cd80 con 0x7f4fc0104bb0 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc01a1c20 con 0x7f4fc0104bb0 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fbd600640 1 -- 172.21.15.102:0/554649337 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fa8010910 con 0x7f4fc0104bb0 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fbd600640 1 -- 172.21.15.102:0/554649337 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa80188f0 con 0x7f4fc0104bb0 2024-04-25T01:54:30.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.815+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc01a2190 con 0x7f4fc0104bb0 2024-04-25T01:54:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.816+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc010ace0 con 0x7f4fc0104bb0 2024-04-25T01:54:30.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.818+0000 7f4fbd600640 1 -- 172.21.15.102:0/554649337 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4fa8010ac0 con 0x7f4fc0104bb0 2024-04-25T01:54:30.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.820+0000 7f4fbd600640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4fa40a2c10 0x7f4fa40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:30.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.820+0000 7f4fbd600640 1 -- 172.21.15.102:0/554649337 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f4fa80cf400 con 0x7f4fc0104bb0 2024-04-25T01:54:30.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.820+0000 7f4fbfe00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4fa40a2c10 0x7f4fa40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:30.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.820+0000 7f4fbfe00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4fa40a2c10 0x7f4fa40a50d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4fb002a9e0 tx=0x7f4fb00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:30.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:30.823+0000 7f4fbd600640 1 -- 172.21.15.102:0/554649337 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4fa809bd20 con 0x7f4fc0104bb0 2024-04-25T01:54:31.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.064+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fc002d170 con 0x7f4fc0104bb0 2024-04-25T01:54:31.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.066+0000 7f4fbd600640 1 -- 172.21.15.102:0/554649337 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11896 (secure 0 0 0) 0x7f4fa80a0020 con 0x7f4fc0104bb0 2024-04-25T01:54:31.068 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:31.068 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:52:49.089206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T01:54:31.068 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 22 2024-04-25T01:54:31.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.068+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4fa40a2c10 msgr2=0x7f4fa40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.068+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4fa40a2c10 0x7f4fa40a50d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4fb002a9e0 tx=0x7f4fb00315b0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.068+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc0104bb0 msgr2=0x7f4fc019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.068+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc0104bb0 0x7f4fc019a260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800e9f0 tx=0x7f4fa800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.069+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 shutdown_connections 2024-04-25T01:54:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.069+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4fa40a2c10 0x7f4fa40a50d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.069+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4fc0105570 0x7f4fc01a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.069+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4fc0104bb0 0x7f4fc019a260 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.069+0000 7f4fc6e00640 1 --2- 172.21.15.102:0/554649337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fc01039c0 0x7f4fc0199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.069+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 >> 172.21.15.102:0/554649337 conn(0x7f4fc00fef30 msgr2=0x7f4fc0107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.069+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 shutdown_connections 2024-04-25T01:54:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.069+0000 7f4fc6e00640 1 -- 172.21.15.102:0/554649337 wait complete. 2024-04-25T01:54:31.089 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:52:43.526145+0000', 'modified': '2024-04-25T01:52:49.089206+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24587, 'mds_1': 24541, 'mds_2': 24533}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6835/4057730311', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6834', 'nonce': 4057730311}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 4057730311}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/1885428891', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 1885428891}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 1885428891}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/3613019309', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 3613019309}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 3613019309}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.110:6837/1533256041', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 1533256041}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 1533256041}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.193:6839/2484219231', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 2484219231}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 2484219231}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.110:6839/4063392981', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6838', 'nonce': 4063392981}, {'type': 'v1', 'addr': '172.21.15.110:6839', 'nonce': 4063392981}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24587, 'qdb_cluster': [24533, 24541, 24587]} 2024-04-25T01:54:31.089 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T01:54:31.090 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T01:54:31.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:30 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2593868685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:31.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:30 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3378174459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:31.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:30 smithi102 ceph-mon[27796]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:31.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:30 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2593868685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:31.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:30 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3378174459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:31.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:30 smithi110 ceph-mon[32848]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:31.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:30 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2593868685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:31.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:30 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3378174459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:54:31.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:30 smithi193 ceph-mon[32981]: pgmap v373: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:31.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.281+0000 7f3810000640 1 -- 172.21.15.102:0/1753011237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38081074c0 msgr2=0x7f38081098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.281+0000 7f3810000640 1 --2- 172.21.15.102:0/1753011237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38081074c0 0x7f38081098b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3800009900 tx=0x7f380002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.281+0000 7f3810000640 1 -- 172.21.15.102:0/1753011237 shutdown_connections 2024-04-25T01:54:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.281+0000 7f3810000640 1 --2- 172.21.15.102:0/1753011237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38081074c0 0x7f38081098b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.281+0000 7f3810000640 1 --2- 172.21.15.102:0/1753011237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3808104b90 0x7f3808106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.281+0000 7f3810000640 1 --2- 172.21.15.102:0/1753011237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3808102260 0x7f3808104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.281+0000 7f3810000640 1 -- 172.21.15.102:0/1753011237 >> 172.21.15.102:0/1753011237 conn(0x7f38080fbbf0 msgr2=0x7f38080fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.281+0000 7f3810000640 1 -- 172.21.15.102:0/1753011237 shutdown_connections 2024-04-25T01:54:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.282+0000 7f3810000640 1 -- 172.21.15.102:0/1753011237 wait complete. 2024-04-25T01:54:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.282+0000 7f3810000640 1 Processor -- start 2024-04-25T01:54:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f3810000640 1 -- start start 2024-04-25T01:54:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f3810000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3808102260 0x7f3808197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f3810000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3808104b90 0x7f3808197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f3810000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38081074c0 0x7f380819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f3810000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f380810d450 con 0x7f38081074c0 2024-04-25T01:54:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f3810000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f380810d2d0 con 0x7f3808104b90 2024-04-25T01:54:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f3810000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f380810d5d0 con 0x7f3808102260 2024-04-25T01:54:31.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38081074c0 0x7f380819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380cc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3808104b90 0x7f3808197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380cc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3808104b90 0x7f3808197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41010/0 (socket says 172.21.15.102:41010) 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380cc00640 1 -- 172.21.15.102:0/2171678972 learned_addr learned my addr 172.21.15.102:0/2171678972 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380d600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3808102260 0x7f3808197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380cc00640 1 -- 172.21.15.102:0/2171678972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3808102260 msgr2=0x7f3808197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380cc00640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3808102260 0x7f3808197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380cc00640 1 -- 172.21.15.102:0/2171678972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38081074c0 msgr2=0x7f380819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380cc00640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38081074c0 0x7f380819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.283+0000 7f380cc00640 1 -- 172.21.15.102:0/2171678972 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380819f6c0 con 0x7f3808104b90 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.284+0000 7f380cc00640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3808104b90 0x7f3808197fe0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f37f400e9c0 tx=0x7f37f400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.284+0000 7f3806000640 1 -- 172.21.15.102:0/2171678972 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f400cd80 con 0x7f3808104b90 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.284+0000 7f3806000640 1 -- 172.21.15.102:0/2171678972 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f4005500 con 0x7f3808104b90 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.284+0000 7f3810000640 1 -- 172.21.15.102:0/2171678972 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f380819f9b0 con 0x7f3808104b90 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.284+0000 7f3806000640 1 -- 172.21.15.102:0/2171678972 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4010690 con 0x7f3808104b90 2024-04-25T01:54:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.284+0000 7f3810000640 1 -- 172.21.15.102:0/2171678972 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f380819ff00 con 0x7f3808104b90 2024-04-25T01:54:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.287+0000 7f3812d95740 1 Processor -- start 2024-04-25T01:54:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.287+0000 7f3806000640 1 -- 172.21.15.102:0/2171678972 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f37f40046c0 con 0x7f3808104b90 2024-04-25T01:54:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f3812d95740 1 -- start start 2024-04-25T01:54:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f3812d95740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bddbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f3812d95740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 0x563c6bde0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f3812d95740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 0x563c6bde3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f3812d95740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x563c6bcdbed0 con 0x563c6bde12c0 2024-04-25T01:54:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f3812d95740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563c6bd3edf0 con 0x563c6bddb760 2024-04-25T01:54:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f3812d95740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c6bce78d0 con 0x563c6bddc150 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f380e000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bddbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f380e000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bddbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41018/0 (socket says 172.21.15.102:41018) 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f380cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 0x563c6bde3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f380e000640 1 -- 172.21.15.102:0/1138213188 learned_addr learned my addr 172.21.15.102:0/1138213188 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.288+0000 7f380d600640 1 --2- 172.21.15.102:0/1138213188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 0x563c6bde0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f380e000640 1 -- 172.21.15.102:0/1138213188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 msgr2=0x563c6bde0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f380e000640 1 --2- 172.21.15.102:0/1138213188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 0x563c6bde0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f380e000640 1 -- 172.21.15.102:0/1138213188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 msgr2=0x563c6bde3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f380e000640 1 --2- 172.21.15.102:0/1138213188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 0x563c6bde3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f380e000640 1 -- 172.21.15.102:0/1138213188 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c6ba532d0 con 0x563c6bddb760 2024-04-25T01:54:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f380e000640 1 --2- 172.21.15.102:0/1138213188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bddbb80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f380002a9e0 tx=0x7f3800005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47757a75c5a0d65c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f37fe600640 1 -- 172.21.15.102:0/1138213188 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800040070 con 0x563c6bddb760 2024-04-25T01:54:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f37fe600640 1 -- 172.21.15.102:0/1138213188 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3800005500 con 0x563c6bddb760 2024-04-25T01:54:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f37fe600640 1 -- 172.21.15.102:0/1138213188 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800035040 con 0x563c6bddb760 2024-04-25T01:54:31.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.289+0000 7f3806000640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f37ec0a2b90 0x7f37ec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3806000640 1 -- 172.21.15.102:0/2171678972 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f37f40cf430 con 0x7f3808104b90 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f380d600640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f37ec0a2b90 0x7f37ec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3812d95740 1 -- 172.21.15.102:0/1138213188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 msgr2=0x563c6bddbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3812d95740 1 --2- 172.21.15.102:0/1138213188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bddbb80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f380002a9e0 tx=0x7f3800005e60 comp rx=0 tx=0).stop 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3812d95740 1 -- 172.21.15.102:0/1138213188 shutdown_connections 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3812d95740 1 --2- 172.21.15.102:0/1138213188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 0x563c6bde3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3812d95740 1 --2- 172.21.15.102:0/1138213188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 0x563c6bde0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3812d95740 1 --2- 172.21.15.102:0/1138213188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bddbb80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3812d95740 1 -- 172.21.15.102:0/1138213188 >> 172.21.15.102:0/1138213188 conn(0x563c6bdd6dd0 msgr2=0x563c6bdd71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3812d95740 1 -- 172.21.15.102:0/1138213188 shutdown_connections 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f3812d95740 1 -- 172.21.15.102:0/1138213188 wait complete. 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.290+0000 7f380d600640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f37ec0a2b90 0x7f37ec0a5050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x563c6bddc550 tx=0x7f37f8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.291+0000 7f3812d95740 1 -- start start 2024-04-25T01:54:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.291+0000 7f3812d95740 1 Processor -- start 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.291+0000 7f3812d95740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.291+0000 7f3812d95740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddc150 0x563c6bdea340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.291+0000 7f3812d95740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bdf1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.291+0000 7f3812d95740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x563c6bcfad20 con 0x563c6bddc150 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.291+0000 7f3812d95740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563c6bd3adb0 con 0x563c6bddb760 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.291+0000 7f3812d95740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c6bde4210 con 0x563c6bde12c0 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.291+0000 7f3812d95740 10 client.0 _ll_register_callbacks cb 0x563c6bd35f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddc150 0x563c6bdea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddc150 0x563c6bdea340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55902/0 (socket says 172.21.15.102:55902) 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380cc00640 1 -- 172.21.15.102:0/2960434974 learned_addr learned my addr 172.21.15.102:0/2960434974 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380e000640 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bdf1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380d600640 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380cc00640 1 -- 172.21.15.102:0/2960434974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 msgr2=0x563c6bdf1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380cc00640 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bdf1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380cc00640 1 -- 172.21.15.102:0/2960434974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 msgr2=0x563c6bde9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380cc00640 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380cc00640 1 -- 172.21.15.102:0/2960434974 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c6bb1d6b0 con 0x563c6bddc150 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f380cc00640 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddc150 0x563c6bdea340 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f37f40980a0 tx=0x7f37f40d29a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ccb0528366dbe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f37f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f37f2a00640 1 -- 172.21.15.102:0/2960434974 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f409ba10 con 0x563c6bddc150 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f37f2a00640 1 -- 172.21.15.102:0/2960434974 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f37f40a4420 con 0x563c6bddc150 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.292+0000 7f3812d95740 1 -- 172.21.15.102:0/2960434974 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3800005500 con 0x563c6bddc150 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f37f2a00640 1 -- 172.21.15.102:0/2960434974 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f37f409b5a0 con 0x563c6bddc150 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f3812d95740 10 client.14889 fetch_fsmap learned FSMap version 22 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f3812d95740 1 -- 172.21.15.102:0/2960434974 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x563c6bdf47c0 con 0x563c6bddc150 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f37f2a00640 1 -- 172.21.15.102:0/2960434974 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3800005500 con 0x563c6bddc150 2024-04-25T01:54:31.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f37f2a00640 1 -- 172.21.15.102:0/2960434974 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f37f409b090 con 0x563c6bddc150 2024-04-25T01:54:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f3812d95740 10 client.14889 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:54:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f3812d95740 10 client.14889 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:54:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f3812d95740 20 client.14889 populate_metadata read hostname 'smithi102' 2024-04-25T01:54:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f3812d95740 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x563c6bdf6e50 0x563c6bdf82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f3812d95740 4 client.14889 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.293+0000 7f3812d95740 1 -- 172.21.15.102:0/2960434974 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c6bd4c330 con 0x563c6bdf6e50 2024-04-25T01:54:31.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.294+0000 7f380d600640 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x563c6bdf6e50 0x563c6bdf82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.294+0000 7f380d600640 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x563c6bdf6e50 0x563c6bdf82c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a6ded26ec21a17cc server_cookie=8542f6b9499e50b6 in_seq=0 out_seq=0 2024-04-25T01:54:31.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.294+0000 7f37f2a00640 10 client.14889 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:54:31.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f37f2a00640 1 -- 172.21.15.102:0/2960434974 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c6bd4c330 con 0x563c6bdf6e50 2024-04-25T01:54:31.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f37f2a00640 10 client.14889 handle_command_reply: tid=0 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 client.14889 shutdown 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 -- 172.21.15.102:0/2960434974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddc150 msgr2=0x563c6bdea340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddc150 0x563c6bdea340 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f37f40980a0 tx=0x7f37f40d29a0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 -- 172.21.15.102:0/2960434974 shutdown_connections 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bdf1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddc150 0x563c6bdea340 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x563c6bdf6e50 0x563c6bdf82c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 --2- 172.21.15.102:0/2960434974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 -- 172.21.15.102:0/2960434974 >> 172.21.15.102:0/2960434974 conn(0x563c6bdd6dd0 msgr2=0x563c6bdd5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f37f2a00640 0 client.14889 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.295+0000 7f3812d95740 1 -- 172.21.15.102:0/2960434974 shutdown_connections 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.296+0000 7f3812d95740 1 -- 172.21.15.102:0/2960434974 wait complete. 2024-04-25T01:54:31.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.296+0000 7f3812d95740 20 client.14889 trim_cache size 0 max 16384 2024-04-25T01:54:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.315+0000 7f3812d95740 1 Processor -- start 2024-04-25T01:54:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.315+0000 7f3812d95740 1 -- start start 2024-04-25T01:54:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f3812d95740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddb760 0x563c6bde8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f3812d95740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddc150 0x563c6bdf1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f3812d95740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bde89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f3812d95740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x563c6bdf6600 con 0x563c6bddb760 2024-04-25T01:54:31.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f3812d95740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563c6bdf6cd0 con 0x563c6bddc150 2024-04-25T01:54:31.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f3812d95740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c6bdf73a0 con 0x563c6bde12c0 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bde89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bde89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53236/0 (socket says 172.21.15.102:53236) 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380e000640 1 -- 172.21.15.102:0/3984664878 learned_addr learned my addr 172.21.15.102:0/3984664878 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380d600640 1 --2- 172.21.15.102:0/3984664878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddb760 0x563c6bde8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380cc00640 1 --2- 172.21.15.102:0/3984664878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddc150 0x563c6bdf1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380e000640 1 -- 172.21.15.102:0/3984664878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddc150 msgr2=0x563c6bdf1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380e000640 1 --2- 172.21.15.102:0/3984664878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddc150 0x563c6bdf1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380e000640 1 -- 172.21.15.102:0/3984664878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddb760 msgr2=0x563c6bde8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380e000640 1 --2- 172.21.15.102:0/3984664878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddb760 0x563c6bde8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.316+0000 7f380e000640 1 -- 172.21.15.102:0/3984664878 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c6bdd5410 con 0x563c6bde12c0 2024-04-25T01:54:31.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.317+0000 7f380e000640 1 --2- 172.21.15.102:0/3984664878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bde89a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3800009900 tx=0x7f380002a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da342f7147603b9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.317+0000 7f37fdc00640 1 -- 172.21.15.102:0/3984664878 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800040040 con 0x563c6bde12c0 2024-04-25T01:54:31.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.317+0000 7f37fdc00640 1 -- 172.21.15.102:0/3984664878 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38000043f0 con 0x563c6bde12c0 2024-04-25T01:54:31.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.317+0000 7f37fdc00640 1 -- 172.21.15.102:0/3984664878 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800035040 con 0x563c6bde12c0 2024-04-25T01:54:31.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 -- 172.21.15.102:0/3984664878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 msgr2=0x563c6bde89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 --2- 172.21.15.102:0/3984664878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bde89a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3800009900 tx=0x7f380002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 -- 172.21.15.102:0/3984664878 shutdown_connections 2024-04-25T01:54:31.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 --2- 172.21.15.102:0/3984664878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bde12c0 0x563c6bde89a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 --2- 172.21.15.102:0/3984664878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddc150 0x563c6bdf1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 --2- 172.21.15.102:0/3984664878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bddb760 0x563c6bde8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 -- 172.21.15.102:0/3984664878 >> 172.21.15.102:0/3984664878 conn(0x563c6bdd6dd0 msgr2=0x563c6bdd71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 -- 172.21.15.102:0/3984664878 shutdown_connections 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 -- 172.21.15.102:0/3984664878 wait complete. 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 -- start start 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.318+0000 7f3812d95740 1 Processor -- start 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f3812d95740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f3812d95740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 0x563c6bde72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f3812d95740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 0x563c6be5adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f3812d95740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x563c6bdf8140 con 0x563c6bde12c0 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f3812d95740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563c6bdf7a70 con 0x563c6bddb760 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f3812d95740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c6bdf8810 con 0x563c6bddc150 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f3812d95740 10 client.0 _ll_register_callbacks cb 0x7f37f4009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f380d600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f380e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 0x563c6be5adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f380d600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41050/0 (socket says 172.21.15.102:41050) 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f380d600640 1 -- 172.21.15.102:0/3535494530 learned_addr learned my addr 172.21.15.102:0/3535494530 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f380cc00640 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 0x563c6bde72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.319+0000 7f380e000640 1 -- 172.21.15.102:0/3535494530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 msgr2=0x563c6bde72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f380e000640 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 0x563c6bde72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f380e000640 1 -- 172.21.15.102:0/3535494530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 msgr2=0x563c6bde6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f380e000640 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f380e000640 1 -- 172.21.15.102:0/3535494530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c6be5b3e0 con 0x563c6bde12c0 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f380d600640 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f380e000640 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 0x563c6be5adf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f380002a9e0 tx=0x7f38000048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbe1758eae80815 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f37f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f37f2a00640 1 -- 172.21.15.102:0/3535494530 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3800035030 con 0x563c6bde12c0 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f37f2a00640 1 -- 172.21.15.102:0/3535494530 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f38000318c0 con 0x563c6bde12c0 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f3812d95740 1 -- 172.21.15.102:0/3535494530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38000043f0 con 0x563c6bde12c0 2024-04-25T01:54:31.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f37f2a00640 1 -- 172.21.15.102:0/3535494530 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3800040040 con 0x563c6bde12c0 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f3812d95740 10 client.14895 fetch_fsmap learned FSMap version 22 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.320+0000 7f3812d95740 1 -- 172.21.15.102:0/3535494530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x563c6be5c6e0 con 0x563c6bde12c0 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.321+0000 7f37f2a00640 1 -- 172.21.15.102:0/3535494530 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38000043f0 con 0x563c6bde12c0 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.321+0000 7f37f2a00640 1 -- 172.21.15.102:0/3535494530 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f3800057030 con 0x563c6bde12c0 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.321+0000 7f3812d95740 10 client.14895 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.321+0000 7f3812d95740 10 client.14895 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.321+0000 7f3812d95740 20 client.14895 populate_metadata read hostname 'smithi102' 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.321+0000 7f3812d95740 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x563c6be60bf0 0x563c6be5f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.321+0000 7f3812d95740 4 client.14895 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.321+0000 7f3812d95740 1 -- 172.21.15.102:0/3535494530 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "status"}) -- 0x563c6bd46680 con 0x563c6be60bf0 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.321+0000 7f380d600640 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x563c6be60bf0 0x563c6be5f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f37f2a00640 10 client.14895 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:54:31.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f380d600640 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x563c6be60bf0 0x563c6be5f900 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a29497bd08a1ebe server_cookie=9d75e0f856349e64 in_seq=0 out_seq=0 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f37f2a00640 1 -- 172.21.15.102:0/3535494530 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563c6bd46680 con 0x563c6be60bf0 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f37f2a00640 10 client.14895 handle_command_reply: tid=0 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 client.14895 shutdown 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 -- 172.21.15.102:0/3535494530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 msgr2=0x563c6be5adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 0x563c6be5adf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f380002a9e0 tx=0x7f38000048f0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 -- 172.21.15.102:0/3535494530 shutdown_connections 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563c6bde12c0 0x563c6be5adf0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6bddc150 0x563c6bde72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x563c6be60bf0 0x563c6be5f900 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 --2- 172.21.15.102:0/3535494530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563c6bddb760 0x563c6bde6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f37f2a00640 0 client.14895 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 -- 172.21.15.102:0/3535494530 >> 172.21.15.102:0/3535494530 conn(0x563c6bdd6dd0 msgr2=0x563c6bdd9e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 -- 172.21.15.102:0/3535494530 shutdown_connections 2024-04-25T01:54:31.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.322+0000 7f3812d95740 1 -- 172.21.15.102:0/3535494530 wait complete. 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3812d95740 20 client.14895 trim_cache size 0 max 16384 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 2, 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24533, 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 103.24435851299999, 2024-04-25T01:54:31.329 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 22, 2024-04-25T01:54:31.330 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 89, 2024-04-25T01:54:31.330 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:54:31.330 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 128.525558542 2024-04-25T01:54:31.330 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:54:31.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 -- 172.21.15.102:0/2171678972 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f37ec0a2b90 msgr2=0x7f37ec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f37ec0a2b90 0x7f37ec0a5050 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x563c6bddc550 tx=0x7f37f8009500 comp rx=0 tx=0).stop 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 -- 172.21.15.102:0/2171678972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3808104b90 msgr2=0x7f3808197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3808104b90 0x7f3808197fe0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f37f400e9c0 tx=0x7f37f400ee80 comp rx=0 tx=0).stop 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 -- 172.21.15.102:0/2171678972 shutdown_connections 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f37ec0a2b90 0x7f37ec0a5050 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38081074c0 0x7f380819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3808104b90 0x7f3808197fe0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 --2- 172.21.15.102:0/2171678972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3808102260 0x7f3808197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 -- 172.21.15.102:0/2171678972 >> 172.21.15.102:0/2171678972 conn(0x7f38080fbbf0 msgr2=0x7f3808102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 -- 172.21.15.102:0/2171678972 shutdown_connections 2024-04-25T01:54:31.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.323+0000 7f3810000640 1 -- 172.21.15.102:0/2171678972 wait complete. 2024-04-25T01:54:31.341 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:54:31.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.526+0000 7f5b66c00640 1 -- 172.21.15.102:0/4005194921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b60104b90 msgr2=0x7f5b60105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.526+0000 7f5b66c00640 1 --2- 172.21.15.102:0/4005194921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b60104b90 0x7f5b60105010 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009860 tx=0x7f5b4802a340 comp rx=0 tx=0).stop 2024-04-25T01:54:31.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.526+0000 7f5b66c00640 1 -- 172.21.15.102:0/4005194921 shutdown_connections 2024-04-25T01:54:31.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.526+0000 7f5b66c00640 1 --2- 172.21.15.102:0/4005194921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b60105550 0x7f5b6010bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.526+0000 7f5b66c00640 1 --2- 172.21.15.102:0/4005194921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b60104b90 0x7f5b60105010 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.526+0000 7f5b66c00640 1 --2- 172.21.15.102:0/4005194921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b601039a0 0x7f5b60103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.526+0000 7f5b66c00640 1 -- 172.21.15.102:0/4005194921 >> 172.21.15.102:0/4005194921 conn(0x7f5b600fef30 msgr2=0x7f5b60101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.527+0000 7f5b66c00640 1 -- 172.21.15.102:0/4005194921 shutdown_connections 2024-04-25T01:54:31.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.527+0000 7f5b66c00640 1 -- 172.21.15.102:0/4005194921 wait complete. 2024-04-25T01:54:31.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.527+0000 7f5b66c00640 1 Processor -- start 2024-04-25T01:54:31.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b66c00640 1 -- start start 2024-04-25T01:54:31.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b66c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b601039a0 0x7f5b60199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b66c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b60104b90 0x7f5b6019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b66c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b60105550 0x7f5b601a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b66c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6010f620 con 0x7f5b60104b90 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b66c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6010f4a0 con 0x7f5b60105550 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b66c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6010f7a0 con 0x7f5b601039a0 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b5fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b601039a0 0x7f5b60199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b57400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b60104b90 0x7f5b6019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b5fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b601039a0 0x7f5b60199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53266/0 (socket says 172.21.15.102:53266) 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b5fe00640 1 -- 172.21.15.102:0/1532646986 learned_addr learned my addr 172.21.15.102:0/1532646986 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.528+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b60105550 0x7f5b601a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b5fe00640 1 -- 172.21.15.102:0/1532646986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b60105550 msgr2=0x7f5b601a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b5fe00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b60105550 0x7f5b601a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b5fe00640 1 -- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b60104b90 msgr2=0x7f5b6019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b5fe00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b60104b90 0x7f5b6019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b5fe00640 1 -- 172.21.15.102:0/1532646986 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b601a1900 con 0x7f5b601039a0 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b60105550 0x7f5b601a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b57400640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b60104b90 0x7f5b6019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b5fe00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b601039a0 0x7f5b60199cf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000c3f0 tx=0x7f5b50009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b5d600640 1 -- 172.21.15.102:0/1532646986 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5001b070 con 0x7f5b601039a0 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b5d600640 1 -- 172.21.15.102:0/1532646986 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b5000bb10 con 0x7f5b601039a0 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b66c00640 1 -- 172.21.15.102:0/1532646986 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b601a1bf0 con 0x7f5b601039a0 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b5d600640 1 -- 172.21.15.102:0/1532646986 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50016d80 con 0x7f5b601039a0 2024-04-25T01:54:31.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.529+0000 7f5b66c00640 1 -- 172.21.15.102:0/1532646986 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b601a2140 con 0x7f5b601039a0 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.532+0000 7f5b69a5e740 1 Processor -- start 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.532+0000 7f5b69a5e740 1 -- start start 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.532+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.532+0000 7f5b5d600640 1 -- 172.21.15.102:0/1532646986 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f5b5001e070 con 0x7f5b601039a0 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.532+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06ba150 0x5571c06bec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06bf2c0 0x5571c06c1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5571c05b9ed0 con 0x5571c06bf2c0 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5571c061cdf0 con 0x5571c06ba150 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5571c05c58d0 con 0x5571c06b9760 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b64c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b57400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06bf2c0 0x5571c06c1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b64c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53270/0 (socket says 172.21.15.102:53270) 2024-04-25T01:54:31.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b64c00640 1 -- 172.21.15.102:0/3902152865 learned_addr learned my addr 172.21.15.102:0/3902152865 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b5fe00640 1 --2- 172.21.15.102:0/3902152865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06ba150 0x5571c06bec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b64c00640 1 -- 172.21.15.102:0/3902152865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06ba150 msgr2=0x5571c06bec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b64c00640 1 --2- 172.21.15.102:0/3902152865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06ba150 0x5571c06bec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b64c00640 1 -- 172.21.15.102:0/3902152865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06bf2c0 msgr2=0x5571c06c1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b64c00640 1 --2- 172.21.15.102:0/3902152865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06bf2c0 0x5571c06c1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.533+0000 7f5b64c00640 1 -- 172.21.15.102:0/3902152865 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571c03312d0 con 0x5571c06b9760 2024-04-25T01:54:31.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.534+0000 7f5b64c00640 1 --2- 172.21.15.102:0/3902152865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b9b80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009cc0 tx=0x7f5b580117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbb2ba64b70e5deb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.534+0000 7f5b55200640 1 -- 172.21.15.102:0/3902152865 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58012450 con 0x5571c06b9760 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.534+0000 7f5b55200640 1 -- 172.21.15.102:0/3902152865 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b58012ad0 con 0x5571c06b9760 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.534+0000 7f5b55200640 1 -- 172.21.15.102:0/3902152865 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5800fc90 con 0x5571c06b9760 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.534+0000 7f5b5d600640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5b440a2b90 0x7f5b440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.534+0000 7f5b5fe00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5b440a2b90 0x7f5b440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.534+0000 7f5b5d600640 1 -- 172.21.15.102:0/1532646986 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f5b500cf070 con 0x7f5b601039a0 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 -- 172.21.15.102:0/3902152865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 msgr2=0x5571c06b9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/3902152865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b9b80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009cc0 tx=0x7f5b580117b0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 -- 172.21.15.102:0/3902152865 shutdown_connections 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/3902152865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06bf2c0 0x5571c06c1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/3902152865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06ba150 0x5571c06bec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/3902152865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b9b80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 -- 172.21.15.102:0/3902152865 >> 172.21.15.102:0/3902152865 conn(0x5571c06b4dd0 msgr2=0x5571c06b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 -- 172.21.15.102:0/3902152865 shutdown_connections 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 -- 172.21.15.102:0/3902152865 wait complete. 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b5fe00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5b440a2b90 0x7f5b440a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000cfd0 tx=0x7f5b500d2ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.535+0000 7f5b69a5e740 1 -- start start 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b69a5e740 1 Processor -- start 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06b9760 0x5571c06c39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06ba150 0x5571c06c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5571c05d8d20 con 0x5571c06b9760 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5571c0618db0 con 0x5571c06bf2c0 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5571c06c2270 con 0x5571c06ba150 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b69a5e740 10 client.0 _ll_register_callbacks cb 0x5571c0613f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:31.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b64c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06b9760 0x5571c06c39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b64c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06b9760 0x5571c06c39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55960/0 (socket says 172.21.15.102:55960) 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b64c00640 1 -- 172.21.15.102:0/1172055838 learned_addr learned my addr 172.21.15.102:0/1172055838 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b57400640 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06ba150 0x5571c06c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.536+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b64c00640 1 -- 172.21.15.102:0/1172055838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06ba150 msgr2=0x5571c06c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06ba150 0x5571c06c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b64c00640 1 -- 172.21.15.102:0/1172055838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 msgr2=0x5571c06caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b64c00640 1 -- 172.21.15.102:0/1172055838 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571c03fb6b0 con 0x5571c06b9760 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06b9760 0x5571c06c39d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009840 tx=0x7f5b5800f4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ee3200e271538a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b4d600640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b4d600640 1 -- 172.21.15.102:0/1172055838 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b580128e0 con 0x5571c06b9760 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b4d600640 1 -- 172.21.15.102:0/1172055838 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f5b58005040 con 0x5571c06b9760 2024-04-25T01:54:31.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1172055838 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b58012ad0 con 0x5571c06b9760 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b4d600640 1 -- 172.21.15.102:0/1172055838 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5b58005240 con 0x5571c06b9760 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b69a5e740 10 client.14907 fetch_fsmap learned FSMap version 22 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.537+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1172055838 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5571c06ce2c0 con 0x5571c06b9760 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.538+0000 7f5b4d600640 1 -- 172.21.15.102:0/1172055838 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b58012ad0 con 0x5571c06b9760 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.538+0000 7f5b4d600640 1 -- 172.21.15.102:0/1172055838 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f5b58034480 con 0x5571c06b9760 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.538+0000 7f5b69a5e740 10 client.14907 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.538+0000 7f5b69a5e740 10 client.14907 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.538+0000 7f5b69a5e740 20 client.14907 populate_metadata read hostname 'smithi102' 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.538+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5571c06d0530 0x5571c06d19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.538+0000 7f5b69a5e740 4 client.14907 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.538+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1172055838 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571c062a330 con 0x5571c06d0530 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.539+0000 7f5b57400640 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5571c06d0530 0x5571c06d19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.539+0000 7f5b57400640 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5571c06d0530 0x5571c06d19a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edf59725f559e935 server_cookie=6338fe7758b2216b in_seq=0 out_seq=0 2024-04-25T01:54:31.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.539+0000 7f5b4d600640 10 client.14907 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b4d600640 1 -- 172.21.15.102:0/1172055838 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571c062a330 con 0x5571c06d0530 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b4d600640 10 client.14907 handle_command_reply: tid=0 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b69a5e740 1 client.14907 shutdown 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1172055838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06b9760 msgr2=0x5571c06c39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06b9760 0x5571c06c39d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009840 tx=0x7f5b5800f4e0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1172055838 shutdown_connections 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5571c06d0530 0x5571c06d19a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06ba150 0x5571c06c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1172055838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06b9760 0x5571c06c39d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b4d600640 0 client.14907 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:54:31.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.540+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1172055838 >> 172.21.15.102:0/1172055838 conn(0x5571c06b4dd0 msgr2=0x5571c06b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.541+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1172055838 shutdown_connections 2024-04-25T01:54:31.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.541+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1172055838 wait complete. 2024-04-25T01:54:31.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.541+0000 7f5b69a5e740 20 client.14907 trim_cache size 0 max 16384 2024-04-25T01:54:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.560+0000 7f5b69a5e740 1 Processor -- start 2024-04-25T01:54:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.560+0000 7f5b69a5e740 1 -- start start 2024-04-25T01:54:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.560+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.560+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06caf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.560+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.560+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5571c06cfce0 con 0x5571c06ba150 2024-04-25T01:54:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.560+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5571c06d03b0 con 0x5571c06bf2c0 2024-04-25T01:54:31.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.560+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5571c06d0a80 con 0x5571c06b9760 2024-04-25T01:54:31.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53286/0 (socket says 172.21.15.102:53286) 2024-04-25T01:54:31.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b57400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 -- 172.21.15.102:0/1269912364 learned_addr learned my addr 172.21.15.102:0/1269912364 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1269912364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 -- 172.21.15.102:0/1269912364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 msgr2=0x5571c06c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1269912364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06c35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 -- 172.21.15.102:0/1269912364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 msgr2=0x5571c06caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1269912364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06caf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 -- 172.21.15.102:0/1269912364 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571c06be7a0 con 0x5571c06b9760 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b64c00640 1 --2- 172.21.15.102:0/1269912364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b51b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009be0 tx=0x7f5b580117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28099ac07cf7e983 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b4fe00640 1 -- 172.21.15.102:0/1269912364 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58034480 con 0x5571c06b9760 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b4fe00640 1 -- 172.21.15.102:0/1269912364 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b5802d070 con 0x5571c06b9760 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.561+0000 7f5b4fe00640 1 -- 172.21.15.102:0/1269912364 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b580126e0 con 0x5571c06b9760 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.562+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1269912364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 msgr2=0x5571c06b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.562+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1269912364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b51b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009be0 tx=0x7f5b580117b0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.562+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1269912364 shutdown_connections 2024-04-25T01:54:31.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.562+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1269912364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.562+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1269912364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.562+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/1269912364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06b51b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.562+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1269912364 >> 172.21.15.102:0/1269912364 conn(0x5571c06b4dd0 msgr2=0x5571c06afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.562+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1269912364 shutdown_connections 2024-04-25T01:54:31.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.562+0000 7f5b69a5e740 1 -- 172.21.15.102:0/1269912364 wait complete. 2024-04-25T01:54:31.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b69a5e740 1 -- start start 2024-04-25T01:54:31.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b69a5e740 1 Processor -- start 2024-04-25T01:54:31.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06c7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b69a5e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06ca020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5571c06d1820 con 0x5571c06ba150 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5571c06d1150 con 0x5571c06bf2c0 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b69a5e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5571c06d1ef0 con 0x5571c06b9760 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b69a5e740 10 client.0 _ll_register_callbacks cb 0x7f5b50009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b57400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.563+0000 7f5b64c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b57400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55990/0 (socket says 172.21.15.102:55990) 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b57400640 1 -- 172.21.15.102:0/2537573417 learned_addr learned my addr 172.21.15.102:0/2537573417 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b64c00640 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b57400640 1 -- 172.21.15.102:0/2537573417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 msgr2=0x5571c06c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b57400640 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b57400640 1 -- 172.21.15.102:0/2537573417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 msgr2=0x5571c06ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b57400640 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b57400640 1 -- 172.21.15.102:0/2537573417 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571c06ca610 con 0x5571c06ba150 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b64c00640 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b57400640 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06c7000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5b48000c00 tx=0x7f5b48005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48129b955aff8898 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b4d600640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b4d600640 1 -- 172.21.15.102:0/2537573417 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b48037040 con 0x5571c06ba150 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b4d600640 1 -- 172.21.15.102:0/2537573417 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f5b48004180 con 0x5571c06ba150 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.564+0000 7f5b69a5e740 1 -- 172.21.15.102:0/2537573417 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b5802d070 con 0x5571c06ba150 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b4d600640 1 -- 172.21.15.102:0/2537573417 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5b48004380 con 0x5571c06ba150 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b69a5e740 10 client.14913 fetch_fsmap learned FSMap version 22 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b69a5e740 1 -- 172.21.15.102:0/2537573417 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5571c073f230 con 0x5571c06ba150 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b4d600640 1 -- 172.21.15.102:0/2537573417 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b5802d070 con 0x5571c06ba150 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b4d600640 1 -- 172.21.15.102:0/2537573417 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f5b480035f0 con 0x5571c06ba150 2024-04-25T01:54:31.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b69a5e740 10 client.14913 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b69a5e740 10 client.14913 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b69a5e740 20 client.14913 populate_metadata read hostname 'smithi102' 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5571c0743e30 0x5571c0742b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b69a5e740 4 client.14913 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.565+0000 7f5b69a5e740 1 -- 172.21.15.102:0/2537573417 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "status"}) -- 0x5571c0624680 con 0x5571c0743e30 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.566+0000 7f5b64c00640 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5571c0743e30 0x5571c0742b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.566+0000 7f5b64c00640 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5571c0743e30 0x5571c0742b40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4896ece636db7d8a server_cookie=ea1cf74a7eee5d50 in_seq=0 out_seq=0 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.566+0000 7f5b4d600640 10 client.14913 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.566+0000 7f5b4d600640 1 -- 172.21.15.102:0/2537573417 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5571c0624680 con 0x5571c0743e30 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.566+0000 7f5b4d600640 10 client.14913 handle_command_reply: tid=0 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.566+0000 7f5b69a5e740 1 client.14913 shutdown 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.566+0000 7f5b69a5e740 1 -- 172.21.15.102:0/2537573417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 msgr2=0x5571c06c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.566+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06c7000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5b48000c00 tx=0x7f5b48005f90 comp rx=0 tx=0).stop 2024-04-25T01:54:31.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b69a5e740 1 -- 172.21.15.102:0/2537573417 shutdown_connections 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5571c0743e30 0x5571c0742b40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571c06bf2c0 0x5571c06ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571c06ba150 0x5571c06c7000 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b69a5e740 1 --2- 172.21.15.102:0/2537573417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571c06b9760 0x5571c06c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b4d600640 0 client.14913 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b69a5e740 1 -- 172.21.15.102:0/2537573417 >> 172.21.15.102:0/2537573417 conn(0x5571c06b4dd0 msgr2=0x5571c06b51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b69a5e740 1 -- 172.21.15.102:0/2537573417 shutdown_connections 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b69a5e740 1 -- 172.21.15.102:0/2537573417 wait complete. 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b69a5e740 20 client.14913 trim_cache size 0 max 16384 2024-04-25T01:54:31.573 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 1, 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24541, 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 104.51609876800001, 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 22, 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 89, 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 125.647637374 2024-04-25T01:54:31.574 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:54:31.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b66c00640 1 -- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5b440a2b90 msgr2=0x7f5b440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b66c00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5b440a2b90 0x7f5b440a5050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000cfd0 tx=0x7f5b500d2ab0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b66c00640 1 -- 172.21.15.102:0/1532646986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b601039a0 msgr2=0x7f5b60199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.567+0000 7f5b66c00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b601039a0 0x7f5b60199cf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000c3f0 tx=0x7f5b50009f90 comp rx=0 tx=0).stop 2024-04-25T01:54:31.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.568+0000 7f5b66c00640 1 -- 172.21.15.102:0/1532646986 shutdown_connections 2024-04-25T01:54:31.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.568+0000 7f5b66c00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5b440a2b90 0x7f5b440a5050 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.568+0000 7f5b66c00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b60105550 0x7f5b601a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.568+0000 7f5b66c00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b60104b90 0x7f5b6019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.568+0000 7f5b66c00640 1 --2- 172.21.15.102:0/1532646986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b601039a0 0x7f5b60199cf0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.568+0000 7f5b66c00640 1 -- 172.21.15.102:0/1532646986 >> 172.21.15.102:0/1532646986 conn(0x7f5b600fef30 msgr2=0x7f5b60107f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.568+0000 7f5b66c00640 1 -- 172.21.15.102:0/1532646986 shutdown_connections 2024-04-25T01:54:31.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.568+0000 7f5b66c00640 1 -- 172.21.15.102:0/1532646986 wait complete. 2024-04-25T01:54:31.587 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:54:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.784+0000 7fb1e2c00640 1 -- 172.21.15.102:0/352710588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc105550 msgr2=0x7fb1dc10bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.784+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/352710588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc105550 0x7fb1dc10bde0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009860 tx=0x7fb1d402a340 comp rx=0 tx=0).stop 2024-04-25T01:54:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.784+0000 7fb1e2c00640 1 -- 172.21.15.102:0/352710588 shutdown_connections 2024-04-25T01:54:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.784+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/352710588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc105550 0x7fb1dc10bde0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.784+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/352710588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1dc104b90 0x7fb1dc105010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.784+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/352710588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1dc1039a0 0x7fb1dc103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.784+0000 7fb1e2c00640 1 -- 172.21.15.102:0/352710588 >> 172.21.15.102:0/352710588 conn(0x7fb1dc0fef30 msgr2=0x7fb1dc101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.784+0000 7fb1e2c00640 1 -- 172.21.15.102:0/352710588 shutdown_connections 2024-04-25T01:54:31.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.784+0000 7fb1e2c00640 1 -- 172.21.15.102:0/352710588 wait complete. 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.785+0000 7fb1e2c00640 1 Processor -- start 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.785+0000 7fb1e2c00640 1 -- start start 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1e2c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1dc1039a0 0x7fb1dc199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1e2c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc104b90 0x7fb1dc19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1e2c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1dc105550 0x7fb1dc1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1e2c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc10f620 con 0x7fb1dc105550 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1e2c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc10f4a0 con 0x7fb1dc1039a0 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1e2c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc10f7a0 con 0x7fb1dc104b90 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1d3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc104b90 0x7fb1dc19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1e0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1dc105550 0x7fb1dc1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1dbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1dc1039a0 0x7fb1dc199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1d3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc104b90 0x7fb1dc19a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53314/0 (socket says 172.21.15.102:53314) 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1d3400640 1 -- 172.21.15.102:0/90225609 learned_addr learned my addr 172.21.15.102:0/90225609 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1d3400640 1 -- 172.21.15.102:0/90225609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1dc1039a0 msgr2=0x7fb1dc199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1d3400640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1dc1039a0 0x7fb1dc199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1d3400640 1 -- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1dc105550 msgr2=0x7fb1dc1a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1d3400640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1dc105550 0x7fb1dc1a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1d3400640 1 -- 172.21.15.102:0/90225609 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc1a18b0 con 0x7fb1dc104b90 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.786+0000 7fb1e0c00640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1dc105550 0x7fb1dc1a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.787+0000 7fb1d3400640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc104b90 0x7fb1dc19a1b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400eb90 tx=0x7fb1c400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.787+0000 7fb1d9600640 1 -- 172.21.15.102:0/90225609 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4004d20 con 0x7fb1dc104b90 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.787+0000 7fb1d9600640 1 -- 172.21.15.102:0/90225609 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c400ee90 con 0x7fb1dc104b90 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.787+0000 7fb1d9600640 1 -- 172.21.15.102:0/90225609 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4010830 con 0x7fb1dc104b90 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.787+0000 7fb1e2c00640 1 -- 172.21.15.102:0/90225609 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1dc1a1ba0 con 0x7fb1dc104b90 2024-04-25T01:54:31.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.787+0000 7fb1e2c00640 1 -- 172.21.15.102:0/90225609 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1dc1a20a0 con 0x7fb1dc104b90 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.790+0000 7fb1e59eb740 1 Processor -- start 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.790+0000 7fb1d9600640 1 -- 172.21.15.102:0/90225609 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb1c4005020 con 0x7fb1dc104b90 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1e59eb740 1 -- start start 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 0x555830a77b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a7cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 0x555830a7f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555830977ed0 con 0x555830a77760 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5558309dadf0 con 0x555830a7d2c0 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5558309838d0 con 0x555830a78150 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1e0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 0x555830a77b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 0x555830a7f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a7cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a7cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53328/0 (socket says 172.21.15.102:53328) 2024-04-25T01:54:31.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1dbe00640 1 -- 172.21.15.102:0/2895821563 learned_addr learned my addr 172.21.15.102:0/2895821563 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1dbe00640 1 -- 172.21.15.102:0/2895821563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 msgr2=0x555830a7f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1dbe00640 1 --2- 172.21.15.102:0/2895821563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 0x555830a7f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1dbe00640 1 -- 172.21.15.102:0/2895821563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 msgr2=0x555830a77b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.791+0000 7fb1dbe00640 1 --2- 172.21.15.102:0/2895821563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 0x555830a77b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.792+0000 7fb1dbe00640 1 -- 172.21.15.102:0/2895821563 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558306ef2d0 con 0x555830a78150 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.792+0000 7fb1dbe00640 1 --2- 172.21.15.102:0/2895821563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a7cc50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009c60 tx=0x7fb1cc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78a7bef43736142f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.792+0000 7fb1d1200640 1 -- 172.21.15.102:0/2895821563 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc005220 con 0x555830a78150 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.792+0000 7fb1d1200640 1 -- 172.21.15.102:0/2895821563 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1cc009e10 con 0x555830a78150 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.792+0000 7fb1d1200640 1 -- 172.21.15.102:0/2895821563 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc019790 con 0x555830a78150 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.792+0000 7fb1d9600640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb1c00a2c10 0x7fb1c00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1d9600640 1 -- 172.21.15.102:0/90225609 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fb1c40cf670 con 0x7fb1dc104b90 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e0c00640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb1c00a2c10 0x7fb1c00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e59eb740 1 -- 172.21.15.102:0/2895821563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 msgr2=0x555830a7cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/2895821563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a7cc50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009c60 tx=0x7fb1cc011810 comp rx=0 tx=0).stop 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e59eb740 1 -- 172.21.15.102:0/2895821563 shutdown_connections 2024-04-25T01:54:31.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/2895821563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 0x555830a7f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/2895821563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a7cc50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/2895821563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 0x555830a77b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e59eb740 1 -- 172.21.15.102:0/2895821563 >> 172.21.15.102:0/2895821563 conn(0x555830a72dd0 msgr2=0x555830a731d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e59eb740 1 -- 172.21.15.102:0/2895821563 shutdown_connections 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e59eb740 1 -- 172.21.15.102:0/2895821563 wait complete. 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.793+0000 7fb1e0c00640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb1c00a2c10 0x7fb1c00a50d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1d40305b0 tx=0x7fb1d40058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.794+0000 7fb1e59eb740 1 -- start start 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.794+0000 7fb1e59eb740 1 Processor -- start 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.794+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 0x555830a81a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.794+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a81f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.794+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 0x555830a88f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.794+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555830996d20 con 0x555830a77760 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.794+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5558309d6db0 con 0x555830a7d2c0 2024-04-25T01:54:31.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.794+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555830a80270 con 0x555830a78150 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.794+0000 7fb1e59eb740 10 client.0 _ll_register_callbacks cb 0x5558309d1f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1d3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a81f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1dbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 0x555830a81a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1d3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a81f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53334/0 (socket says 172.21.15.102:53334) 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1d3400640 1 -- 172.21.15.102:0/3508610656 learned_addr learned my addr 172.21.15.102:0/3508610656 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1e0c00640 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 0x555830a88f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1d3400640 1 -- 172.21.15.102:0/3508610656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 msgr2=0x555830a88f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1d3400640 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 0x555830a88f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1d3400640 1 -- 172.21.15.102:0/3508610656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 msgr2=0x555830a81a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1d3400640 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 0x555830a81a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1d3400640 1 -- 172.21.15.102:0/3508610656 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558307b96b0 con 0x555830a78150 2024-04-25T01:54:31.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1dbe00640 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 0x555830a81a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1d3400640 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a81f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40a45c0 tx=0x7fb1c40099b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42b9cf47dec8cc35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1c9600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:54:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.795+0000 7fb1c9600640 1 -- 172.21.15.102:0/3508610656 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c409b9c0 con 0x555830a78150 2024-04-25T01:54:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1c9600640 1 -- 172.21.15.102:0/3508610656 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fb1c4005590 con 0x555830a78150 2024-04-25T01:54:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1e59eb740 1 -- 172.21.15.102:0/3508610656 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1cc009e10 con 0x555830a78150 2024-04-25T01:54:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1c9600640 1 -- 172.21.15.102:0/3508610656 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb1c409b7e0 con 0x555830a78150 2024-04-25T01:54:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1e59eb740 10 client.25018 fetch_fsmap learned FSMap version 22 2024-04-25T01:54:31.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1e59eb740 1 -- 172.21.15.102:0/3508610656 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555830a8c380 con 0x555830a78150 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1c9600640 1 -- 172.21.15.102:0/3508610656 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1cc009e10 con 0x555830a78150 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1c9600640 1 -- 172.21.15.102:0/3508610656 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fb1c40a46f0 con 0x555830a78150 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1e59eb740 10 client.25018 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1e59eb740 10 client.25018 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1e59eb740 20 client.25018 populate_metadata read hostname 'smithi102' 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555830a8e960 0x555830a8fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1e59eb740 4 client.25018 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.796+0000 7fb1e59eb740 1 -- 172.21.15.102:0/3508610656 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558309e8330 con 0x555830a8e960 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.797+0000 7fb1dbe00640 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555830a8e960 0x555830a8fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.797+0000 7fb1dbe00640 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555830a8e960 0x555830a8fdd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb01c1e1f4bb7e57 server_cookie=36049a4678476696 in_seq=0 out_seq=0 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.797+0000 7fb1c9600640 10 client.25018 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.798+0000 7fb1c9600640 1 -- 172.21.15.102:0/3508610656 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5558309e8330 con 0x555830a8e960 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.798+0000 7fb1c9600640 10 client.25018 handle_command_reply: tid=0 2024-04-25T01:54:31.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 client.25018 shutdown 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 -- 172.21.15.102:0/3508610656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 msgr2=0x555830a81f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a81f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40a45c0 tx=0x7fb1c40099b0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 -- 172.21.15.102:0/3508610656 shutdown_connections 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555830a8e960 0x555830a8fdd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a7d2c0 0x555830a88f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a81f70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/3508610656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a77760 0x555830a81a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 -- 172.21.15.102:0/3508610656 >> 172.21.15.102:0/3508610656 conn(0x555830a72dd0 msgr2=0x555830a71f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1c9600640 0 client.25018 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 -- 172.21.15.102:0/3508610656 shutdown_connections 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 1 -- 172.21.15.102:0/3508610656 wait complete. 2024-04-25T01:54:31.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.799+0000 7fb1e59eb740 20 client.25018 trim_cache size 0 max 16384 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.819+0000 7fb1e59eb740 1 Processor -- start 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.819+0000 7fb1e59eb740 1 -- start start 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.819+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 0x555830a81640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.819+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a88f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.819+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a81b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.819+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555830a8e110 con 0x555830a7d2c0 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.819+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555830a8e7e0 con 0x555830a77760 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.819+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555830a8eeb0 con 0x555830a78150 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1e0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a81b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1dbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 0x555830a81640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1e0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a81b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56024/0 (socket says 172.21.15.102:56024) 2024-04-25T01:54:31.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1e0c00640 1 -- 172.21.15.102:0/1918298757 learned_addr learned my addr 172.21.15.102:0/1918298757 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1d3400640 1 --2- 172.21.15.102:0/1918298757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a88f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1e0c00640 1 -- 172.21.15.102:0/1918298757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 msgr2=0x555830a88f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1e0c00640 1 --2- 172.21.15.102:0/1918298757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a88f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1e0c00640 1 -- 172.21.15.102:0/1918298757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 msgr2=0x555830a81640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1e0c00640 1 --2- 172.21.15.102:0/1918298757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 0x555830a81640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1e0c00640 1 -- 172.21.15.102:0/1918298757 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555830a71410 con 0x555830a7d2c0 2024-04-25T01:54:31.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.820+0000 7fb1e0c00640 1 --2- 172.21.15.102:0/1918298757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a81b80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009830 tx=0x7fb1d403d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44b55a20a9c6323 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1cbe00640 1 -- 172.21.15.102:0/1918298757 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d404e070 con 0x555830a7d2c0 2024-04-25T01:54:31.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1cbe00640 1 -- 172.21.15.102:0/1918298757 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d4003cd0 con 0x555830a7d2c0 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1cbe00640 1 -- 172.21.15.102:0/1918298757 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4037040 con 0x555830a7d2c0 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1918298757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 msgr2=0x555830a81b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1918298757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a81b80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009830 tx=0x7fb1d403d000 comp rx=0 tx=0).stop 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1918298757 shutdown_connections 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1918298757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a81b80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1918298757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a88f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1918298757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 0x555830a81640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1918298757 >> 172.21.15.102:0/1918298757 conn(0x555830a72dd0 msgr2=0x555830a6db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.821+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1918298757 shutdown_connections 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.822+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1918298757 wait complete. 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.822+0000 7fb1e59eb740 1 -- start start 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.822+0000 7fb1e59eb740 1 Processor -- start 2024-04-25T01:54:31.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.822+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 0x555830a84e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.822+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a85340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.822+0000 7fb1e59eb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a88310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.822+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555830a8fc50 con 0x555830a7d2c0 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.822+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555830a8f580 con 0x555830a77760 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.822+0000 7fb1e59eb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555830a90320 con 0x555830a78150 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1e59eb740 10 client.0 _ll_register_callbacks cb 0x7fb1c4004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1d3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a85340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1d3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a85340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:53356/0 (socket says 172.21.15.102:53356) 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1d3400640 1 -- 172.21.15.102:0/1431634875 learned_addr learned my addr 172.21.15.102:0/1431634875 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1e0c00640 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a88310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1dbe00640 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 0x555830a84e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1d3400640 1 -- 172.21.15.102:0/1431634875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 msgr2=0x555830a84e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1d3400640 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 0x555830a84e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1d3400640 1 -- 172.21.15.102:0/1431634875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 msgr2=0x555830a88310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1d3400640 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a88310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1d3400640 1 -- 172.21.15.102:0/1431634875 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555830a88900 con 0x555830a78150 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1d3400640 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a85340 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4004af0 tx=0x7fb1c409b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34c7db63870a0eb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1c9600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.823+0000 7fb1c9600640 1 -- 172.21.15.102:0/1431634875 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c40a4af0 con 0x555830a78150 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1c9600640 1 -- 172.21.15.102:0/1431634875 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fb1c40cd1a0 con 0x555830a78150 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1431634875 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d4003cd0 con 0x555830a78150 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1c9600640 1 -- 172.21.15.102:0/1431634875 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb1c409c580 con 0x555830a78150 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1e59eb740 10 client.25024 fetch_fsmap learned FSMap version 22 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1431634875 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555830afd910 con 0x555830a78150 2024-04-25T01:54:31.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1c9600640 1 -- 172.21.15.102:0/1431634875 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d4003cd0 con 0x555830a78150 2024-04-25T01:54:31.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1c9600640 1 -- 172.21.15.102:0/1431634875 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fb1c4097bd0 con 0x555830a78150 2024-04-25T01:54:31.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1e59eb740 10 client.25024 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:54:31.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1e59eb740 10 client.25024 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:54:31.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1e59eb740 20 client.25024 populate_metadata read hostname 'smithi102' 2024-04-25T01:54:31.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555830b02350 0x555830b01060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:31.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1e59eb740 4 client.25024 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:54:31.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.824+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1431634875 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "status"}) -- 0x5558309e2680 con 0x555830b02350 2024-04-25T01:54:31.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.825+0000 7fb1dbe00640 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555830b02350 0x555830b01060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:31.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.825+0000 7fb1dbe00640 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555830b02350 0x555830b01060 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=546971d95bbe928e server_cookie=a37c47d2dab02f3b in_seq=0 out_seq=0 2024-04-25T01:54:31.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.825+0000 7fb1c9600640 10 client.25024 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:54:31.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1c9600640 1 -- 172.21.15.102:0/1431634875 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5558309e2680 con 0x555830b02350 2024-04-25T01:54:31.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1c9600640 10 client.25024 handle_command_reply: tid=0 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 client.25024 shutdown 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1431634875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 msgr2=0x555830a85340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a85340 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4004af0 tx=0x7fb1c409b9c0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1431634875 shutdown_connections 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555830b02350 0x555830b01060 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555830a7d2c0 0x555830a88310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1c9600640 0 client.25024 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555830a78150 0x555830a85340 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 --2- 172.21.15.102:0/1431634875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555830a77760 0x555830a84e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1431634875 >> 172.21.15.102:0/1431634875 conn(0x555830a72dd0 msgr2=0x555830a81640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1431634875 shutdown_connections 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 1 -- 172.21.15.102:0/1431634875 wait complete. 2024-04-25T01:54:31.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.826+0000 7fb1e59eb740 20 client.25024 trim_cache size 0 max 16384 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24587, 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 107.730482264, 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 22, 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 89, 2024-04-25T01:54:31.833 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:54:31.834 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 115.04251824799999 2024-04-25T01:54:31.834 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:54:31.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 -- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb1c00a2c10 msgr2=0x7fb1c00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb1c00a2c10 0x7fb1c00a50d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb1d40305b0 tx=0x7fb1d40058b0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 -- 172.21.15.102:0/90225609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc104b90 msgr2=0x7fb1dc19a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:31.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc104b90 0x7fb1dc19a1b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400eb90 tx=0x7fb1c400c6a0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 -- 172.21.15.102:0/90225609 shutdown_connections 2024-04-25T01:54:31.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fb1c00a2c10 0x7fb1c00a50d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1dc105550 0x7fb1dc1a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc104b90 0x7fb1dc19a1b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 --2- 172.21.15.102:0/90225609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1dc1039a0 0x7fb1dc199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:31.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 -- 172.21.15.102:0/90225609 >> 172.21.15.102:0/90225609 conn(0x7fb1dc0fef30 msgr2=0x7fb1dc107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:31.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 -- 172.21.15.102:0/90225609 shutdown_connections 2024-04-25T01:54:31.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:31.827+0000 7fb1e2c00640 1 -- 172.21.15.102:0/90225609 wait complete. 2024-04-25T01:54:31.846 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T01:54:31.846 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:54:31.924 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T01:54:31.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T01:54:32.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:31 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/554649337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:32.237 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:31 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/554649337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:32.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:31 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/554649337' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:54:32.367 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:33.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:32 smithi102 ceph-mon[27796]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:33.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:32 smithi110 ceph-mon[32848]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:33.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:32 smithi193 ceph-mon[32981]: pgmap v374: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:34.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.786+0000 7fe983e00700 1 -- 172.21.15.102:0/2309292473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 msgr2=0x7fe9840746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.786+0000 7fe983e00700 1 --2- 172.21.15.102:0/2309292473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 0x7fe9840746c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe974009990 tx=0x7fe974009ca0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.786+0000 7fe983e00700 1 -- 172.21.15.102:0/2309292473 shutdown_connections 2024-04-25T01:54:34.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.786+0000 7fe983e00700 1 --2- 172.21.15.102:0/2309292473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 0x7fe9840746c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.786+0000 7fe983e00700 1 --2- 172.21.15.102:0/2309292473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe98410a090 0x7fe98410c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.786+0000 7fe983e00700 1 --2- 172.21.15.102:0/2309292473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe984074c90 0x7fe984075110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.786+0000 7fe983e00700 1 -- 172.21.15.102:0/2309292473 >> 172.21.15.102:0/2309292473 conn(0x7fe9840fd760 msgr2=0x7fe9840ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:34.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.786+0000 7fe983e00700 1 -- 172.21.15.102:0/2309292473 shutdown_connections 2024-04-25T01:54:34.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.786+0000 7fe983e00700 1 -- 172.21.15.102:0/2309292473 wait complete. 2024-04-25T01:54:34.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.787+0000 7fe983e00700 1 Processor -- start 2024-04-25T01:54:34.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe983e00700 1 -- start start 2024-04-25T01:54:34.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe983e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe984074c90 0x7fe98419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe983e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 0x7fe98419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe983e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe98410a090 0x7fe9841a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe983e00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe98410ffd0 con 0x7fe98410a090 2024-04-25T01:54:34.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe983e00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe98410fe50 con 0x7fe984076260 2024-04-25T01:54:34.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe983e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe984110150 con 0x7fe984074c90 2024-04-25T01:54:34.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe983400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe98410a090 0x7fe9841a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe982000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 0x7fe98419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe982a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe984074c90 0x7fe98419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe982000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 0x7fe98419eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56222/0 (socket says 172.21.15.102:56222) 2024-04-25T01:54:34.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe982a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe984074c90 0x7fe98419e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:57500/0 (socket says 172.21.15.102:57500) 2024-04-25T01:54:34.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.788+0000 7fe982000700 1 -- 172.21.15.102:0/1652811299 learned_addr learned my addr 172.21.15.102:0/1652811299 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.789+0000 7fe982000700 1 -- 172.21.15.102:0/1652811299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe984074c90 msgr2=0x7fe98419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.789+0000 7fe982000700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe984074c90 0x7fe98419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.789+0000 7fe982000700 1 -- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe98410a090 msgr2=0x7fe9841a5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.789+0000 7fe982000700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe98410a090 0x7fe9841a5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.789+0000 7fe982000700 1 -- 172.21.15.102:0/1652811299 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9841a6610 con 0x7fe984076260 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.789+0000 7fe982a00700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe984074c90 0x7fe98419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.789+0000 7fe983400700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe98410a090 0x7fe9841a5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.789+0000 7fe982000700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 0x7fe98419eef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe97000e970 tx=0x7fe97000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.790+0000 7fe97b400700 1 -- 172.21.15.102:0/1652811299 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe970004220 con 0x7fe984076260 2024-04-25T01:54:34.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.790+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9841a6900 con 0x7fe984076260 2024-04-25T01:54:34.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.790+0000 7fe97b400700 1 -- 172.21.15.102:0/1652811299 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9700043c0 con 0x7fe984076260 2024-04-25T01:54:34.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.790+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9841a6e90 con 0x7fe984076260 2024-04-25T01:54:34.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.790+0000 7fe97b400700 1 -- 172.21.15.102:0/1652811299 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe970003ce0 con 0x7fe984076260 2024-04-25T01:54:34.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.791+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe984068f20 con 0x7fe984076260 2024-04-25T01:54:34.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.794+0000 7fe97b400700 1 -- 172.21.15.102:0/1652811299 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe97000c7e0 con 0x7fe984076260 2024-04-25T01:54:34.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.795+0000 7fe97b400700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe9680a2ca0 0x7fe9680a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:34.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.795+0000 7fe97b400700 1 -- 172.21.15.102:0/1652811299 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fe9700cf300 con 0x7fe984076260 2024-04-25T01:54:34.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.795+0000 7fe982a00700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe9680a2ca0 0x7fe9680a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:34.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.796+0000 7fe982a00700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe9680a2ca0 0x7fe9680a5160 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe974000c00 tx=0x7fe97400b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:34.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:34.796+0000 7fe97b400700 1 -- 172.21.15.102:0/1652811299 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe97009c2a0 con 0x7fe984076260 2024-04-25T01:54:35.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.036+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fe984062670 con 0x7fe9680a2ca0 2024-04-25T01:54:35.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.038+0000 7fe97b400700 1 -- 172.21.15.102:0/1652811299 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fe984062670 con 0x7fe9680a2ca0 2024-04-25T01:54:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:Backend: cephadm 2024-04-25T01:54:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:Available: Yes 2024-04-25T01:54:35.040 INFO:teuthology.orchestra.run.smithi102.stdout:Paused: No 2024-04-25T01:54:35.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe9680a2ca0 msgr2=0x7fe9680a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe9680a2ca0 0x7fe9680a5160 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe974000c00 tx=0x7fe97400b880 comp rx=0 tx=0).stop 2024-04-25T01:54:35.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 msgr2=0x7fe98419eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:35.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 0x7fe98419eef0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe97000e970 tx=0x7fe97000ec80 comp rx=0 tx=0).stop 2024-04-25T01:54:35.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 shutdown_connections 2024-04-25T01:54:35.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe9680a2ca0 0x7fe9680a5160 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe98410a090 0x7fe9841a5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe984074c90 0x7fe98419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 --2- 172.21.15.102:0/1652811299 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe984076260 0x7fe98419eef0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:35.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.042+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 >> 172.21.15.102:0/1652811299 conn(0x7fe9840fd760 msgr2=0x7fe984101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:35.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.043+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 shutdown_connections 2024-04-25T01:54:35.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:35.043+0000 7fe983e00700 1 -- 172.21.15.102:0/1652811299 wait complete. 2024-04-25T01:54:35.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T01:54:36.077 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:35 smithi102 ceph-mon[27796]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:36.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:35 smithi110 ceph-mon[32848]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:36.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:35 smithi193 ceph-mon[32981]: pgmap v375: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:36.352 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:37.139 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:36 smithi110 ceph-mon[32848]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:37.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:36 smithi102 ceph-mon[27796]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:37.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:36 smithi193 ceph-mon[32981]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:38.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:37 smithi102 ceph-mon[27796]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:38.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:37 smithi110 ceph-mon[32848]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:38.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:37 smithi193 ceph-mon[32981]: pgmap v376: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.634+0000 7f900ea00700 1 -- 172.21.15.102:0/275750348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9008102d30 msgr2=0x7f90081095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.634+0000 7f900ea00700 1 --2- 172.21.15.102:0/275750348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9008102d30 0x7f90081095e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f900000b0d0 tx=0x7f900000b3e0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.634+0000 7f900ea00700 1 -- 172.21.15.102:0/275750348 shutdown_connections 2024-04-25T01:54:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.634+0000 7f900ea00700 1 --2- 172.21.15.102:0/275750348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9008102d30 0x7f90081095e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.634+0000 7f900ea00700 1 --2- 172.21.15.102:0/275750348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9008101190 0x7f90081015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.634+0000 7f900ea00700 1 --2- 172.21.15.102:0/275750348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9008102390 0x7f90081027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.634+0000 7f900ea00700 1 -- 172.21.15.102:0/275750348 >> 172.21.15.102:0/275750348 conn(0x7f90080fc730 msgr2=0x7f90080feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.634+0000 7f900ea00700 1 -- 172.21.15.102:0/275750348 shutdown_connections 2024-04-25T01:54:38.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.634+0000 7f900ea00700 1 -- 172.21.15.102:0/275750348 wait complete. 2024-04-25T01:54:38.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.635+0000 7f900ea00700 1 Processor -- start 2024-04-25T01:54:38.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f900ea00700 1 -- start start 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f900ea00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9008101190 0x7f90081984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f900ea00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9008102390 0x7f9008198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f900ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9008102d30 0x7f900819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f900ea00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f900810cde0 con 0x7f9008102390 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f900ea00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f900810cc60 con 0x7f9008101190 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f900ea00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f900810cf60 con 0x7f9008102d30 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f9007e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9008101190 0x7f90081984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f9007e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9008101190 0x7f90081984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56252/0 (socket says 172.21.15.102:56252) 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f9007e00700 1 -- 172.21.15.102:0/3796549080 learned_addr learned my addr 172.21.15.102:0/3796549080 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.636+0000 7f9007400700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9008102390 0x7f9008198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.637+0000 7f900ca00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9008102d30 0x7f900819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.637+0000 7f9007e00700 1 -- 172.21.15.102:0/3796549080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9008102d30 msgr2=0x7f900819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.637+0000 7f9007e00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9008102d30 0x7f900819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.637+0000 7f9007e00700 1 -- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9008102390 msgr2=0x7f9008198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:38.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.637+0000 7f9007e00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9008102390 0x7f9008198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.637+0000 7f9007e00700 1 -- 172.21.15.102:0/3796549080 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90081a0140 con 0x7f9008101190 2024-04-25T01:54:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.637+0000 7f9007e00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9008101190 0x7f90081984e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800e950 tx=0x7f8ff800ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.638+0000 7f9004c00700 1 -- 172.21.15.102:0/3796549080 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8004230 con 0x7f9008101190 2024-04-25T01:54:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.638+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90081a0430 con 0x7f9008101190 2024-04-25T01:54:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.638+0000 7f9004c00700 1 -- 172.21.15.102:0/3796549080 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ff80043d0 con 0x7f9008101190 2024-04-25T01:54:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.638+0000 7f9004c00700 1 -- 172.21.15.102:0/3796549080 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8003dc0 con 0x7f9008101190 2024-04-25T01:54:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.638+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90081a0970 con 0x7f9008101190 2024-04-25T01:54:38.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.639+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9008068f20 con 0x7f9008101190 2024-04-25T01:54:38.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.641+0000 7f9004c00700 1 -- 172.21.15.102:0/3796549080 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8ff8044b30 con 0x7f9008101190 2024-04-25T01:54:38.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.642+0000 7f9004c00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8fec0a2c20 0x7f8fec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:38.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.643+0000 7f9004c00700 1 -- 172.21.15.102:0/3796549080 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f8ff80cf270 con 0x7f9008101190 2024-04-25T01:54:38.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.643+0000 7f9007400700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8fec0a2c20 0x7f8fec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:38.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.643+0000 7f9007400700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8fec0a2c20 0x7f8fec0a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009c60 tx=0x7f8ff4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.645+0000 7f9004c00700 1 -- 172.21.15.102:0/3796549080 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8ff8044e20 con 0x7f9008101190 2024-04-25T01:54:38.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.879+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f90080fe8b0 con 0x7f8fec0a2c20 2024-04-25T01:54:38.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.920+0000 7f9004c00700 1 -- 172.21.15.102:0/3796549080 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f90080fe8b0 con 0x7f8fec0a2c20 2024-04-25T01:54:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T01:54:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:mds.a smithi102 running (2m) 26s ago 2m 12.2M - 19.0.0-2445-gd2c45d97 f68e37b807ef dbf63e9e5b10 2024-04-25T01:54:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:mds.b smithi110 running (2m) 53s ago 2m 26.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef 0735d4cfb609 2024-04-25T01:54:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:mds.c smithi193 running (2m) 53s ago 2m 17.9M - 19.0.0-2445-gd2c45d97 f68e37b807ef ae9e095acf8c 2024-04-25T01:54:38.923 INFO:teuthology.orchestra.run.smithi102.stdout:mds.d smithi102 running (2m) 26s ago 2m 12.3M - 19.0.0-2445-gd2c45d97 f68e37b807ef 404ae44b30b4 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.e smithi110 running (2m) 53s ago 2m 19.2M - 19.0.0-2445-gd2c45d97 f68e37b807ef 29a3871a9cba 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.f smithi193 running (2m) 53s ago 2m 18.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef 899614771ba1 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.g smithi102 running (2m) 26s ago 2m 12.2M - 19.0.0-2445-gd2c45d97 f68e37b807ef 04cd9af40f24 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.h smithi110 running (2m) 53s ago 2m 19.7M - 19.0.0-2445-gd2c45d97 f68e37b807ef 57e29242caf0 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.i smithi193 running (2m) 53s ago 2m 17.7M - 19.0.0-2445-gd2c45d97 f68e37b807ef 8e755f7c76b7 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.j smithi102 running (2m) 26s ago 2m 12.4M - 19.0.0-2445-gd2c45d97 f68e37b807ef fcb3f2944bff 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.k smithi110 running (2m) 53s ago 2m 14.0M - 19.0.0-2445-gd2c45d97 f68e37b807ef 91adea0f106b 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mds.l smithi193 running (2m) 53s ago 2m 13.1M - 19.0.0-2445-gd2c45d97 f68e37b807ef e5133fdb413b 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi102 *:9283,8765 running (10m) 26s ago 10m 516M - 19.0.0-2445-gd2c45d97 f68e37b807ef 5b9d2d220f12 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi110 *:8443,8765 running (8m) 53s ago 8m 453M - 19.0.0-2445-gd2c45d97 f68e37b807ef 9c2040968d4d 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.z smithi193 *:8443,8765 running (8m) 53s ago 8m 454M - 19.0.0-2445-gd2c45d97 f68e37b807ef 5416e1fa7c80 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (10m) 26s ago 11m 47.7M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 7e663022620a 2024-04-25T01:54:38.924 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi110 running (8m) 53s ago 8m 39.0M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef b5efaa94d32a 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi193 running (9m) 53s ago 9m 38.4M 2048M 19.0.0-2445-gd2c45d97 f68e37b807ef 0ded0404a1ee 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (8m) 26s ago 8m 56.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 5bb9e2e67e68 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi110 running (7m) 53s ago 7m 48.7M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef ed52a3260d4f 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi193 running (7m) 53s ago 7m 59.9M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef d57a39582c23 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (6m) 26s ago 6m 53.2M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 39cbdd876e58 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi110 running (6m) 53s ago 6m 46.4M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 277050e002be 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi193 running (5m) 53s ago 5m 46.7M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 3187d04a345a 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi102 running (5m) 26s ago 5m 43.9M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 4658a2f6896d 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi110 running (4m) 53s ago 4m 45.0M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 18eb98db8ad0 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.8 smithi193 running (4m) 53s ago 4m 53.6M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 24c050192233 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.9 smithi102 running (3m) 26s ago 3m 43.5M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 87313dc88377 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.10 smithi110 running (3m) 53s ago 3m 54.4M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 3b754627aeb2 2024-04-25T01:54:38.925 INFO:teuthology.orchestra.run.smithi102.stdout:osd.11 smithi193 running (3m) 53s ago 3m 52.6M 4272M 19.0.0-2445-gd2c45d97 f68e37b807ef 67fe9cfcbb31 2024-04-25T01:54:38.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8fec0a2c20 msgr2=0x7f8fec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:38.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8fec0a2c20 0x7f8fec0a50e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009c60 tx=0x7f8ff4009260 comp rx=0 tx=0).stop 2024-04-25T01:54:38.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9008101190 msgr2=0x7f90081984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:38.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9008101190 0x7f90081984e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800e950 tx=0x7f8ff800ec60 comp rx=0 tx=0).stop 2024-04-25T01:54:38.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 shutdown_connections 2024-04-25T01:54:38.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f8fec0a2c20 0x7f8fec0a50e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9008101190 0x7f90081984e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9008102d30 0x7f900819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 --2- 172.21.15.102:0/3796549080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9008102390 0x7f9008198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:38.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 >> 172.21.15.102:0/3796549080 conn(0x7f90080fc730 msgr2=0x7f90080fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:38.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 shutdown_connections 2024-04-25T01:54:38.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:38.924+0000 7f900ea00700 1 -- 172.21.15.102:0/3796549080 wait complete. 2024-04-25T01:54:39.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T01:54:40.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:39 smithi102 ceph-mon[27796]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:40.196 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:40.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:39 smithi110 ceph-mon[32848]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:40.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:39 smithi193 ceph-mon[32981]: pgmap v377: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:41.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:40 smithi102 ceph-mon[27796]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:41.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:40 smithi110 ceph-mon[32848]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:41.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:40 smithi193 ceph-mon[32981]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:41.923 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:41 smithi102 ceph-mon[27796]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:42.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:41 smithi110 ceph-mon[32848]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:42.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:41 smithi193 ceph-mon[32981]: pgmap v378: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:42.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.479+0000 7f645d400700 1 -- 172.21.15.102:0/3196933852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f645810a170 msgr2=0x7f645810c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:42.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.479+0000 7f645d400700 1 --2- 172.21.15.102:0/3196933852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f645810a170 0x7f645810c600 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f645000b0a0 tx=0x7f645000b3b0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.479+0000 7f645d400700 1 -- 172.21.15.102:0/3196933852 shutdown_connections 2024-04-25T01:54:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.479+0000 7f645d400700 1 --2- 172.21.15.102:0/3196933852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f645810a170 0x7f645810c600 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.479+0000 7f645d400700 1 --2- 172.21.15.102:0/3196933852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6458076200 0x7f6458074660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.479+0000 7f645d400700 1 --2- 172.21.15.102:0/3196933852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6458074c30 0x7f64580750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.479+0000 7f645d400700 1 -- 172.21.15.102:0/3196933852 >> 172.21.15.102:0/3196933852 conn(0x7f64580fd830 msgr2=0x7f64580ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.479+0000 7f645d400700 1 -- 172.21.15.102:0/3196933852 shutdown_connections 2024-04-25T01:54:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.479+0000 7f645d400700 1 -- 172.21.15.102:0/3196933852 wait complete. 2024-04-25T01:54:42.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.480+0000 7f645d400700 1 Processor -- start 2024-04-25T01:54:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f645d400700 1 -- start start 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f645d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6458074c30 0x7f645819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f645d400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6458076200 0x7f645819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f645d400700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645810a170 0x7f64581a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f645d400700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6458110060 con 0x7f645810a170 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f645d400700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f645810fee0 con 0x7f6458076200 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f645d400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f64581101e0 con 0x7f6458074c30 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f6456800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6458074c30 0x7f645819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f6457200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645810a170 0x7f64581a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f6456800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6458074c30 0x7f645819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:57532/0 (socket says 172.21.15.102:57532) 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f6456800700 1 -- 172.21.15.102:0/3743142892 learned_addr learned my addr 172.21.15.102:0/3743142892 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.481+0000 7f6455e00700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6458076200 0x7f645819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:42.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f6456800700 1 -- 172.21.15.102:0/3743142892 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6458076200 msgr2=0x7f645819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f6456800700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6458076200 0x7f645819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f6456800700 1 -- 172.21.15.102:0/3743142892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645810a170 msgr2=0x7f64581a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f6456800700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645810a170 0x7f64581a1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f6456800700 1 -- 172.21.15.102:0/3743142892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64581a22b0 con 0x7f6458074c30 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f6456800700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6458074c30 0x7f645819a650 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f644800e9f0 tx=0x7f644800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f644f400700 1 -- 172.21.15.102:0/3743142892 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644800cbd0 con 0x7f6458074c30 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64581a25a0 con 0x7f6458074c30 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f644f400700 1 -- 172.21.15.102:0/3743142892 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f644800ee90 con 0x7f6458074c30 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.482+0000 7f644f400700 1 -- 172.21.15.102:0/3743142892 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64480056c0 con 0x7f6458074c30 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.483+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64581a2ae0 con 0x7f6458074c30 2024-04-25T01:54:42.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.484+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6458068f20 con 0x7f6458074c30 2024-04-25T01:54:42.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.486+0000 7f644f400700 1 -- 172.21.15.102:0/3743142892 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6448004750 con 0x7f6458074c30 2024-04-25T01:54:42.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.487+0000 7f644f400700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f643c0a2c20 0x7f643c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:42.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.487+0000 7f644f400700 1 -- 172.21.15.102:0/3743142892 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f64480cf500 con 0x7f6458074c30 2024-04-25T01:54:42.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.488+0000 7f6455e00700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f643c0a2c20 0x7f643c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:42.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.488+0000 7f6455e00700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f643c0a2c20 0x7f643c0a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6444009c60 tx=0x7f6444009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:42.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.489+0000 7f644f400700 1 -- 172.21.15.102:0/3743142892 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6448053740 con 0x7f6458074c30 2024-04-25T01:54:42.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.721+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f6458062670 con 0x7f643c0a2c20 2024-04-25T01:54:42.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.729+0000 7f644f400700 1 -- 172.21.15.102:0/3743142892 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f6458062670 con 0x7f643c0a2c20 2024-04-25T01:54:42.732 INFO:teuthology.orchestra.run.smithi102.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T01:54:42.732 INFO:teuthology.orchestra.run.smithi102.stdout:mds.all 12/12 57s ago 2m smithi102=a;smithi102=d;smithi102=g;smithi102=j;smithi110=b;smithi110=e;smithi110=h;smithi110=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-04-25T01:54:42.732 INFO:teuthology.orchestra.run.smithi102.stdout:mgr 3/3 57s ago 8m smithi102=x;smithi110=y;smithi193=z;count:3 2024-04-25T01:54:42.732 INFO:teuthology.orchestra.run.smithi102.stdout:mon 3/3 57s ago 9m smithi102:172.21.15.102=a;smithi110:172.21.15.110=b;smithi193:172.21.15.193=c;count:3 2024-04-25T01:54:42.732 INFO:teuthology.orchestra.run.smithi102.stdout:osd 12 57s ago - 2024-04-25T01:54:42.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f643c0a2c20 msgr2=0x7f643c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f643c0a2c20 0x7f643c0a50e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6444009c60 tx=0x7f6444009260 comp rx=0 tx=0).stop 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6458074c30 msgr2=0x7f645819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6458074c30 0x7f645819a650 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f644800e9f0 tx=0x7f644800ed00 comp rx=0 tx=0).stop 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 shutdown_connections 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f643c0a2c20 0x7f643c0a50e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645810a170 0x7f64581a1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6458074c30 0x7f645819a650 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 --2- 172.21.15.102:0/3743142892 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6458076200 0x7f645819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 >> 172.21.15.102:0/3743142892 conn(0x7f64580fd830 msgr2=0x7f64581060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.732+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 shutdown_connections 2024-04-25T01:54:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:42.733+0000 7f645d400700 1 -- 172.21.15.102:0/3743142892 wait complete. 2024-04-25T01:54:43.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:42 smithi102 ceph-mon[27796]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:43.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:42 smithi102 ceph-mon[27796]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:43.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:42 smithi110 ceph-mon[32848]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:43.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:42 smithi110 ceph-mon[32848]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:43.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:42 smithi193 ceph-mon[32981]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:43.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:42 smithi193 ceph-mon[32981]: pgmap v379: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:54:43.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T01:54:44.195 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:45.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:45 smithi102 ceph-mon[27796]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:46.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:45 smithi110 ceph-mon[32848]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:46.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:45 smithi193 ceph-mon[32981]: pgmap v380: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:46.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.487+0000 7faacae00700 1 -- 172.21.15.102:0/1678846422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faac4103e30 msgr2=0x7faac410a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:46.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.487+0000 7faacae00700 1 --2- 172.21.15.102:0/1678846422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faac4103e30 0x7faac410a6e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faabc009fc0 tx=0x7faabc00b330 comp rx=0 tx=0).stop 2024-04-25T01:54:46.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.487+0000 7faacae00700 1 -- 172.21.15.102:0/1678846422 shutdown_connections 2024-04-25T01:54:46.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.487+0000 7faacae00700 1 --2- 172.21.15.102:0/1678846422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faac4103e30 0x7faac410a6e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:46.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.487+0000 7faacae00700 1 --2- 172.21.15.102:0/1678846422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faac4102290 0x7faac41026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:46.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.487+0000 7faacae00700 1 --2- 172.21.15.102:0/1678846422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faac4103490 0x7faac41038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:46.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.487+0000 7faacae00700 1 -- 172.21.15.102:0/1678846422 >> 172.21.15.102:0/1678846422 conn(0x7faac40fd830 msgr2=0x7faac40ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:46.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.487+0000 7faacae00700 1 -- 172.21.15.102:0/1678846422 shutdown_connections 2024-04-25T01:54:46.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.487+0000 7faacae00700 1 -- 172.21.15.102:0/1678846422 wait complete. 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.488+0000 7faacae00700 1 Processor -- start 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faacae00700 1 -- start start 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faacae00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faac4102290 0x7faac419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faacae00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faac4103490 0x7faac419abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faacae00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faac4103e30 0x7faac41a1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faacae00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faac410dec0 con 0x7faac4102290 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faacae00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faac410dd40 con 0x7faac4103e30 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faacae00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faac410e040 con 0x7faac4103490 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faac3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faac4102290 0x7faac419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faac3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faac4103490 0x7faac419abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faac8e00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faac4103e30 0x7faac41a1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faac3e00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faac4102290 0x7faac419a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39458/0 (socket says 172.21.15.102:39458) 2024-04-25T01:54:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faac3e00700 1 -- 172.21.15.102:0/2217951634 learned_addr learned my addr 172.21.15.102:0/2217951634 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:46.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.489+0000 7faac3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faac4103490 0x7faac419abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33222/0 (socket says 172.21.15.102:33222) 2024-04-25T01:54:46.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faac8e00700 1 -- 172.21.15.102:0/2217951634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faac4103490 msgr2=0x7faac419abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:46.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faac8e00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faac4103490 0x7faac419abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:46.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faac8e00700 1 -- 172.21.15.102:0/2217951634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faac4102290 msgr2=0x7faac419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:46.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faac8e00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faac4102290 0x7faac419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:46.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faac8e00700 1 -- 172.21.15.102:0/2217951634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac41a22e0 con 0x7faac4103e30 2024-04-25T01:54:46.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faac8e00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faac4103e30 0x7faac41a1c40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faabc007c50 tx=0x7faabc007c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faac0c00700 1 -- 172.21.15.102:0/2217951634 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc01f070 con 0x7faac4103e30 2024-04-25T01:54:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faac0c00700 1 -- 172.21.15.102:0/2217951634 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faabc003e10 con 0x7faac4103e30 2024-04-25T01:54:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac41a2570 con 0x7faac4103e30 2024-04-25T01:54:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faac0c00700 1 -- 172.21.15.102:0/2217951634 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc01b730 con 0x7faac4103e30 2024-04-25T01:54:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.490+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac41a2a20 con 0x7faac4103e30 2024-04-25T01:54:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.492+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac4068f20 con 0x7faac4103e30 2024-04-25T01:54:46.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.494+0000 7faac0c00700 1 -- 172.21.15.102:0/2217951634 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7faabc022070 con 0x7faac4103e30 2024-04-25T01:54:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.495+0000 7faac0c00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7faaa80a2ca0 0x7faaa80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:46.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.495+0000 7faac0c00700 1 -- 172.21.15.102:0/2217951634 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7faabc00f020 con 0x7faac4103e30 2024-04-25T01:54:46.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.495+0000 7faac3e00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7faaa80a2ca0 0x7faaa80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:46.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.496+0000 7faac3e00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7faaa80a2ca0 0x7faaa80a5160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faab4005950 tx=0x7faab4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:46.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.497+0000 7faac0c00700 1 -- 172.21.15.102:0/2217951634 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faabc029020 con 0x7faac4103e30 2024-04-25T01:54:46.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.729+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7faac4075b80 con 0x7faaa80a2ca0 2024-04-25T01:54:46.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.732+0000 7faac0c00700 1 -- 172.21.15.102:0/2217951634 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7faac4075b80 con 0x7faaa80a2ca0 2024-04-25T01:54:46.733 INFO:teuthology.orchestra.run.smithi102.stdout:HOST ADDR LABELS STATUS 2024-04-25T01:54:46.733 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 172.21.15.102 2024-04-25T01:54:46.733 INFO:teuthology.orchestra.run.smithi102.stdout:smithi110 172.21.15.110 2024-04-25T01:54:46.734 INFO:teuthology.orchestra.run.smithi102.stdout:smithi193 172.21.15.193 2024-04-25T01:54:46.734 INFO:teuthology.orchestra.run.smithi102.stdout:3 hosts in cluster 2024-04-25T01:54:46.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.735+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7faaa80a2ca0 msgr2=0x7faaa80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.735+0000 7faacae00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7faaa80a2ca0 0x7faaa80a5160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faab4005950 tx=0x7faab4009260 comp rx=0 tx=0).stop 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.735+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faac4103e30 msgr2=0x7faac41a1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.735+0000 7faacae00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faac4103e30 0x7faac41a1c40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7faabc007c50 tx=0x7faabc007c80 comp rx=0 tx=0).stop 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.735+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 shutdown_connections 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.735+0000 7faacae00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7faaa80a2ca0 0x7faaa80a5160 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.735+0000 7faacae00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faac4103e30 0x7faac41a1c40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.736+0000 7faacae00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faac4102290 0x7faac419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.736+0000 7faacae00700 1 --2- 172.21.15.102:0/2217951634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faac4103490 0x7faac419abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.736+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 >> 172.21.15.102:0/2217951634 conn(0x7faac40fd830 msgr2=0x7faac4074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.736+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 shutdown_connections 2024-04-25T01:54:46.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:46.736+0000 7faacae00700 1 -- 172.21.15.102:0/2217951634 wait complete. 2024-04-25T01:54:47.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T01:54:48.059 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:48.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:47 smithi102 ceph-mon[27796]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:48.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:47 smithi102 ceph-mon[27796]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:48.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:47 smithi110 ceph-mon[32848]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:48.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:47 smithi110 ceph-mon[32848]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:48.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:47 smithi193 ceph-mon[32981]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:48.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:47 smithi193 ceph-mon[32981]: pgmap v381: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:50.065 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:49 smithi102 ceph-mon[27796]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:50.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.230+0000 7fd7b2200700 1 -- 172.21.15.102:0/2938141026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7ac1056b0 msgr2=0x7fd7ac107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.230+0000 7fd7b2200700 1 --2- 172.21.15.102:0/2938141026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7ac1056b0 0x7fd7ac107aa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd794009990 tx=0x7fd794009ca0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.230+0000 7fd7b2200700 1 -- 172.21.15.102:0/2938141026 shutdown_connections 2024-04-25T01:54:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.230+0000 7fd7b2200700 1 --2- 172.21.15.102:0/2938141026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7ac107fe0 0x7fd7ac10a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.230+0000 7fd7b2200700 1 --2- 172.21.15.102:0/2938141026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac102d80 0x7fd7ac105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.230+0000 7fd7b2200700 1 --2- 172.21.15.102:0/2938141026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7ac1056b0 0x7fd7ac107aa0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.230+0000 7fd7b2200700 1 -- 172.21.15.102:0/2938141026 >> 172.21.15.102:0/2938141026 conn(0x7fd7ac0fc730 msgr2=0x7fd7ac0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.230+0000 7fd7b2200700 1 -- 172.21.15.102:0/2938141026 shutdown_connections 2024-04-25T01:54:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.230+0000 7fd7b2200700 1 -- 172.21.15.102:0/2938141026 wait complete. 2024-04-25T01:54:50.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.231+0000 7fd7b2200700 1 Processor -- start 2024-04-25T01:54:50.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7b2200700 1 -- start start 2024-04-25T01:54:50.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7b2200700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7ac102d80 0x7fd7ac19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7b2200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac1056b0 0x7fd7ac19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7b2200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7ac107fe0 0x7fd7ac1a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7b2200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac10def0 con 0x7fd7ac107fe0 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7b2200700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac10dd70 con 0x7fd7ac102d80 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7b2200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac10e070 con 0x7fd7ac1056b0 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7ab400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7ac102d80 0x7fd7ac19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7abe00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7ac107fe0 0x7fd7ac1a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7ab400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7ac102d80 0x7fd7ac19a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:55830/0 (socket says 172.21.15.102:55830) 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.232+0000 7fd7ab400700 1 -- 172.21.15.102:0/726621563 learned_addr learned my addr 172.21.15.102:0/726621563 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.233+0000 7fd7a2a00700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac1056b0 0x7fd7ac19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.233+0000 7fd7ab400700 1 -- 172.21.15.102:0/726621563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac1056b0 msgr2=0x7fd7ac19ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.233+0000 7fd7ab400700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac1056b0 0x7fd7ac19ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.233+0000 7fd7ab400700 1 -- 172.21.15.102:0/726621563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7ac107fe0 msgr2=0x7fd7ac1a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.233+0000 7fd7ab400700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7ac107fe0 0x7fd7ac1a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.233+0000 7fd7ab400700 1 -- 172.21.15.102:0/726621563 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac1a23b0 con 0x7fd7ac102d80 2024-04-25T01:54:50.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.233+0000 7fd7ab400700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7ac102d80 0x7fd7ac19a750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00ef10 tx=0x7fd79c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:50.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.233+0000 7fd7a8c00700 1 -- 172.21.15.102:0/726621563 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c00ca70 con 0x7fd7ac102d80 2024-04-25T01:54:50.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.234+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ac1a2640 con 0x7fd7ac102d80 2024-04-25T01:54:50.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.234+0000 7fd7a8c00700 1 -- 172.21.15.102:0/726621563 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd79c019070 con 0x7fd7ac102d80 2024-04-25T01:54:50.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.234+0000 7fd7a8c00700 1 -- 172.21.15.102:0/726621563 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0146e0 con 0x7fd7ac102d80 2024-04-25T01:54:50.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.234+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ac1a2b80 con 0x7fd7ac102d80 2024-04-25T01:54:50.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.235+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ac068f20 con 0x7fd7ac102d80 2024-04-25T01:54:50.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:49 smithi110 ceph-mon[32848]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:50.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.237+0000 7fd7a8c00700 1 -- 172.21.15.102:0/726621563 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd79c012050 con 0x7fd7ac102d80 2024-04-25T01:54:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.238+0000 7fd7a8c00700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7900a2c20 0x7fd7900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.239+0000 7fd7a8c00700 1 -- 172.21.15.102:0/726621563 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fd79c0cf110 con 0x7fd7ac102d80 2024-04-25T01:54:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.239+0000 7fd7a2a00700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7900a2c20 0x7fd7900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:50.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.239+0000 7fd7a2a00700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7900a2c20 0x7fd7900a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd79400b7b0 tx=0x7fd79401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:50.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.243+0000 7fd7a8c00700 1 -- 172.21.15.102:0/726621563 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd79c010040 con 0x7fd7ac102d80 2024-04-25T01:54:50.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:49 smithi193 ceph-mon[32981]: pgmap v382: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:50.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.474+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 --> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fd7ac062670 con 0x7fd7900a2c20 2024-04-25T01:54:50.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.478+0000 7fd7a8c00700 1 -- 172.21.15.102:0/726621563 <== mgr.14152 v2:172.21.15.102:6800/83904525 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fd7ac062670 con 0x7fd7900a2c20 2024-04-25T01:54:50.480 INFO:teuthology.orchestra.run.smithi102.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T01:54:50.480 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:54:50.480 INFO:teuthology.orchestra.run.smithi102.stdout:smithi110 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002T480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:54:50.480 INFO:teuthology.orchestra.run.smithi102.stdout:smithi193 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500HB400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7900a2c20 msgr2=0x7fd7900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7900a2c20 0x7fd7900a50e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd79400b7b0 tx=0x7fd79401a040 comp rx=0 tx=0).stop 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7ac102d80 msgr2=0x7fd7ac19a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7ac102d80 0x7fd7ac19a750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00ef10 tx=0x7fd79c00c3a0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 shutdown_connections 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7900a2c20 0x7fd7900a50e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7ac102d80 0x7fd7ac19a750 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7ac107fe0 0x7fd7ac1a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 --2- 172.21.15.102:0/726621563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac1056b0 0x7fd7ac19ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.481+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 >> 172.21.15.102:0/726621563 conn(0x7fd7ac0fc730 msgr2=0x7fd7ac102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.482+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 shutdown_connections 2024-04-25T01:54:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:50.482+0000 7fd7b2200700 1 -- 172.21.15.102:0/726621563 wait complete. 2024-04-25T01:54:51.356 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T01:54:51.365 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T01:54:51.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T01:54:51.825 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:52.144 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:51 smithi102 ceph-mon[27796]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:52.144 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:51 smithi102 ceph-mon[27796]: pgmap v383: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:52.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:51 smithi110 ceph-mon[32848]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:52.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:51 smithi110 ceph-mon[32848]: pgmap v383: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:52.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:51 smithi193 ceph-mon[32981]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:54:52.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:51 smithi193 ceph-mon[32981]: pgmap v383: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:54:54.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.009+0000 7f5d61200700 1 -- 172.21.15.102:0/234508412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d5c1055d0 msgr2=0x7f5d5c105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:54.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.009+0000 7f5d61200700 1 --2- 172.21.15.102:0/234508412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d5c1055d0 0x7f5d5c105a30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d480099c0 tx=0x7f5d48009cd0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.009+0000 7f5d61200700 1 -- 172.21.15.102:0/234508412 shutdown_connections 2024-04-25T01:54:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.009+0000 7f5d61200700 1 --2- 172.21.15.102:0/234508412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d5c1043d0 0x7f5d5c1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.009+0000 7f5d61200700 1 --2- 172.21.15.102:0/234508412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d5c105f70 0x7f5d5c10c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.009+0000 7f5d61200700 1 --2- 172.21.15.102:0/234508412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d5c1055d0 0x7f5d5c105a30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.009+0000 7f5d61200700 1 -- 172.21.15.102:0/234508412 >> 172.21.15.102:0/234508412 conn(0x7f5d5c0ff970 msgr2=0x7f5d5c101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.009+0000 7f5d61200700 1 -- 172.21.15.102:0/234508412 shutdown_connections 2024-04-25T01:54:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.009+0000 7f5d61200700 1 -- 172.21.15.102:0/234508412 wait complete. 2024-04-25T01:54:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.010+0000 7f5d61200700 1 Processor -- start 2024-04-25T01:54:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.010+0000 7f5d61200700 1 -- start start 2024-04-25T01:54:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.010+0000 7f5d61200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d5c1043d0 0x7f5d5c19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d61200700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d5c1055d0 0x7f5d5c19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d61200700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d5c105f70 0x7f5d5c1a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d61200700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c110000 con 0x7f5d5c105f70 2024-04-25T01:54:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d61200700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c10fe80 con 0x7f5d5c1055d0 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d61200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c110180 con 0x7f5d5c1043d0 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d5c1043d0 0x7f5d5c19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5be00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d5c105f70 0x7f5d5c1a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d5c1043d0 0x7f5d5c19a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:32986/0 (socket says 172.21.15.102:32986) 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5b400700 1 -- 172.21.15.102:0/2971766066 learned_addr learned my addr 172.21.15.102:0/2971766066 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5aa00700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d5c1055d0 0x7f5d5c19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5b400700 1 -- 172.21.15.102:0/2971766066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d5c1055d0 msgr2=0x7f5d5c19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5b400700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d5c1055d0 0x7f5d5c19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5b400700 1 -- 172.21.15.102:0/2971766066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d5c105f70 msgr2=0x7f5d5c1a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5b400700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d5c105f70 0x7f5d5c1a1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5b400700 1 -- 172.21.15.102:0/2971766066 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5c1a2220 con 0x7f5d5c1043d0 2024-04-25T01:54:54.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.011+0000 7f5d5aa00700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d5c1055d0 0x7f5d5c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:54:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.012+0000 7f5d5b400700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d5c1043d0 0x7f5d5c19a5c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00e9a0 tx=0x7f5d4c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.012+0000 7f5d53e00700 1 -- 172.21.15.102:0/2971766066 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c00cbd0 con 0x7f5d5c1043d0 2024-04-25T01:54:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.012+0000 7f5d53e00700 1 -- 172.21.15.102:0/2971766066 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d4c004500 con 0x7f5d5c1043d0 2024-04-25T01:54:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.012+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5c1a2510 con 0x7f5d5c1043d0 2024-04-25T01:54:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.012+0000 7f5d53e00700 1 -- 172.21.15.102:0/2971766066 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c003e00 con 0x7f5d5c1043d0 2024-04-25T01:54:54.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.012+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d5c1a2a50 con 0x7f5d5c1043d0 2024-04-25T01:54:54.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.013+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d5c068f20 con 0x7f5d5c1043d0 2024-04-25T01:54:54.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.015+0000 7f5d53e00700 1 -- 172.21.15.102:0/2971766066 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f5d4c003fe0 con 0x7f5d5c1043d0 2024-04-25T01:54:54.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.016+0000 7f5d53e00700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5d400a2c20 0x7f5d400a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:54.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.017+0000 7f5d53e00700 1 -- 172.21.15.102:0/2971766066 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f5d4c0cf3a0 con 0x7f5d5c1043d0 2024-04-25T01:54:54.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.017+0000 7f5d5aa00700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5d400a2c20 0x7f5d400a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:54.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.017+0000 7f5d5aa00700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5d400a2c20 0x7f5d400a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5d48000c00 tx=0x7f5d48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:54.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.021+0000 7f5d53e00700 1 -- 172.21.15.102:0/2971766066 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5d4c09c2b0 con 0x7f5d5c1043d0 2024-04-25T01:54:54.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:53 smithi102 ceph-mon[27796]: pgmap v384: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:54.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:53 smithi110 ceph-mon[32848]: pgmap v384: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:54.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:53 smithi193 ceph-mon[32981]: pgmap v384: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.290+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f5d5c062e40 con 0x7f5d5c1043d0 2024-04-25T01:54:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.291+0000 7f5d53e00700 1 -- 172.21.15.102:0/2971766066 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) ==== 76+0+3066 (secure 0 0 0) 0x7f5d4c09bbb0 con 0x7f5d5c1043d0 2024-04-25T01:54:54.292 INFO:teuthology.orchestra.run.smithi102.stdout:e22 2024-04-25T01:54:54.293 INFO:teuthology.orchestra.run.smithi102.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T01:54:54.293 INFO:teuthology.orchestra.run.smithi102.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:54:54.293 INFO:teuthology.orchestra.run.smithi102.stdout:legacy client fscid: 1 2024-04-25T01:54:54.293 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:54.293 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem 'cephfs' (1) 2024-04-25T01:54:54.293 INFO:teuthology.orchestra.run.smithi102.stdout:fs_name cephfs 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 22 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-04-25T01:52:43.526145+0000 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-04-25T01:52:49.089206+0000 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:tableserver 0 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:root 0 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:session_timeout 300 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:session_autoclose 300 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:max_file_size 1099511627776 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:max_xattr_size 65536 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:required_client_features {} 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure 0 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:last_failure_osd_epoch 0 2024-04-25T01:54:54.294 INFO:teuthology.orchestra.run.smithi102.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:max_mds 3 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:in 0,1,2 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:up {0=24587,1=24541,2=24533} 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:failed 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:damaged 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:stopped 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:data_pools [3] 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:metadata_pool 2 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:inline_data disabled 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:balancer 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:bal_rank_mask -1 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:standby_count_wanted 0 2024-04-25T01:54:54.295 INFO:teuthology.orchestra.run.smithi102.stdout:qdb_cluster leader: 24587 members: 24533,24541,24587 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b{0:24587} state up:active seq 3 addr [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c{0:24520} state up:standby-replay seq 1 addr [v2:172.21.15.193:6836/3613019309,v1:172.21.15.193:6837/3613019309] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.i{1:24541} state up:active seq 7 addr [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.f{1:24508} state up:standby-replay seq 1 addr [v2:172.21.15.193:6834/1885428891,v1:172.21.15.193:6835/1885428891] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.h{2:24533} state up:active seq 8 addr [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.e{2:24506} state up:standby-replay seq 1 addr [v2:172.21.15.110:6834/4057730311,v1:172.21.15.110:6835/4057730311] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:Standby daemons: 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.j{-1:14394} state up:standby seq 1 addr [v2:172.21.15.102:6834/2866248981,v1:172.21.15.102:6835/2866248981] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d{-1:14412} state up:standby seq 1 addr [v2:172.21.15.102:6836/3271665645,v1:172.21.15.102:6837/3271665645] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.g{-1:14457} state up:standby seq 1 addr [v2:172.21.15.102:6838/4134663811,v1:172.21.15.102:6839/4134663811] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a{-1:14475} state up:standby seq 1 addr [v2:172.21.15.102:6840/3724889216,v1:172.21.15.102:6841/3724889216] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.297 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.l{-1:24448} state up:standby seq 1 addr [v2:172.21.15.193:6832/3303248011,v1:172.21.15.193:6833/3303248011] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.297 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.k{-1:24461} state up:standby seq 1 addr [v2:172.21.15.110:6832/3320624567,v1:172.21.15.110:6833/3320624567] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5d400a2c20 msgr2=0x7f5d400a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5d400a2c20 0x7f5d400a50e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5d48000c00 tx=0x7f5d48005fd0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d5c1043d0 msgr2=0x7f5d5c19a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d5c1043d0 0x7f5d5c19a5c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c00e9a0 tx=0x7f5d4c00ecb0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 shutdown_connections 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f5d400a2c20 0x7f5d400a50e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d5c105f70 0x7f5d5c1a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d5c1043d0 0x7f5d5c19a5c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 --2- 172.21.15.102:0/2971766066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d5c1055d0 0x7f5d5c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 >> 172.21.15.102:0/2971766066 conn(0x7f5d5c0ff970 msgr2=0x7f5d5c108950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 shutdown_connections 2024-04-25T01:54:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:54.294+0000 7f5d61200700 1 -- 172.21.15.102:0/2971766066 wait complete. 2024-04-25T01:54:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 22 2024-04-25T01:54:55.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:54 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2971766066' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:54:55.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:54 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2971766066' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:54:55.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:54 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2971766066' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T01:54:55.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T01:54:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:54:56.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:55 smithi102 ceph-mon[27796]: pgmap v385: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:56.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:55 smithi110 ceph-mon[32848]: pgmap v385: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:56.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:55 smithi193 ceph-mon[32981]: pgmap v385: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:56.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:56 smithi102 ceph-mon[27796]: pgmap v386: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:57.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:56 smithi110 ceph-mon[32848]: pgmap v386: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:57.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:56 smithi193 ceph-mon[32981]: pgmap v386: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T01:54:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.817+0000 7f060ce00700 1 -- 172.21.15.102:0/1738475274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0608074c30 msgr2=0x7f06080750b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.817+0000 7f060ce00700 1 --2- 172.21.15.102:0/1738475274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0608074c30 0x7f06080750b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f05f00098f0 tx=0x7f05f0009c00 comp rx=0 tx=0).stop 2024-04-25T01:54:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.818+0000 7f060ce00700 1 -- 172.21.15.102:0/1738475274 shutdown_connections 2024-04-25T01:54:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.818+0000 7f060ce00700 1 --2- 172.21.15.102:0/1738475274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0608076200 0x7f0608074660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.818+0000 7f060ce00700 1 --2- 172.21.15.102:0/1738475274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060810a1c0 0x7f060810c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.818+0000 7f060ce00700 1 --2- 172.21.15.102:0/1738475274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0608074c30 0x7f06080750b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.818+0000 7f060ce00700 1 -- 172.21.15.102:0/1738475274 >> 172.21.15.102:0/1738475274 conn(0x7f06080fd880 msgr2=0x7f06080ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.818+0000 7f060ce00700 1 -- 172.21.15.102:0/1738475274 shutdown_connections 2024-04-25T01:54:57.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.818+0000 7f060ce00700 1 -- 172.21.15.102:0/1738475274 wait complete. 2024-04-25T01:54:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.819+0000 7f060ce00700 1 Processor -- start 2024-04-25T01:54:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.819+0000 7f060ce00700 1 -- start start 2024-04-25T01:54:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.819+0000 7f060ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0608074c30 0x7f060819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:57.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.819+0000 7f060ce00700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0608076200 0x7f060819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.819+0000 7f060ce00700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060810a1c0 0x7f06081a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.819+0000 7f060ce00700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f06081100e0 con 0x7f060810a1c0 2024-04-25T01:54:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.819+0000 7f060ce00700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f060810ff60 con 0x7f0608076200 2024-04-25T01:54:57.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.819+0000 7f060ce00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0608110260 con 0x7f0608074c30 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0605400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0608076200 0x7f060819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0605e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0608074c30 0x7f060819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0606800700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060810a1c0 0x7f06081a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0605400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0608076200 0x7f060819ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:57912/0 (socket says 172.21.15.102:57912) 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0605400700 1 -- 172.21.15.102:0/2683563032 learned_addr learned my addr 172.21.15.102:0/2683563032 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0605400700 1 -- 172.21.15.102:0/2683563032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0608074c30 msgr2=0x7f060819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0605400700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0608074c30 0x7f060819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0605400700 1 -- 172.21.15.102:0/2683563032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060810a1c0 msgr2=0x7f06081a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0605400700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060810a1c0 0x7f06081a1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.820+0000 7f0605400700 1 -- 172.21.15.102:0/2683563032 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06081a2340 con 0x7f0608076200 2024-04-25T01:54:57.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.821+0000 7f0605400700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0608076200 0x7f060819ac20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009810 tx=0x7f05f0005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.821+0000 7f05fea00700 1 -- 172.21.15.102:0/2683563032 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0005440 con 0x7f0608076200 2024-04-25T01:54:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.821+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06081a25d0 con 0x7f0608076200 2024-04-25T01:54:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.821+0000 7f05fea00700 1 -- 172.21.15.102:0/2683563032 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05f00055e0 con 0x7f0608076200 2024-04-25T01:54:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.821+0000 7f05fea00700 1 -- 172.21.15.102:0/2683563032 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0003bc0 con 0x7f0608076200 2024-04-25T01:54:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.821+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06081a2a80 con 0x7f0608076200 2024-04-25T01:54:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.823+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e4005360 con 0x7f0608076200 2024-04-25T01:54:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.824+0000 7f05fea00700 1 -- 172.21.15.102:0/2683563032 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f05f000b560 con 0x7f0608076200 2024-04-25T01:54:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.826+0000 7f05fea00700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05ec0a2c20 0x7f05ec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.826+0000 7f05fea00700 1 -- 172.21.15.102:0/2683563032 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f05f00cffe0 con 0x7f0608076200 2024-04-25T01:54:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.826+0000 7f0605e00700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05ec0a2c20 0x7f05ec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.826+0000 7f0605e00700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05ec0a2c20 0x7f05ec0a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009c60 tx=0x7f05f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:57.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:57.830+0000 7f05fea00700 1 -- 172.21.15.102:0/2683563032 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f05f0013030 con 0x7f0608076200 2024-04-25T01:54:58.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.053+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f05e4005e60 con 0x7f0608076200 2024-04-25T01:54:58.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.055+0000 7f05fea00700 1 -- 172.21.15.102:0/2683563032 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) ==== 56+0+5301 (secure 0 0 0) 0x7f05f009cfa0 con 0x7f0608076200 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 89 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-04-25T01:43:34.661160+0000 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:modified 2024-04-25T01:53:43.851432+0000 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:crush_version 26 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:full_ratio 0.95 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:backfillfull_ratio 0.9 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:nearfull_ratio 0.85 2024-04-25T01:54:58.057 INFO:teuthology.orchestra.run.smithi102.stdout:require_min_compat_client luminous 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:min_compat_client luminous 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:require_osd_release squid 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:stretch_mode_enabled false 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:max_osd 12 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 up in weight 1 up_from 8 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6802/417549417,v1:172.21.15.102:6803/417549417] [v2:172.21.15.102:6804/417549417,v1:172.21.15.102:6805/417549417] exists,up caf505e1-b2d5-43da-848f-89b616616084 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6800/2162810099,v1:172.21.15.110:6801/2162810099] [v2:172.21.15.110:6802/2162810099,v1:172.21.15.110:6803/2162810099] exists,up 437ccdd6-3c94-47c1-9350-b252f8aec22f 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6800/602865964,v1:172.21.15.193:6801/602865964] [v2:172.21.15.193:6802/602865964,v1:172.21.15.193:6803/602865964] exists,up c730312e-3b4f-4839-bcc4-587ad4c5c576 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 up in weight 1 up_from 27 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6810/3993205287,v1:172.21.15.102:6811/3993205287] [v2:172.21.15.102:6812/3993205287,v1:172.21.15.102:6813/3993205287] exists,up 240b29df-2fe9-4354-83c2-6a319aecefac 2024-04-25T01:54:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 up in weight 1 up_from 35 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6808/4200899007,v1:172.21.15.110:6809/4200899007] [v2:172.21.15.110:6810/4200899007,v1:172.21.15.110:6811/4200899007] exists,up 63705793-8fc3-4a74-80d0-41e70f84c5bf 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 up in weight 1 up_from 40 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6808/3543273137,v1:172.21.15.193:6809/3543273137] [v2:172.21.15.193:6810/3543273137,v1:172.21.15.193:6811/3543273137] exists,up 8109f99d-f1f9-424a-8d2e-d1ddb0626f0c 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 up in weight 1 up_from 48 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6818/2856826689,v1:172.21.15.102:6819/2856826689] [v2:172.21.15.102:6820/2856826689,v1:172.21.15.102:6821/2856826689] exists,up 92fb32af-539d-474a-87ff-2305e626f89a 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 up in weight 1 up_from 53 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6816/2471551369,v1:172.21.15.110:6817/2471551369] [v2:172.21.15.110:6818/2471551369,v1:172.21.15.110:6819/2471551369] exists,up 99d65788-2491-4fc8-b15c-1861814a488a 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.8 up in weight 1 up_from 61 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6816/3634064328,v1:172.21.15.193:6817/3634064328] [v2:172.21.15.193:6818/3634064328,v1:172.21.15.193:6819/3634064328] exists,up bc7180db-d4e2-468d-a845-3846caa7a043 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.9 up in weight 1 up_from 69 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6826/1305468681,v1:172.21.15.102:6827/1305468681] [v2:172.21.15.102:6828/1305468681,v1:172.21.15.102:6829/1305468681] exists,up d15d150e-08b4-45c2-8f9d-cd64dba163fb 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.10 up in weight 1 up_from 74 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6824/851854931,v1:172.21.15.110:6825/851854931] [v2:172.21.15.110:6826/851854931,v1:172.21.15.110:6827/851854931] exists,up eb0f9f59-1254-4fbb-9f87-566e43a05d06 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:osd.11 up in weight 1 up_from 82 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6824/40469709,v1:172.21.15.193:6825/40469709] [v2:172.21.15.193:6826/40469709,v1:172.21.15.193:6827/40469709] exists,up 0e8f1ff1-80f8-459e-8fea-733a1b7e54ac 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 2.5 [8,11] 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 2.c [2,4] 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 2.10 [8,11] 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 2.14 [2,4] 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 3.1 [4,9] 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 3.1d [5,9] 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 3.27 [4,8] 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:pg_upmap_items 3.2c [4,9] 2024-04-25T01:54:58.059 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/3113170519 expires 2024-04-26T01:44:42.581235+0000 2024-04-25T01:54:58.060 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6801/1124203809 expires 2024-04-26T01:44:42.581235+0000 2024-04-25T01:54:58.060 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6800/1124203809 expires 2024-04-26T01:44:42.581235+0000 2024-04-25T01:54:58.060 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6801/2757023783 expires 2024-04-26T01:44:08.232557+0000 2024-04-25T01:54:58.060 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:6800/2757023783 expires 2024-04-26T01:44:08.232557+0000 2024-04-25T01:54:58.060 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/411628693 expires 2024-04-26T01:44:42.581235+0000 2024-04-25T01:54:58.060 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/1716050795 expires 2024-04-26T01:44:08.232557+0000 2024-04-25T01:54:58.060 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/2344465952 expires 2024-04-26T01:44:42.581235+0000 2024-04-25T01:54:58.060 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/4034021546 expires 2024-04-26T01:44:08.232557+0000 2024-04-25T01:54:58.060 INFO:teuthology.orchestra.run.smithi102.stdout:blocklist 172.21.15.102:0/316635575 expires 2024-04-26T01:44:08.232557+0000 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05ec0a2c20 msgr2=0x7f05ec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05ec0a2c20 0x7f05ec0a50e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009c60 tx=0x7f05f8009260 comp rx=0 tx=0).stop 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0608076200 msgr2=0x7f060819ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0608076200 0x7f060819ac20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009810 tx=0x7f05f0005250 comp rx=0 tx=0).stop 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 shutdown_connections 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f05ec0a2c20 0x7f05ec0a50e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f060810a1c0 0x7f06081a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0608074c30 0x7f060819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 --2- 172.21.15.102:0/2683563032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0608076200 0x7f060819ac20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 >> 172.21.15.102:0/2683563032 conn(0x7f06080fd880 msgr2=0x7f0608106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 shutdown_connections 2024-04-25T01:54:58.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:58.058+0000 7f060ce00700 1 -- 172.21.15.102:0/2683563032 wait complete. 2024-04-25T01:54:58.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:58 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2683563032' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:54:58.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:58 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2683563032' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:54:58.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:58 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2683563032' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T01:54:59.024 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T01:54:59.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:54:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.248+0000 7fe548400640 1 -- 172.21.15.102:0/2336831748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe540077340 msgr2=0x7fe5400777a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.248+0000 7fe548400640 1 --2- 172.21.15.102:0/2336831748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe540077340 0x7fe5400777a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009900 tx=0x7fe52c02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.248+0000 7fe548400640 1 -- 172.21.15.102:0/2336831748 shutdown_connections 2024-04-25T01:54:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.248+0000 7fe548400640 1 --2- 172.21.15.102:0/2336831748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540077ce0 0x7fe540078580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.248+0000 7fe548400640 1 --2- 172.21.15.102:0/2336831748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe540077340 0x7fe5400777a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.248+0000 7fe548400640 1 --2- 172.21.15.102:0/2336831748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe540076100 0x7fe540076520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.248+0000 7fe548400640 1 -- 172.21.15.102:0/2336831748 >> 172.21.15.102:0/2336831748 conn(0x7fe5400fef30 msgr2=0x7fe540101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:59.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.248+0000 7fe548400640 1 -- 172.21.15.102:0/2336831748 shutdown_connections 2024-04-25T01:54:59.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.248+0000 7fe548400640 1 -- 172.21.15.102:0/2336831748 wait complete. 2024-04-25T01:54:59.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.249+0000 7fe548400640 1 Processor -- start 2024-04-25T01:54:59.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.249+0000 7fe548400640 1 -- start start 2024-04-25T01:54:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe548400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540076100 0x7fe54019e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe548400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe540077340 0x7fe54019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe548400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe540077ce0 0x7fe5401a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:59.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe548400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe540113a10 con 0x7fe540076100 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe548400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe540113890 con 0x7fe540077ce0 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe548400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe540113b90 con 0x7fe540077340 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe545000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe540077340 0x7fe54019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe545000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe540077340 0x7fe54019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33020/0 (socket says 172.21.15.102:33020) 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe545000640 1 -- 172.21.15.102:0/2758760970 learned_addr learned my addr 172.21.15.102:0/2758760970 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe546400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe540077ce0 0x7fe5401a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.250+0000 7fe545a00640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540076100 0x7fe54019e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe546400640 1 -- 172.21.15.102:0/2758760970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe540077340 msgr2=0x7fe54019e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe546400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe540077340 0x7fe54019e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe546400640 1 -- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540076100 msgr2=0x7fe54019e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe546400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540076100 0x7fe54019e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe546400640 1 -- 172.21.15.102:0/2758760970 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5401a5d00 con 0x7fe540077ce0 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe545000640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe540077340 0x7fe54019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe546400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe540077ce0 0x7fe5401a5690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe53800da10 tx=0x7fe53800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:59.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe53e000640 1 -- 172.21.15.102:0/2758760970 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53800bd70 con 0x7fe540077ce0 2024-04-25T01:54:59.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5401a5ff0 con 0x7fe540077ce0 2024-04-25T01:54:59.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe53e000640 1 -- 172.21.15.102:0/2758760970 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe538003e50 con 0x7fe540077ce0 2024-04-25T01:54:59.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe53e000640 1 -- 172.21.15.102:0/2758760970 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5380106a0 con 0x7fe540077ce0 2024-04-25T01:54:59.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.251+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5401a6460 con 0x7fe540077ce0 2024-04-25T01:54:59.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.252+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe540110530 con 0x7fe540077ce0 2024-04-25T01:54:59.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.254+0000 7fe53e000640 1 -- 172.21.15.102:0/2758760970 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe538003970 con 0x7fe540077ce0 2024-04-25T01:54:59.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.256+0000 7fe53e000640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe5280a2b90 0x7fe5280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.256+0000 7fe53e000640 1 -- 172.21.15.102:0/2758760970 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fe5380cf440 con 0x7fe540077ce0 2024-04-25T01:54:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.256+0000 7fe545a00640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe5280a2b90 0x7fe5280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.256+0000 7fe545a00640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe5280a2b90 0x7fe5280a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe53400bbe0 tx=0x7fe53400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.260+0000 7fe53e000640 1 -- 172.21.15.102:0/2758760970 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe53809c350 con 0x7fe540077ce0 2024-04-25T01:54:59.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:54:59 smithi102 ceph-mon[27796]: pgmap v387: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:59.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.462+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe540062370 con 0x7fe540077ce0 2024-04-25T01:54:59.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.464+0000 7fe53e000640 1 -- 172.21.15.102:0/2758760970 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22686 (secure 0 0 0) 0x7fe53809bc50 con 0x7fe540077ce0 2024-04-25T01:54:59.466 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:54:59.466 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:53:43.851432+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:54:59.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe5280a2b90 msgr2=0x7fe5280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe5280a2b90 0x7fe5280a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe53400bbe0 tx=0x7fe53400b390 comp rx=0 tx=0).stop 2024-04-25T01:54:59.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe540077ce0 msgr2=0x7fe5401a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe540077ce0 0x7fe5401a5690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe53800da10 tx=0x7fe53800ded0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 shutdown_connections 2024-04-25T01:54:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fe5280a2b90 0x7fe5280a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe540077ce0 0x7fe5401a5690 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe540077340 0x7fe54019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 --2- 172.21.15.102:0/2758760970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540076100 0x7fe54019e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 >> 172.21.15.102:0/2758760970 conn(0x7fe5400fef30 msgr2=0x7fe5401006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 shutdown_connections 2024-04-25T01:54:59.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.467+0000 7fe548400640 1 -- 172.21.15.102:0/2758760970 wait complete. 2024-04-25T01:54:59.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:41.585137+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:43.022666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:54:59.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:54:59.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:54:59 smithi110 ceph-mon[32848]: pgmap v387: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:59.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:54:59 smithi193 ceph-mon[32981]: pgmap v387: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:54:59.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.689+0000 7f1bc3a00640 1 -- 172.21.15.102:0/322895327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bbc0741a0 msgr2=0x7f1bbc074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.689+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/322895327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bbc0741a0 0x7f1bbc074620 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009860 tx=0x7f1ba802a340 comp rx=0 tx=0).stop 2024-04-25T01:54:59.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.690+0000 7f1bc3a00640 1 -- 172.21.15.102:0/322895327 shutdown_connections 2024-04-25T01:54:59.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.690+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/322895327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bbc109650 0x7f1bbc10bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.690+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/322895327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bbc0741a0 0x7f1bbc074620 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.690+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/322895327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bbc075780 0x7f1bbc073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.690+0000 7f1bc3a00640 1 -- 172.21.15.102:0/322895327 >> 172.21.15.102:0/322895327 conn(0x7f1bbc0fcd20 msgr2=0x7f1bbc0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:59.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.690+0000 7f1bc3a00640 1 -- 172.21.15.102:0/322895327 shutdown_connections 2024-04-25T01:54:59.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.690+0000 7f1bc3a00640 1 -- 172.21.15.102:0/322895327 wait complete. 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc3a00640 1 Processor -- start 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc3a00640 1 -- start start 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc3a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bbc0741a0 0x7f1bbc199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc3a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bbc075780 0x7f1bbc19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc3a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bbc109650 0x7f1bbc1a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc3a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc10f5f0 con 0x7f1bbc109650 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc3a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc10f470 con 0x7f1bbc0741a0 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc3a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc10f770 con 0x7f1bbc075780 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc1000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bbc0741a0 0x7f1bbc199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bbc109650 0x7f1bbc1a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc1000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bbc0741a0 0x7f1bbc199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:57954/0 (socket says 172.21.15.102:57954) 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bc1000640 1 -- 172.21.15.102:0/2275202406 learned_addr learned my addr 172.21.15.102:0/2275202406 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.691+0000 7f1bbbe00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bbc075780 0x7f1bbc19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bc1000640 1 -- 172.21.15.102:0/2275202406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bbc075780 msgr2=0x7f1bbc19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bc1000640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bbc075780 0x7f1bbc19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bc1000640 1 -- 172.21.15.102:0/2275202406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bbc109650 msgr2=0x7f1bbc1a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bc1000640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bbc109650 0x7f1bbc1a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bc1000640 1 -- 172.21.15.102:0/2275202406 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bbc1a1940 con 0x7f1bbc0741a0 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bbbe00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bbc075780 0x7f1bbc19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bc1000640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bbc0741a0 0x7f1bbc199d30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000e9f0 tx=0x7f1bb000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bb9600640 1 -- 172.21.15.102:0/2275202406 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000cd80 con 0x7f1bbc0741a0 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bb9600640 1 -- 172.21.15.102:0/2275202406 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bb0005500 con 0x7f1bbc0741a0 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bbc1a1c30 con 0x7f1bbc0741a0 2024-04-25T01:54:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.692+0000 7f1bb9600640 1 -- 172.21.15.102:0/2275202406 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0010690 con 0x7f1bbc0741a0 2024-04-25T01:54:59.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.693+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bbc1a2130 con 0x7f1bbc0741a0 2024-04-25T01:54:59.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.694+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bbc108380 con 0x7f1bbc0741a0 2024-04-25T01:54:59.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.696+0000 7f1bb9600640 1 -- 172.21.15.102:0/2275202406 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1bb00046c0 con 0x7f1bbc0741a0 2024-04-25T01:54:59.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.697+0000 7f1bb9600640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1ba40a2b90 0x7f1ba40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:54:59.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.697+0000 7f1bb9600640 1 -- 172.21.15.102:0/2275202406 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f1bb00cf540 con 0x7f1bbc0741a0 2024-04-25T01:54:59.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.697+0000 7f1bbbe00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1ba40a2b90 0x7f1ba40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:54:59.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.698+0000 7f1bbbe00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1ba40a2b90 0x7f1ba40a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1ba802a930 tx=0x7f1ba8030570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:54:59.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.701+0000 7f1bb9600640 1 -- 172.21.15.102:0/2275202406 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1bb009c450 con 0x7f1bbc0741a0 2024-04-25T01:54:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.900+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1bbc0623e0 con 0x7f1bbc0741a0 2024-04-25T01:54:59.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.901+0000 7f1bb9600640 1 -- 172.21.15.102:0/2275202406 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f1bb009bd50 con 0x7f1bbc0741a0 2024-04-25T01:54:59.902 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:54:59.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1ba40a2b90 msgr2=0x7f1ba40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1ba40a2b90 0x7f1ba40a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1ba802a930 tx=0x7f1ba8030570 comp rx=0 tx=0).stop 2024-04-25T01:54:59.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bbc0741a0 msgr2=0x7f1bbc199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:54:59.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bbc0741a0 0x7f1bbc199d30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000e9f0 tx=0x7f1bb000edf0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 shutdown_connections 2024-04-25T01:54:59.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1ba40a2b90 0x7f1ba40a5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bbc109650 0x7f1bbc1a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bbc075780 0x7f1bbc19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 --2- 172.21.15.102:0/2275202406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bbc0741a0 0x7f1bbc199d30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:54:59.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 >> 172.21.15.102:0/2275202406 conn(0x7f1bbc0fcd20 msgr2=0x7f1bbc105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:54:59.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 shutdown_connections 2024-04-25T01:54:59.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:54:59.904+0000 7f1bc3a00640 1 -- 172.21.15.102:0/2275202406 wait complete. 2024-04-25T01:54:59.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:55:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.129+0000 7fd811400640 1 -- 172.21.15.102:0/4123379322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80c107530 msgr2=0x7fd80c109920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.129+0000 7fd811400640 1 --2- 172.21.15.102:0/4123379322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80c107530 0x7fd80c109920 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd804009ca0 tx=0x7fd80402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.129+0000 7fd811400640 1 -- 172.21.15.102:0/4123379322 shutdown_connections 2024-04-25T01:55:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.129+0000 7fd811400640 1 --2- 172.21.15.102:0/4123379322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80c107530 0x7fd80c109920 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.129+0000 7fd811400640 1 --2- 172.21.15.102:0/4123379322 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd80c104c00 0x7fd80c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.129+0000 7fd811400640 1 --2- 172.21.15.102:0/4123379322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd80c1022d0 0x7fd80c1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.129+0000 7fd811400640 1 -- 172.21.15.102:0/4123379322 >> 172.21.15.102:0/4123379322 conn(0x7fd80c0fbc00 msgr2=0x7fd80c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.129+0000 7fd811400640 1 -- 172.21.15.102:0/4123379322 shutdown_connections 2024-04-25T01:55:00.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.129+0000 7fd811400640 1 -- 172.21.15.102:0/4123379322 wait complete. 2024-04-25T01:55:00.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.130+0000 7fd811400640 1 Processor -- start 2024-04-25T01:55:00.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.130+0000 7fd811400640 1 -- start start 2024-04-25T01:55:00.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd811400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd80c1022d0 0x7fd80c199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:00.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd811400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd80c104c00 0x7fd80c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd811400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80c107530 0x7fd80c1a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd811400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd80c10d470 con 0x7fd80c1022d0 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd811400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd80c10d2f0 con 0x7fd80c104c00 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd811400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd80c10d5f0 con 0x7fd80c107530 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd80a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd80c1022d0 0x7fd80c199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd80a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd80c1022d0 0x7fd80c199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58694/0 (socket says 172.21.15.102:58694) 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd80a800640 1 -- 172.21.15.102:0/3957619969 learned_addr learned my addr 172.21.15.102:0/3957619969 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd809e00640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd80c104c00 0x7fd80c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd80b200640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80c107530 0x7fd80c1a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd80a800640 1 -- 172.21.15.102:0/3957619969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80c107530 msgr2=0x7fd80c1a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd80a800640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80c107530 0x7fd80c1a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd80a800640 1 -- 172.21.15.102:0/3957619969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd80c104c00 msgr2=0x7fd80c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd80a800640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd80c104c00 0x7fd80c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd80a800640 1 -- 172.21.15.102:0/3957619969 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80c1a1910 con 0x7fd80c1022d0 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.131+0000 7fd809e00640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd80c104c00 0x7fd80c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.132+0000 7fd80a800640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd80c1022d0 0x7fd80c199d00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00e9f0 tx=0x7fd7fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.132+0000 7fd803400640 1 -- 172.21.15.102:0/3957619969 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc00cd80 con 0x7fd80c1022d0 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.132+0000 7fd803400640 1 -- 172.21.15.102:0/3957619969 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7fc010910 con 0x7fd80c1022d0 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.132+0000 7fd803400640 1 -- 172.21.15.102:0/3957619969 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc018990 con 0x7fd80c1022d0 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.132+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80c1a1c00 con 0x7fd80c1022d0 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.132+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd80c1a21a0 con 0x7fd80c1022d0 2024-04-25T01:55:00.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.133+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd80c109020 con 0x7fd80c1022d0 2024-04-25T01:55:00.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.135+0000 7fd803400640 1 -- 172.21.15.102:0/3957619969 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd7fc010ac0 con 0x7fd80c1022d0 2024-04-25T01:55:00.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.136+0000 7fd803400640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:00.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.137+0000 7fd803400640 1 -- 172.21.15.102:0/3957619969 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fd7fc0cf390 con 0x7fd80c1022d0 2024-04-25T01:55:00.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.137+0000 7fd809e00640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:00.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.137+0000 7fd809e00640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd80c06a4b0 tx=0x7fd7f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:00.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.141+0000 7fd803400640 1 -- 172.21.15.102:0/3957619969 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd7fc09bc50 con 0x7fd80c1022d0 2024-04-25T01:55:00.173 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:00 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2758760970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:00.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:00 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2275202406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:00.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.340+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd80c062370 con 0x7fd80c1022d0 2024-04-25T01:55:00.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.341+0000 7fd803400640 1 -- 172.21.15.102:0/3957619969 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fd7fc09ff50 con 0x7fd80c1022d0 2024-04-25T01:55:00.342 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-25T01:55:00.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.343+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7f00a2c10 msgr2=0x7fd7f00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.343+0000 7fd811400640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd80c06a4b0 tx=0x7fd7f8005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.343+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd80c1022d0 msgr2=0x7fd80c199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.343+0000 7fd811400640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd80c1022d0 0x7fd80c199d00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00e9f0 tx=0x7fd7fc00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.344+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 shutdown_connections 2024-04-25T01:55:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.344+0000 7fd811400640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7f00a2c10 0x7fd7f00a50d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.344+0000 7fd811400640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd80c107530 0x7fd80c1a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.344+0000 7fd811400640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd80c104c00 0x7fd80c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.344+0000 7fd811400640 1 --2- 172.21.15.102:0/3957619969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd80c1022d0 0x7fd80c199d00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.344+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 >> 172.21.15.102:0/3957619969 conn(0x7fd80c0fbc00 msgr2=0x7fd80c101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:00.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.344+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 shutdown_connections 2024-04-25T01:55:00.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.344+0000 7fd811400640 1 -- 172.21.15.102:0/3957619969 wait complete. 2024-04-25T01:55:00.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:55:00.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:00 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2758760970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:00.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:00 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2275202406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:00.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:00 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2758760970' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:00.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:00 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2275202406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:00.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.570+0000 7fd047200640 1 -- 172.21.15.102:0/2044776535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0481028c0 msgr2=0x7fd048102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.570+0000 7fd047200640 1 --2- 172.21.15.102:0/2044776535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0481028c0 0x7fd048102d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd030009900 tx=0x7fd03002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.570+0000 7fd047200640 1 -- 172.21.15.102:0/2044776535 shutdown_connections 2024-04-25T01:55:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.570+0000 7fd047200640 1 --2- 172.21.15.102:0/2044776535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048103280 0x7fd048109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.570+0000 7fd047200640 1 --2- 172.21.15.102:0/2044776535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0481028c0 0x7fd048102d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.570+0000 7fd047200640 1 --2- 172.21.15.102:0/2044776535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0481016d0 0x7fd048101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.570+0000 7fd047200640 1 -- 172.21.15.102:0/2044776535 >> 172.21.15.102:0/2044776535 conn(0x7fd0480fcc40 msgr2=0x7fd0480ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.570+0000 7fd047200640 1 -- 172.21.15.102:0/2044776535 shutdown_connections 2024-04-25T01:55:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.570+0000 7fd047200640 1 -- 172.21.15.102:0/2044776535 wait complete. 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.571+0000 7fd047200640 1 Processor -- start 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd047200640 1 -- start start 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd047200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0481016d0 0x7fd048199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd047200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0481028c0 0x7fd04819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd047200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd048103280 0x7fd0481a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd047200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd04810d2f0 con 0x7fd0481016d0 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd047200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd04810d170 con 0x7fd0481028c0 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd047200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd04810d470 con 0x7fd048103280 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd045e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0481016d0 0x7fd048199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd045e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0481016d0 0x7fd048199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58726/0 (socket says 172.21.15.102:58726) 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd045e00640 1 -- 172.21.15.102:0/502210731 learned_addr learned my addr 172.21.15.102:0/502210731 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd046800640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd048103280 0x7fd0481a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.572+0000 7fd03d400640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0481028c0 0x7fd04819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd045e00640 1 -- 172.21.15.102:0/502210731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd048103280 msgr2=0x7fd0481a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd045e00640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd048103280 0x7fd0481a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd045e00640 1 -- 172.21.15.102:0/502210731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0481028c0 msgr2=0x7fd04819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd045e00640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0481028c0 0x7fd04819a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd045e00640 1 -- 172.21.15.102:0/502210731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0481a17d0 con 0x7fd0481016d0 2024-04-25T01:55:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd045e00640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0481016d0 0x7fd048199bb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd03800eb70 tx=0x7fd03800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd03f400640 1 -- 172.21.15.102:0/502210731 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038003b70 con 0x7fd0481016d0 2024-04-25T01:55:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0481a1ac0 con 0x7fd0481016d0 2024-04-25T01:55:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd03f400640 1 -- 172.21.15.102:0/502210731 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd03800ee70 con 0x7fd0481016d0 2024-04-25T01:55:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd03f400640 1 -- 172.21.15.102:0/502210731 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038010830 con 0x7fd0481016d0 2024-04-25T01:55:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.573+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0481a1f90 con 0x7fd0481016d0 2024-04-25T01:55:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.575+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd048068480 con 0x7fd0481016d0 2024-04-25T01:55:00.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.576+0000 7fd03f400640 1 -- 172.21.15.102:0/502210731 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd038004020 con 0x7fd0481016d0 2024-04-25T01:55:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.578+0000 7fd03f400640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd02c0a2b90 0x7fd02c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.578+0000 7fd03f400640 1 -- 172.21.15.102:0/502210731 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fd0380cf720 con 0x7fd0481016d0 2024-04-25T01:55:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.578+0000 7fd03d400640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd02c0a2b90 0x7fd02c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.579+0000 7fd03d400640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd02c0a2b90 0x7fd02c0a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd0300314e0 tx=0x7fd030000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:00.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.582+0000 7fd03f400640 1 -- 172.21.15.102:0/502210731 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd03809c630 con 0x7fd0481016d0 2024-04-25T01:55:00.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.780+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd0481a2900 con 0x7fd0481016d0 2024-04-25T01:55:00.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.781+0000 7fd03f400640 1 -- 172.21.15.102:0/502210731 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fd03809bf30 con 0x7fd0481016d0 2024-04-25T01:55:00.782 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-25T01:55:00.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.783+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd02c0a2b90 msgr2=0x7fd02c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.783+0000 7fd047200640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd02c0a2b90 0x7fd02c0a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd0300314e0 tx=0x7fd030000bc0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.783+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0481016d0 msgr2=0x7fd048199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:00.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.783+0000 7fd047200640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0481016d0 0x7fd048199bb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd03800eb70 tx=0x7fd03800c6a0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.784+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 shutdown_connections 2024-04-25T01:55:00.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.784+0000 7fd047200640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd02c0a2b90 0x7fd02c0a5050 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.784+0000 7fd047200640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd048103280 0x7fd0481a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.784+0000 7fd047200640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0481028c0 0x7fd04819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.784+0000 7fd047200640 1 --2- 172.21.15.102:0/502210731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0481016d0 0x7fd048199bb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.784+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 >> 172.21.15.102:0/502210731 conn(0x7fd0480fcc40 msgr2=0x7fd048073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.784+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 shutdown_connections 2024-04-25T01:55:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:00.784+0000 7fd047200640 1 -- 172.21.15.102:0/502210731 wait complete. 2024-04-25T01:55:00.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:55:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.004+0000 7f87b1e00640 1 -- 172.21.15.102:0/4153882214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87ac1049b0 msgr2=0x7f87ac104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.004+0000 7f87b1e00640 1 --2- 172.21.15.102:0/4153882214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87ac1049b0 0x7f87ac104e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8798009900 tx=0x7f879802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.005+0000 7f87b1e00640 1 -- 172.21.15.102:0/4153882214 shutdown_connections 2024-04-25T01:55:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.005+0000 7f87b1e00640 1 --2- 172.21.15.102:0/4153882214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac105370 0x7f87ac10bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.005+0000 7f87b1e00640 1 --2- 172.21.15.102:0/4153882214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87ac1049b0 0x7f87ac104e30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.005+0000 7f87b1e00640 1 --2- 172.21.15.102:0/4153882214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87ac1037c0 0x7f87ac103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.005+0000 7f87b1e00640 1 -- 172.21.15.102:0/4153882214 >> 172.21.15.102:0/4153882214 conn(0x7f87ac0fed50 msgr2=0x7f87ac101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:01.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.005+0000 7f87b1e00640 1 -- 172.21.15.102:0/4153882214 shutdown_connections 2024-04-25T01:55:01.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.005+0000 7f87b1e00640 1 -- 172.21.15.102:0/4153882214 wait complete. 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.006+0000 7f87b1e00640 1 Processor -- start 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.006+0000 7f87b1e00640 1 -- start start 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.006+0000 7f87b1e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87ac1037c0 0x7f87ac199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.006+0000 7f87b1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac1049b0 0x7f87ac19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.006+0000 7f87b1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87ac105370 0x7f87ac1a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.006+0000 7f87b1e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac10f440 con 0x7f87ac1049b0 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.006+0000 7f87b1e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac10f2c0 con 0x7f87ac1037c0 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.006+0000 7f87b1e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac10f5c0 con 0x7f87ac105370 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.006+0000 7f87abe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac1049b0 0x7f87ac19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87abe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac1049b0 0x7f87ac19a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58740/0 (socket says 172.21.15.102:58740) 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87abe00640 1 -- 172.21.15.102:0/3441179543 learned_addr learned my addr 172.21.15.102:0/3441179543 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87b0a00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87ac1037c0 0x7f87ac199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87b1400640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87ac105370 0x7f87ac1a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87abe00640 1 -- 172.21.15.102:0/3441179543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87ac105370 msgr2=0x7f87ac1a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87abe00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87ac105370 0x7f87ac1a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87abe00640 1 -- 172.21.15.102:0/3441179543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87ac1037c0 msgr2=0x7f87ac199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87abe00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87ac1037c0 0x7f87ac199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87abe00640 1 -- 172.21.15.102:0/3441179543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ac1a1730 con 0x7f87ac1049b0 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87b0a00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87ac1037c0 0x7f87ac199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.007+0000 7f87abe00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac1049b0 0x7f87ac19a050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8798009230 tx=0x7f87980042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.008+0000 7f87a9600640 1 -- 172.21.15.102:0/3441179543 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879803f030 con 0x7f87ac1049b0 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.008+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ac1a19c0 con 0x7f87ac1049b0 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.008+0000 7f87a9600640 1 -- 172.21.15.102:0/3441179543 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8798003ce0 con 0x7f87ac1049b0 2024-04-25T01:55:01.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.008+0000 7f87a9600640 1 -- 172.21.15.102:0/3441179543 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798035070 con 0x7f87ac1049b0 2024-04-25T01:55:01.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.008+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ac1a1eb0 con 0x7f87ac1049b0 2024-04-25T01:55:01.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.009+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87ac10ab00 con 0x7f87ac1049b0 2024-04-25T01:55:01.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.011+0000 7f87a9600640 1 -- 172.21.15.102:0/3441179543 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8798042030 con 0x7f87ac1049b0 2024-04-25T01:55:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.013+0000 7f87a9600640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f87940a2b90 0x7f87940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.013+0000 7f87a9600640 1 -- 172.21.15.102:0/3441179543 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f87980ef5c0 con 0x7f87ac1049b0 2024-04-25T01:55:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.013+0000 7f87b0a00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f87940a2b90 0x7f87940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.014+0000 7f87b0a00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f87940a2b90 0x7f87940a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f87ac104810 tx=0x7f87a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.016+0000 7f87a9600640 1 -- 172.21.15.102:0/3441179543 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f87980bc4d0 con 0x7f87ac1049b0 2024-04-25T01:55:01.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.254+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87ac062340 con 0x7f87ac1049b0 2024-04-25T01:55:01.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.255+0000 7f87a9600640 1 -- 172.21.15.102:0/3441179543 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11896 (secure 0 0 0) 0x7f87980bbdd0 con 0x7f87ac1049b0 2024-04-25T01:55:01.256 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:01.256 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:52:49.089206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T01:55:01.256 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 22 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.257+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f87940a2b90 msgr2=0x7f87940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.257+0000 7f87b1e00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f87940a2b90 0x7f87940a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f87ac104810 tx=0x7f87a0005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.257+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac1049b0 msgr2=0x7f87ac19a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.257+0000 7f87b1e00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac1049b0 0x7f87ac19a050 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8798009230 tx=0x7f87980042f0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.257+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 shutdown_connections 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.257+0000 7f87b1e00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f87940a2b90 0x7f87940a5050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.257+0000 7f87b1e00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87ac105370 0x7f87ac1a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.257+0000 7f87b1e00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87ac1049b0 0x7f87ac19a050 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.258+0000 7f87b1e00640 1 --2- 172.21.15.102:0/3441179543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87ac1037c0 0x7f87ac199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.258+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 >> 172.21.15.102:0/3441179543 conn(0x7f87ac0fed50 msgr2=0x7f87ac107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.258+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 shutdown_connections 2024-04-25T01:55:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.258+0000 7f87b1e00640 1 -- 172.21.15.102:0/3441179543 wait complete. 2024-04-25T01:55:01.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:55:01.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:01 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3957619969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:01.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:01 smithi102 ceph-mon[27796]: pgmap v388: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:55:01.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:01 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/502210731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:01.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.483+0000 7fbfd8400640 1 -- 172.21.15.102:0/4205679588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0107da0 msgr2=0x7fbfd01081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.483+0000 7fbfd8400640 1 --2- 172.21.15.102:0/4205679588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0107da0 0x7fbfd01081c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009900 tx=0x7fbfc002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.483+0000 7fbfd8400640 1 -- 172.21.15.102:0/4205679588 shutdown_connections 2024-04-25T01:55:01.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.483+0000 7fbfd8400640 1 --2- 172.21.15.102:0/4205679588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd0109950 0x7fbfd01101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.483+0000 7fbfd8400640 1 --2- 172.21.15.102:0/4205679588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfd0108f90 0x7fbfd0109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.483+0000 7fbfd8400640 1 --2- 172.21.15.102:0/4205679588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0107da0 0x7fbfd01081c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.483+0000 7fbfd8400640 1 -- 172.21.15.102:0/4205679588 >> 172.21.15.102:0/4205679588 conn(0x7fbfd0076d60 msgr2=0x7fbfd0079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:01.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.483+0000 7fbfd8400640 1 -- 172.21.15.102:0/4205679588 shutdown_connections 2024-04-25T01:55:01.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.483+0000 7fbfd8400640 1 -- 172.21.15.102:0/4205679588 wait complete. 2024-04-25T01:55:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd8400640 1 Processor -- start 2024-04-25T01:55:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd8400640 1 -- start start 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd8400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfd0107da0 0x7fbfd019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd8400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0108f90 0x7fbfd019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd8400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd0109950 0x7fbfd01a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd8400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0113a10 con 0x7fbfd0108f90 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd8400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0113890 con 0x7fbfd0109950 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd8400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0113b90 con 0x7fbfd0107da0 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0108f90 0x7fbfd019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0108f90 0x7fbfd019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58758/0 (socket says 172.21.15.102:58758) 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd5000640 1 -- 172.21.15.102:0/155801525 learned_addr learned my addr 172.21.15.102:0/155801525 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd6400640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd0109950 0x7fbfd01a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.485+0000 7fbfd5a00640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfd0107da0 0x7fbfd019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfd5000640 1 -- 172.21.15.102:0/155801525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfd0107da0 msgr2=0x7fbfd019e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfd5000640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfd0107da0 0x7fbfd019e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfd5000640 1 -- 172.21.15.102:0/155801525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd0109950 msgr2=0x7fbfd01a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfd5000640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd0109950 0x7fbfd01a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfd5000640 1 -- 172.21.15.102:0/155801525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd01a5d60 con 0x7fbfd0108f90 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfd5a00640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfd0107da0 0x7fbfd019e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfd6400640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd0109950 0x7fbfd01a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfd5000640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0108f90 0x7fbfd019e680 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc00eb90 tx=0x7fbfbc00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfce000640 1 -- 172.21.15.102:0/155801525 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc004b20 con 0x7fbfd0108f90 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfce000640 1 -- 172.21.15.102:0/155801525 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfbc004cc0 con 0x7fbfd0108f90 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfce000640 1 -- 172.21.15.102:0/155801525 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc010720 con 0x7fbfd0108f90 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.486+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd01a5ff0 con 0x7fbfd0108f90 2024-04-25T01:55:01.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.487+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd01a64c0 con 0x7fbfd0108f90 2024-04-25T01:55:01.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:01 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3957619969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:01.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:01 smithi110 ceph-mon[32848]: pgmap v388: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:55:01.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:01 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/502210731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:01.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.488+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd010f130 con 0x7fbfd0108f90 2024-04-25T01:55:01.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.490+0000 7fbfce000640 1 -- 172.21.15.102:0/155801525 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbfbc005020 con 0x7fbfd0108f90 2024-04-25T01:55:01.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.492+0000 7fbfce000640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbfb40a2b90 0x7fbfb40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.492+0000 7fbfce000640 1 -- 172.21.15.102:0/155801525 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fbfbc0cf740 con 0x7fbfd0108f90 2024-04-25T01:55:01.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.492+0000 7fbfd5a00640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbfb40a2b90 0x7fbfb40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.493+0000 7fbfd5a00640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbfb40a2b90 0x7fbfb40a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0006010 tx=0x7fbfc0032cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:01.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.496+0000 7fbfce000640 1 -- 172.21.15.102:0/155801525 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbfbc09c650 con 0x7fbfd0108f90 2024-04-25T01:55:01.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:01 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3957619969' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:01.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:01 smithi193 ceph-mon[32981]: pgmap v388: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T01:55:01.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:01 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/502210731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:01.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.696+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbfd00623e0 con 0x7fbfd0108f90 2024-04-25T01:55:01.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.698+0000 7fbfce000640 1 -- 172.21.15.102:0/155801525 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22686 (secure 0 0 0) 0x7fbfbc09bf50 con 0x7fbfd0108f90 2024-04-25T01:55:01.700 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:01.700 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:53:43.851432+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:55:01.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbfb40a2b90 msgr2=0x7fbfb40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbfb40a2b90 0x7fbfb40a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0006010 tx=0x7fbfc0032cf0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0108f90 msgr2=0x7fbfd019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0108f90 0x7fbfd019e680 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc00eb90 tx=0x7fbfbc00ef90 comp rx=0 tx=0).stop 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 shutdown_connections 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fbfb40a2b90 0x7fbfb40a5050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfd0109950 0x7fbfd01a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd0108f90 0x7fbfd019e680 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 --2- 172.21.15.102:0/155801525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbfd0107da0 0x7fbfd019e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 >> 172.21.15.102:0/155801525 conn(0x7fbfd0076d60 msgr2=0x7fbfd010c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 shutdown_connections 2024-04-25T01:55:01.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.701+0000 7fbfd8400640 1 -- 172.21.15.102:0/155801525 wait complete. 2024-04-25T01:55:01.722 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:41.585137+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:43.022666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:55:01.722 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:55:01.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.923+0000 7fa87ea00640 1 -- 172.21.15.102:0/351489327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 msgr2=0x7fa878109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.923+0000 7fa87ea00640 1 --2- 172.21.15.102:0/351489327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 0x7fa878109410 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa868009ca0 tx=0x7fa86802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.924+0000 7fa87ea00640 1 -- 172.21.15.102:0/351489327 shutdown_connections 2024-04-25T01:55:01.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.924+0000 7fa87ea00640 1 --2- 172.21.15.102:0/351489327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa878109950 0x7fa8781101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.924+0000 7fa87ea00640 1 --2- 172.21.15.102:0/351489327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 0x7fa878109410 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.924+0000 7fa87ea00640 1 --2- 172.21.15.102:0/351489327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa878107da0 0x7fa8781081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.924+0000 7fa87ea00640 1 -- 172.21.15.102:0/351489327 >> 172.21.15.102:0/351489327 conn(0x7fa878076d40 msgr2=0x7fa878079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:01.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.924+0000 7fa87ea00640 1 -- 172.21.15.102:0/351489327 shutdown_connections 2024-04-25T01:55:01.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.924+0000 7fa87ea00640 1 -- 172.21.15.102:0/351489327 wait complete. 2024-04-25T01:55:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa87ea00640 1 Processor -- start 2024-04-25T01:55:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa87ea00640 1 -- start start 2024-04-25T01:55:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa87ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa878107da0 0x7fa87819e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa87ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 0x7fa87819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa87ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa878109950 0x7fa8781a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa87ea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa878113a50 con 0x7fa878108f90 2024-04-25T01:55:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa87ea00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa8781138d0 con 0x7fa878109950 2024-04-25T01:55:01.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa87ea00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa878113bd0 con 0x7fa878107da0 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa877400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 0x7fa87819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa877400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 0x7fa87819e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58780/0 (socket says 172.21.15.102:58780) 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa877400640 1 -- 172.21.15.102:0/2905719771 learned_addr learned my addr 172.21.15.102:0/2905719771 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa877e00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa878107da0 0x7fa87819e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.925+0000 7fa87ca00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa878109950 0x7fa8781a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.926+0000 7fa877400640 1 -- 172.21.15.102:0/2905719771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa878107da0 msgr2=0x7fa87819e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.926+0000 7fa877400640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa878107da0 0x7fa87819e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.926+0000 7fa877400640 1 -- 172.21.15.102:0/2905719771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa878109950 msgr2=0x7fa8781a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.926+0000 7fa877400640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa878109950 0x7fa8781a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.926+0000 7fa877400640 1 -- 172.21.15.102:0/2905719771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8781a5ce0 con 0x7fa878108f90 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.926+0000 7fa87ca00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa878109950 0x7fa8781a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.926+0000 7fa877400640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 0x7fa87819e600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa868000f80 tx=0x7fa8680319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.926+0000 7fa874c00640 1 -- 172.21.15.102:0/2905719771 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868003b70 con 0x7fa878108f90 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.927+0000 7fa874c00640 1 -- 172.21.15.102:0/2905719771 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa868003d10 con 0x7fa878108f90 2024-04-25T01:55:01.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.927+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8781a5f70 con 0x7fa878108f90 2024-04-25T01:55:01.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.927+0000 7fa874c00640 1 -- 172.21.15.102:0/2905719771 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86802fc70 con 0x7fa878108f90 2024-04-25T01:55:01.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.927+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8781a6480 con 0x7fa878108f90 2024-04-25T01:55:01.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.928+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa87810f0c0 con 0x7fa878108f90 2024-04-25T01:55:01.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.930+0000 7fa874c00640 1 -- 172.21.15.102:0/2905719771 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa868004040 con 0x7fa878108f90 2024-04-25T01:55:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.931+0000 7fa874c00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa85c0a2c10 0x7fa85c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.931+0000 7fa874c00640 1 -- 172.21.15.102:0/2905719771 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fa8680f04b0 con 0x7fa878108f90 2024-04-25T01:55:01.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.931+0000 7fa877e00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa85c0a2c10 0x7fa85c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:01.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.932+0000 7fa877e00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa85c0a2c10 0x7fa85c0a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa878108df0 tx=0x7fa860005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:01.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:01.935+0000 7fa874c00640 1 -- 172.21.15.102:0/2905719771 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa8680bd4e0 con 0x7fa878108f90 2024-04-25T01:55:02.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.137+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa8780623e0 con 0x7fa878108f90 2024-04-25T01:55:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.140+0000 7fa874c00640 1 -- 172.21.15.102:0/2905719771 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fa8680bcde0 con 0x7fa878108f90 2024-04-25T01:55:02.141 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:55:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.143+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa85c0a2c10 msgr2=0x7fa85c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.143+0000 7fa87ea00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa85c0a2c10 0x7fa85c0a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa878108df0 tx=0x7fa860005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.143+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 msgr2=0x7fa87819e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.143+0000 7fa87ea00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 0x7fa87819e600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa868000f80 tx=0x7fa8680319f0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.144+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 shutdown_connections 2024-04-25T01:55:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.144+0000 7fa87ea00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa85c0a2c10 0x7fa85c0a50d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.144+0000 7fa87ea00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa878109950 0x7fa8781a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.144+0000 7fa87ea00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa878108f90 0x7fa87819e600 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.144+0000 7fa87ea00640 1 --2- 172.21.15.102:0/2905719771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa878107da0 0x7fa87819e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.144+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 >> 172.21.15.102:0/2905719771 conn(0x7fa878076d40 msgr2=0x7fa87810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:02.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.144+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 shutdown_connections 2024-04-25T01:55:02.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.144+0000 7fa87ea00640 1 -- 172.21.15.102:0/2905719771 wait complete. 2024-04-25T01:55:02.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:55:02.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.369+0000 7f0d14600640 1 -- 172.21.15.102:0/534531879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d0c102260 msgr2=0x7f0d0c104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.369+0000 7f0d14600640 1 --2- 172.21.15.102:0/534531879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d0c102260 0x7f0d0c104650 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0d00009900 tx=0x7f0d0002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.369+0000 7f0d14600640 1 -- 172.21.15.102:0/534531879 shutdown_connections 2024-04-25T01:55:02.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.369+0000 7f0d14600640 1 --2- 172.21.15.102:0/534531879 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d0c1074c0 0x7f0d0c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.369+0000 7f0d14600640 1 --2- 172.21.15.102:0/534531879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c104b90 0x7f0d0c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.369+0000 7f0d14600640 1 --2- 172.21.15.102:0/534531879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d0c102260 0x7f0d0c104650 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.369+0000 7f0d14600640 1 -- 172.21.15.102:0/534531879 >> 172.21.15.102:0/534531879 conn(0x7f0d0c0fbbf0 msgr2=0x7f0d0c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:02.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.369+0000 7f0d14600640 1 -- 172.21.15.102:0/534531879 shutdown_connections 2024-04-25T01:55:02.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.369+0000 7f0d14600640 1 -- 172.21.15.102:0/534531879 wait complete. 2024-04-25T01:55:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.370+0000 7f0d14600640 1 Processor -- start 2024-04-25T01:55:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.370+0000 7f0d14600640 1 -- start start 2024-04-25T01:55:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d14600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d0c102260 0x7f0d0c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d14600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c104b90 0x7f0d0c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d14600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d0c1074c0 0x7f0d0c1a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d14600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c10d470 con 0x7f0d0c1074c0 2024-04-25T01:55:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d14600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c10d2f0 con 0x7f0d0c102260 2024-04-25T01:55:02.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d14600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0c10d5f0 con 0x7f0d0c104b90 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d11c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d0c102260 0x7f0d0c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d11200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c104b90 0x7f0d0c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d11c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d0c102260 0x7f0d0c199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58058/0 (socket says 172.21.15.102:58058) 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d11c00640 1 -- 172.21.15.102:0/1356172385 learned_addr learned my addr 172.21.15.102:0/1356172385 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d12600640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d0c1074c0 0x7f0d0c1a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d11c00640 1 -- 172.21.15.102:0/1356172385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c104b90 msgr2=0x7f0d0c19a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d11c00640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c104b90 0x7f0d0c19a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d11c00640 1 -- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d0c1074c0 msgr2=0x7f0d0c1a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d11c00640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d0c1074c0 0x7f0d0c1a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.371+0000 7f0d11c00640 1 -- 172.21.15.102:0/1356172385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0c1a19d0 con 0x7f0d0c102260 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.372+0000 7f0d11c00640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d0c102260 0x7f0d0c199dc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0d00031440 tx=0x7f0d00032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.372+0000 7f0d0a000640 1 -- 172.21.15.102:0/1356172385 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0003f030 con 0x7f0d0c102260 2024-04-25T01:55:02.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.372+0000 7f0d0a000640 1 -- 172.21.15.102:0/1356172385 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d00003ce0 con 0x7f0d0c102260 2024-04-25T01:55:02.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.372+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0c1a1c60 con 0x7f0d0c102260 2024-04-25T01:55:02.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.372+0000 7f0d0a000640 1 -- 172.21.15.102:0/1356172385 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00035070 con 0x7f0d0c102260 2024-04-25T01:55:02.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.372+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d0c1a2150 con 0x7f0d0c102260 2024-04-25T01:55:02.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.373+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d0c108fb0 con 0x7f0d0c102260 2024-04-25T01:55:02.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.375+0000 7f0d0a000640 1 -- 172.21.15.102:0/1356172385 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0d00042030 con 0x7f0d0c102260 2024-04-25T01:55:02.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.377+0000 7f0d0a000640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0cf40a2b90 0x7f0cf40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:02.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.377+0000 7f0d0a000640 1 -- 172.21.15.102:0/1356172385 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f0d000ef5c0 con 0x7f0d0c102260 2024-04-25T01:55:02.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.377+0000 7f0d11200640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0cf40a2b90 0x7f0cf40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:02.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.378+0000 7f0d11200640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0cf40a2b90 0x7f0cf40a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0693d0 tx=0x7f0cf8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:02.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.381+0000 7f0d0a000640 1 -- 172.21.15.102:0/1356172385 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0d000bc4d0 con 0x7f0d0c102260 2024-04-25T01:55:02.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:02 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3441179543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:02.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:02 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/155801525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:02 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3441179543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:02.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:02 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/155801525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:02.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:02 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3441179543' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:02.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:02 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/155801525' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:02.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.579+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0d0c062340 con 0x7f0d0c102260 2024-04-25T01:55:02.581 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-25T01:55:02.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.580+0000 7f0d0a000640 1 -- 172.21.15.102:0/1356172385 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f0d000bbdd0 con 0x7f0d0c102260 2024-04-25T01:55:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.582+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0cf40a2b90 msgr2=0x7f0cf40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.582+0000 7f0d14600640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0cf40a2b90 0x7f0cf40a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0693d0 tx=0x7f0cf8005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.582+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d0c102260 msgr2=0x7f0d0c199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.582+0000 7f0d14600640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d0c102260 0x7f0d0c199dc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0d00031440 tx=0x7f0d00032ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.583+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 shutdown_connections 2024-04-25T01:55:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.583+0000 7f0d14600640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f0cf40a2b90 0x7f0cf40a5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.583+0000 7f0d14600640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d0c1074c0 0x7f0d0c1a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.583+0000 7f0d14600640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c104b90 0x7f0d0c19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.583+0000 7f0d14600640 1 --2- 172.21.15.102:0/1356172385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d0c102260 0x7f0d0c199dc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.583+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 >> 172.21.15.102:0/1356172385 conn(0x7f0d0c0fbbf0 msgr2=0x7f0d0c101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:02.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.583+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 shutdown_connections 2024-04-25T01:55:02.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.583+0000 7f0d14600640 1 -- 172.21.15.102:0/1356172385 wait complete. 2024-04-25T01:55:02.603 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:55:02.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.816+0000 7f7cbbe00640 1 -- 172.21.15.102:0/182006022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb4104b90 msgr2=0x7f7cb4105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.816+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/182006022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb4104b90 0x7f7cb4105010 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009860 tx=0x7f7ca002a340 comp rx=0 tx=0).stop 2024-04-25T01:55:02.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.816+0000 7f7cbbe00640 1 -- 172.21.15.102:0/182006022 shutdown_connections 2024-04-25T01:55:02.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.816+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/182006022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb4105550 0x7f7cb410bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.816+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/182006022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb4104b90 0x7f7cb4105010 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.816+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/182006022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb41039a0 0x7f7cb4103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.816+0000 7f7cbbe00640 1 -- 172.21.15.102:0/182006022 >> 172.21.15.102:0/182006022 conn(0x7f7cb40fef30 msgr2=0x7f7cb4101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:02.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.816+0000 7f7cbbe00640 1 -- 172.21.15.102:0/182006022 shutdown_connections 2024-04-25T01:55:02.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.816+0000 7f7cbbe00640 1 -- 172.21.15.102:0/182006022 wait complete. 2024-04-25T01:55:02.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.818+0000 7f7cbbe00640 1 Processor -- start 2024-04-25T01:55:02.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.818+0000 7f7cbbe00640 1 -- start start 2024-04-25T01:55:02.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.818+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb41039a0 0x7f7cb4199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:02.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.818+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb4104b90 0x7f7cb419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:02.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.818+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4105550 0x7f7cb41a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:02.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cbbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb410f620 con 0x7f7cb4105550 2024-04-25T01:55:02.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cbbe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb410f4a0 con 0x7f7cb4104b90 2024-04-25T01:55:02.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cbbe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb410f7a0 con 0x7f7cb41039a0 2024-04-25T01:55:02.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb41039a0 0x7f7cb4199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb41039a0 0x7f7cb4199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44822/0 (socket says 172.21.15.102:44822) 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9400640 1 -- 172.21.15.102:0/775487224 learned_addr learned my addr 172.21.15.102:0/775487224 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb8a00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb4104b90 0x7f7cb419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9e00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4105550 0x7f7cb41a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9400640 1 -- 172.21.15.102:0/775487224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb4104b90 msgr2=0x7f7cb419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9400640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb4104b90 0x7f7cb419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9400640 1 -- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4105550 msgr2=0x7f7cb41a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9400640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4105550 0x7f7cb41a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9400640 1 -- 172.21.15.102:0/775487224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb41a1900 con 0x7f7cb41039a0 2024-04-25T01:55:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.819+0000 7f7cb9400640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb41039a0 0x7f7cb4199cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800e9f0 tx=0x7f7ca800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:02.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.820+0000 7f7cb2000640 1 -- 172.21.15.102:0/775487224 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca800ccc0 con 0x7f7cb41039a0 2024-04-25T01:55:02.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.820+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb41a1bf0 con 0x7f7cb41039a0 2024-04-25T01:55:02.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.820+0000 7f7cb2000640 1 -- 172.21.15.102:0/775487224 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ca80035f0 con 0x7f7cb41039a0 2024-04-25T01:55:02.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.820+0000 7f7cb2000640 1 -- 172.21.15.102:0/775487224 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca800ce60 con 0x7f7cb41039a0 2024-04-25T01:55:02.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.820+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb41a2160 con 0x7f7cb41039a0 2024-04-25T01:55:02.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.821+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb410acc0 con 0x7f7cb41039a0 2024-04-25T01:55:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.823+0000 7f7cb2000640 1 -- 172.21.15.102:0/775487224 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f7ca80037a0 con 0x7f7cb41039a0 2024-04-25T01:55:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.824+0000 7f7cb2000640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c9c0a2b90 0x7f7c9c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.825+0000 7f7cb2000640 1 -- 172.21.15.102:0/775487224 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f7ca80cf2b0 con 0x7f7cb41039a0 2024-04-25T01:55:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.825+0000 7f7cb8a00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c9c0a2b90 0x7f7c9c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.825+0000 7f7cb8a00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c9c0a2b90 0x7f7c9c0a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009830 tx=0x7f7ca0030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:02.829+0000 7f7cb2000640 1 -- 172.21.15.102:0/775487224 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7ca809c1c0 con 0x7f7cb41039a0 2024-04-25T01:55:03.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.029+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7cb402d170 con 0x7f7cb41039a0 2024-04-25T01:55:03.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.030+0000 7f7cb2000640 1 -- 172.21.15.102:0/775487224 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f7ca809bac0 con 0x7f7cb41039a0 2024-04-25T01:55:03.031 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-25T01:55:03.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c9c0a2b90 msgr2=0x7f7c9c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c9c0a2b90 0x7f7c9c0a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009830 tx=0x7f7ca0030550 comp rx=0 tx=0).stop 2024-04-25T01:55:03.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb41039a0 msgr2=0x7f7cb4199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb41039a0 0x7f7cb4199cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800e9f0 tx=0x7f7ca800eeb0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 shutdown_connections 2024-04-25T01:55:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f7c9c0a2b90 0x7f7c9c0a5050 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cb4105550 0x7f7cb41a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb4104b90 0x7f7cb419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 --2- 172.21.15.102:0/775487224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cb41039a0 0x7f7cb4199cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 >> 172.21.15.102:0/775487224 conn(0x7f7cb40fef30 msgr2=0x7f7cb4107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 shutdown_connections 2024-04-25T01:55:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.033+0000 7f7cbbe00640 1 -- 172.21.15.102:0/775487224 wait complete. 2024-04-25T01:55:03.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:55:03.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.252+0000 7fde52000640 1 -- 172.21.15.102:0/401500747 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c030ac0 con 0x7fde5410b820 2024-04-25T01:55:03.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.252+0000 7fde5a200640 1 -- 172.21.15.102:0/401500747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5410b820 msgr2=0x7fde5410dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.252+0000 7fde5a200640 1 --2- 172.21.15.102:0/401500747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5410b820 0x7fde5410dc10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fde3c00aa10 tx=0x7fde3c02b720 comp rx=0 tx=0).stop 2024-04-25T01:55:03.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.253+0000 7fde5a200640 1 -- 172.21.15.102:0/401500747 shutdown_connections 2024-04-25T01:55:03.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.253+0000 7fde5a200640 1 --2- 172.21.15.102:0/401500747 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde5410e280 0x7fde54110740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.253+0000 7fde5a200640 1 --2- 172.21.15.102:0/401500747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5410b820 0x7fde5410dc10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.253+0000 7fde5a200640 1 --2- 172.21.15.102:0/401500747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde5406a590 0x7fde5410b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.253+0000 7fde5a200640 1 -- 172.21.15.102:0/401500747 >> 172.21.15.102:0/401500747 conn(0x7fde5406e980 msgr2=0x7fde54070da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.253+0000 7fde5a200640 1 -- 172.21.15.102:0/401500747 shutdown_connections 2024-04-25T01:55:03.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.253+0000 7fde5a200640 1 -- 172.21.15.102:0/401500747 wait complete. 2024-04-25T01:55:03.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde5a200640 1 Processor -- start 2024-04-25T01:55:03.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde5a200640 1 -- start start 2024-04-25T01:55:03.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde5a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5406a590 0x7fde541a2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde5a200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde5410b820 0x7fde541a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde5a200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde5410e280 0x7fde541a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde5a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde54113560 con 0x7fde5406a590 2024-04-25T01:55:03.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde5a200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fde541133e0 con 0x7fde5410b820 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde5a200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde541136e0 con 0x7fde5410e280 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde53400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5406a590 0x7fde541a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde53400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5406a590 0x7fde541a2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35194/0 (socket says 172.21.15.102:35194) 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde53400640 1 -- 172.21.15.102:0/1021369720 learned_addr learned my addr 172.21.15.102:0/1021369720 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde53e00640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde5410e280 0x7fde541a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.254+0000 7fde52a00640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde5410b820 0x7fde541a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde53400640 1 -- 172.21.15.102:0/1021369720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde5410e280 msgr2=0x7fde541a9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde53400640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde5410e280 0x7fde541a9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde53400640 1 -- 172.21.15.102:0/1021369720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde5410b820 msgr2=0x7fde541a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde53400640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde5410b820 0x7fde541a2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde53400640 1 -- 172.21.15.102:0/1021369720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde541aa130 con 0x7fde5406a590 2024-04-25T01:55:03.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde53400640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5406a590 0x7fde541a2560 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fde4400d950 tx=0x7fde4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde4be00640 1 -- 172.21.15.102:0/1021369720 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde44017070 con 0x7fde5406a590 2024-04-25T01:55:03.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde4be00640 1 -- 172.21.15.102:0/1021369720 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde44009c30 con 0x7fde5406a590 2024-04-25T01:55:03.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde541aa420 con 0x7fde5406a590 2024-04-25T01:55:03.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde4be00640 1 -- 172.21.15.102:0/1021369720 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4400f600 con 0x7fde5406a590 2024-04-25T01:55:03.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.255+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde541aa940 con 0x7fde5406a590 2024-04-25T01:55:03.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.257+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde541aaba0 con 0x7fde5406a590 2024-04-25T01:55:03.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.258+0000 7fde4be00640 1 -- 172.21.15.102:0/1021369720 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fde4401a070 con 0x7fde5406a590 2024-04-25T01:55:03.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.260+0000 7fde4be00640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fde380a2c10 0x7fde380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.260+0000 7fde4be00640 1 -- 172.21.15.102:0/1021369720 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fde440ce9a0 con 0x7fde5406a590 2024-04-25T01:55:03.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.260+0000 7fde52a00640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fde380a2c10 0x7fde380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.261+0000 7fde52a00640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fde380a2c10 0x7fde380a50d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fde3c005ab0 tx=0x7fde3c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.264+0000 7fde4be00640 1 -- 172.21.15.102:0/1021369720 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fde44009de0 con 0x7fde5406a590 2024-04-25T01:55:03.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:03 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2905719771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:03.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:03 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1356172385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:03.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:03 smithi102 ceph-mon[27796]: pgmap v389: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:55:03.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:03 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/775487224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:03.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:03 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2905719771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:03.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:03 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1356172385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:03.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:03 smithi110 ceph-mon[32848]: pgmap v389: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:55:03.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:03 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/775487224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:03.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.503+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde541aae10 con 0x7fde5406a590 2024-04-25T01:55:03.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.505+0000 7fde4be00640 1 -- 172.21.15.102:0/1021369720 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11896 (secure 0 0 0) 0x7fde4409b2c0 con 0x7fde5406a590 2024-04-25T01:55:03.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:03 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2905719771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:03.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:03 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1356172385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:03.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:03 smithi193 ceph-mon[32981]: pgmap v389: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T01:55:03.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:03 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/775487224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:03.507 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:03.507 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:52:49.089206+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T01:55:03.507 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 22 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fde380a2c10 msgr2=0x7fde380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fde380a2c10 0x7fde380a50d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fde3c005ab0 tx=0x7fde3c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5406a590 msgr2=0x7fde541a2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5406a590 0x7fde541a2560 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fde4400d950 tx=0x7fde4400de10 comp rx=0 tx=0).stop 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 shutdown_connections 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fde380a2c10 0x7fde380a50d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde5410e280 0x7fde541a9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde5410b820 0x7fde541a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 --2- 172.21.15.102:0/1021369720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde5406a590 0x7fde541a2560 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.507+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 >> 172.21.15.102:0/1021369720 conn(0x7fde5406e980 msgr2=0x7fde5410e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.508+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 shutdown_connections 2024-04-25T01:55:03.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.508+0000 7fde5a200640 1 -- 172.21.15.102:0/1021369720 wait complete. 2024-04-25T01:55:03.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:55:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.714+0000 7f15b9a00640 1 -- 172.21.15.102:0/647409093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b41037e0 msgr2=0x7f15b4103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.714+0000 7f15b9a00640 1 --2- 172.21.15.102:0/647409093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b41037e0 0x7f15b4103c00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009900 tx=0x7f15a402a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.714+0000 7f15b9a00640 1 -- 172.21.15.102:0/647409093 shutdown_connections 2024-04-25T01:55:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.714+0000 7f15b9a00640 1 --2- 172.21.15.102:0/647409093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15b4105390 0x7f15b410bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.714+0000 7f15b9a00640 1 --2- 172.21.15.102:0/647409093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15b41049d0 0x7f15b4104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.714+0000 7f15b9a00640 1 --2- 172.21.15.102:0/647409093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b41037e0 0x7f15b4103c00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.714+0000 7f15b9a00640 1 -- 172.21.15.102:0/647409093 >> 172.21.15.102:0/647409093 conn(0x7f15b40fed50 msgr2=0x7f15b4101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.714+0000 7f15b9a00640 1 -- 172.21.15.102:0/647409093 shutdown_connections 2024-04-25T01:55:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.714+0000 7f15b9a00640 1 -- 172.21.15.102:0/647409093 wait complete. 2024-04-25T01:55:03.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.715+0000 7f15b9a00640 1 Processor -- start 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b9a00640 1 -- start start 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b9a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15b41037e0 0x7f15b4199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15b41049d0 0x7f15b419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b9a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b4105390 0x7f15b41a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b9a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f15b410f480 con 0x7f15b41049d0 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b9a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f15b410f300 con 0x7f15b41037e0 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b9a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f15b410f600 con 0x7f15b4105390 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15b41049d0 0x7f15b419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15b41049d0 0x7f15b419a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35210/0 (socket says 172.21.15.102:35210) 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b3400640 1 -- 172.21.15.102:0/902718176 learned_addr learned my addr 172.21.15.102:0/902718176 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b3e00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15b41037e0 0x7f15b4199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b9000640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b4105390 0x7f15b41a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b3400640 1 -- 172.21.15.102:0/902718176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b4105390 msgr2=0x7f15b41a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b3400640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b4105390 0x7f15b41a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b3400640 1 -- 172.21.15.102:0/902718176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15b41037e0 msgr2=0x7f15b4199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b3400640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15b41037e0 0x7f15b4199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b3400640 1 -- 172.21.15.102:0/902718176 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b41a1710 con 0x7f15b41049d0 2024-04-25T01:55:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.716+0000 7f15b9000640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b4105390 0x7f15b41a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.717+0000 7f15b3400640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15b41049d0 0x7f15b419a030 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f15a000e9f0 tx=0x7f15a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.717+0000 7f15b0c00640 1 -- 172.21.15.102:0/902718176 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a000cd80 con 0x7f15b41049d0 2024-04-25T01:55:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.717+0000 7f15b0c00640 1 -- 172.21.15.102:0/902718176 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a0010910 con 0x7f15b41049d0 2024-04-25T01:55:03.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.717+0000 7f15b0c00640 1 -- 172.21.15.102:0/902718176 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a0018990 con 0x7f15b41049d0 2024-04-25T01:55:03.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.717+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b41a1a00 con 0x7f15b41049d0 2024-04-25T01:55:03.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.717+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b41a1f50 con 0x7f15b41049d0 2024-04-25T01:55:03.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.718+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b410ab00 con 0x7f15b41049d0 2024-04-25T01:55:03.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.721+0000 7f15b0c00640 1 -- 172.21.15.102:0/902718176 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f15a0010ac0 con 0x7f15b41049d0 2024-04-25T01:55:03.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.722+0000 7f15b0c00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f15980a2ce0 0x7f15980a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:03.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.722+0000 7f15b0c00640 1 -- 172.21.15.102:0/902718176 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f15a00cf2c0 con 0x7f15b41049d0 2024-04-25T01:55:03.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.722+0000 7f15b3e00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f15980a2ce0 0x7f15980a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:03.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.723+0000 7f15b3e00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f15980a2ce0 0x7f15980a51a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f15a402a9e0 tx=0x7f15a40315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:03.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.727+0000 7f15b0c00640 1 -- 172.21.15.102:0/902718176 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f15a009c230 con 0x7f15b41049d0 2024-04-25T01:55:03.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.928+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15b4062340 con 0x7f15b41049d0 2024-04-25T01:55:03.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.930+0000 7f15b0c00640 1 -- 172.21.15.102:0/902718176 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+22686 (secure 0 0 0) 0x7f15a009bb30 con 0x7f15b41049d0 2024-04-25T01:55:03.932 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:03.932 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:53:43.851432+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f15980a2ce0 msgr2=0x7f15980a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f15980a2ce0 0x7f15980a51a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f15a402a9e0 tx=0x7f15a40315b0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15b41049d0 msgr2=0x7f15b419a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15b41049d0 0x7f15b419a030 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f15a000e9f0 tx=0x7f15a000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 shutdown_connections 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f15980a2ce0 0x7f15980a51a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b4105390 0x7f15b41a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15b41049d0 0x7f15b419a030 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 --2- 172.21.15.102:0/902718176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15b41037e0 0x7f15b4199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.933+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 >> 172.21.15.102:0/902718176 conn(0x7f15b40fed50 msgr2=0x7f15b4107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.934+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 shutdown_connections 2024-04-25T01:55:03.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:03.934+0000 7f15b9a00640 1 -- 172.21.15.102:0/902718176 wait complete. 2024-04-25T01:55:03.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T01:52:43.526145+0000', 'modified': '2024-04-25T01:52:49.089206+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24587, 'mds_1': 24541, 'mds_2': 24533}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6835/4057730311', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6834', 'nonce': 4057730311}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 4057730311}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/1885428891', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 1885428891}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 1885428891}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/3613019309', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 3613019309}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 3613019309}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.110:6837/1533256041', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 1533256041}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 1533256041}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.193:6839/2484219231', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 2484219231}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 2484219231}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24587': {'gid': 24587, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.110:6839/4063392981', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6838', 'nonce': 4063392981}, {'type': 'v1', 'addr': '172.21.15.110:6839', 'nonce': 4063392981}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24587, 'qdb_cluster': [24533, 24541, 24587]} 2024-04-25T01:55:03.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T01:55:03.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T01:55:04.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.156+0000 7f723fe00640 1 -- 172.21.15.102:0/2203994660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7240107c80 msgr2=0x7f72401080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.156+0000 7f723fe00640 1 --2- 172.21.15.102:0/2203994660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7240107c80 0x7f72401080a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7230009900 tx=0x7f723002a410 comp rx=0 tx=0).stop 2024-04-25T01:55:04.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.156+0000 7f723fe00640 1 -- 172.21.15.102:0/2203994660 shutdown_connections 2024-04-25T01:55:04.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.156+0000 7f723fe00640 1 --2- 172.21.15.102:0/2203994660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240109830 0x7f72401100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.156+0000 7f723fe00640 1 --2- 172.21.15.102:0/2203994660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7240108e70 0x7f72401092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.156+0000 7f723fe00640 1 --2- 172.21.15.102:0/2203994660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7240107c80 0x7f72401080a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.156+0000 7f723fe00640 1 -- 172.21.15.102:0/2203994660 >> 172.21.15.102:0/2203994660 conn(0x7f7240076c10 msgr2=0x7f7240079050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.156+0000 7f723fe00640 1 -- 172.21.15.102:0/2203994660 shutdown_connections 2024-04-25T01:55:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.156+0000 7f723fe00640 1 -- 172.21.15.102:0/2203994660 wait complete. 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.157+0000 7f723fe00640 1 Processor -- start 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.157+0000 7f723fe00640 1 -- start start 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.157+0000 7f723fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240107c80 0x7f724019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.157+0000 7f723fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7240108e70 0x7f724019e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.157+0000 7f723fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7240109830 0x7f72401a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.157+0000 7f723fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7240113920 con 0x7f7240107c80 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.157+0000 7f723fe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f72401137a0 con 0x7f7240109830 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.157+0000 7f723fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7240113aa0 con 0x7f7240108e70 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.157+0000 7f723ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240107c80 0x7f724019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240107c80 0x7f724019df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35232/0 (socket says 172.21.15.102:35232) 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723ea00640 1 -- 172.21.15.102:0/3047136641 learned_addr learned my addr 172.21.15.102:0/3047136641 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723e000640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7240108e70 0x7f724019e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723f400640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7240109830 0x7f72401a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723ea00640 1 -- 172.21.15.102:0/3047136641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7240108e70 msgr2=0x7f724019e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723ea00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7240108e70 0x7f724019e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723ea00640 1 -- 172.21.15.102:0/3047136641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7240109830 msgr2=0x7f72401a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723ea00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7240109830 0x7f72401a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723ea00640 1 -- 172.21.15.102:0/3047136641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72401a5b50 con 0x7f7240107c80 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.158+0000 7f723ea00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240107c80 0x7f724019df80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7230031440 tx=0x7f7230032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.159+0000 7f7237400640 1 -- 172.21.15.102:0/3047136641 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723003f030 con 0x7f7240107c80 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.159+0000 7f7237400640 1 -- 172.21.15.102:0/3047136641 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7230003ce0 con 0x7f7240107c80 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.159+0000 7f723fe00640 1 -- 172.21.15.102:0/3047136641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72401a5de0 con 0x7f7240107c80 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.159+0000 7f7237400640 1 -- 172.21.15.102:0/3047136641 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230035070 con 0x7f7240107c80 2024-04-25T01:55:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.159+0000 7f723fe00640 1 -- 172.21.15.102:0/3047136641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72401a6240 con 0x7f7240107c80 2024-04-25T01:55:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.162+0000 7f7237400640 1 -- 172.21.15.102:0/3047136641 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f7230042030 con 0x7f7240107c80 2024-04-25T01:55:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.162+0000 7f7248bdc740 1 Processor -- start 2024-04-25T01:55:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.162+0000 7f7248bdc740 1 -- start start 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.162+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.162+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e967f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.162+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.162+0000 7f7248bdc740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555e966f4ed0 con 0x555e967fa4d0 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.162+0000 7f7248bdc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555e96757df0 con 0x555e967f4970 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.162+0000 7f7248bdc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555e967008d0 con 0x555e967f5360 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967f4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:37188/0 (socket says 172.21.15.102:37188) 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723f400640 1 -- 172.21.15.102:0/3389501722 learned_addr learned my addr 172.21.15.102:0/3389501722 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723ea00640 1 --2- 172.21.15.102:0/3389501722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e967f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723f400640 1 -- 172.21.15.102:0/3389501722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 msgr2=0x555e967f9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723f400640 1 --2- 172.21.15.102:0/3389501722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e967f9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723f400640 1 -- 172.21.15.102:0/3389501722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 msgr2=0x555e967fc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723f400640 1 --2- 172.21.15.102:0/3389501722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723f400640 1 -- 172.21.15.102:0/3389501722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e9646c2d0 con 0x555e967f4970 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723e000640 1 --2- 172.21.15.102:0/3389501722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f7237400640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f72240a2c10 0x7f72240a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f723f400640 1 --2- 172.21.15.102:0/3389501722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967f4d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7238009fd0 tx=0x7f72380118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5355735fd0bed5d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.163+0000 7f7237400640 1 -- 172.21.15.102:0/3047136641 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f72300ef820 con 0x7f7240107c80 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f723ea00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f72240a2c10 0x7f72240a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f722be00640 1 -- 172.21.15.102:0/3389501722 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238004d20 con 0x555e967f4970 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f722be00640 1 -- 172.21.15.102:0/3389501722 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7238004050 con 0x555e967f4970 2024-04-25T01:55:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f722be00640 1 -- 172.21.15.102:0/3389501722 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238005260 con 0x555e967f4970 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f7248bdc740 1 -- 172.21.15.102:0/3389501722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 msgr2=0x555e967f4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f723ea00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f72240a2c10 0x7f72240a50d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f72300b8540 tx=0x7f72300f5530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f7248bdc740 1 --2- 172.21.15.102:0/3389501722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967f4d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7238009fd0 tx=0x7f72380118d0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f7248bdc740 1 -- 172.21.15.102:0/3389501722 shutdown_connections 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f7248bdc740 1 --2- 172.21.15.102:0/3389501722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f7248bdc740 1 --2- 172.21.15.102:0/3389501722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e967f9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f7248bdc740 1 --2- 172.21.15.102:0/3389501722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967f4d90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f7248bdc740 1 -- 172.21.15.102:0/3389501722 >> 172.21.15.102:0/3389501722 conn(0x555e967effe0 msgr2=0x555e967f03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.164+0000 7f7248bdc740 1 -- 172.21.15.102:0/3389501722 shutdown_connections 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.165+0000 7f7248bdc740 1 -- 172.21.15.102:0/3389501722 wait complete. 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.165+0000 7f7248bdc740 1 -- start start 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.165+0000 7f7248bdc740 1 Processor -- start 2024-04-25T01:55:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.165+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f4970 0x555e967feca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.165+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e967ff1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967fa4d0 0x555e96806210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f7248bdc740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555e96713d20 con 0x555e967f4970 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f7248bdc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555e96753db0 con 0x555e967fa4d0 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f7248bdc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555e967fd480 con 0x555e967f5360 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f7248bdc740 10 client.0 _ll_register_callbacks cb 0x555e9674ef10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f4970 0x555e967feca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f4970 0x555e967feca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35260/0 (socket says 172.21.15.102:35260) 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 -- 172.21.15.102:0/855450363 learned_addr learned my addr 172.21.15.102:0/855450363 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723e000640 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e967ff1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967fa4d0 0x555e96806210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 -- 172.21.15.102:0/855450363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 msgr2=0x555e967ff1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e967ff1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 -- 172.21.15.102:0/855450363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967fa4d0 msgr2=0x555e96806210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967fa4d0 0x555e96806210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 -- 172.21.15.102:0/855450363 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e965366b0 con 0x555e967f4970 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f723f400640 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f4970 0x555e967feca0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7238011dd0 tx=0x7f7238009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8ec03797a3abf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.166+0000 7f7228c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.167+0000 7f7228c00640 1 -- 172.21.15.102:0/855450363 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238005040 con 0x555e967f4970 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.167+0000 7f7228c00640 1 -- 172.21.15.102:0/855450363 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f72380051e0 con 0x555e967f4970 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.167+0000 7f7248bdc740 1 -- 172.21.15.102:0/855450363 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7238004050 con 0x555e967f4970 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.167+0000 7f7228c00640 1 -- 172.21.15.102:0/855450363 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7238005400 con 0x555e967f4970 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.167+0000 7f7248bdc740 10 client.15051 fetch_fsmap learned FSMap version 22 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.167+0000 7f7248bdc740 1 -- 172.21.15.102:0/855450363 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555e96809580 con 0x555e967f4970 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.167+0000 7f7228c00640 1 -- 172.21.15.102:0/855450363 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7238004050 con 0x555e967f4970 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.167+0000 7f7228c00640 1 -- 172.21.15.102:0/855450363 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f723800fae0 con 0x555e967f4970 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.167+0000 7f7248bdc740 10 client.15051 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.168+0000 7f7248bdc740 10 client.15051 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.168+0000 7f7248bdc740 20 client.15051 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.168+0000 7f7248bdc740 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555e9680b990 0x555e9680ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.168+0000 7f7248bdc740 4 client.15051 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:04.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.168+0000 7f7248bdc740 1 -- 172.21.15.102:0/855450363 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e96765330 con 0x555e9680b990 2024-04-25T01:55:04.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.168+0000 7f723e000640 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555e9680b990 0x555e9680ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.168+0000 7f723e000640 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555e9680b990 0x555e9680ce00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=67a7f22be77a374a server_cookie=7733617f3fc02fdf in_seq=0 out_seq=0 2024-04-25T01:55:04.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.168+0000 7f7228c00640 10 client.15051 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.169+0000 7f7228c00640 1 -- 172.21.15.102:0/855450363 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e96765330 con 0x555e9680b990 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.169+0000 7f7228c00640 10 client.15051 handle_command_reply: tid=0 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.169+0000 7f7248bdc740 1 client.15051 shutdown 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 -- 172.21.15.102:0/855450363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f4970 msgr2=0x555e967feca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f4970 0x555e967feca0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7238011dd0 tx=0x7f7238009c80 comp rx=0 tx=0).stop 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 -- 172.21.15.102:0/855450363 shutdown_connections 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555e9680b990 0x555e9680ce00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967fa4d0 0x555e96806210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7228c00640 0 client.15051 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:55:04.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e967ff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 --2- 172.21.15.102:0/855450363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f4970 0x555e967feca0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 -- 172.21.15.102:0/855450363 >> 172.21.15.102:0/855450363 conn(0x555e967effe0 msgr2=0x555e967ef120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 -- 172.21.15.102:0/855450363 shutdown_connections 2024-04-25T01:55:04.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 1 -- 172.21.15.102:0/855450363 wait complete. 2024-04-25T01:55:04.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.170+0000 7f7248bdc740 20 client.15051 trim_cache size 0 max 16384 2024-04-25T01:55:04.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f7248bdc740 1 Processor -- start 2024-04-25T01:55:04.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f7248bdc740 1 -- start start 2024-04-25T01:55:04.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e968060e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f7248bdc740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555e9680b140 con 0x555e967fa4d0 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f7248bdc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555e9680b810 con 0x555e967f4970 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f7248bdc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555e9680bee0 con 0x555e967f5360 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f723f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f723f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35272/0 (socket says 172.21.15.102:35272) 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.189+0000 7f723f400640 1 -- 172.21.15.102:0/3537418351 learned_addr learned my addr 172.21.15.102:0/3537418351 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f723f400640 1 --2- 172.21.15.102:0/3537418351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f723e000640 1 --2- 172.21.15.102:0/3537418351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e968060e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f723f400640 1 -- 172.21.15.102:0/3537418351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 msgr2=0x555e968060e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f723f400640 1 --2- 172.21.15.102:0/3537418351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e968060e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f723f400640 1 -- 172.21.15.102:0/3537418351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 msgr2=0x555e967fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f723f400640 1 --2- 172.21.15.102:0/3537418351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967fe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f723f400640 1 -- 172.21.15.102:0/3537418351 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e967efb80 con 0x555e967fa4d0 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f723f400640 1 --2- 172.21.15.102:0/3537418351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fedf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7238005d70 tx=0x7f72380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800f5aec5ff95b1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f722b400640 1 -- 172.21.15.102:0/3537418351 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238005240 con 0x555e967fa4d0 2024-04-25T01:55:04.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f722b400640 1 -- 172.21.15.102:0/3537418351 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f723802c070 con 0x555e967fa4d0 2024-04-25T01:55:04.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.190+0000 7f722b400640 1 -- 172.21.15.102:0/3537418351 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238033900 con 0x555e967fa4d0 2024-04-25T01:55:04.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 -- 172.21.15.102:0/3537418351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 msgr2=0x555e967fedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 --2- 172.21.15.102:0/3537418351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fedf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7238005d70 tx=0x7f72380118d0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 -- 172.21.15.102:0/3537418351 shutdown_connections 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 --2- 172.21.15.102:0/3537418351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967fa4d0 0x555e967fedf0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 --2- 172.21.15.102:0/3537418351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967f5360 0x555e968060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 --2- 172.21.15.102:0/3537418351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e967fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 -- 172.21.15.102:0/3537418351 >> 172.21.15.102:0/3537418351 conn(0x555e967effe0 msgr2=0x555e967f03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 -- 172.21.15.102:0/3537418351 shutdown_connections 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 -- 172.21.15.102:0/3537418351 wait complete. 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.191+0000 7f7248bdc740 1 -- start start 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f7248bdc740 1 Processor -- start 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e96802310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f5360 0x555e96802850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f7248bdc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967fa4d0 0x555e96805690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f7248bdc740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555e9680cc80 con 0x555e967f5360 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f7248bdc740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555e9680c5b0 con 0x555e967f4970 2024-04-25T01:55:04.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f7248bdc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555e9680d350 con 0x555e967fa4d0 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f7248bdc740 10 client.0 _ll_register_callbacks cb 0x7f722c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f723e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f5360 0x555e96802850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f723f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e96802310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f723f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e96802310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:37214/0 (socket says 172.21.15.102:37214) 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f723f400640 1 -- 172.21.15.102:0/622609028 learned_addr learned my addr 172.21.15.102:0/622609028 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.192+0000 7f723f400640 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967fa4d0 0x555e96805690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f723e000640 1 -- 172.21.15.102:0/622609028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967fa4d0 msgr2=0x555e96805690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f723e000640 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967fa4d0 0x555e96805690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f723e000640 1 -- 172.21.15.102:0/622609028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 msgr2=0x555e96802310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f723e000640 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e96802310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f723e000640 1 -- 172.21.15.102:0/622609028 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e96805c80 con 0x555e967f5360 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f723f400640 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e96802310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f723e000640 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f5360 0x555e96802850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x555e9680dc10 tx=0x7f722c0044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993a175504bda649 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f7228c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f7228c00640 1 -- 172.21.15.102:0/622609028 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c024020 con 0x555e967f5360 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f7228c00640 1 -- 172.21.15.102:0/622609028 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f722c0200e0 con 0x555e967f5360 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f7248bdc740 1 -- 172.21.15.102:0/622609028 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f723802c070 con 0x555e967f5360 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.193+0000 7f7228c00640 1 -- 172.21.15.102:0/622609028 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f722c0202e0 con 0x555e967f5360 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7248bdc740 10 client.15057 fetch_fsmap learned FSMap version 22 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7248bdc740 1 -- 172.21.15.102:0/622609028 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x555e96872370 con 0x555e967f5360 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7228c00640 1 -- 172.21.15.102:0/622609028 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f723802c070 con 0x555e967f5360 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7228c00640 1 -- 172.21.15.102:0/622609028 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f722c010c60 con 0x555e967f5360 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7248bdc740 10 client.15057 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7248bdc740 10 client.15057 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7248bdc740 20 client.15057 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7248bdc740 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555e96876fc0 0x555e96875cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7248bdc740 4 client.15057 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:55:04.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.194+0000 7f7248bdc740 1 -- 172.21.15.102:0/622609028 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "status"}) -- 0x555e9675f680 con 0x555e96876fc0 2024-04-25T01:55:04.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f723f400640 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555e96876fc0 0x555e96875cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f723f400640 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555e96876fc0 0x555e96875cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=404dae4fd6a85a09 server_cookie=5e1094bdace9a002 in_seq=0 out_seq=0 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7228c00640 10 client.15057 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7228c00640 1 -- 172.21.15.102:0/622609028 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x555e9675f680 con 0x555e96876fc0 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7228c00640 10 client.15057 handle_command_reply: tid=0 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7248bdc740 1 client.15057 shutdown 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7248bdc740 1 -- 172.21.15.102:0/622609028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f5360 msgr2=0x555e96802850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7248bdc740 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f5360 0x555e96802850 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x555e9680dc10 tx=0x7f722c0044d0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7248bdc740 1 -- 172.21.15.102:0/622609028 shutdown_connections 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7248bdc740 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e967fa4d0 0x555e96805690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7248bdc740 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e967f5360 0x555e96802850 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7248bdc740 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555e96876fc0 0x555e96875cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7248bdc740 1 --2- 172.21.15.102:0/622609028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e967f4970 0x555e96802310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7248bdc740 1 -- 172.21.15.102:0/622609028 >> 172.21.15.102:0/622609028 conn(0x555e967effe0 msgr2=0x555e967fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.195+0000 7f7228c00640 0 client.15057 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f7248bdc740 1 -- 172.21.15.102:0/622609028 shutdown_connections 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f7248bdc740 1 -- 172.21.15.102:0/622609028 wait complete. 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f7248bdc740 20 client.15057 trim_cache size 0 max 16384 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 2, 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24533, 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-25T01:55:04.201 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-25T01:55:04.202 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-25T01:55:04.202 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 136.11661725299999, 2024-04-25T01:55:04.202 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 22, 2024-04-25T01:55:04.202 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 89, 2024-04-25T01:55:04.202 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:55:04.202 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 161.39781697800001 2024-04-25T01:55:04.202 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 -- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f72240a2c10 msgr2=0x7f72240a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f72240a2c10 0x7f72240a50d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f72300b8540 tx=0x7f72300f5530 comp rx=0 tx=0).stop 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 -- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240107c80 msgr2=0x7f724019df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240107c80 0x7f724019df80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7230031440 tx=0x7f7230032ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 -- 172.21.15.102:0/3047136641 shutdown_connections 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f72240a2c10 0x7f72240a50d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7240109830 0x7f72401a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7240108e70 0x7f724019e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 --2- 172.21.15.102:0/3047136641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240107c80 0x7f724019df80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 -- 172.21.15.102:0/3047136641 >> 172.21.15.102:0/3047136641 conn(0x7f7240076c10 msgr2=0x7f724010c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.196+0000 7f723fe00640 1 -- 172.21.15.102:0/3047136641 shutdown_connections 2024-04-25T01:55:04.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.197+0000 7f723fe00640 1 -- 172.21.15.102:0/3047136641 wait complete. 2024-04-25T01:55:04.212 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:04 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1021369720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:04.212 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:04 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/902718176' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:04.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T01:55:04.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.403+0000 7fd746e00640 1 -- 172.21.15.102:0/3138355894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd740103880 msgr2=0x7fd740103ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.403+0000 7fd746e00640 1 --2- 172.21.15.102:0/3138355894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd740103880 0x7fd740103ca0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd730009900 tx=0x7fd73002a410 comp rx=0 tx=0).stop 2024-04-25T01:55:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.404+0000 7fd746e00640 1 -- 172.21.15.102:0/3138355894 shutdown_connections 2024-04-25T01:55:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.404+0000 7fd746e00640 1 --2- 172.21.15.102:0/3138355894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd740105430 0x7fd74010bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.404+0000 7fd746e00640 1 --2- 172.21.15.102:0/3138355894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd740104a70 0x7fd740104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.404+0000 7fd746e00640 1 --2- 172.21.15.102:0/3138355894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd740103880 0x7fd740103ca0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.404+0000 7fd746e00640 1 -- 172.21.15.102:0/3138355894 >> 172.21.15.102:0/3138355894 conn(0x7fd7400fedf0 msgr2=0x7fd740101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.404+0000 7fd746e00640 1 -- 172.21.15.102:0/3138355894 shutdown_connections 2024-04-25T01:55:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.404+0000 7fd746e00640 1 -- 172.21.15.102:0/3138355894 wait complete. 2024-04-25T01:55:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.404+0000 7fd746e00640 1 Processor -- start 2024-04-25T01:55:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd746e00640 1 -- start start 2024-04-25T01:55:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd746e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd740103880 0x7fd740199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd746e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd740104a70 0x7fd74019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd746e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd740105430 0x7fd7401a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd746e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd74010f4d0 con 0x7fd740103880 2024-04-25T01:55:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd746e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd74010f350 con 0x7fd740104a70 2024-04-25T01:55:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd746e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd74010f650 con 0x7fd740105430 2024-04-25T01:55:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd745a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd740103880 0x7fd740199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd745a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd740103880 0x7fd740199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35292/0 (socket says 172.21.15.102:35292) 2024-04-25T01:55:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd745a00640 1 -- 172.21.15.102:0/1868476212 learned_addr learned my addr 172.21.15.102:0/1868476212 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd745000640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd740104a70 0x7fd74019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.405+0000 7fd746400640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd740105430 0x7fd7401a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd745a00640 1 -- 172.21.15.102:0/1868476212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd740105430 msgr2=0x7fd7401a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd745a00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd740105430 0x7fd7401a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd745a00640 1 -- 172.21.15.102:0/1868476212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd740104a70 msgr2=0x7fd74019a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd745a00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd740104a70 0x7fd74019a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd745a00640 1 -- 172.21.15.102:0/1868476212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7401a17c0 con 0x7fd740103880 2024-04-25T01:55:04.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd745a00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd740103880 0x7fd740199ba0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd730031610 tx=0x7fd730004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd73e000640 1 -- 172.21.15.102:0/1868476212 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73003f030 con 0x7fd740103880 2024-04-25T01:55:04.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd73e000640 1 -- 172.21.15.102:0/1868476212 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd730003ad0 con 0x7fd740103880 2024-04-25T01:55:04.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd746e00640 1 -- 172.21.15.102:0/1868476212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7401a1a50 con 0x7fd740103880 2024-04-25T01:55:04.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd73e000640 1 -- 172.21.15.102:0/1868476212 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd730035070 con 0x7fd740103880 2024-04-25T01:55:04.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.406+0000 7fd746e00640 1 -- 172.21.15.102:0/1868476212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7401a1f40 con 0x7fd740103880 2024-04-25T01:55:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd73e000640 1 -- 172.21.15.102:0/1868476212 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd730042030 con 0x7fd740103880 2024-04-25T01:55:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd74b1b1740 1 Processor -- start 2024-04-25T01:55:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd74b1b1740 1 -- start start 2024-04-25T01:55:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae6bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6c360 0x55cadae70e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae73960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cadad6bed0 con 0x55cadae6b970 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cadadcedf0 con 0x55cadae714d0 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cadad778d0 con 0x55cadae6c360 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.410+0000 7fd746400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae6bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd746400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae6bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35306/0 (socket says 172.21.15.102:35306) 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd746400640 1 -- 172.21.15.102:0/1764025966 learned_addr learned my addr 172.21.15.102:0/1764025966 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd745a00640 1 --2- 172.21.15.102:0/1764025966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6c360 0x55cadae70e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd745000640 1 --2- 172.21.15.102:0/1764025966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae73960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd746400640 1 -- 172.21.15.102:0/1764025966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6c360 msgr2=0x55cadae70e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd746400640 1 --2- 172.21.15.102:0/1764025966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6c360 0x55cadae70e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd746400640 1 -- 172.21.15.102:0/1764025966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 msgr2=0x55cadae73960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd746400640 1 --2- 172.21.15.102:0/1764025966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae73960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd746400640 1 -- 172.21.15.102:0/1764025966 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cadaae32d0 con 0x55cadae6b970 2024-04-25T01:55:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd746400640 1 --2- 172.21.15.102:0/1764025966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae6bd90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd738009fd0 tx=0x7fd7380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b2a0de4bf51946 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.411+0000 7fd73e000640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7280a2b90 0x7fd7280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.412+0000 7fd72ea00640 1 -- 172.21.15.102:0/1764025966 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738004d20 con 0x55cadae6b970 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.412+0000 7fd73e000640 1 -- 172.21.15.102:0/1868476212 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fd7300ef5c0 con 0x7fd740103880 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.412+0000 7fd72ea00640 1 -- 172.21.15.102:0/1764025966 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd738004050 con 0x55cadae6b970 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.412+0000 7fd72ea00640 1 -- 172.21.15.102:0/1764025966 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738005280 con 0x55cadae6b970 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.412+0000 7fd745a00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7280a2b90 0x7fd7280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.412+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1764025966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 msgr2=0x55cadae6bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.412+0000 7fd745a00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7280a2b90 0x7fd7280a5050 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd7300ed440 tx=0x7fd7300f5110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.412+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1764025966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae6bd90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd738009fd0 tx=0x7fd7380118d0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.413+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1764025966 shutdown_connections 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.413+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1764025966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae73960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.413+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1764025966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6c360 0x55cadae70e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.413+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1764025966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae6bd90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.413+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1764025966 >> 172.21.15.102:0/1764025966 conn(0x55cadae66fe0 msgr2=0x55cadae673e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.413+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1764025966 shutdown_connections 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.413+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1764025966 wait complete. 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.413+0000 7fd74b1b1740 1 -- start start 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.413+0000 7fd74b1b1740 1 Processor -- start 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae75be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae76120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae7d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cadad8ad20 con 0x55cadae6c360 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cadadcadb0 con 0x55cadae714d0 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cadae74480 con 0x55cadae6b970 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd74b1b1740 10 client.0 _ll_register_callbacks cb 0x55cadadc5f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd745000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae76120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd745000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae76120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35320/0 (socket says 172.21.15.102:35320) 2024-04-25T01:55:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd745000640 1 -- 172.21.15.102:0/1812007003 learned_addr learned my addr 172.21.15.102:0/1812007003 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd746400640 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae75be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd746400640 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae7d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd745000640 1 -- 172.21.15.102:0/1812007003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 msgr2=0x55cadae75be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd745000640 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae75be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd745000640 1 -- 172.21.15.102:0/1812007003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 msgr2=0x55cadae7d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd745000640 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae7d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd745000640 1 -- 172.21.15.102:0/1812007003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cadabad6b0 con 0x55cadae6c360 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.414+0000 7fd746400640 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae7d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.415+0000 7fd745000640 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae76120 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd734012700 tx=0x7fd734012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d367e1754469b5ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.415+0000 7fd723400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.415+0000 7fd723400640 1 -- 172.21.15.102:0/1812007003 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73400f8c0 con 0x55cadae6c360 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.415+0000 7fd723400640 1 -- 172.21.15.102:0/1812007003 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fd73400fa60 con 0x55cadae6c360 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.415+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1812007003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd738004050 con 0x55cadae6c360 2024-04-25T01:55:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.415+0000 7fd723400640 1 -- 172.21.15.102:0/1812007003 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd73400fc60 con 0x55cadae6c360 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.415+0000 7fd74b1b1740 10 client.15069 fetch_fsmap learned FSMap version 22 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.415+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1812007003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55cadae805f0 con 0x55cadae6c360 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.415+0000 7fd723400640 1 -- 172.21.15.102:0/1812007003 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd738004050 con 0x55cadae6c360 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.416+0000 7fd723400640 1 -- 172.21.15.102:0/1812007003 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd73401c420 con 0x55cadae6c360 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.416+0000 7fd74b1b1740 10 client.15069 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.416+0000 7fd74b1b1740 10 client.15069 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.416+0000 7fd74b1b1740 20 client.15069 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.416+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55cadae82c90 0x55cadae84100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.416+0000 7fd74b1b1740 4 client.15069 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.416+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1812007003 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cadaddc330 con 0x55cadae82c90 2024-04-25T01:55:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.416+0000 7fd746400640 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55cadae82c90 0x55cadae84100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.417+0000 7fd746400640 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55cadae82c90 0x55cadae84100 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c331e13ef2b9e6b server_cookie=63cd0dd7c75cf568 in_seq=0 out_seq=0 2024-04-25T01:55:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.417+0000 7fd723400640 10 client.15069 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:55:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd723400640 1 -- 172.21.15.102:0/1812007003 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cadaddc330 con 0x55cadae82c90 2024-04-25T01:55:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd723400640 10 client.15069 handle_command_reply: tid=0 2024-04-25T01:55:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 client.15069 shutdown 2024-04-25T01:55:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1812007003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 msgr2=0x55cadae76120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae76120 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd734012700 tx=0x7fd734012bc0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1812007003 shutdown_connections 2024-04-25T01:55:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55cadae82c90 0x55cadae84100 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae7d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae76120 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd723400640 0 client.15069 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:55:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/1812007003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae75be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1812007003 >> 172.21.15.102:0/1812007003 conn(0x55cadae66fe0 msgr2=0x55cadae66120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1812007003 shutdown_connections 2024-04-25T01:55:04.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 1 -- 172.21.15.102:0/1812007003 wait complete. 2024-04-25T01:55:04.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.418+0000 7fd74b1b1740 20 client.15069 trim_cache size 0 max 16384 2024-04-25T01:55:04.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.437+0000 7fd74b1b1740 1 Processor -- start 2024-04-25T01:55:04.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.437+0000 7fd74b1b1740 1 -- start start 2024-04-25T01:55:04.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.437+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae65fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.437+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae7d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.437+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae8b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.437+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cadae82440 con 0x55cadae6c360 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.437+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cadae82b10 con 0x55cadae714d0 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.437+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cadae831e0 con 0x55cadae6b970 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae65fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae65fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44972/0 (socket says 172.21.15.102:44972) 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 -- 172.21.15.102:0/2120407553 learned_addr learned my addr 172.21.15.102:0/2120407553 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 --2- 172.21.15.102:0/2120407553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae8b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd745000640 1 --2- 172.21.15.102:0/2120407553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae7d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 -- 172.21.15.102:0/2120407553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 msgr2=0x55cadae8b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 --2- 172.21.15.102:0/2120407553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae8b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 -- 172.21.15.102:0/2120407553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 msgr2=0x55cadae7d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 --2- 172.21.15.102:0/2120407553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae7d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 -- 172.21.15.102:0/2120407553 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cadae6c740 con 0x55cadae6b970 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd746400640 1 --2- 172.21.15.102:0/2120407553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae65fa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd738004970 tx=0x7fd738005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f7b376666f87d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd72e000640 1 -- 172.21.15.102:0/2120407553 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73802fd50 con 0x55cadae6b970 2024-04-25T01:55:04.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.438+0000 7fd72e000640 1 -- 172.21.15.102:0/2120407553 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd738003a30 con 0x55cadae6b970 2024-04-25T01:55:04.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd72e000640 1 -- 172.21.15.102:0/2120407553 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73802f540 con 0x55cadae6b970 2024-04-25T01:55:04.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd74b1b1740 1 -- 172.21.15.102:0/2120407553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 msgr2=0x55cadae65fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/2120407553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae65fa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd738004970 tx=0x7fd738005fd0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd74b1b1740 1 -- 172.21.15.102:0/2120407553 shutdown_connections 2024-04-25T01:55:04.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/2120407553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae714d0 0x55cadae8b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/2120407553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6c360 0x55cadae7d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/2120407553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae6b970 0x55cadae65fa0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd74b1b1740 1 -- 172.21.15.102:0/2120407553 >> 172.21.15.102:0/2120407553 conn(0x55cadae66fe0 msgr2=0x55cadae65890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd74b1b1740 1 -- 172.21.15.102:0/2120407553 shutdown_connections 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.439+0000 7fd74b1b1740 1 -- 172.21.15.102:0/2120407553 wait complete. 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd74b1b1740 1 -- start start 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd74b1b1740 1 Processor -- start 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae7b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae6c360 0x55cadae7b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd74b1b1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae714d0 0x55cadae7bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cadae83f80 con 0x55cadae6b970 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cadae838b0 con 0x55cadae6c360 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd74b1b1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cadae84650 con 0x55cadae714d0 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd74b1b1740 10 client.0 _ll_register_callbacks cb 0x7fd734009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd746400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae7b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd746400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae7b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35328/0 (socket says 172.21.15.102:35328) 2024-04-25T01:55:04.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd746400640 1 -- 172.21.15.102:0/3679270374 learned_addr learned my addr 172.21.15.102:0/3679270374 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.440+0000 7fd745000640 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae6c360 0x55cadae7b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd746400640 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae714d0 0x55cadae7bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd746400640 1 -- 172.21.15.102:0/3679270374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae714d0 msgr2=0x55cadae7bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd746400640 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae714d0 0x55cadae7bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd746400640 1 -- 172.21.15.102:0/3679270374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae6c360 msgr2=0x55cadae7b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd746400640 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae6c360 0x55cadae7b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd746400640 1 -- 172.21.15.102:0/3679270374 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cadae7c550 con 0x55cadae6b970 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd745000640 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae6c360 0x55cadae7b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd746400640 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae7b360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd73800a010 tx=0x7fd738000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fc8332be24d226 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd723400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd723400640 1 -- 172.21.15.102:0/3679270374 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738040060 con 0x55cadae6b970 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd723400640 1 -- 172.21.15.102:0/3679270374 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fd738004480 con 0x55cadae6b970 2024-04-25T01:55:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.441+0000 7fd74b1b1740 1 -- 172.21.15.102:0/3679270374 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd738003a30 con 0x55cadae6b970 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd723400640 1 -- 172.21.15.102:0/3679270374 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fd73800f4e0 con 0x55cadae6b970 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd74b1b1740 10 client.15075 fetch_fsmap learned FSMap version 22 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd74b1b1740 1 -- 172.21.15.102:0/3679270374 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55cadaee8de0 con 0x55cadae6b970 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd723400640 1 -- 172.21.15.102:0/3679270374 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd738003a30 con 0x55cadae6b970 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd723400640 1 -- 172.21.15.102:0/3679270374 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fd738025040 con 0x55cadae6b970 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd74b1b1740 10 client.15075 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd74b1b1740 10 client.15075 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd74b1b1740 20 client.15075 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55cadaeed280 0x55cadaeebf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd74b1b1740 4 client.15075 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.442+0000 7fd74b1b1740 1 -- 172.21.15.102:0/3679270374 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "status"}) -- 0x55cadadd6680 con 0x55cadaeed280 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.443+0000 7fd745000640 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55cadaeed280 0x55cadaeebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.443+0000 7fd745000640 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55cadaeed280 0x55cadaeebf90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c53843b2ccf6ce1 server_cookie=d05e3615f0ec4a95 in_seq=0 out_seq=0 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.443+0000 7fd723400640 10 client.15075 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.443+0000 7fd723400640 1 -- 172.21.15.102:0/3679270374 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55cadadd6680 con 0x55cadaeed280 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.443+0000 7fd723400640 10 client.15075 handle_command_reply: tid=0 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.443+0000 7fd74b1b1740 1 client.15075 shutdown 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 -- 172.21.15.102:0/3679270374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 msgr2=0x55cadae7b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae7b360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd73800a010 tx=0x7fd738000f40 comp rx=0 tx=0).stop 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 -- 172.21.15.102:0/3679270374 shutdown_connections 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55cadaeed280 0x55cadaeebf90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cadae714d0 0x55cadae7bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cadae6c360 0x55cadae7b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd723400640 0 client.15075 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 --2- 172.21.15.102:0/3679270374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cadae6b970 0x55cadae7b360 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 -- 172.21.15.102:0/3679270374 >> 172.21.15.102:0/3679270374 conn(0x55cadae66fe0 msgr2=0x55cadae66120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 -- 172.21.15.102:0/3679270374 shutdown_connections 2024-04-25T01:55:04.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 1 -- 172.21.15.102:0/3679270374 wait complete. 2024-04-25T01:55:04.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd74b1b1740 20 client.15075 trim_cache size 0 max 16384 2024-04-25T01:55:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:55:04.450 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:55:04.450 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 1, 2024-04-25T01:55:04.450 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24541, 2024-04-25T01:55:04.450 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-25T01:55:04.450 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-25T01:55:04.450 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-25T01:55:04.451 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 137.392158053, 2024-04-25T01:55:04.451 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 22, 2024-04-25T01:55:04.451 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 89, 2024-04-25T01:55:04.451 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:55:04.451 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 158.523695659 2024-04-25T01:55:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:55:04.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd746e00640 1 -- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7280a2b90 msgr2=0x7fd7280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd746e00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7280a2b90 0x7fd7280a5050 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd7300ed440 tx=0x7fd7300f5110 comp rx=0 tx=0).stop 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd746e00640 1 -- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd740103880 msgr2=0x7fd740199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.444+0000 7fd746e00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd740103880 0x7fd740199ba0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd730031610 tx=0x7fd730004270 comp rx=0 tx=0).stop 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.445+0000 7fd746e00640 1 -- 172.21.15.102:0/1868476212 shutdown_connections 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.445+0000 7fd746e00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fd7280a2b90 0x7fd7280a5050 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.445+0000 7fd746e00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd740105430 0x7fd7401a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.445+0000 7fd746e00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd740104a70 0x7fd74019a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.445+0000 7fd746e00640 1 --2- 172.21.15.102:0/1868476212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd740103880 0x7fd740199ba0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.445+0000 7fd746e00640 1 -- 172.21.15.102:0/1868476212 >> 172.21.15.102:0/1868476212 conn(0x7fd7400fedf0 msgr2=0x7fd740107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.445+0000 7fd746e00640 1 -- 172.21.15.102:0/1868476212 shutdown_connections 2024-04-25T01:55:04.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.445+0000 7fd746e00640 1 -- 172.21.15.102:0/1868476212 wait complete. 2024-04-25T01:55:04.462 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T01:55:04.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:04 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1021369720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:04.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:04 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/902718176' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:04.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:04 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1021369720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:04.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:04 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/902718176' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.656+0000 7fa7db200640 1 -- 172.21.15.102:0/3426730317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d4108740 msgr2=0x7fa7d410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.656+0000 7fa7db200640 1 --2- 172.21.15.102:0/3426730317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d4108740 0x7fa7d410ab70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4009900 tx=0x7fa7c402a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.657+0000 7fa7db200640 1 -- 172.21.15.102:0/3426730317 shutdown_connections 2024-04-25T01:55:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.657+0000 7fa7db200640 1 --2- 172.21.15.102:0/3426730317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7d410b530 0x7fa7d41143d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.657+0000 7fa7db200640 1 --2- 172.21.15.102:0/3426730317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7d410b0b0 0x7fa7d4111e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.657+0000 7fa7db200640 1 --2- 172.21.15.102:0/3426730317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d4108740 0x7fa7d410ab70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.657+0000 7fa7db200640 1 -- 172.21.15.102:0/3426730317 >> 172.21.15.102:0/3426730317 conn(0x7fa7d406e970 msgr2=0x7fa7d4070d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.657+0000 7fa7db200640 1 -- 172.21.15.102:0/3426730317 shutdown_connections 2024-04-25T01:55:04.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.657+0000 7fa7db200640 1 -- 172.21.15.102:0/3426730317 wait complete. 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.658+0000 7fa7db200640 1 Processor -- start 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.658+0000 7fa7db200640 1 -- start start 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.658+0000 7fa7db200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d4108740 0x7fa7d41a2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.658+0000 7fa7db200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7d410b0b0 0x7fa7d41a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.658+0000 7fa7db200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7d410b530 0x7fa7d41a9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.658+0000 7fa7db200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d4117e10 con 0x7fa7d410b0b0 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.658+0000 7fa7db200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d4117c90 con 0x7fa7d4108740 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.658+0000 7fa7db200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d4117f90 con 0x7fa7d410b530 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7d410b0b0 0x7fa7d41a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7d410b0b0 0x7fa7d41a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35346/0 (socket says 172.21.15.102:35346) 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d4108740 0x7fa7d41a2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7cbe00640 1 -- 172.21.15.102:0/4167438560 learned_addr learned my addr 172.21.15.102:0/4167438560 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7d9200640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7d410b530 0x7fa7d41a9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7cbe00640 1 -- 172.21.15.102:0/4167438560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7d410b530 msgr2=0x7fa7d41a9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7d410b530 0x7fa7d41a9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7cbe00640 1 -- 172.21.15.102:0/4167438560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d4108740 msgr2=0x7fa7d41a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d4108740 0x7fa7d41a2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7cbe00640 1 -- 172.21.15.102:0/4167438560 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d41aa140 con 0x7fa7d410b0b0 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d4108740 0x7fa7d41a2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.659+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7d410b0b0 0x7fa7d41a2a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00e9c0 tx=0x7fa7bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.660+0000 7fa7d1600640 1 -- 172.21.15.102:0/4167438560 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc00cd80 con 0x7fa7d410b0b0 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.660+0000 7fa7db200640 1 -- 172.21.15.102:0/4167438560 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d41aa430 con 0x7fa7d410b0b0 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.660+0000 7fa7d1600640 1 -- 172.21.15.102:0/4167438560 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7bc0035f0 con 0x7fa7d410b0b0 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.660+0000 7fa7d1600640 1 -- 172.21.15.102:0/4167438560 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc010690 con 0x7fa7d410b0b0 2024-04-25T01:55:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.660+0000 7fa7db200640 1 -- 172.21.15.102:0/4167438560 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d41aa930 con 0x7fa7d410b0b0 2024-04-25T01:55:04.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.663+0000 7fa7d1600640 1 -- 172.21.15.102:0/4167438560 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa7bc0037a0 con 0x7fa7d410b0b0 2024-04-25T01:55:04.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.663+0000 7fa7de034740 1 Processor -- start 2024-04-25T01:55:04.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.663+0000 7fa7de034740 1 -- start start 2024-04-25T01:55:04.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.663+0000 7fa7de034740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768ddb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.663+0000 7fa7de034740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768e2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.663+0000 7fa7de034740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x5627768e5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.663+0000 7fa7de034740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5627767dded0 con 0x5627768e32c0 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.663+0000 7fa7de034740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562776840df0 con 0x5627768de150 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.663+0000 7fa7de034740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5627767e98d0 con 0x5627768dd760 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768e2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768e2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:37326/0 (socket says 172.21.15.102:37326) 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x5627768e5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d3e00640 1 -- 172.21.15.102:0/2545643027 learned_addr learned my addr 172.21.15.102:0/2545643027 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d9200640 1 --2- 172.21.15.102:0/2545643027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768ddb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d3e00640 1 -- 172.21.15.102:0/2545643027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 msgr2=0x5627768ddb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/2545643027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768ddb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d3e00640 1 -- 172.21.15.102:0/2545643027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 msgr2=0x5627768e5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/2545643027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x5627768e5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d3e00640 1 -- 172.21.15.102:0/2545643027 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627765552d0 con 0x5627768de150 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/2545643027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768e2c50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa7c402a910 tx=0x7fa7c4009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=618aa051a21db85b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.664+0000 7fa7d1600640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa7b80a2b90 0x7fa7b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7c9600640 1 -- 172.21.15.102:0/2545643027 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4040070 con 0x5627768de150 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7c9600640 1 -- 172.21.15.102:0/2545643027 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7c40319b0 con 0x5627768de150 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7c9600640 1 -- 172.21.15.102:0/2545643027 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4035040 con 0x5627768de150 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7d1600640 1 -- 172.21.15.102:0/4167438560 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fa7bc0cf3e0 con 0x7fa7d410b0b0 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7d9200640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa7b80a2b90 0x7fa7b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7de034740 1 -- 172.21.15.102:0/2545643027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 msgr2=0x5627768e2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7de034740 1 --2- 172.21.15.102:0/2545643027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768e2c50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa7c402a910 tx=0x7fa7c4009680 comp rx=0 tx=0).stop 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7d9200640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa7b80a2b90 0x7fa7b80a5050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5627768de9b0 tx=0x7fa7cc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7de034740 1 -- 172.21.15.102:0/2545643027 shutdown_connections 2024-04-25T01:55:04.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7de034740 1 --2- 172.21.15.102:0/2545643027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x5627768e5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7de034740 1 --2- 172.21.15.102:0/2545643027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768e2c50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7de034740 1 --2- 172.21.15.102:0/2545643027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768ddb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.665+0000 7fa7de034740 1 -- 172.21.15.102:0/2545643027 >> 172.21.15.102:0/2545643027 conn(0x5627768d8dd0 msgr2=0x5627768d91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.666+0000 7fa7de034740 1 -- 172.21.15.102:0/2545643027 shutdown_connections 2024-04-25T01:55:04.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.666+0000 7fa7de034740 1 -- 172.21.15.102:0/2545643027 wait complete. 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.666+0000 7fa7de034740 1 -- start start 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.666+0000 7fa7de034740 1 Processor -- start 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7de034740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7de034740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768e7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7de034740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 0x5627768eef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7de034740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5627767fcd20 con 0x5627768de150 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7de034740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56277683cdb0 con 0x5627768e32c0 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7de034740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5627768e6210 con 0x5627768dd760 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7de034740 10 client.0 _ll_register_callbacks cb 0x562776837f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768e7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768e7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35364/0 (socket says 172.21.15.102:35364) 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7cbe00640 1 -- 172.21.15.102:0/3164533584 learned_addr learned my addr 172.21.15.102:0/3164533584 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7d9200640 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 0x5627768eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7cbe00640 1 -- 172.21.15.102:0/3164533584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 msgr2=0x5627768e7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768e7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7cbe00640 1 -- 172.21.15.102:0/3164533584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 msgr2=0x5627768eef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 0x5627768eef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7cbe00640 1 -- 172.21.15.102:0/3164533584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56277661f6b0 con 0x5627768de150 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.667+0000 7fa7d9200640 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 0x5627768eef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768e7f40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0a4330 tx=0x7fa7bc097830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a8de52cc973276 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7c2000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7c2000640 1 -- 172.21.15.102:0/3164533584 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc09b550 con 0x5627768de150 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7c2000640 1 -- 172.21.15.102:0/3164533584 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fa7bc0a43d0 con 0x5627768de150 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7de034740 1 -- 172.21.15.102:0/3164533584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7c40319b0 con 0x5627768de150 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7c2000640 1 -- 172.21.15.102:0/3164533584 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa7bc09c2f0 con 0x5627768de150 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7de034740 10 client.15087 fetch_fsmap learned FSMap version 22 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7de034740 1 -- 172.21.15.102:0/3164533584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5627768f2360 con 0x5627768de150 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7c2000640 1 -- 172.21.15.102:0/3164533584 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7c40319b0 con 0x5627768de150 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7c2000640 1 -- 172.21.15.102:0/3164533584 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fa7bc09b040 con 0x5627768de150 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.668+0000 7fa7de034740 10 client.15087 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.669+0000 7fa7de034740 10 client.15087 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.669+0000 7fa7de034740 20 client.15087 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.669+0000 7fa7de034740 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5627768f4990 0x5627768f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.669+0000 7fa7de034740 4 client.15087 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:04.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.669+0000 7fa7de034740 1 -- 172.21.15.102:0/3164533584 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56277684e330 con 0x5627768f4990 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.669+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5627768f4990 0x5627768f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.670+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5627768f4990 0x5627768f5e00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=482855e7ff9fddad server_cookie=9a7a51223ee28579 in_seq=0 out_seq=0 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.670+0000 7fa7c2000640 10 client.15087 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7c2000640 1 -- 172.21.15.102:0/3164533584 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56277684e330 con 0x5627768f4990 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7c2000640 10 client.15087 handle_command_reply: tid=0 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 client.15087 shutdown 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 -- 172.21.15.102:0/3164533584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 msgr2=0x5627768e7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768e7f40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0a4330 tx=0x7fa7bc097830 comp rx=0 tx=0).stop 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 -- 172.21.15.102:0/3164533584 shutdown_connections 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5627768f4990 0x5627768f5e00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 0x5627768eef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768e7f40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7c2000640 0 client.15087 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 --2- 172.21.15.102:0/3164533584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 -- 172.21.15.102:0/3164533584 >> 172.21.15.102:0/3164533584 conn(0x5627768d8dd0 msgr2=0x5627768d7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 -- 172.21.15.102:0/3164533584 shutdown_connections 2024-04-25T01:55:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.671+0000 7fa7de034740 1 -- 172.21.15.102:0/3164533584 wait complete. 2024-04-25T01:55:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.672+0000 7fa7de034740 20 client.15087 trim_cache size 0 max 16384 2024-04-25T01:55:04.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7de034740 1 Processor -- start 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7de034740 1 -- start start 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7de034740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768d3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7de034740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768eef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7de034740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 0x5627768e65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7de034740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5627768f4140 con 0x5627768de150 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7de034740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5627768f4810 con 0x5627768e32c0 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7de034740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5627768f4ee0 con 0x5627768dd760 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768eef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768eef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35378/0 (socket says 172.21.15.102:35378) 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.691+0000 7fa7cbe00640 1 -- 172.21.15.102:0/2723136791 learned_addr learned my addr 172.21.15.102:0/2723136791 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7d9200640 1 --2- 172.21.15.102:0/2723136791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 0x5627768e65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/2723136791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768d3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7cbe00640 1 -- 172.21.15.102:0/2723136791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 msgr2=0x5627768d3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/2723136791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768d3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7cbe00640 1 -- 172.21.15.102:0/2723136791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 msgr2=0x5627768e65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/2723136791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 0x5627768e65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7cbe00640 1 -- 172.21.15.102:0/2723136791 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627768de9e0 con 0x5627768de150 2024-04-25T01:55:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/2723136791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768eef40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0985c0 tx=0x7fa7bc005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703406bf0e7f6b8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7c8c00640 1 -- 172.21.15.102:0/2723136791 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc09b550 con 0x5627768de150 2024-04-25T01:55:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7c8c00640 1 -- 172.21.15.102:0/2723136791 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7bc086050 con 0x5627768de150 2024-04-25T01:55:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.692+0000 7fa7c8c00640 1 -- 172.21.15.102:0/2723136791 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0a43d0 con 0x5627768de150 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 -- 172.21.15.102:0/2723136791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 msgr2=0x5627768eef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 --2- 172.21.15.102:0/2723136791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768eef40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc0985c0 tx=0x7fa7bc005e30 comp rx=0 tx=0).stop 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 -- 172.21.15.102:0/2723136791 shutdown_connections 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 --2- 172.21.15.102:0/2723136791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768e32c0 0x5627768e65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 --2- 172.21.15.102:0/2723136791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768de150 0x5627768eef40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 --2- 172.21.15.102:0/2723136791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768d3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 -- 172.21.15.102:0/2723136791 >> 172.21.15.102:0/2723136791 conn(0x5627768d8dd0 msgr2=0x5627768d91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 -- 172.21.15.102:0/2723136791 shutdown_connections 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 -- 172.21.15.102:0/2723136791 wait complete. 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 -- start start 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.693+0000 7fa7de034740 1 Processor -- start 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7de034740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768ea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7de034740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768ea730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7de034740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x562776958930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7de034740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5627768f5c80 con 0x5627768e32c0 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7de034740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5627768f55b0 con 0x5627768de150 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7de034740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5627768f6350 con 0x5627768dd760 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7de034740 10 client.0 _ll_register_callbacks cb 0x7fa7bc0d2b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7d9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x562776958930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7d9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x562776958930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35390/0 (socket says 172.21.15.102:35390) 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7d9200640 1 -- 172.21.15.102:0/2218447930 learned_addr learned my addr 172.21.15.102:0/2218447930 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768ea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7d9200640 1 -- 172.21.15.102:0/2218447930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 msgr2=0x5627768ea1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7d9200640 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768ea1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7d9200640 1 -- 172.21.15.102:0/2218447930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 msgr2=0x5627768ea730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7d9200640 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768ea730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7d9200640 1 -- 172.21.15.102:0/2218447930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562776958ef0 con 0x5627768e32c0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.694+0000 7fa7cbe00640 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768ea730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.695+0000 7fa7d9200640 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x562776958930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc004650 tx=0x7fa7cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5054faed34f92881 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.695+0000 7fa7c2000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.695+0000 7fa7c2000640 1 -- 172.21.15.102:0/2218447930 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc004b20 con 0x5627768e32c0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.695+0000 7fa7de034740 1 -- 172.21.15.102:0/2218447930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7bc086050 con 0x5627768e32c0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.695+0000 7fa7c2000640 1 -- 172.21.15.102:0/2218447930 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7fa7cc004cc0 con 0x5627768e32c0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.695+0000 7fa7c2000640 1 -- 172.21.15.102:0/2218447930 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fa7cc005020 con 0x5627768e32c0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.695+0000 7fa7de034740 10 client.15093 fetch_fsmap learned FSMap version 22 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.695+0000 7fa7de034740 1 -- 172.21.15.102:0/2218447930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x56277695a310 con 0x5627768e32c0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.695+0000 7fa7c2000640 1 -- 172.21.15.102:0/2218447930 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7bc086050 con 0x5627768e32c0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.696+0000 7fa7c2000640 1 -- 172.21.15.102:0/2218447930 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fa7cc01e550 con 0x5627768e32c0 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.696+0000 7fa7de034740 10 client.15093 fetch_fsmap finished waiting for FSMap version 22 2024-04-25T01:55:04.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.696+0000 7fa7de034740 10 client.15093 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T01:55:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.696+0000 7fa7de034740 20 client.15093 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.696+0000 7fa7de034740 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56277695ef80 0x56277695dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.696+0000 7fa7de034740 4 client.15093 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T01:55:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.696+0000 7fa7de034740 1 -- 172.21.15.102:0/2218447930 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "status"}) -- 0x562776848680 con 0x56277695ef80 2024-04-25T01:55:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.696+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56277695ef80 0x56277695dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7d3e00640 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56277695ef80 0x56277695dc90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea58afaa1f8a87a8 server_cookie=d717b986d0cf2325 in_seq=0 out_seq=0 2024-04-25T01:55:04.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7c2000640 10 client.15093 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7c2000640 1 -- 172.21.15.102:0/2218447930 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x562776848680 con 0x56277695ef80 2024-04-25T01:55:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7c2000640 10 client.15093 handle_command_reply: tid=0 2024-04-25T01:55:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 client.15093 shutdown 2024-04-25T01:55:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 -- 172.21.15.102:0/2218447930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 msgr2=0x562776958930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x562776958930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc004650 tx=0x7fa7cc004a40 comp rx=0 tx=0).stop 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 -- 172.21.15.102:0/2218447930 shutdown_connections 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56277695ef80 0x56277695dc90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5627768e32c0 0x562776958930 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5627768de150 0x5627768ea730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 --2- 172.21.15.102:0/2218447930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627768dd760 0x5627768ea1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7c2000640 0 client.15093 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 -- 172.21.15.102:0/2218447930 >> 172.21.15.102:0/2218447930 conn(0x5627768d8dd0 msgr2=0x5627768e65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 -- 172.21.15.102:0/2218447930 shutdown_connections 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 1 -- 172.21.15.102:0/2218447930 wait complete. 2024-04-25T01:55:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.697+0000 7fa7de034740 20 client.15093 trim_cache size 0 max 16384 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "145ca8d0-02a5-11ef-bc93-c7b262605968", 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 24587, 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 140.600694667, 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 22, 2024-04-25T01:55:04.706 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 89, 2024-04-25T01:55:04.707 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 87, 2024-04-25T01:55:04.707 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 147.91276198200001 2024-04-25T01:55:04.707 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 -- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa7b80a2b90 msgr2=0x7fa7b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa7b80a2b90 0x7fa7b80a5050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5627768de9b0 tx=0x7fa7cc009500 comp rx=0 tx=0).stop 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 -- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7d410b0b0 msgr2=0x7fa7d41a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7d410b0b0 0x7fa7d41a2a60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00e9c0 tx=0x7fa7bc00ee80 comp rx=0 tx=0).stop 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 -- 172.21.15.102:0/4167438560 shutdown_connections 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7fa7b80a2b90 0x7fa7b80a5050 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa7d410b530 0x7fa7d41a9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7d410b0b0 0x7fa7d41a2a60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 --2- 172.21.15.102:0/4167438560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d4108740 0x7fa7d41a2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 -- 172.21.15.102:0/4167438560 >> 172.21.15.102:0/4167438560 conn(0x7fa7d406e970 msgr2=0x7fa7d406f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 -- 172.21.15.102:0/4167438560 shutdown_connections 2024-04-25T01:55:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.698+0000 7fa7db200640 1 -- 172.21.15.102:0/4167438560 wait complete. 2024-04-25T01:55:04.715 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T01:55:04.728 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T01:55:04.728 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T01:55:04.729 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T01:55:04.729 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-25T01:55:04.730 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:55:04.730 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T01:55:04.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-25T01:55:04.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.912+0000 7f8506e00640 1 -- 172.21.15.102:0/3311960798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 msgr2=0x7f8500101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.912+0000 7f8506e00640 1 --2- 172.21.15.102:0/3311960798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 0x7f8500101bc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009ca0 tx=0x7f84f802a8c0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.912+0000 7f8506e00640 1 -- 172.21.15.102:0/3311960798 shutdown_connections 2024-04-25T01:55:04.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.912+0000 7f8506e00640 1 --2- 172.21.15.102:0/3311960798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8500103350 0x7f8500109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.912+0000 7f8506e00640 1 --2- 172.21.15.102:0/3311960798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8500102990 0x7f8500102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.912+0000 7f8506e00640 1 --2- 172.21.15.102:0/3311960798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 0x7f8500101bc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.912+0000 7f8506e00640 1 -- 172.21.15.102:0/3311960798 >> 172.21.15.102:0/3311960798 conn(0x7f85000fcd30 msgr2=0x7f85000ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:04.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.912+0000 7f8506e00640 1 -- 172.21.15.102:0/3311960798 shutdown_connections 2024-04-25T01:55:04.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.912+0000 7f8506e00640 1 -- 172.21.15.102:0/3311960798 wait complete. 2024-04-25T01:55:04.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.913+0000 7f8506e00640 1 Processor -- start 2024-04-25T01:55:04.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.913+0000 7f8506e00640 1 -- start start 2024-04-25T01:55:04.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.913+0000 7f8506e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 0x7f8500197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.913+0000 7f8506e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8500102990 0x7f8500197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.913+0000 7f8506e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8500103350 0x7f850019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f8506e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f850010d420 con 0x7f8500102990 2024-04-25T01:55:04.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f8506e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f850010d2a0 con 0x7f8500103350 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f8506e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f850010d5a0 con 0x7f85001017a0 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f84ffe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 0x7f8500197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f84ff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8500102990 0x7f8500197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f84ffe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 0x7f8500197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:45068/0 (socket says 172.21.15.102:45068) 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f84ffe00640 1 -- 172.21.15.102:0/3443046352 learned_addr learned my addr 172.21.15.102:0/3443046352 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f8504e00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8500103350 0x7f850019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f84ffe00640 1 -- 172.21.15.102:0/3443046352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8500103350 msgr2=0x7f850019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f84ffe00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8500103350 0x7f850019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f84ffe00640 1 -- 172.21.15.102:0/3443046352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8500102990 msgr2=0x7f8500197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f84ffe00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8500102990 0x7f8500197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.914+0000 7f84ffe00640 1 -- 172.21.15.102:0/3443046352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850019f620 con 0x7f85001017a0 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.915+0000 7f84ffe00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 0x7f8500197a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009bf0 tx=0x7f84f8031ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.915+0000 7f84fcc00640 1 -- 172.21.15.102:0/3443046352 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f802fa20 con 0x7f85001017a0 2024-04-25T01:55:04.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.915+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850019f8b0 con 0x7f85001017a0 2024-04-25T01:55:04.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.915+0000 7f84fcc00640 1 -- 172.21.15.102:0/3443046352 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84f802fbc0 con 0x7f85001017a0 2024-04-25T01:55:04.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.915+0000 7f84fcc00640 1 -- 172.21.15.102:0/3443046352 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8046750 con 0x7f85001017a0 2024-04-25T01:55:04.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.915+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850019fe50 con 0x7f85001017a0 2024-04-25T01:55:04.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.917+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8500108a60 con 0x7f85001017a0 2024-04-25T01:55:04.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.919+0000 7f84fcc00640 1 -- 172.21.15.102:0/3443046352 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f84f802fd70 con 0x7f85001017a0 2024-04-25T01:55:04.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.920+0000 7f84fcc00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f84e40a2cb0 0x7f84e40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:04.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.921+0000 7f84fcc00640 1 -- 172.21.15.102:0/3443046352 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f84f80f0310 con 0x7f85001017a0 2024-04-25T01:55:04.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.921+0000 7f84ff400640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f84e40a2cb0 0x7f84e40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:04.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.922+0000 7f84ff400640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f84e40a2cb0 0x7f84e40a5170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8500198f10 tx=0x7f84e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:04.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:04.925+0000 7f84fcc00640 1 -- 172.21.15.102:0/3443046352 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84f80bd220 con 0x7f85001017a0 2024-04-25T01:55:05.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:05.178+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f8500062340 con 0x7f85001017a0 2024-04-25T01:55:05.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:05 smithi102 ceph-mon[27796]: pgmap v390: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:55:05.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:05 smithi110 ceph-mon[32848]: pgmap v390: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:55:05.505 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:05 smithi193 ceph-mon[32981]: pgmap v390: 97 pgs: 97 active+clean; 641 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T01:55:06.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.171+0000 7f84fcc00640 1 -- 172.21.15.102:0/3443046352 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f84f80b9190 con 0x7f85001017a0 2024-04-25T01:55:06.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.173+0000 7f84fcc00640 1 -- 172.21.15.102:0/3443046352 <== mon.1 v2:172.21.15.193:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f84f8097de0 con 0x7f85001017a0 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f84e40a2cb0 msgr2=0x7f84e40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f84e40a2cb0 0x7f84e40a5170 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8500198f10 tx=0x7f84e8005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 msgr2=0x7f8500197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 0x7f8500197a00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009bf0 tx=0x7f84f8031ee0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 shutdown_connections 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f84e40a2cb0 0x7f84e40a5170 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8500103350 0x7f850019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8500102990 0x7f8500197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 --2- 172.21.15.102:0/3443046352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85001017a0 0x7f8500197a00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 >> 172.21.15.102:0/3443046352 conn(0x7f85000fcd30 msgr2=0x7f8500105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 shutdown_connections 2024-04-25T01:55:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.180+0000 7f8506e00640 1 -- 172.21.15.102:0/3443046352 wait complete. 2024-04-25T01:55:06.206 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-25T01:55:06.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.406+0000 7f68b1e00640 1 -- 172.21.15.102:0/305125463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac104bb0 msgr2=0x7f68ac105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.406+0000 7f68b1e00640 1 --2- 172.21.15.102:0/305125463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac104bb0 0x7f68ac105030 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6898009900 tx=0x7f689802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.406+0000 7f68b1e00640 1 -- 172.21.15.102:0/305125463 shutdown_connections 2024-04-25T01:55:06.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.406+0000 7f68b1e00640 1 --2- 172.21.15.102:0/305125463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac105570 0x7f68ac10be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.406+0000 7f68b1e00640 1 --2- 172.21.15.102:0/305125463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac104bb0 0x7f68ac105030 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.406+0000 7f68b1e00640 1 --2- 172.21.15.102:0/305125463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac1039c0 0x7f68ac103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.406+0000 7f68b1e00640 1 -- 172.21.15.102:0/305125463 >> 172.21.15.102:0/305125463 conn(0x7f68ac0fef30 msgr2=0x7f68ac101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.406+0000 7f68b1e00640 1 -- 172.21.15.102:0/305125463 shutdown_connections 2024-04-25T01:55:06.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.406+0000 7f68b1e00640 1 -- 172.21.15.102:0/305125463 wait complete. 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68b1e00640 1 Processor -- start 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68b1e00640 1 -- start start 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68b1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac1039c0 0x7f68ac199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68b1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac104bb0 0x7f68ac19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68b1e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac105570 0x7f68ac1a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68b1e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac10f690 con 0x7f68ac1039c0 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68b1e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac10f510 con 0x7f68ac105570 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68b1e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac10f810 con 0x7f68ac104bb0 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac1039c0 0x7f68ac199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68aa000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac104bb0 0x7f68ac19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.407+0000 7f68ab400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac105570 0x7f68ac1a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68aa000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac104bb0 0x7f68ac19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:45098/0 (socket says 172.21.15.102:45098) 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68ab400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac105570 0x7f68ac1a11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:37388/0 (socket says 172.21.15.102:37388) 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68aa000640 1 -- 172.21.15.102:0/2724755273 learned_addr learned my addr 172.21.15.102:0/2724755273 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68aa000640 1 -- 172.21.15.102:0/2724755273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac105570 msgr2=0x7f68ac1a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68aa000640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac105570 0x7f68ac1a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68aa000640 1 -- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac1039c0 msgr2=0x7f68ac199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68aa000640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac1039c0 0x7f68ac199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68aa000640 1 -- 172.21.15.102:0/2724755273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68ac1a1870 con 0x7f68ac104bb0 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68ab400640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac105570 0x7f68ac1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.408+0000 7f68aa000640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac104bb0 0x7f68ac19a1a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6898031440 tx=0x7f6898032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.409+0000 7f68b0a00640 1 -- 172.21.15.102:0/2724755273 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689803f030 con 0x7f68ac104bb0 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.409+0000 7f68b0a00640 1 -- 172.21.15.102:0/2724755273 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6898003ce0 con 0x7f68ac104bb0 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.409+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68ac1a1b00 con 0x7f68ac104bb0 2024-04-25T01:55:06.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.409+0000 7f68b0a00640 1 -- 172.21.15.102:0/2724755273 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898035070 con 0x7f68ac104bb0 2024-04-25T01:55:06.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.409+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68ac1a1ff0 con 0x7f68ac104bb0 2024-04-25T01:55:06.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.410+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68ac10ac80 con 0x7f68ac104bb0 2024-04-25T01:55:06.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.413+0000 7f68b0a00640 1 -- 172.21.15.102:0/2724755273 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f6898042030 con 0x7f68ac104bb0 2024-04-25T01:55:06.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.415+0000 7f68b0a00640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68900a2c80 0x7f68900a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.415+0000 7f68aaa00640 1 -- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68900a2c80 msgr2=0x7f68900a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/83904525 2024-04-25T01:55:06.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.415+0000 7f68b0a00640 1 -- 172.21.15.102:0/2724755273 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f68980ef6e0 con 0x7f68ac104bb0 2024-04-25T01:55:06.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.415+0000 7f68aaa00640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68900a2c80 0x7f68900a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:55:06.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.419+0000 7f68b0a00640 1 -- 172.21.15.102:0/2724755273 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68980bbfa0 con 0x7f68ac104bb0 2024-04-25T01:55:06.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:06 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3443046352' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:55:06.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:06 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:55:06.489 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: ignoring --setuser ceph since I am not root 2024-04-25T01:55:06.489 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: ignoring --setgroup ceph since I am not root 2024-04-25T01:55:06.489 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:06 smithi110 ceph-mgr[34243]: -- 172.21.15.110:0/4161161315 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557f7074e5a0 con 0x557f70722800 2024-04-25T01:55:06.489 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:06.341+0000 7fe671d43200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:55:06.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:06 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3443046352' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:55:06.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:06 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:55:06.506 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: ignoring --setuser ceph since I am not root 2024-04-25T01:55:06.506 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: ignoring --setgroup ceph since I am not root 2024-04-25T01:55:06.506 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:06 smithi193 ceph-mgr[34771]: -- 172.21.15.193:0/903190349 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cf0e3405a0 con 0x55cf0e314c00 2024-04-25T01:55:06.506 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:06.338+0000 7efe61e27200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:55:06.528 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:06 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3443046352' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:55:06.528 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:06 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T01:55:06.529 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: ignoring --setuser ceph since I am not root 2024-04-25T01:55:06.529 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: ignoring --setgroup ceph since I am not root 2024-04-25T01:55:06.529 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:06.526+0000 7f50ea592200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T01:55:06.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.616+0000 7f68aaa00640 1 -- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68900a2c80 msgr2=0x7f68900a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/83904525 2024-04-25T01:55:06.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.616+0000 7f68aaa00640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68900a2c80 0x7f68900a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:55:06.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.622+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f68ac062340 con 0x7f68ac104bb0 2024-04-25T01:55:06.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.629+0000 7f68b0a00640 1 -- 172.21.15.102:0/2724755273 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f68980c02a0 con 0x7f68ac104bb0 2024-04-25T01:55:06.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68900a2c80 msgr2=0x7f68900a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:55:06.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68900a2c80 0x7f68900a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac104bb0 msgr2=0x7f68ac19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac104bb0 0x7f68ac19a1a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6898031440 tx=0x7f6898032ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 shutdown_connections 2024-04-25T01:55:06.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f68900a2c80 0x7f68900a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68ac105570 0x7f68ac1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac104bb0 0x7f68ac19a1a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 --2- 172.21.15.102:0/2724755273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68ac1039c0 0x7f68ac199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 >> 172.21.15.102:0/2724755273 conn(0x7f68ac0fef30 msgr2=0x7f68ac107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 shutdown_connections 2024-04-25T01:55:06.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.632+0000 7f68b1e00640 1 -- 172.21.15.102:0/2724755273 wait complete. 2024-04-25T01:55:06.651 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-25T01:55:06.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.834+0000 7f14c3200640 1 -- 172.21.15.102:0/898940105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c4075780 msgr2=0x7f14c4073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.834+0000 7f14c3200640 1 --2- 172.21.15.102:0/898940105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c4075780 0x7f14c4073bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009860 tx=0x7f14b402a340 comp rx=0 tx=0).stop 2024-04-25T01:55:06.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.834+0000 7f14c3200640 1 -- 172.21.15.102:0/898940105 shutdown_connections 2024-04-25T01:55:06.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.834+0000 7f14c3200640 1 --2- 172.21.15.102:0/898940105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14c410dba0 0x7f14c410ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.834+0000 7f14c3200640 1 --2- 172.21.15.102:0/898940105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c40741a0 0x7f14c4074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.834+0000 7f14c3200640 1 --2- 172.21.15.102:0/898940105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c4075780 0x7f14c4073bd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.834+0000 7f14c3200640 1 -- 172.21.15.102:0/898940105 >> 172.21.15.102:0/898940105 conn(0x7f14c40fcb40 msgr2=0x7f14c40fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:06.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.834+0000 7f14c3200640 1 -- 172.21.15.102:0/898940105 shutdown_connections 2024-04-25T01:55:06.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.834+0000 7f14c3200640 1 -- 172.21.15.102:0/898940105 wait complete. 2024-04-25T01:55:06.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.835+0000 7f14c3200640 1 Processor -- start 2024-04-25T01:55:06.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.835+0000 7f14c3200640 1 -- start start 2024-04-25T01:55:06.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c3200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c40741a0 0x7f14c419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c3200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14c4075780 0x7f14c419e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c410dba0 0x7f14c41a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f14c4113840 con 0x7f14c410dba0 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c3200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f14c41136c0 con 0x7f14c4075780 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c3200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14c41139c0 con 0x7f14c40741a0 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c40741a0 0x7f14c419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c1400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14c4075780 0x7f14c419e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c410dba0 0x7f14c41a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c40741a0 0x7f14c419df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:45108/0 (socket says 172.21.15.102:45108) 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.836+0000 7f14c1e00640 1 -- 172.21.15.102:0/1921000121 learned_addr learned my addr 172.21.15.102:0/1921000121 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14c1400640 1 -- 172.21.15.102:0/1921000121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c40741a0 msgr2=0x7f14c419df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14c1400640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c40741a0 0x7f14c419df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14c1400640 1 -- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c410dba0 msgr2=0x7f14c41a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14c1400640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c410dba0 0x7f14c41a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14c1400640 1 -- 172.21.15.102:0/1921000121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c41a5af0 con 0x7f14c4075780 2024-04-25T01:55:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14c1e00640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c40741a0 0x7f14c419df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14c1400640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14c4075780 0x7f14c419e460 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f14b000e790 tx=0x7f14b000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14baa00640 1 -- 172.21.15.102:0/1921000121 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b000eea0 con 0x7f14c4075780 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c41a5de0 con 0x7f14c4075780 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14baa00640 1 -- 172.21.15.102:0/1921000121 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b000ccc0 con 0x7f14c4075780 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14baa00640 1 -- 172.21.15.102:0/1921000121 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0018760 con 0x7f14c4075780 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.837+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c41a62a0 con 0x7f14c4075780 2024-04-25T01:55:06.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.838+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c4103d70 con 0x7f14c4075780 2024-04-25T01:55:06.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.841+0000 7f14baa00640 1 -- 172.21.15.102:0/1921000121 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f14b0010430 con 0x7f14c4075780 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.842+0000 7f14baa00640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14a80a2bd0 0x7f14a80a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.842+0000 7f14c1e00640 1 -- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14a80a2bd0 msgr2=0x7f14a80a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/83904525 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.842+0000 7f14baa00640 1 -- 172.21.15.102:0/1921000121 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f14b00cf240 con 0x7f14c4075780 2024-04-25T01:55:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.842+0000 7f14c1e00640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14a80a2bd0 0x7f14a80a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:55:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:06.846+0000 7f14baa00640 1 -- 172.21.15.102:0/1921000121 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f14b009bb00 con 0x7f14c4075780 2024-04-25T01:55:06.914 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:06.774+0000 7f50ea592200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:55:06.914 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:06.912+0000 7f50ea592200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:55:06.936 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:06.525+0000 7fe671d43200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:55:06.937 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:06.626+0000 7fe671d43200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:55:06.937 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:06.522+0000 7efe61e27200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:55:06.937 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:06.625+0000 7efe61e27200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T01:55:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.043+0000 7f14c1e00640 1 -- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14a80a2bd0 msgr2=0x7f14a80a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/83904525 2024-04-25T01:55:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.043+0000 7f14c1e00640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14a80a2bd0 0x7f14a80a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:55:07.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.047+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f14c4062450 con 0x7f14c4075780 2024-04-25T01:55:07.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.054+0000 7f14baa00640 1 -- 172.21.15.102:0/1921000121 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f14b009fe00 con 0x7f14c4075780 2024-04-25T01:55:07.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.056+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14a80a2bd0 msgr2=0x7f14a80a5090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:55:07.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.056+0000 7f14c3200640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14a80a2bd0 0x7f14a80a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.056+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14c4075780 msgr2=0x7f14c419e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:07.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.056+0000 7f14c3200640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14c4075780 0x7f14c419e460 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f14b000e790 tx=0x7f14b000ec50 comp rx=0 tx=0).stop 2024-04-25T01:55:07.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.057+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 shutdown_connections 2024-04-25T01:55:07.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.057+0000 7f14c3200640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f14a80a2bd0 0x7f14a80a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.057+0000 7f14c3200640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14c410dba0 0x7f14c41a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.057+0000 7f14c3200640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14c4075780 0x7f14c419e460 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.057+0000 7f14c3200640 1 --2- 172.21.15.102:0/1921000121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14c40741a0 0x7f14c419df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:07.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.057+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 >> 172.21.15.102:0/1921000121 conn(0x7f14c40fcb40 msgr2=0x7f14c4100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:07.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.057+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 shutdown_connections 2024-04-25T01:55:07.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:07.057+0000 7f14c3200640 1 -- 172.21.15.102:0/1921000121 wait complete. 2024-04-25T01:55:07.076 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T01:55:07.100 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T01:55:07.100 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T01:55:07.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:07.100 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-04-25T01:55:07.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:07.148 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-04-25T01:55:07.187 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:07 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:55:07.187 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:06.934+0000 7fe671d43200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:55:07.187 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:07 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:07.018+0000 7fe671d43200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:55:07.188 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:06.934+0000 7efe61e27200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:55:07.188 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:07.020+0000 7efe61e27200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:55:07.188 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:07 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:55:07.188 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:07 smithi193 ceph-mon[32981]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-25T01:55:07.188 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:07 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' 2024-04-25T01:55:07.188 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:07 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' 2024-04-25T01:55:07.278 INFO:teuthology.orchestra.run.smithi102.stderr:Cannot find device "ceph-brx" 2024-04-25T01:55:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T01:55:07.283 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v1', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-04-25T01:55:07.283 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T01:55:07.284 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-04-25T01:55:07.284 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-04-25T01:55:07.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T01:55:07.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:07 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T01:55:07.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:07 smithi102 ceph-mon[27796]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-25T01:55:07.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:07 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' 2024-04-25T01:55:07.444 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:07 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' 2024-04-25T01:55:07.444 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:07 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:07.325+0000 7f50ea592200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T01:55:07.444 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:07 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:07.442+0000 7f50ea592200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T01:55:07.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:07 smithi110 ceph-mon[32848]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-25T01:55:07.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:07 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' 2024-04-25T01:55:07.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:07 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' 2024-04-25T01:55:07.689 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T01:55:08.464 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:08.198+0000 7fe671d43200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:55:08.464 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:08.375+0000 7fe671d43200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:55:08.506 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:08.208+0000 7efe61e27200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:55:08.506 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:08.378+0000 7efe61e27200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:55:08.738 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:08.552+0000 7fe671d43200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:55:08.837 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:08.549+0000 7efe61e27200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:55:09.133 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:08.835+0000 7efe61e27200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:55:09.134 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:08.926+0000 7efe61e27200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:55:09.134 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:09.131+0000 7efe61e27200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:55:09.156 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:08.852+0000 7fe671d43200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:55:09.156 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:08.947+0000 7fe671d43200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:55:09.416 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:09.154+0000 7fe671d43200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:55:09.416 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:09.351+0000 7fe671d43200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:55:09.424 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:09.026+0000 7f50ea592200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T01:55:09.424 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:09.256+0000 7f50ea592200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T01:55:09.506 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:09.324+0000 7efe61e27200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:55:09.841 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:09.479+0000 7f50ea592200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T01:55:09.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.995+0000 7f1a89000700 1 -- 172.21.15.102:0/1702240514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a841024e0 msgr2=0x7f1a841028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:09.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.995+0000 7f1a89000700 1 --2- 172.21.15.102:0/1702240514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a841024e0 0x7f1a841028c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1a740098f0 tx=0x7f1a74009c00 comp rx=0 tx=0).stop 2024-04-25T01:55:09.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.995+0000 7f1a89000700 1 -- 172.21.15.102:0/1702240514 shutdown_connections 2024-04-25T01:55:09.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.995+0000 7f1a89000700 1 --2- 172.21.15.102:0/1702240514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a841024e0 0x7f1a841028c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:09.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.995+0000 7f1a89000700 1 --2- 172.21.15.102:0/1702240514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a84107570 0x7f1a8410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:09.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.995+0000 7f1a89000700 1 --2- 172.21.15.102:0/1702240514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a84102e90 0x7f1a84106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:09.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.995+0000 7f1a89000700 1 -- 172.21.15.102:0/1702240514 >> 172.21.15.102:0/1702240514 conn(0x7f1a84079390 msgr2=0x7f1a840797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:09.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.996+0000 7f1a89000700 1 -- 172.21.15.102:0/1702240514 shutdown_connections 2024-04-25T01:55:09.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.996+0000 7f1a89000700 1 -- 172.21.15.102:0/1702240514 wait complete. 2024-04-25T01:55:09.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.997+0000 7f1a89000700 1 Processor -- start 2024-04-25T01:55:09.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.997+0000 7f1a89000700 1 -- start start 2024-04-25T01:55:09.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.997+0000 7f1a89000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a841024e0 0x7f1a841a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:09.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.997+0000 7f1a89000700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a84102e90 0x7f1a841a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:09.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.997+0000 7f1a89000700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a84107570 0x7f1a8419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:09.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.997+0000 7f1a89000700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1a84112580 con 0x7f1a84107570 2024-04-25T01:55:09.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.997+0000 7f1a89000700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1a84112400 con 0x7f1a84102e90 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.997+0000 7f1a89000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a84112700 con 0x7f1a841024e0 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a81400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a84102e90 0x7f1a841a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a81400700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a84102e90 0x7f1a841a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:37430/0 (socket says 172.21.15.102:37430) 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a81400700 1 -- 172.21.15.102:0/698963640 learned_addr learned my addr 172.21.15.102:0/698963640 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a82800700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a84107570 0x7f1a8419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a81400700 1 -- 172.21.15.102:0/698963640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a841024e0 msgr2=0x7f1a841a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a81400700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a841024e0 0x7f1a841a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a81400700 1 -- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a84107570 msgr2=0x7f1a8419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a81400700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a84107570 0x7f1a8419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:10.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a81400700 1 -- 172.21.15.102:0/698963640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8419b230 con 0x7f1a84102e90 2024-04-25T01:55:10.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a81400700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a84102e90 0x7f1a841a0e50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000e9f0 tx=0x7f1a7000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:10.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.998+0000 7f1a7aa00700 1 -- 172.21.15.102:0/698963640 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7000c9c0 con 0x7f1a84102e90 2024-04-25T01:55:10.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.999+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8419b520 con 0x7f1a84102e90 2024-04-25T01:55:10.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.999+0000 7f1a7aa00700 1 -- 172.21.15.102:0/698963640 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a7000cb60 con 0x7f1a84102e90 2024-04-25T01:55:10.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.999+0000 7f1a7aa00700 1 -- 172.21.15.102:0/698963640 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70018680 con 0x7f1a84102e90 2024-04-25T01:55:10.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:09.999+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a841a77e0 con 0x7f1a84102e90 2024-04-25T01:55:10.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.000+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a84068f20 con 0x7f1a84102e90 2024-04-25T01:55:10.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.002+0000 7f1a7aa00700 1 -- 172.21.15.102:0/698963640 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f1a7000cd10 con 0x7f1a84102e90 2024-04-25T01:55:10.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.003+0000 7f1a7aa00700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1a680a2c70 0x7f1a680a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:10.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.004+0000 7f1a81e00700 1 -- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1a680a2c70 msgr2=0x7f1a680a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/83904525 2024-04-25T01:55:10.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.004+0000 7f1a7aa00700 1 -- 172.21.15.102:0/698963640 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f1a700cf840 con 0x7f1a84102e90 2024-04-25T01:55:10.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.004+0000 7f1a81e00700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1a680a2c70 0x7f1a680a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:55:10.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.005+0000 7f1a7aa00700 1 -- 172.21.15.102:0/698963640 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1a7009c750 con 0x7f1a84102e90 2024-04-25T01:55:10.174 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:09.867+0000 7f50ea592200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T01:55:10.174 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:09.996+0000 7f50ea592200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T01:55:10.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.205+0000 7f1a81e00700 1 -- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1a680a2c70 msgr2=0x7f1a680a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/83904525 2024-04-25T01:55:10.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.205+0000 7f1a81e00700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1a680a2c70 0x7f1a680a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:55:10.238 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:09.945+0000 7fe671d43200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:55:10.256 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:09.923+0000 7efe61e27200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:55:10.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.278+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1a841a7a10 con 0x7f1a84102e90 2024-04-25T01:55:10.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.279+0000 7f1a7aa00700 1 -- 172.21.15.102:0/698963640 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) ==== 53+0+73 (secure 0 0 0) 0x7f1a7009c050 con 0x7f1a84102e90 2024-04-25T01:55:10.280 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T01:55:10.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1a680a2c70 msgr2=0x7f1a680a5130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:55:10.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1a680a2c70 0x7f1a680a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:10.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a84102e90 msgr2=0x7f1a841a0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:10.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a84102e90 0x7f1a841a0e50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000e9f0 tx=0x7f1a7000ed00 comp rx=0 tx=0).stop 2024-04-25T01:55:10.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 shutdown_connections 2024-04-25T01:55:10.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f1a680a2c70 0x7f1a680a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:10.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a84107570 0x7f1a8419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:10.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a841024e0 0x7f1a841a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:10.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 --2- 172.21.15.102:0/698963640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a84102e90 0x7f1a841a0e50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:10.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 >> 172.21.15.102:0/698963640 conn(0x7f1a84079390 msgr2=0x7f1a841086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:10.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 shutdown_connections 2024-04-25T01:55:10.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:10.282+0000 7f1a89000700 1 -- 172.21.15.102:0/698963640 wait complete. 2024-04-25T01:55:10.530 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:10 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/698963640' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:55:10.530 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:10 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:10.258+0000 7f50ea592200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T01:55:10.640 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:10 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/698963640' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:55:10.640 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:10.344+0000 7efe61e27200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:55:10.640 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:10.440+0000 7efe61e27200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:55:10.640 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:10.637+0000 7efe61e27200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:55:10.667 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:10 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/698963640' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T01:55:10.667 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:10.369+0000 7fe671d43200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:55:10.667 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:10.465+0000 7fe671d43200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:55:10.924 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:10 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:10.528+0000 7f50ea592200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T01:55:10.988 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:10.665+0000 7fe671d43200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:55:11.154 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T01:55:11.154 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T01:55:11.154 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-04-25T01:55:11.154 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T01:55:11.154 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-04-25T01:55:11.154 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T01:55:11.154 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T01:55:11.154 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T01:55:11.154 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-04-25T01:55:11.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:11.155 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:55:11.202 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-04-25T01:55:11.203 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-04-25T01:55:11.203 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T01:55:11.203 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:55:11.203 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T01:55:11.203 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-04-25T01:55:11.203 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T01:55:11.203 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42013sec preferred_lft 42013sec 2024-04-25T01:55:11.203 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-04-25T01:55:11.203 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-04-25T01:55:11.203 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T01:55:11.204 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:55:11.204 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-25T01:55:11.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2024-04-25T01:55:11.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2024-04-25T01:55:11.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2024-04-25T01:55:11.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T01:55:11.204 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-25T01:55:11.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:55:11.488 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:11.182+0000 7fe671d43200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:55:11.488 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:11.273+0000 7fe671d43200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:55:11.494 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:11.161+0000 7efe61e27200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:55:11.494 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:11.252+0000 7efe61e27200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:55:11.625 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:11 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:11.375+0000 7f50ea592200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T01:55:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:55:11.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:11.700 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T01:55:11.756 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:11.492+0000 7efe61e27200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:55:11.756 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:11.624+0000 7efe61e27200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:55:11.756 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:11.717+0000 7efe61e27200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:55:11.761 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-04-25T01:55:11.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:11.765 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-04-25T01:55:11.819 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-04-25T01:55:11.819 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T01:55:11.819 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T01:55:11.819 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T01:55:11.819 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T01:55:11.819 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:55:11.819 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-25T01:55:11.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T01:55:11.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T01:55:11.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T01:55:11.820 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-25T01:55:11.854 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:11.512+0000 7fe671d43200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:55:11.854 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:11.644+0000 7fe671d43200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:55:11.854 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:11.738+0000 7fe671d43200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:55:11.854 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:11.851+0000 7fe671d43200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:55:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:55:12.171 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:11 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:11.967+0000 7f50ea592200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T01:55:12.171 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:12 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:12.099+0000 7f50ea592200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T01:55:12.238 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:11 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:11.942+0000 7fe671d43200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:55:12.256 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:11.833+0000 7efe61e27200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:55:12.256 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:11.925+0000 7efe61e27200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:55:12.424 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:12 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:12.369+0000 7f50ea592200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T01:55:12.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:55:12.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:12.566 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-04-25T01:55:12.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:12.610 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-04-25T01:55:12.702 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:55:12.702 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-25T01:55:12.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--tmp-cephfs 2024-04-25T01:55:12.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-04-25T01:55:12.703 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-25T01:55:12.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:55:12.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:55:12.867 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-04-25T01:55:12.868 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:55:12.868 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-25T01:55:12.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-04-25T01:55:12.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T01:55:12.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-04-25T01:55:12.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-04-25T01:55:12.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-04-25T01:55:12.868 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-25T01:55:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:55:13.006 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:12.738+0000 7efe61e27200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:55:13.006 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 01:55:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z[34733]: 2024-04-25T01:55:12.952+0000 7efe61e27200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:55:13.028 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:12.748+0000 7fe671d43200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:55:13.028 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 01:55:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y[34239]: 2024-04-25T01:55:12.962+0000 7fe671d43200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: Standby manager daemon z restarted 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: Standby manager daemon z started 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: Standby manager daemon y restarted 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: Standby manager daemon y started 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:55:13.070 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-mon[27796]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:55:13.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:55:13.145 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T01:55:13.145 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-04-25T01:55:13.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-04-25T01:55:13.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T01:55:13.145 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-04-25T01:55:13.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:55:13.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:55:13.327 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:55:13.327 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /tmp/cephfs 2024-04-25T01:55:13.364 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/tmp/cephfs' 2024-04-25T01:55:13.365 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T01:55:13.365 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /tmp/cephfs 2024-04-25T01:55:13.424 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:13.078+0000 7f50ea592200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T01:55:13.424 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:13.205+0000 7f50ea592200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T01:55:13.450 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T01:55:13.451 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T01:55:13.451 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76 2024-04-25T01:55:13.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:13.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76 /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,context=system_u:object_r:postgresql_db_t:s0 2024-04-25T01:55:13.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: Standby manager daemon z restarted 2024-04-25T01:55:13.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: Standby manager daemon z started 2024-04-25T01:55:13.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:55:13.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:55:13.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:55:13.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:55:13.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: Standby manager daemon y restarted 2024-04-25T01:55:13.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: Standby manager daemon y started 2024-04-25T01:55:13.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:55:13.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:55:13.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:55:13.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:13 smithi110 ceph-mon[32848]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:55:13.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: Standby manager daemon z restarted 2024-04-25T01:55:13.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: Standby manager daemon z started 2024-04-25T01:55:13.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T01:55:13.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:55:13.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T01:55:13.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.193:0/795325409' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:55:13.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: Standby manager daemon y restarted 2024-04-25T01:55:13.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: Standby manager daemon y started 2024-04-25T01:55:13.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T01:55:13.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T01:55:13.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T01:55:13.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:13 smithi193 ceph-mon[32981]: from='mgr.? 172.21.15.110:0/3992745888' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T01:55:13.636 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,context="system_u:object_r:postgresql_db_t:s0" 2024-04-25T01:55:13.636 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-25T01:55:13.637 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-04-25T01:55:13.909 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:13.575+0000 7f50ea592200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T01:55:13.909 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:13.775+0000 7f50ea592200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T01:55:14.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:14 smithi102 ceph-mon[27796]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T01:55:14.174 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:13.906+0000 7f50ea592200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T01:55:14.174 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:14.066+0000 7f50ea592200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T01:55:14.404 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,context="system_u:object_r:postgresql_db_t:s0" 2024-04-25T01:55:14.405 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-04-25T01:55:14.405 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-04-25T01:55:14.405 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.110:6789,172.21.15.193:6789" 2024-04-25T01:55:14.405 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.110:6789,172.21.15.193:6789:/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76 2024-04-25T01:55:14.405 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,context="system_u:object_r:postgresql_db_t:s0",key=0,fsid=145ca8d0-02a5-11ef-bc93-c7b262605968" will pass to kernel 2024-04-25T01:55:14.411 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T01:55:14.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:14.411 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /tmp/cephfs 2024-04-25T01:55:14.467 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:14.195+0000 7f50ea592200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T01:55:14.481 DEBUG:teuthology.orchestra.run.smithi102:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:55:14.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:14 smithi110 ceph-mon[32848]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T01:55:14.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:14 smithi193 ceph-mon[32981]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T01:55:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:55:14.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:55:14.596 INFO:teuthology.orchestra.run.smithi102.stdout:client25235 2024-04-25T01:55:14.596 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T01:55:14.596 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T01:55:14.642 DEBUG:teuthology.orchestra.run.smithi102:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:55:14.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:55:14.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:55:14.796 INFO:teuthology.orchestra.run.smithi102.stdout:client25235 2024-04-25T01:55:14.796 DEBUG:teuthology.orchestra.run.smithi102:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T01:55:14.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T01:55:14.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:55:14.898 INFO:teuthology.orchestra.run.smithi102.stdout:client25235 2024-04-25T01:55:14.898 DEBUG:teuthology.orchestra.run.smithi102:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968.client25235/status') 2024-04-25T01:55:14.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T01:55:15.019 INFO:teuthology.orchestra.run.smithi102.stdout:instance: client.25235 (3)172.21.15.102:0/2309142047 2024-04-25T01:55:15.019 INFO:teuthology.orchestra.run.smithi102.stdout:blocklisted: false 2024-04-25T01:55:15.020 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records in 2024-04-25T01:55:15.020 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records out 2024-04-25T01:55:15.020 INFO:teuthology.orchestra.run.smithi102.stderr:72 bytes copied, 8.4237e-05 s, 855 kB/s 2024-04-25T01:55:15.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T01:55:15.032 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:55:15.041 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:55:15.041 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T01:55:15.041 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-25T01:55:15.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.227+0000 7f4def200640 1 -- 172.21.15.102:0/2392005888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4de80ffa90 msgr2=0x7f4de80ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:15.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.227+0000 7f4def200640 1 --2- 172.21.15.102:0/2392005888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4de80ffa90 0x7f4de80ffeb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8009900 tx=0x7f4dd802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.228+0000 7f4def200640 1 -- 172.21.15.102:0/2392005888 shutdown_connections 2024-04-25T01:55:15.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.228+0000 7f4def200640 1 --2- 172.21.15.102:0/2392005888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de81011b0 0x7f4de8105a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.228+0000 7f4def200640 1 --2- 172.21.15.102:0/2392005888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de81007f0 0x7f4de8100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.228+0000 7f4def200640 1 --2- 172.21.15.102:0/2392005888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4de80ffa90 0x7f4de80ffeb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.228+0000 7f4def200640 1 -- 172.21.15.102:0/2392005888 >> 172.21.15.102:0/2392005888 conn(0x7f4de80fb3c0 msgr2=0x7f4de80fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:15.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.228+0000 7f4def200640 1 -- 172.21.15.102:0/2392005888 shutdown_connections 2024-04-25T01:55:15.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.228+0000 7f4def200640 1 -- 172.21.15.102:0/2392005888 wait complete. 2024-04-25T01:55:15.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4def200640 1 Processor -- start 2024-04-25T01:55:15.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4def200640 1 -- start start 2024-04-25T01:55:15.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4def200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4de80ffa90 0x7f4de819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:15.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4def200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de81007f0 0x7f4de819e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:15.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4def200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de81011b0 0x7f4de81a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:15.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4def200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4de8113ac0 con 0x7f4de81011b0 2024-04-25T01:55:15.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4def200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4de8113940 con 0x7f4de81007f0 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4def200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4de8113c40 con 0x7f4de80ffa90 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4ded200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de81011b0 0x7f4de81a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4de7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4de80ffa90 0x7f4de819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4de7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4de80ffa90 0x7f4de819dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35366/0 (socket says 172.21.15.102:35366) 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4ded200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de81011b0 0x7f4de81a5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33632/0 (socket says 172.21.15.102:33632) 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4de7e00640 1 -- 172.21.15.102:0/2921816294 learned_addr learned my addr 172.21.15.102:0/2921816294 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.229+0000 7f4de7400640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de81007f0 0x7f4de819e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.230+0000 7f4de7400640 1 -- 172.21.15.102:0/2921816294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4de80ffa90 msgr2=0x7f4de819dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.230+0000 7f4de7400640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4de80ffa90 0x7f4de819dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.230+0000 7f4de7400640 1 -- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de81011b0 msgr2=0x7f4de81a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.230+0000 7f4de7400640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de81011b0 0x7f4de81a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.230+0000 7f4de7400640 1 -- 172.21.15.102:0/2921816294 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4de81a5c00 con 0x7f4de81007f0 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.230+0000 7f4ded200640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de81011b0 0x7f4de81a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:15.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.230+0000 7f4de7400640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de81007f0 0x7f4de819e520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400e9c0 tx=0x7f4dd400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:15.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.230+0000 7f4de4c00640 1 -- 172.21.15.102:0/2921816294 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd400ccc0 con 0x7f4de81007f0 2024-04-25T01:55:15.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.231+0000 7f4de4c00640 1 -- 172.21.15.102:0/2921816294 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dd40035f0 con 0x7f4de81007f0 2024-04-25T01:55:15.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.231+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4de81a5ef0 con 0x7f4de81007f0 2024-04-25T01:55:15.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.231+0000 7f4de4c00640 1 -- 172.21.15.102:0/2921816294 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd400ce60 con 0x7f4de81007f0 2024-04-25T01:55:15.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.231+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4de81a63b0 con 0x7f4de81007f0 2024-04-25T01:55:15.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.232+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4de8104860 con 0x7f4de81007f0 2024-04-25T01:55:15.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.237+0000 7f4de4c00640 1 -- 172.21.15.102:0/2921816294 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 20) ==== 137851+0+0 (secure 0 0 0) 0x7f4dd40037a0 con 0x7f4de81007f0 2024-04-25T01:55:15.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.238+0000 7f4de4c00640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dcc0a2c80 0x7f4dcc0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:15.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.238+0000 7f4de7e00640 1 -- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dcc0a2c80 msgr2=0x7f4dcc0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/83904525 2024-04-25T01:55:15.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.238+0000 7f4de4c00640 1 -- 172.21.15.102:0/2921816294 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8211+0+0 (secure 0 0 0) 0x7f4dd40cf580 con 0x7f4de81007f0 2024-04-25T01:55:15.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.238+0000 7f4de7e00640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dcc0a2c80 0x7f4dcc0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T01:55:15.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.238+0000 7f4de4c00640 1 -- 172.21.15.102:0/2921816294 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4dd4098490 con 0x7f4de81007f0 2024-04-25T01:55:15.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.439+0000 7f4de7e00640 1 -- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dcc0a2c80 msgr2=0x7f4dcc0a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.102:6800/83904525 2024-04-25T01:55:15.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.439+0000 7f4de7e00640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dcc0a2c80 0x7f4dcc0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T01:55:15.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.478+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f4de802d170 con 0x7f4de81007f0 2024-04-25T01:55:15.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.536+0000 7f4de4c00640 1 -- 172.21.15.102:0/2921816294 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v23) ==== 117+0+0 (secure 0 0 0) 0x7f4dd409c520 con 0x7f4de81007f0 2024-04-25T01:55:15.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.539+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dcc0a2c80 msgr2=0x7f4dcc0a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:55:15.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.539+0000 7f4def200640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dcc0a2c80 0x7f4dcc0a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.539+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de81007f0 msgr2=0x7f4de819e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:15.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.539+0000 7f4def200640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de81007f0 0x7f4de819e520 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400e9c0 tx=0x7f4dd400ee80 comp rx=0 tx=0).stop 2024-04-25T01:55:15.541 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.261+0000 7f50ea592200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T01:55:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.540+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 shutdown_connections 2024-04-25T01:55:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.540+0000 7f4def200640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:6800/83904525,v1:172.21.15.102:6801/83904525] conn(0x7f4dcc0a2c80 0x7f4dcc0a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.540+0000 7f4def200640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de81011b0 0x7f4de81a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.540+0000 7f4def200640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4de81007f0 0x7f4de819e520 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.540+0000 7f4def200640 1 --2- 172.21.15.102:0/2921816294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4de80ffa90 0x7f4de819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.540+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 >> 172.21.15.102:0/2921816294 conn(0x7f4de80fb3c0 msgr2=0x7f4de8101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.540+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 shutdown_connections 2024-04-25T01:55:15.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.540+0000 7f4def200640 1 -- 172.21.15.102:0/2921816294 wait complete. 2024-04-25T01:55:15.561 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T01:55:15.572 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-25T01:55:15.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:55:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.793+0000 7f7d51c00640 1 -- 172.21.15.102:0/1877112860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c0ffe30 msgr2=0x7f7d4c102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.793+0000 7f7d51c00640 1 --2- 172.21.15.102:0/1877112860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c0ffe30 0x7f7d4c102260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009900 tx=0x7f7d3402a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.793+0000 7f7d51c00640 1 -- 172.21.15.102:0/1877112860 shutdown_connections 2024-04-25T01:55:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.793+0000 7f7d51c00640 1 --2- 172.21.15.102:0/1877112860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d4c105150 0x7f7d4c1055d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.793+0000 7f7d51c00640 1 --2- 172.21.15.102:0/1877112860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d4c1027a0 0x7f7d4c104c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.793+0000 7f7d51c00640 1 --2- 172.21.15.102:0/1877112860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c0ffe30 0x7f7d4c102260 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.793+0000 7f7d51c00640 1 -- 172.21.15.102:0/1877112860 >> 172.21.15.102:0/1877112860 conn(0x7f7d4c0fba10 msgr2=0x7f7d4c0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.793+0000 7f7d51c00640 1 -- 172.21.15.102:0/1877112860 shutdown_connections 2024-04-25T01:55:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.793+0000 7f7d51c00640 1 -- 172.21.15.102:0/1877112860 wait complete. 2024-04-25T01:55:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.794+0000 7f7d51c00640 1 Processor -- start 2024-04-25T01:55:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.794+0000 7f7d51c00640 1 -- start start 2024-04-25T01:55:15.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d51c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d4c0ffe30 0x7f7d4c10e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d51c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c1027a0 0x7f7d4c10eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d51c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d4c105150 0x7f7d4c10fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d51c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c106d50 con 0x7f7d4c0ffe30 2024-04-25T01:55:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d51c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c106bd0 con 0x7f7d4c1027a0 2024-04-25T01:55:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d51c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c106ed0 con 0x7f7d4c105150 2024-04-25T01:55:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d43e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c1027a0 0x7f7d4c10eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d43e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c1027a0 0x7f7d4c10eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32924/0 (socket says 172.21.15.102:32924) 2024-04-25T01:55:15.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d43e00640 1 -- 172.21.15.102:0/3018038286 learned_addr learned my addr 172.21.15.102:0/3018038286 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:15.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d51200640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d4c105150 0x7f7d4c10fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:15.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d4be00640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d4c0ffe30 0x7f7d4c10e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d43e00640 1 -- 172.21.15.102:0/3018038286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d4c105150 msgr2=0x7f7d4c10fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d43e00640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d4c105150 0x7f7d4c10fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d43e00640 1 -- 172.21.15.102:0/3018038286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d4c0ffe30 msgr2=0x7f7d4c10e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d43e00640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d4c0ffe30 0x7f7d4c10e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.795+0000 7f7d43e00640 1 -- 172.21.15.102:0/3018038286 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4c1102b0 con 0x7f7d4c1027a0 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.796+0000 7f7d51200640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d4c105150 0x7f7d4c10fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.796+0000 7f7d43e00640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c1027a0 0x7f7d4c10eca0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00ea60 tx=0x7f7d3c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.796+0000 7f7d49600640 1 -- 172.21.15.102:0/3018038286 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c00cde0 con 0x7f7d4c1027a0 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.796+0000 7f7d49600640 1 -- 172.21.15.102:0/3018038286 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d3c0035f0 con 0x7f7d4c1027a0 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.796+0000 7f7d49600640 1 -- 172.21.15.102:0/3018038286 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c010670 con 0x7f7d4c1027a0 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.796+0000 7f7d51c00640 1 -- 172.21.15.102:0/3018038286 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4c1105a0 con 0x7f7d4c1027a0 2024-04-25T01:55:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.796+0000 7f7d51c00640 1 -- 172.21.15.102:0/3018038286 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d4c1aecb0 con 0x7f7d4c1027a0 2024-04-25T01:55:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.798+0000 7f7d51c00640 1 -- 172.21.15.102:0/3018038286 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d4c108690 con 0x7f7d4c1027a0 2024-04-25T01:55:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.801+0000 7f7d49600640 1 -- 172.21.15.102:0/3018038286 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 21) ==== 137518+0+0 (secure 0 0 0) 0x7f7d3c0037a0 con 0x7f7d4c1027a0 2024-04-25T01:55:15.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.804+0000 7f7d49600640 1 -- 172.21.15.102:0/3018038286 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7d3c0cee30 con 0x7f7d4c1027a0 2024-04-25T01:55:15.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:15.805+0000 7f7d49600640 1 -- 172.21.15.102:0/3018038286 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7d3c09c0c0 con 0x7f7d4c1027a0 2024-04-25T01:55:15.811 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:15 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2921816294' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:55:15.811 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:15 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:55:15.826 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:15 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2921816294' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:55:15.827 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:15 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:55:15.843 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2921816294' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:55:15.844 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T01:55:15.844 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.563+0000 7f50ea592200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T01:55:16.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.076+0000 7f7d51c00640 1 -- 172.21.15.102:0/3018038286 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d4c1af040 con 0x7f7d4c1027a0 2024-04-25T01:55:16.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.078+0000 7f7d49600640 1 -- 172.21.15.102:0/3018038286 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23064 (secure 0 0 0) 0x7f7d3c09b9c0 con 0x7f7d4c1027a0 2024-04-25T01:55:16.081 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:16.081 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":90,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:55:15.567792+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6801/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/459072323":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/2875896366":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1835499087":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/511834001":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:55:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.082+0000 7f7d51c00640 1 -- 172.21.15.102:0/3018038286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c1027a0 msgr2=0x7f7d4c10eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.082+0000 7f7d51c00640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c1027a0 0x7f7d4c10eca0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00ea60 tx=0x7f7d3c00ef20 comp rx=0 tx=0).stop 2024-04-25T01:55:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.082+0000 7f7d51c00640 1 -- 172.21.15.102:0/3018038286 shutdown_connections 2024-04-25T01:55:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.082+0000 7f7d51c00640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d4c105150 0x7f7d4c10fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.082+0000 7f7d51c00640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d4c1027a0 0x7f7d4c10eca0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.082+0000 7f7d51c00640 1 --2- 172.21.15.102:0/3018038286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d4c0ffe30 0x7f7d4c10e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.082+0000 7f7d51c00640 1 -- 172.21.15.102:0/3018038286 >> 172.21.15.102:0/3018038286 conn(0x7f7d4c0fba10 msgr2=0x7f7d4c0fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.082+0000 7f7d51c00640 1 -- 172.21.15.102:0/3018038286 shutdown_connections 2024-04-25T01:55:16.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.082+0000 7f7d51c00640 1 -- 172.21.15.102:0/3018038286 wait complete. 2024-04-25T01:55:16.106 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.907+0000 7f503ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.106 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.907+0000 7f503ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.106 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.907+0000 7f503ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.106 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.907+0000 7f503ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.106 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.907+0000 7f503ce00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.107 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.909+0000 7f5040000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.107 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.909+0000 7f5040000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.107 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.909+0000 7f5040000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.107 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.909+0000 7f5040000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.107 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:55:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:55:15.909+0000 7f5040000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:55:16.109 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:41.585137+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:43.022666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:55:16.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:55:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.280+0000 7f3211800640 1 -- 172.21.15.102:0/560876825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c101390 msgr2=0x7f320c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.280+0000 7f3211800640 1 --2- 172.21.15.102:0/560876825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c101390 0x7f320c101810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3204009ca0 tx=0x7f320402a950 comp rx=0 tx=0).stop 2024-04-25T01:55:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.280+0000 7f3211800640 1 -- 172.21.15.102:0/560876825 shutdown_connections 2024-04-25T01:55:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.280+0000 7f3211800640 1 --2- 172.21.15.102:0/560876825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c101390 0x7f320c101810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.280+0000 7f3211800640 1 --2- 172.21.15.102:0/560876825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f320c1009f0 0x7f320c100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.280+0000 7f3211800640 1 --2- 172.21.15.102:0/560876825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f320c100090 0x7f320c1004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.280+0000 7f3211800640 1 -- 172.21.15.102:0/560876825 >> 172.21.15.102:0/560876825 conn(0x7f320c0fbc10 msgr2=0x7f320c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.280+0000 7f3211800640 1 -- 172.21.15.102:0/560876825 shutdown_connections 2024-04-25T01:55:16.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.280+0000 7f3211800640 1 -- 172.21.15.102:0/560876825 wait complete. 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f3211800640 1 Processor -- start 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f3211800640 1 -- start start 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f3211800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c100090 0x7f320c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f3211800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f320c1009f0 0x7f320c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f3211800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f320c101390 0x7f320c1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f3211800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f320c10f670 con 0x7f320c1009f0 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f3211800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f320c10f4f0 con 0x7f320c100090 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f3211800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f320c10f7f0 con 0x7f320c101390 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f320aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c100090 0x7f320c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f320aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c100090 0x7f320c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33000/0 (socket says 172.21.15.102:33000) 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f3203e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f320c1009f0 0x7f320c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f320aa00640 1 -- 172.21.15.102:0/1046635764 learned_addr learned my addr 172.21.15.102:0/1046635764 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.281+0000 7f320b400640 1 --2- 172.21.15.102:0/1046635764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f320c101390 0x7f320c1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f320aa00640 1 -- 172.21.15.102:0/1046635764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f320c101390 msgr2=0x7f320c1a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f320aa00640 1 --2- 172.21.15.102:0/1046635764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f320c101390 0x7f320c1a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f320aa00640 1 -- 172.21.15.102:0/1046635764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f320c1009f0 msgr2=0x7f320c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f320aa00640 1 --2- 172.21.15.102:0/1046635764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f320c1009f0 0x7f320c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f320aa00640 1 -- 172.21.15.102:0/1046635764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320c1a1880 con 0x7f320c100090 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f320aa00640 1 --2- 172.21.15.102:0/1046635764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c100090 0x7f320c199c70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00e9f0 tx=0x7f31fc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f3203400640 1 -- 172.21.15.102:0/1046635764 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00cd80 con 0x7f320c100090 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f3203400640 1 -- 172.21.15.102:0/1046635764 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31fc005500 con 0x7f320c100090 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f3211800640 1 -- 172.21.15.102:0/1046635764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f320c1a1b70 con 0x7f320c100090 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f3203400640 1 -- 172.21.15.102:0/1046635764 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc010690 con 0x7f320c100090 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.282+0000 7f3211800640 1 -- 172.21.15.102:0/1046635764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f320c1a2070 con 0x7f320c100090 2024-04-25T01:55:16.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.283+0000 7f3211800640 1 -- 172.21.15.102:0/1046635764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f320c10aee0 con 0x7f320c100090 2024-04-25T01:55:16.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.285+0000 7f3203400640 1 -- 172.21.15.102:0/1046635764 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 21) ==== 137518+0+0 (secure 0 0 0) 0x7f31fc0046c0 con 0x7f320c100090 2024-04-25T01:55:16.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.287+0000 7f3203400640 1 -- 172.21.15.102:0/1046635764 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f31fc0cf090 con 0x7f320c100090 2024-04-25T01:55:16.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.290+0000 7f3203400640 1 -- 172.21.15.102:0/1046635764 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f31fc09c2a0 con 0x7f320c100090 2024-04-25T01:55:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.486+0000 7f3211800640 1 -- 172.21.15.102:0/1046635764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f320c0623e0 con 0x7f320c100090 2024-04-25T01:55:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.487+0000 7f3203400640 1 -- 172.21.15.102:0/1046635764 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f31fc09bba0 con 0x7f320c100090 2024-04-25T01:55:16.489 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:55:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.490+0000 7f3211800640 1 -- 172.21.15.102:0/1046635764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c100090 msgr2=0x7f320c199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.490+0000 7f3211800640 1 --2- 172.21.15.102:0/1046635764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c100090 0x7f320c199c70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00e9f0 tx=0x7f31fc00edf0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.490+0000 7f3211800640 1 -- 172.21.15.102:0/1046635764 shutdown_connections 2024-04-25T01:55:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.490+0000 7f3211800640 1 --2- 172.21.15.102:0/1046635764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f320c101390 0x7f320c1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.490+0000 7f3211800640 1 --2- 172.21.15.102:0/1046635764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f320c1009f0 0x7f320c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.490+0000 7f3211800640 1 --2- 172.21.15.102:0/1046635764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f320c100090 0x7f320c199c70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.490+0000 7f3211800640 1 -- 172.21.15.102:0/1046635764 >> 172.21.15.102:0/1046635764 conn(0x7f320c0fbc10 msgr2=0x7f320c0fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.490+0000 7f3211800640 1 -- 172.21.15.102:0/1046635764 shutdown_connections 2024-04-25T01:55:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.490+0000 7f3211800640 1 -- 172.21.15.102:0/1046635764 wait complete. 2024-04-25T01:55:16.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:55:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.706+0000 7fd6dfe00640 1 -- 172.21.15.102:0/657666868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0104a70 msgr2=0x7fd6e0104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.706+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/657666868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0104a70 0x7fd6e0104ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc009900 tx=0x7fd6cc02a410 comp rx=0 tx=0).stop 2024-04-25T01:55:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.706+0000 7fd6dfe00640 1 -- 172.21.15.102:0/657666868 shutdown_connections 2024-04-25T01:55:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.706+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/657666868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6e0105430 0x7fd6e010bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.706+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/657666868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0104a70 0x7fd6e0104ef0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.706+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/657666868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e0103880 0x7fd6e0103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.706+0000 7fd6dfe00640 1 -- 172.21.15.102:0/657666868 >> 172.21.15.102:0/657666868 conn(0x7fd6e00fedf0 msgr2=0x7fd6e0101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.706+0000 7fd6dfe00640 1 -- 172.21.15.102:0/657666868 shutdown_connections 2024-04-25T01:55:16.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.706+0000 7fd6dfe00640 1 -- 172.21.15.102:0/657666868 wait complete. 2024-04-25T01:55:16.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6dfe00640 1 Processor -- start 2024-04-25T01:55:16.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6dfe00640 1 -- start start 2024-04-25T01:55:16.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6dfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e0103880 0x7fd6e0199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6dfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6e0104a70 0x7fd6e019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6dfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0105430 0x7fd6e01a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6dfe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e010f530 con 0x7fd6e0104a70 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6dfe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e010f3b0 con 0x7fd6e0105430 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6dfe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e010f6b0 con 0x7fd6e0103880 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6df400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0105430 0x7fd6e01a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6de000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6e0104a70 0x7fd6e019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6df400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0105430 0x7fd6e01a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33012/0 (socket says 172.21.15.102:33012) 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6df400640 1 -- 172.21.15.102:0/2306829916 learned_addr learned my addr 172.21.15.102:0/2306829916 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.708+0000 7fd6dea00640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e0103880 0x7fd6e0199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6df400640 1 -- 172.21.15.102:0/2306829916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e0103880 msgr2=0x7fd6e0199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6df400640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e0103880 0x7fd6e0199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6df400640 1 -- 172.21.15.102:0/2306829916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6e0104a70 msgr2=0x7fd6e019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6df400640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6e0104a70 0x7fd6e019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6df400640 1 -- 172.21.15.102:0/2306829916 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e01a1760 con 0x7fd6e0105430 2024-04-25T01:55:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6df400640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0105430 0x7fd6e01a10f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800eaa0 tx=0x7fd6d800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6d7400640 1 -- 172.21.15.102:0/2306829916 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d800cdb0 con 0x7fd6e0105430 2024-04-25T01:55:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6d7400640 1 -- 172.21.15.102:0/2306829916 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6d8005020 con 0x7fd6e0105430 2024-04-25T01:55:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6d7400640 1 -- 172.21.15.102:0/2306829916 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d80116d0 con 0x7fd6e0105430 2024-04-25T01:55:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e01a1a50 con 0x7fd6e0105430 2024-04-25T01:55:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.709+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e01a1f70 con 0x7fd6e0105430 2024-04-25T01:55:16.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.710+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e0068480 con 0x7fd6e0105430 2024-04-25T01:55:16.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.712+0000 7fd6d7400640 1 -- 172.21.15.102:0/2306829916 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7fd6d8005540 con 0x7fd6e0105430 2024-04-25T01:55:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.713+0000 7fd6d7400640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd6c40a2a10 0x7fd6c40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.714+0000 7fd6d7400640 1 -- 172.21.15.102:0/2306829916 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd6d80cee60 con 0x7fd6e0105430 2024-04-25T01:55:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.714+0000 7fd6dea00640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd6c40a2a10 0x7fd6c40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.714+0000 7fd6dea00640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd6c40a2a10 0x7fd6c40a4ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0005fd0 tx=0x7fd6d000b410 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:16.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.717+0000 7fd6d7400640 1 -- 172.21.15.102:0/2306829916 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd6d8016030 con 0x7fd6e0105430 2024-04-25T01:55:16.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: Active manager daemon x restarted 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: Activating manager daemon x 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: mgrmap e21: x(active, starting, since 0.013394s), standbys: z, y 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:55:16.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:55:16.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: Manager daemon x is now available 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:55:16.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:55:16.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3018038286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:16.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: [25/Apr/2024:01:55:16] ENGINE Bus STARTING 2024-04-25T01:55:16.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: [25/Apr/2024:01:55:16] ENGINE Serving on https://172.21.15.102:7150 2024-04-25T01:55:16.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:16 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1046635764' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:16.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T01:55:16.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:16.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: Active manager daemon x restarted 2024-04-25T01:55:16.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: Activating manager daemon x 2024-04-25T01:55:16.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:55:16.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: mgrmap e21: x(active, starting, since 0.013394s), standbys: z, y 2024-04-25T01:55:16.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:55:16.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:55:16.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:55:16.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: Manager daemon x is now available 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3018038286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: [25/Apr/2024:01:55:16] ENGINE Bus STARTING 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: [25/Apr/2024:01:55:16] ENGINE Serving on https://172.21.15.102:7150 2024-04-25T01:55:16.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:16 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1046635764' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:16.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.925+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd6e0062340 con 0x7fd6e0105430 2024-04-25T01:55:16.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.925+0000 7fd6d7400640 1 -- 172.21.15.102:0/2306829916 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fd6d809b930 con 0x7fd6e0105430 2024-04-25T01:55:16.928 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.927+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd6c40a2a10 msgr2=0x7fd6c40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.927+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd6c40a2a10 0x7fd6c40a4ed0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0005fd0 tx=0x7fd6d000b410 comp rx=0 tx=0).stop 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.927+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0105430 msgr2=0x7fd6e01a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.927+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0105430 0x7fd6e01a10f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800eaa0 tx=0x7fd6d800ef60 comp rx=0 tx=0).stop 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.928+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 shutdown_connections 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.928+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd6c40a2a10 0x7fd6c40a4ed0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.928+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6e0105430 0x7fd6e01a10f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.928+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6e0104a70 0x7fd6e019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.928+0000 7fd6dfe00640 1 --2- 172.21.15.102:0/2306829916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e0103880 0x7fd6e0199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.928+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 >> 172.21.15.102:0/2306829916 conn(0x7fd6e00fedf0 msgr2=0x7fd6e0107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:16.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.928+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 shutdown_connections 2024-04-25T01:55:16.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:16.928+0000 7fd6dfe00640 1 -- 172.21.15.102:0/2306829916 wait complete. 2024-04-25T01:55:16.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:55:16.988 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: Active manager daemon x restarted 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: Activating manager daemon x 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: mgrmap e21: x(active, starting, since 0.013394s), standbys: z, y 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T01:55:16.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T01:55:16.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: Manager daemon x is now available 2024-04-25T01:55:16.991 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:55:16.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:55:16.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T01:55:16.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:55:16.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T01:55:16.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3018038286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:16.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: [25/Apr/2024:01:55:16] ENGINE Bus STARTING 2024-04-25T01:55:16.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: [25/Apr/2024:01:55:16] ENGINE Serving on https://172.21.15.102:7150 2024-04-25T01:55:16.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:16 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1046635764' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:17.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.124+0000 7f638c800640 1 -- 172.21.15.102:0/2614956712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384102200 msgr2=0x7f63841045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.124+0000 7f638c800640 1 --2- 172.21.15.102:0/2614956712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384102200 0x7f63841045f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6378009900 tx=0x7f637802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.124+0000 7f638c800640 1 -- 172.21.15.102:0/2614956712 shutdown_connections 2024-04-25T01:55:17.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.124+0000 7f638c800640 1 --2- 172.21.15.102:0/2614956712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6384107460 0x7f63841098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.124+0000 7f638c800640 1 --2- 172.21.15.102:0/2614956712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6384104b30 0x7f6384106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.124+0000 7f638c800640 1 --2- 172.21.15.102:0/2614956712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384102200 0x7f63841045f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.124+0000 7f638c800640 1 -- 172.21.15.102:0/2614956712 >> 172.21.15.102:0/2614956712 conn(0x7f63840fbc30 msgr2=0x7f63840fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:17.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.124+0000 7f638c800640 1 -- 172.21.15.102:0/2614956712 shutdown_connections 2024-04-25T01:55:17.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.124+0000 7f638c800640 1 -- 172.21.15.102:0/2614956712 wait complete. 2024-04-25T01:55:17.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.125+0000 7f638c800640 1 Processor -- start 2024-04-25T01:55:17.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.125+0000 7f638c800640 1 -- start start 2024-04-25T01:55:17.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.125+0000 7f638c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6384102200 0x7f6384197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:17.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.125+0000 7f638c800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6384104b30 0x7f6384198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.125+0000 7f638c800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384107460 0x7f638419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.125+0000 7f638c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f638410d450 con 0x7f6384102200 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.125+0000 7f638c800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f638410d2d0 con 0x7f6384107460 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.125+0000 7f638c800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f638410d5d0 con 0x7f6384104b30 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f638a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384107460 0x7f638419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f6389e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6384102200 0x7f6384197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f638a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384107460 0x7f638419f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33020/0 (socket says 172.21.15.102:33020) 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f638a800640 1 -- 172.21.15.102:0/1168681653 learned_addr learned my addr 172.21.15.102:0/1168681653 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f6389400640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6384104b30 0x7f6384198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f638a800640 1 -- 172.21.15.102:0/1168681653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6384104b30 msgr2=0x7f6384198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f638a800640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6384104b30 0x7f6384198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f638a800640 1 -- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6384102200 msgr2=0x7f6384197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f638a800640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6384102200 0x7f6384197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f638a800640 1 -- 172.21.15.102:0/1168681653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638419f740 con 0x7f6384107460 2024-04-25T01:55:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f6389400640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6384104b30 0x7f6384198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f6389e00640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6384102200 0x7f6384197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.126+0000 7f638a800640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384107460 0x7f638419f0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f638000eb10 tx=0x7f638000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.127+0000 7f637ea00640 1 -- 172.21.15.102:0/1168681653 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6380004b20 con 0x7f6384107460 2024-04-25T01:55:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.127+0000 7f637ea00640 1 -- 172.21.15.102:0/1168681653 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6380004cc0 con 0x7f6384107460 2024-04-25T01:55:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.127+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6384100050 con 0x7f6384107460 2024-04-25T01:55:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.127+0000 7f637ea00640 1 -- 172.21.15.102:0/1168681653 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6380010760 con 0x7f6384107460 2024-04-25T01:55:17.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.127+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6384100550 con 0x7f6384107460 2024-04-25T01:55:17.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.128+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63840fd720 con 0x7f6384107460 2024-04-25T01:55:17.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.130+0000 7f637ea00640 1 -- 172.21.15.102:0/1168681653 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f6380005020 con 0x7f6384107460 2024-04-25T01:55:17.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.131+0000 7f637ea00640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f636c0a2990 0x7f636c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:17.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.131+0000 7f637ea00640 1 -- 172.21.15.102:0/1168681653 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f63800cf320 con 0x7f6384107460 2024-04-25T01:55:17.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.131+0000 7f6389e00640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f636c0a2990 0x7f636c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:17.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.132+0000 7f6389e00640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f636c0a2990 0x7f636c0a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6378031440 tx=0x7f63780315b0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:17.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.135+0000 7f637ea00640 1 -- 172.21.15.102:0/1168681653 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f638009c440 con 0x7f6384107460 2024-04-25T01:55:17.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.323+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6384101990 con 0x7f6384107460 2024-04-25T01:55:17.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.324+0000 7f637ea00640 1 -- 172.21.15.102:0/1168681653 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f638009bd40 con 0x7f6384107460 2024-04-25T01:55:17.327 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-25T01:55:17.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.327+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f636c0a2990 msgr2=0x7f636c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.327+0000 7f638c800640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f636c0a2990 0x7f636c0a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6378031440 tx=0x7f63780315b0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.327+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384107460 msgr2=0x7f638419f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.327+0000 7f638c800640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384107460 0x7f638419f0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f638000eb10 tx=0x7f638000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.328+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 shutdown_connections 2024-04-25T01:55:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.328+0000 7f638c800640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f636c0a2990 0x7f636c0a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.328+0000 7f638c800640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6384107460 0x7f638419f0d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.328+0000 7f638c800640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6384104b30 0x7f6384198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.328+0000 7f638c800640 1 --2- 172.21.15.102:0/1168681653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6384102200 0x7f6384197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.328+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 >> 172.21.15.102:0/1168681653 conn(0x7f63840fbc30 msgr2=0x7f6384102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:17.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.328+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 shutdown_connections 2024-04-25T01:55:17.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.328+0000 7f638c800640 1 -- 172.21.15.102:0/1168681653 wait complete. 2024-04-25T01:55:17.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:55:17.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.540+0000 7f0d4de00640 1 -- 172.21.15.102:0/1127476447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48104b40 msgr2=0x7f0d48106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.540+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1127476447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48104b40 0x7f0d48106f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009900 tx=0x7f0d3402a400 comp rx=0 tx=0).stop 2024-04-25T01:55:17.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.540+0000 7f0d4de00640 1 -- 172.21.15.102:0/1127476447 shutdown_connections 2024-04-25T01:55:17.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.540+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1127476447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48107470 0x7f0d481098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.540+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1127476447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48104b40 0x7f0d48106f30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.540+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1127476447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d48102210 0x7f0d48104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.540+0000 7f0d4de00640 1 -- 172.21.15.102:0/1127476447 >> 172.21.15.102:0/1127476447 conn(0x7f0d480fbc00 msgr2=0x7f0d480fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:17.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.541+0000 7f0d4de00640 1 -- 172.21.15.102:0/1127476447 shutdown_connections 2024-04-25T01:55:17.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.541+0000 7f0d4de00640 1 -- 172.21.15.102:0/1127476447 wait complete. 2024-04-25T01:55:17.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.542+0000 7f0d4de00640 1 Processor -- start 2024-04-25T01:55:17.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.542+0000 7f0d4de00640 1 -- start start 2024-04-25T01:55:17.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.542+0000 7f0d4de00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d48102210 0x7f0d48101b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:17.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.542+0000 7f0d4de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48104b40 0x7f0d481001c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:17.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.542+0000 7f0d4de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48107470 0x7f0d48100700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:17.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.542+0000 7f0d4de00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4810d400 con 0x7f0d48104b40 2024-04-25T01:55:17.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.542+0000 7f0d4de00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4810d280 con 0x7f0d48102210 2024-04-25T01:55:17.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.542+0000 7f0d4de00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4810d580 con 0x7f0d48107470 2024-04-25T01:55:17.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.543+0000 7f0d46000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48104b40 0x7f0d481001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:17.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.543+0000 7f0d47400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48107470 0x7f0d48100700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:17.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.543+0000 7f0d46000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48104b40 0x7f0d481001c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33848/0 (socket says 172.21.15.102:33848) 2024-04-25T01:55:17.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.543+0000 7f0d46000640 1 -- 172.21.15.102:0/1435950321 learned_addr learned my addr 172.21.15.102:0/1435950321 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:17.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.543+0000 7f0d47400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48107470 0x7f0d48100700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35616/0 (socket says 172.21.15.102:35616) 2024-04-25T01:55:17.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.543+0000 7f0d46a00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d48102210 0x7f0d48101b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:17.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.543+0000 7f0d46000640 1 -- 172.21.15.102:0/1435950321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48107470 msgr2=0x7f0d48100700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.543+0000 7f0d46000640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48107470 0x7f0d48100700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.543+0000 7f0d46000640 1 -- 172.21.15.102:0/1435950321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d48102210 msgr2=0x7f0d48101b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.544+0000 7f0d46000640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d48102210 0x7f0d48101b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.544+0000 7f0d46000640 1 -- 172.21.15.102:0/1435950321 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d48100f90 con 0x7f0d48104b40 2024-04-25T01:55:17.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.544+0000 7f0d47400640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48107470 0x7f0d48100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:17.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.544+0000 7f0d46a00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d48102210 0x7f0d48101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:17.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.544+0000 7f0d46000640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48104b40 0x7f0d481001c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d34031610 tx=0x7f0d34004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:17.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.544+0000 7f0d4ca00640 1 -- 172.21.15.102:0/1435950321 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3403f030 con 0x7f0d48104b40 2024-04-25T01:55:17.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.544+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d481a1ba0 con 0x7f0d48104b40 2024-04-25T01:55:17.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.544+0000 7f0d4ca00640 1 -- 172.21.15.102:0/1435950321 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d34003ad0 con 0x7f0d48104b40 2024-04-25T01:55:17.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.544+0000 7f0d4ca00640 1 -- 172.21.15.102:0/1435950321 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d34035070 con 0x7f0d48104b40 2024-04-25T01:55:17.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.545+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d481a20f0 con 0x7f0d48104b40 2024-04-25T01:55:17.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.546+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d480fd6f0 con 0x7f0d48104b40 2024-04-25T01:55:17.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.548+0000 7f0d4ca00640 1 -- 172.21.15.102:0/1435950321 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7f0d34042030 con 0x7f0d48104b40 2024-04-25T01:55:17.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.550+0000 7f0d4ca00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d2c0a2990 0x7f0d2c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:17.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.550+0000 7f0d4ca00640 1 -- 172.21.15.102:0/1435950321 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0d340eedd0 con 0x7f0d48104b40 2024-04-25T01:55:17.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.550+0000 7f0d46a00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d2c0a2990 0x7f0d2c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:17.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.551+0000 7f0d46a00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d2c0a2990 0x7f0d2c0a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009fc0 tx=0x7f0d38005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:17.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.555+0000 7f0d4ca00640 1 -- 172.21.15.102:0/1435950321 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0d340bbef0 con 0x7f0d48104b40 2024-04-25T01:55:17.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: [25/Apr/2024:01:55:16] ENGINE Serving on http://172.21.15.102:8765 2024-04-25T01:55:17.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: [25/Apr/2024:01:55:16] ENGINE Bus STARTED 2024-04-25T01:55:17.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: mgrmap e22: x(active, since 1.02139s), standbys: z, y 2024-04-25T01:55:17.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:55:17.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:17.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:17.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:17.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:17.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2306829916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:17.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:17 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1168681653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: [25/Apr/2024:01:55:16] ENGINE Serving on http://172.21.15.102:8765 2024-04-25T01:55:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: [25/Apr/2024:01:55:16] ENGINE Bus STARTED 2024-04-25T01:55:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: mgrmap e22: x(active, since 1.02139s), standbys: z, y 2024-04-25T01:55:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:55:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:17.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:17.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:17.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:17.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2306829916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:17.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:17 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1168681653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:17.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.793+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d48062340 con 0x7f0d48104b40 2024-04-25T01:55:17.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.795+0000 7f0d4ca00640 1 -- 172.21.15.102:0/1435950321 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23064 (secure 0 0 0) 0x7f0d340bb7f0 con 0x7f0d48104b40 2024-04-25T01:55:17.799 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:17.799 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":90,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:55:15.567792+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6801/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/459072323":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/2875896366":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1835499087":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/511834001":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:55:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d2c0a2990 msgr2=0x7f0d2c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d2c0a2990 0x7f0d2c0a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d38009fc0 tx=0x7f0d38005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48104b40 msgr2=0x7f0d481001c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48104b40 0x7f0d481001c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d34031610 tx=0x7f0d34004270 comp rx=0 tx=0).stop 2024-04-25T01:55:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 shutdown_connections 2024-04-25T01:55:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d2c0a2990 0x7f0d2c0a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d48107470 0x7f0d48100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d48104b40 0x7f0d481001c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 --2- 172.21.15.102:0/1435950321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d48102210 0x7f0d48101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:17.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 >> 172.21.15.102:0/1435950321 conn(0x7f0d480fbc00 msgr2=0x7f0d48102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:17.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 shutdown_connections 2024-04-25T01:55:17.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.798+0000 7f0d4de00640 1 -- 172.21.15.102:0/1435950321 wait complete. 2024-04-25T01:55:17.818 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:41.585137+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:43.022666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:55:17.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:55:17.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.997+0000 7ff4e6200640 1 -- 172.21.15.102:0/4203657958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e006cc80 msgr2=0x7ff4e006b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:17.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.997+0000 7ff4e6200640 1 --2- 172.21.15.102:0/4203657958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e006cc80 0x7ff4e006b0d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009ca0 tx=0x7ff4d002a950 comp rx=0 tx=0).stop 2024-04-25T01:55:18.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.997+0000 7ff4e6200640 1 -- 172.21.15.102:0/4203657958 shutdown_connections 2024-04-25T01:55:18.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.997+0000 7ff4e6200640 1 --2- 172.21.15.102:0/4203657958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4e0111e10 0x7ff4e01142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.997+0000 7ff4e6200640 1 --2- 172.21.15.102:0/4203657958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4e006b6a0 0x7ff4e006bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.997+0000 7ff4e6200640 1 --2- 172.21.15.102:0/4203657958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e006cc80 0x7ff4e006b0d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.997+0000 7ff4e6200640 1 -- 172.21.15.102:0/4203657958 >> 172.21.15.102:0/4203657958 conn(0x7ff4e0070710 msgr2=0x7ff4e0072b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:18.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.997+0000 7ff4e6200640 1 -- 172.21.15.102:0/4203657958 shutdown_connections 2024-04-25T01:55:18.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.997+0000 7ff4e6200640 1 -- 172.21.15.102:0/4203657958 wait complete. 2024-04-25T01:55:18.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.998+0000 7ff4e6200640 1 Processor -- start 2024-04-25T01:55:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.998+0000 7ff4e6200640 1 -- start start 2024-04-25T01:55:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.998+0000 7ff4e6200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4e006b6a0 0x7ff4e01a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4e6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4e006cc80 0x7ff4e01a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4e6200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e0111e10 0x7ff4e01a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4e6200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e0117e60 con 0x7ff4e006cc80 2024-04-25T01:55:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4e6200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e0117ce0 con 0x7ff4e0111e10 2024-04-25T01:55:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4e6200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e0117fe0 con 0x7ff4e006b6a0 2024-04-25T01:55:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4e006b6a0 0x7ff4e01a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4e006b6a0 0x7ff4e01a24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35642/0 (socket says 172.21.15.102:35642) 2024-04-25T01:55:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4df400640 1 -- 172.21.15.102:0/3045616342 learned_addr learned my addr 172.21.15.102:0/3045616342 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4d6a00640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4e006cc80 0x7ff4e01a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4dfe00640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e0111e10 0x7ff4e01a9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4df400640 1 -- 172.21.15.102:0/3045616342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e0111e10 msgr2=0x7ff4e01a9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4df400640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e0111e10 0x7ff4e01a9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4df400640 1 -- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4e006cc80 msgr2=0x7ff4e01a2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4df400640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4e006cc80 0x7ff4e01a2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4df400640 1 -- 172.21.15.102:0/3045616342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e01aa100 con 0x7ff4e006b6a0 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4dfe00640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e0111e10 0x7ff4e01a9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:17.999+0000 7ff4d6a00640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4e006cc80 0x7ff4e01a2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.000+0000 7ff4df400640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4e006b6a0 0x7ff4e01a24e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff4d002f870 tx=0x7ff4d002fd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.000+0000 7ff4dcc00640 1 -- 172.21.15.102:0/3045616342 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0034040 con 0x7ff4e006b6a0 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.000+0000 7ff4dcc00640 1 -- 172.21.15.102:0/3045616342 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0030920 con 0x7ff4e006b6a0 2024-04-25T01:55:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.000+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e01aa390 con 0x7ff4e006b6a0 2024-04-25T01:55:18.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.000+0000 7ff4dcc00640 1 -- 172.21.15.102:0/3045616342 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0045480 con 0x7ff4e006b6a0 2024-04-25T01:55:18.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.000+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e01aa880 con 0x7ff4e006b6a0 2024-04-25T01:55:18.009 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: [25/Apr/2024:01:55:16] ENGINE Serving on http://172.21.15.102:8765 2024-04-25T01:55:18.009 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: [25/Apr/2024:01:55:16] ENGINE Bus STARTED 2024-04-25T01:55:18.009 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: mgrmap e22: x(active, since 1.02139s), standbys: z, y 2024-04-25T01:55:18.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:55:18.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:18.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:18.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:18.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:18.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2306829916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:18.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:17 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1168681653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.001+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4e0110bd0 con 0x7ff4e006b6a0 2024-04-25T01:55:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.003+0000 7ff4dcc00640 1 -- 172.21.15.102:0/3045616342 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 23) ==== 137954+0+0 (secure 0 0 0) 0x7ff4d0037020 con 0x7ff4e006b6a0 2024-04-25T01:55:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.005+0000 7ff4dcc00640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff4c40a2d80 0x7ff4c40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.005+0000 7ff4dcc00640 1 -- 172.21.15.102:0/3045616342 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff4d00f0320 con 0x7ff4e006b6a0 2024-04-25T01:55:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.005+0000 7ff4d6a00640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff4c40a2d80 0x7ff4c40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.006+0000 7ff4d6a00640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff4c40a2d80 0x7ff4c40a5240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff4e01a39f0 tx=0x7ff4c8005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.007+0000 7ff4dcc00640 1 -- 172.21.15.102:0/3045616342 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff4d00bd020 con 0x7ff4e006b6a0 2024-04-25T01:55:18.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.207+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff4e0062340 con 0x7ff4e006b6a0 2024-04-25T01:55:18.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.208+0000 7ff4dcc00640 1 -- 172.21.15.102:0/3045616342 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7ff4d00bc920 con 0x7ff4e006b6a0 2024-04-25T01:55:18.210 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:55:18.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff4c40a2d80 msgr2=0x7ff4c40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff4c40a2d80 0x7ff4c40a5240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff4e01a39f0 tx=0x7ff4c8005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:18.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4e006b6a0 msgr2=0x7ff4e01a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4e006b6a0 0x7ff4e01a24e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff4d002f870 tx=0x7ff4d002fd60 comp rx=0 tx=0).stop 2024-04-25T01:55:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 shutdown_connections 2024-04-25T01:55:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff4c40a2d80 0x7ff4c40a5240 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4e0111e10 0x7ff4e01a9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4e006cc80 0x7ff4e01a2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 --2- 172.21.15.102:0/3045616342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff4e006b6a0 0x7ff4e01a24e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 >> 172.21.15.102:0/3045616342 conn(0x7ff4e0070710 msgr2=0x7ff4e010dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 shutdown_connections 2024-04-25T01:55:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.211+0000 7ff4e6200640 1 -- 172.21.15.102:0/3045616342 wait complete. 2024-04-25T01:55:18.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:55:18.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.410+0000 7fbee6c00640 1 -- 172.21.15.102:0/2131684594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbee01051e0 msgr2=0x7fbee0105660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.410+0000 7fbee6c00640 1 --2- 172.21.15.102:0/2131684594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbee01051e0 0x7fbee0105660 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009ca0 tx=0x7fbed802a8b0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.410+0000 7fbee6c00640 1 -- 172.21.15.102:0/2131684594 shutdown_connections 2024-04-25T01:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.410+0000 7fbee6c00640 1 --2- 172.21.15.102:0/2131684594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbee01051e0 0x7fbee0105660 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.410+0000 7fbee6c00640 1 --2- 172.21.15.102:0/2131684594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbee0102830 0x7fbee0104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.410+0000 7fbee6c00640 1 --2- 172.21.15.102:0/2131684594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbee00ffec0 0x7fbee01022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.410+0000 7fbee6c00640 1 -- 172.21.15.102:0/2131684594 >> 172.21.15.102:0/2131684594 conn(0x7fbee00fbae0 msgr2=0x7fbee00fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.410+0000 7fbee6c00640 1 -- 172.21.15.102:0/2131684594 shutdown_connections 2024-04-25T01:55:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.410+0000 7fbee6c00640 1 -- 172.21.15.102:0/2131684594 wait complete. 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.411+0000 7fbee6c00640 1 Processor -- start 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.411+0000 7fbee6c00640 1 -- start start 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbee00ffec0 0x7fbee019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbee0102830 0x7fbee019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbee01051e0 0x7fbee01a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbee01138c0 con 0x7fbee0102830 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbee0113740 con 0x7fbee01051e0 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbee0113a40 con 0x7fbee00ffec0 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbee01051e0 0x7fbee01a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbee00ffec0 0x7fbee019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbee01051e0 0x7fbee01a54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33102/0 (socket says 172.21.15.102:33102) 2024-04-25T01:55:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6200640 1 -- 172.21.15.102:0/4104926530 learned_addr learned my addr 172.21.15.102:0/4104926530 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee4e00640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbee0102830 0x7fbee019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6200640 1 -- 172.21.15.102:0/4104926530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbee00ffec0 msgr2=0x7fbee019def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6200640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbee00ffec0 0x7fbee019def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6200640 1 -- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbee0102830 msgr2=0x7fbee019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6200640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbee0102830 0x7fbee019e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.412+0000 7fbee6200640 1 -- 172.21.15.102:0/4104926530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee01a5b10 con 0x7fbee01051e0 2024-04-25T01:55:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.413+0000 7fbee6200640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbee01051e0 0x7fbee01a54a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009dc0 tx=0x7fbed8003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:18.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.413+0000 7fbede000640 1 -- 172.21.15.102:0/4104926530 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8004450 con 0x7fbee01051e0 2024-04-25T01:55:18.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.413+0000 7fbede000640 1 -- 172.21.15.102:0/4104926530 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbed8040070 con 0x7fbee01051e0 2024-04-25T01:55:18.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.413+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee01a5da0 con 0x7fbee01051e0 2024-04-25T01:55:18.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.413+0000 7fbede000640 1 -- 172.21.15.102:0/4104926530 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8045d10 con 0x7fbee01051e0 2024-04-25T01:55:18.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.413+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee01a6260 con 0x7fbee01051e0 2024-04-25T01:55:18.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.414+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee00fd5d0 con 0x7fbee01051e0 2024-04-25T01:55:18.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.416+0000 7fbede000640 1 -- 172.21.15.102:0/4104926530 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 23) ==== 137954+0+0 (secure 0 0 0) 0x7fbed8045470 con 0x7fbee01051e0 2024-04-25T01:55:18.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.417+0000 7fbede000640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbec80a2d00 0x7fbec80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.417+0000 7fbede000640 1 -- 172.21.15.102:0/4104926530 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fbed80f0630 con 0x7fbee01051e0 2024-04-25T01:55:18.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.418+0000 7fbee5800640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbec80a2d00 0x7fbec80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.418+0000 7fbee5800640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbec80a2d00 0x7fbec80a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbed4009bd0 tx=0x7fbed4005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:18.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.421+0000 7fbede000640 1 -- 172.21.15.102:0/4104926530 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbed80bd330 con 0x7fbee01051e0 2024-04-25T01:55:18.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.613+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbee002d170 con 0x7fbee01051e0 2024-04-25T01:55:18.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.614+0000 7fbede000640 1 -- 172.21.15.102:0/4104926530 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fbed80bcc30 con 0x7fbee01051e0 2024-04-25T01:55:18.616 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-25T01:55:18.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbec80a2d00 msgr2=0x7fbec80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbec80a2d00 0x7fbec80a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbed4009bd0 tx=0x7fbed4005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:18.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbee01051e0 msgr2=0x7fbee01a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbee01051e0 0x7fbee01a54a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009dc0 tx=0x7fbed8003c20 comp rx=0 tx=0).stop 2024-04-25T01:55:18.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 shutdown_connections 2024-04-25T01:55:18.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbec80a2d00 0x7fbec80a51c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbee01051e0 0x7fbee01a54a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbee0102830 0x7fbee019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 --2- 172.21.15.102:0/4104926530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbee00ffec0 0x7fbee019def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 >> 172.21.15.102:0/4104926530 conn(0x7fbee00fbae0 msgr2=0x7fbee0100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:18.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 shutdown_connections 2024-04-25T01:55:18.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.616+0000 7fbee6c00640 1 -- 172.21.15.102:0/4104926530 wait complete. 2024-04-25T01:55:18.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:55:18.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.838+0000 7f0301a00640 1 -- 172.21.15.102:0/2969199996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02fc107390 msgr2=0x7f02fc109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.838+0000 7f0301a00640 1 --2- 172.21.15.102:0/2969199996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02fc107390 0x7f02fc109780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009ca0 tx=0x7f02f402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.838+0000 7f0301a00640 1 -- 172.21.15.102:0/2969199996 shutdown_connections 2024-04-25T01:55:18.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.838+0000 7f0301a00640 1 --2- 172.21.15.102:0/2969199996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02fc107390 0x7f02fc109780 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.838+0000 7f0301a00640 1 --2- 172.21.15.102:0/2969199996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02fc104a60 0x7f02fc106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.838+0000 7f0301a00640 1 --2- 172.21.15.102:0/2969199996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02fc102130 0x7f02fc104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.838+0000 7f0301a00640 1 -- 172.21.15.102:0/2969199996 >> 172.21.15.102:0/2969199996 conn(0x7f02fc0fbae0 msgr2=0x7f02fc0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:18.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.838+0000 7f0301a00640 1 -- 172.21.15.102:0/2969199996 shutdown_connections 2024-04-25T01:55:18.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.838+0000 7f0301a00640 1 -- 172.21.15.102:0/2969199996 wait complete. 2024-04-25T01:55:18.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.839+0000 7f0301a00640 1 Processor -- start 2024-04-25T01:55:18.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f0301a00640 1 -- start start 2024-04-25T01:55:18.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f0301a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02fc102130 0x7f02fc199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f0301a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02fc104a60 0x7f02fc19a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f0301a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02fc107390 0x7f02fc1a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f0301a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc10d330 con 0x7f02fc104a60 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f0301a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc10d1b0 con 0x7f02fc102130 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f0301a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc10d4b0 con 0x7f02fc107390 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02fc102130 0x7f02fc199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02fc102130 0x7f02fc199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33118/0 (socket says 172.21.15.102:33118) 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fbe00640 1 -- 172.21.15.102:0/2946896041 learned_addr learned my addr 172.21.15.102:0/2946896041 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f0301000640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02fc107390 0x7f02fc1a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fb400640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02fc104a60 0x7f02fc19a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fbe00640 1 -- 172.21.15.102:0/2946896041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02fc107390 msgr2=0x7f02fc1a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fbe00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02fc107390 0x7f02fc1a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fbe00640 1 -- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02fc104a60 msgr2=0x7f02fc19a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fbe00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02fc104a60 0x7f02fc19a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fbe00640 1 -- 172.21.15.102:0/2946896041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc1a1700 con 0x7f02fc102130 2024-04-25T01:55:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.840+0000 7f02fb400640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02fc104a60 0x7f02fc19a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.841+0000 7f02fbe00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02fc102130 0x7f02fc199ae0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.841+0000 7f02f8c00640 1 -- 172.21.15.102:0/2946896041 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec00cd80 con 0x7f02fc102130 2024-04-25T01:55:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.841+0000 7f02f8c00640 1 -- 172.21.15.102:0/2946896041 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ec010910 con 0x7f02fc102130 2024-04-25T01:55:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.841+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc1a19f0 con 0x7f02fc102130 2024-04-25T01:55:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.841+0000 7f02f8c00640 1 -- 172.21.15.102:0/2946896041 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec018990 con 0x7f02fc102130 2024-04-25T01:55:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.841+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02fc1a1f90 con 0x7f02fc102130 2024-04-25T01:55:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.842+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02fc108e80 con 0x7f02fc102130 2024-04-25T01:55:18.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.844+0000 7f02f8c00640 1 -- 172.21.15.102:0/2946896041 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 23) ==== 137954+0+0 (secure 0 0 0) 0x7f02ec010ac0 con 0x7f02fc102130 2024-04-25T01:55:18.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.845+0000 7f02f8c00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f02e00a2d80 0x7f02e00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.846+0000 7f02f8c00640 1 -- 172.21.15.102:0/2946896041 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f02ec0cf7a0 con 0x7f02fc102130 2024-04-25T01:55:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.846+0000 7f02fb400640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f02e00a2d80 0x7f02e00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.846+0000 7f02fb400640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f02e00a2d80 0x7f02e00a5240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f02fc19aff0 tx=0x7f02e8005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:18.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:18.849+0000 7f02f8c00640 1 -- 172.21.15.102:0/2946896041 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f02ec09c390 con 0x7f02fc102130 2024-04-25T01:55:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:18 smithi102 ceph-mon[27796]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:55:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:18 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:18 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:18 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1435950321' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:18 smithi102 ceph-mon[27796]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T01:55:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:18 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3045616342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:18 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:18 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:18.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:18 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/4104926530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:19.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.100+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f02fc062340 con 0x7f02fc102130 2024-04-25T01:55:19.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.100+0000 7f02f8c00640 1 -- 172.21.15.102:0/2946896041 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f02ec09bc90 con 0x7f02fc102130 2024-04-25T01:55:19.103 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-25T01:55:19.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f02e00a2d80 msgr2=0x7f02e00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f02e00a2d80 0x7f02e00a5240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f02fc19aff0 tx=0x7f02e8005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02fc102130 msgr2=0x7f02fc199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02fc102130 0x7f02fc199ae0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 shutdown_connections 2024-04-25T01:55:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f02e00a2d80 0x7f02e00a5240 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02fc107390 0x7f02fc1a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02fc104a60 0x7f02fc19a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 --2- 172.21.15.102:0/2946896041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02fc102130 0x7f02fc199ae0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 >> 172.21.15.102:0/2946896041 conn(0x7f02fc0fbae0 msgr2=0x7f02fc101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:19.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 shutdown_connections 2024-04-25T01:55:19.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.103+0000 7f0301a00640 1 -- 172.21.15.102:0/2946896041 wait complete. 2024-04-25T01:55:19.125 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-25T01:55:19.126 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:55:19.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:18 smithi110 ceph-mon[32848]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:55:19.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:18 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:18 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:18 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1435950321' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:19.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:18 smithi110 ceph-mon[32848]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T01:55:19.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:18 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3045616342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:19.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:18 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:18 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:18 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/4104926530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:19.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:18 smithi193 ceph-mon[32981]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:55:19.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:18 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:18 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:18 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1435950321' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:19.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:18 smithi193 ceph-mon[32981]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-25T01:55:19.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:18 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3045616342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:19.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:18 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:18 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:18 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/4104926530' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.318+0000 7fdbcc400640 1 -- 172.21.15.102:0/2104605921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100050 msgr2=0x7fdbc4100470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.318+0000 7fdbcc400640 1 --2- 172.21.15.102:0/2104605921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100050 0x7fdbc4100470 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009900 tx=0x7fdbb802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.318+0000 7fdbcc400640 1 -- 172.21.15.102:0/2104605921 shutdown_connections 2024-04-25T01:55:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.318+0000 7fdbcc400640 1 --2- 172.21.15.102:0/2104605921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc4101350 0x7fdbc41017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.318+0000 7fdbcc400640 1 --2- 172.21.15.102:0/2104605921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc41009b0 0x7fdbc4100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.318+0000 7fdbcc400640 1 --2- 172.21.15.102:0/2104605921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc4100050 0x7fdbc4100470 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.318+0000 7fdbcc400640 1 -- 172.21.15.102:0/2104605921 >> 172.21.15.102:0/2104605921 conn(0x7fdbc40fbbf0 msgr2=0x7fdbc40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.319+0000 7fdbcc400640 1 -- 172.21.15.102:0/2104605921 shutdown_connections 2024-04-25T01:55:19.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.319+0000 7fdbcc400640 1 -- 172.21.15.102:0/2104605921 wait complete. 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.319+0000 7fdbcc400640 1 Processor -- start 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbcc400640 1 -- start start 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbcc400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc4100050 0x7fdbc4199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbcc400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc41009b0 0x7fdbc419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbcc400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc4101350 0x7fdbc41a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbcc400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc410f5c0 con 0x7fdbc4100050 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbcc400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc410f440 con 0x7fdbc4101350 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbcc400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc410f740 con 0x7fdbc41009b0 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbc9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc4100050 0x7fdbc4199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:19.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbc9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc4100050 0x7fdbc4199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33924/0 (socket says 172.21.15.102:33924) 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbc9a00640 1 -- 172.21.15.102:0/1115765941 learned_addr learned my addr 172.21.15.102:0/1115765941 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbca400640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc4101350 0x7fdbc41a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.320+0000 7fdbc9000640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc41009b0 0x7fdbc419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbc9a00640 1 -- 172.21.15.102:0/1115765941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc41009b0 msgr2=0x7fdbc419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbc9a00640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc41009b0 0x7fdbc419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbc9a00640 1 -- 172.21.15.102:0/1115765941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc4101350 msgr2=0x7fdbc41a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbc9a00640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc4101350 0x7fdbc41a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbc9a00640 1 -- 172.21.15.102:0/1115765941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc41a18b0 con 0x7fdbc4100050 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbca400640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc4101350 0x7fdbc41a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbc9a00640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc4100050 0x7fdbc4199c70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009230 tx=0x7fdbb8031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbc2000640 1 -- 172.21.15.102:0/1115765941 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb803f030 con 0x7fdbc4100050 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbc2000640 1 -- 172.21.15.102:0/1115765941 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbb8003ad0 con 0x7fdbc4100050 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc41a1b40 con 0x7fdbc4100050 2024-04-25T01:55:19.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbc2000640 1 -- 172.21.15.102:0/1115765941 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8035070 con 0x7fdbc4100050 2024-04-25T01:55:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.321+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc41a2030 con 0x7fdbc4100050 2024-04-25T01:55:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.323+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc410af80 con 0x7fdbc4100050 2024-04-25T01:55:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.325+0000 7fdbc2000640 1 -- 172.21.15.102:0/1115765941 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 23) ==== 137954+0+0 (secure 0 0 0) 0x7fdbb8042030 con 0x7fdbc4100050 2024-04-25T01:55:19.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.326+0000 7fdbc2000640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdbac0a2d00 0x7fdbac0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:19.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.326+0000 7fdbc2000640 1 -- 172.21.15.102:0/1115765941 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fdbb80ef610 con 0x7fdbc4100050 2024-04-25T01:55:19.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.326+0000 7fdbc9000640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdbac0a2d00 0x7fdbac0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:19.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.327+0000 7fdbc9000640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdbac0a2d00 0x7fdbac0a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdbc419b170 tx=0x7fdbb0005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:19.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.331+0000 7fdbc2000640 1 -- 172.21.15.102:0/1115765941 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdbb80bbbb0 con 0x7fdbc4100050 2024-04-25T01:55:19.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.619+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbc402d370 con 0x7fdbc4100050 2024-04-25T01:55:19.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.620+0000 7fdbc2000640 1 -- 172.21.15.102:0/1115765941 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11896 (secure 0 0 0) 0x7fdbb80bfeb0 con 0x7fdbc4100050 2024-04-25T01:55:19.622 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:19.622 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:55:15.481210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587,24533,24541]},"id":1}]} 2024-04-25T01:55:19.622 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 23 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdbac0a2d00 msgr2=0x7fdbac0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdbac0a2d00 0x7fdbac0a51c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdbc419b170 tx=0x7fdbb0005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc4100050 msgr2=0x7fdbc4199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc4100050 0x7fdbc4199c70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009230 tx=0x7fdbb8031b60 comp rx=0 tx=0).stop 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 shutdown_connections 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdbac0a2d00 0x7fdbac0a51c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc4101350 0x7fdbc41a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbc41009b0 0x7fdbc419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 --2- 172.21.15.102:0/1115765941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbc4100050 0x7fdbc4199c70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.623+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 >> 172.21.15.102:0/1115765941 conn(0x7fdbc40fbbf0 msgr2=0x7fdbc40fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:19.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.624+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 shutdown_connections 2024-04-25T01:55:19.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.624+0000 7fdbcc400640 1 -- 172.21.15.102:0/1115765941 wait complete. 2024-04-25T01:55:19.644 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-25T01:55:19.644 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-25T01:55:19.645 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:55:19.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.827+0000 7fd013400640 1 -- 172.21.15.102:0/3895554563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 msgr2=0x7fd00c100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.827+0000 7fd013400640 1 --2- 172.21.15.102:0/3895554563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 0x7fd00c100ae0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd000009900 tx=0x7fd00002a400 comp rx=0 tx=0).stop 2024-04-25T01:55:19.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.827+0000 7fd013400640 1 -- 172.21.15.102:0/3895554563 shutdown_connections 2024-04-25T01:55:19.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.827+0000 7fd013400640 1 --2- 172.21.15.102:0/3895554563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd00c102270 0x7fd00c108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.827+0000 7fd013400640 1 --2- 172.21.15.102:0/3895554563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c1018b0 0x7fd00c101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.827+0000 7fd013400640 1 --2- 172.21.15.102:0/3895554563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 0x7fd00c100ae0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.827+0000 7fd013400640 1 -- 172.21.15.102:0/3895554563 >> 172.21.15.102:0/3895554563 conn(0x7fd00c0fbc30 msgr2=0x7fd00c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:19.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.827+0000 7fd013400640 1 -- 172.21.15.102:0/3895554563 shutdown_connections 2024-04-25T01:55:19.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.827+0000 7fd013400640 1 -- 172.21.15.102:0/3895554563 wait complete. 2024-04-25T01:55:19.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.828+0000 7fd013400640 1 Processor -- start 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.828+0000 7fd013400640 1 -- start start 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.828+0000 7fd013400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 0x7fd00c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.828+0000 7fd013400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd00c1018b0 0x7fd00c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd013400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c102270 0x7fd00c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd013400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c10c390 con 0x7fd00c102270 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd013400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c10c210 con 0x7fd00c1006c0 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd013400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c10c510 con 0x7fd00c1018b0 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd011400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c102270 0x7fd00c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd010a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 0x7fd00c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd010a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 0x7fd00c197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33146/0 (socket says 172.21.15.102:33146) 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd010a00640 1 -- 172.21.15.102:0/3562899338 learned_addr learned my addr 172.21.15.102:0/3562899338 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:19.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd011400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c102270 0x7fd00c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33928/0 (socket says 172.21.15.102:33928) 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd00be00640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd00c1018b0 0x7fd00c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd010a00640 1 -- 172.21.15.102:0/3562899338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd00c1018b0 msgr2=0x7fd00c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd010a00640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd00c1018b0 0x7fd00c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd010a00640 1 -- 172.21.15.102:0/3562899338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c102270 msgr2=0x7fd00c19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd010a00640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c102270 0x7fd00c19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.829+0000 7fd010a00640 1 -- 172.21.15.102:0/3562899338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00c19f720 con 0x7fd00c1006c0 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.830+0000 7fd010a00640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 0x7fd00c197b00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd000031610 tx=0x7fd000032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.830+0000 7fd009600640 1 -- 172.21.15.102:0/3562899338 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00003f030 con 0x7fd00c1006c0 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.830+0000 7fd009600640 1 -- 172.21.15.102:0/3562899338 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd000003ce0 con 0x7fd00c1006c0 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.830+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00c19f9b0 con 0x7fd00c1006c0 2024-04-25T01:55:19.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.830+0000 7fd009600640 1 -- 172.21.15.102:0/3562899338 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000035070 con 0x7fd00c1006c0 2024-04-25T01:55:19.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.830+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00c19fef0 con 0x7fd00c1006c0 2024-04-25T01:55:19.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.831+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd00c1a0120 con 0x7fd00c1006c0 2024-04-25T01:55:19.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.833+0000 7fd009600640 1 -- 172.21.15.102:0/3562899338 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 23) ==== 137954+0+0 (secure 0 0 0) 0x7fd000042030 con 0x7fd00c1006c0 2024-04-25T01:55:19.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.834+0000 7fd009600640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcff40a2d80 0x7fcff40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:19.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.834+0000 7fd009600640 1 -- 172.21.15.102:0/3562899338 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd0000ef840 con 0x7fd00c1006c0 2024-04-25T01:55:19.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.835+0000 7fd00be00640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcff40a2d80 0x7fcff40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:19.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.835+0000 7fd00be00640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcff40a2d80 0x7fcff40a5240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcff80095f0 tx=0x7fcff8005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:19.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:19.838+0000 7fd009600640 1 -- 172.21.15.102:0/3562899338 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd0000bc430 con 0x7fd00c1006c0 2024-04-25T01:55:19.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2946896041' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:19.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:19 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1115765941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:20.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.038+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 --> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd00c0fdde0 con 0x7fcff40a2d80 2024-04-25T01:55:20.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.047+0000 7fd009600640 1 -- 172.21.15.102:0/3562899338 <== mgr.25214 v2:172.21.15.102:6800/855543702 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257771 (secure 0 0 0) 0x7fd00c0fdde0 con 0x7fcff40a2d80 2024-04-25T01:55:20.054 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:20.055 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcff40a2d80 msgr2=0x7fcff40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcff40a2d80 0x7fcff40a5240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcff80095f0 tx=0x7fcff8005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 msgr2=0x7fd00c197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 0x7fd00c197b00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd000031610 tx=0x7fd000032ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 shutdown_connections 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcff40a2d80 0x7fcff40a5240 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c102270 0x7fd00c19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd00c1018b0 0x7fd00c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 --2- 172.21.15.102:0/3562899338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd00c1006c0 0x7fd00c197b00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 >> 172.21.15.102:0/3562899338 conn(0x7fd00c0fbc30 msgr2=0x7fd00c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:20.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 shutdown_connections 2024-04-25T01:55:20.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.054+0000 7fd013400640 1 -- 172.21.15.102:0/3562899338 wait complete. 2024-04-25T01:55:20.075 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T01:55:19.596440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188884,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1136,"num_read_kb":728,"num_write":470,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1242498,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384436,"kb_used_data":14388,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557388,"statfs":{"total":1151940427776,"available":1151546765312,"internally_reserved":0,"allocated":14733312,"data_stored":8484918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":378713403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.984677"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904984+0000","last_change":"2024-04-25T01:52:44.102143+0000","last_active":"2024-04-25T01:55:15.904984+0000","last_peered":"2024-04-25T01:55:15.904984+0000","last_clean":"2024-04-25T01:55:15.904984+0000","last_became_active":"2024-04-25T01:52:44.101810+0000","last_became_peered":"2024-04-25T01:52:44.101810+0000","last_unstale":"2024-04-25T01:55:15.904984+0000","last_undegraded":"2024-04-25T01:55:15.904984+0000","last_fullsized":"2024-04-25T01:55:15.904984+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:02.356294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906327+0000","last_change":"2024-04-25T01:52:44.103074+0000","last_active":"2024-04-25T01:55:15.906327+0000","last_peered":"2024-04-25T01:55:15.906327+0000","last_clean":"2024-04-25T01:55:15.906327+0000","last_became_active":"2024-04-25T01:52:44.102103+0000","last_became_peered":"2024-04-25T01:52:44.102103+0000","last_unstale":"2024-04-25T01:55:15.906327+0000","last_undegraded":"2024-04-25T01:55:15.906327+0000","last_fullsized":"2024-04-25T01:55:15.906327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:36:43.016756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906006+0000","last_change":"2024-04-25T01:52:44.101467+0000","last_active":"2024-04-25T01:55:15.906006+0000","last_peered":"2024-04-25T01:55:15.906006+0000","last_clean":"2024-04-25T01:55:15.906006+0000","last_became_active":"2024-04-25T01:52:44.100911+0000","last_became_peered":"2024-04-25T01:52:44.100911+0000","last_unstale":"2024-04-25T01:55:15.906006+0000","last_undegraded":"2024-04-25T01:55:15.906006+0000","last_fullsized":"2024-04-25T01:55:15.906006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:58.533213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904287+0000","last_change":"2024-04-25T01:52:44.088592+0000","last_active":"2024-04-25T01:55:15.904287+0000","last_peered":"2024-04-25T01:55:15.904287+0000","last_clean":"2024-04-25T01:55:15.904287+0000","last_became_active":"2024-04-25T01:52:44.087651+0000","last_became_peered":"2024-04-25T01:52:44.087651+0000","last_unstale":"2024-04-25T01:55:15.904287+0000","last_undegraded":"2024-04-25T01:55:15.904287+0000","last_fullsized":"2024-04-25T01:55:15.904287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:52.240818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702409+0000","last_change":"2024-04-25T01:52:44.095033+0000","last_active":"2024-04-25T01:55:15.702409+0000","last_peered":"2024-04-25T01:55:15.702409+0000","last_clean":"2024-04-25T01:55:15.702409+0000","last_became_active":"2024-04-25T01:52:44.094719+0000","last_became_peered":"2024-04-25T01:52:44.094719+0000","last_unstale":"2024-04-25T01:55:15.702409+0000","last_undegraded":"2024-04-25T01:55:15.702409+0000","last_fullsized":"2024-04-25T01:55:15.702409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:52.422544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.717941+0000","last_change":"2024-04-25T01:52:44.102574+0000","last_active":"2024-04-25T01:55:15.717941+0000","last_peered":"2024-04-25T01:55:15.717941+0000","last_clean":"2024-04-25T01:55:15.717941+0000","last_became_active":"2024-04-25T01:52:44.102158+0000","last_became_peered":"2024-04-25T01:52:44.102158+0000","last_unstale":"2024-04-25T01:55:15.717941+0000","last_undegraded":"2024-04-25T01:55:15.717941+0000","last_fullsized":"2024-04-25T01:55:15.717941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:20.931392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918311+0000","last_change":"2024-04-25T01:52:44.107662+0000","last_active":"2024-04-25T01:55:15.918311+0000","last_peered":"2024-04-25T01:55:15.918311+0000","last_clean":"2024-04-25T01:55:15.918311+0000","last_became_active":"2024-04-25T01:52:44.107304+0000","last_became_peered":"2024-04-25T01:52:44.107304+0000","last_unstale":"2024-04-25T01:55:15.918311+0000","last_undegraded":"2024-04-25T01:55:15.918311+0000","last_fullsized":"2024-04-25T01:55:15.918311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:42:54.636950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583336+0000","last_change":"2024-04-25T01:52:44.100273+0000","last_active":"2024-04-25T01:55:15.583336+0000","last_peered":"2024-04-25T01:55:15.583336+0000","last_clean":"2024-04-25T01:55:15.583336+0000","last_became_active":"2024-04-25T01:52:44.098223+0000","last_became_peered":"2024-04-25T01:52:44.098223+0000","last_unstale":"2024-04-25T01:55:15.583336+0000","last_undegraded":"2024-04-25T01:55:15.583336+0000","last_fullsized":"2024-04-25T01:55:15.583336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:05.003265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576770+0000","last_change":"2024-04-25T01:52:44.089658+0000","last_active":"2024-04-25T01:55:15.576770+0000","last_peered":"2024-04-25T01:55:15.576770+0000","last_clean":"2024-04-25T01:55:15.576770+0000","last_became_active":"2024-04-25T01:52:44.089232+0000","last_became_peered":"2024-04-25T01:52:44.089232+0000","last_unstale":"2024-04-25T01:55:15.576770+0000","last_undegraded":"2024-04-25T01:55:15.576770+0000","last_fullsized":"2024-04-25T01:55:15.576770+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:54.385782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.718285+0000","last_change":"2024-04-25T01:52:44.086660+0000","last_active":"2024-04-25T01:55:15.718285+0000","last_peered":"2024-04-25T01:55:15.718285+0000","last_clean":"2024-04-25T01:55:15.718285+0000","last_became_active":"2024-04-25T01:52:44.086306+0000","last_became_peered":"2024-04-25T01:52:44.086306+0000","last_unstale":"2024-04-25T01:55:15.718285+0000","last_undegraded":"2024-04-25T01:55:15.718285+0000","last_fullsized":"2024-04-25T01:55:15.718285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:33:20.726616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.919001+0000","last_change":"2024-04-25T01:52:44.090239+0000","last_active":"2024-04-25T01:55:15.919001+0000","last_peered":"2024-04-25T01:55:15.919001+0000","last_clean":"2024-04-25T01:55:15.919001+0000","last_became_active":"2024-04-25T01:52:44.089736+0000","last_became_peered":"2024-04-25T01:52:44.089736+0000","last_unstale":"2024-04-25T01:55:15.919001+0000","last_undegraded":"2024-04-25T01:55:15.919001+0000","last_fullsized":"2024-04-25T01:55:15.919001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:22.954011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904432+0000","last_change":"2024-04-25T01:52:44.088878+0000","last_active":"2024-04-25T01:55:15.904432+0000","last_peered":"2024-04-25T01:55:15.904432+0000","last_clean":"2024-04-25T01:55:15.904432+0000","last_became_active":"2024-04-25T01:52:44.088250+0000","last_became_peered":"2024-04-25T01:52:44.088250+0000","last_unstale":"2024-04-25T01:55:15.904432+0000","last_undegraded":"2024-04-25T01:55:15.904432+0000","last_fullsized":"2024-04-25T01:55:15.904432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:15.490131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575344+0000","last_change":"2024-04-25T01:52:44.094675+0000","last_active":"2024-04-25T01:55:15.575344+0000","last_peered":"2024-04-25T01:55:15.575344+0000","last_clean":"2024-04-25T01:55:15.575344+0000","last_became_active":"2024-04-25T01:52:44.094219+0000","last_became_peered":"2024-04-25T01:52:44.094219+0000","last_unstale":"2024-04-25T01:55:15.575344+0000","last_undegraded":"2024-04-25T01:55:15.575344+0000","last_fullsized":"2024-04-25T01:55:15.575344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:26.160156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905482+0000","last_change":"2024-04-25T01:52:44.111131+0000","last_active":"2024-04-25T01:55:15.905482+0000","last_peered":"2024-04-25T01:55:15.905482+0000","last_clean":"2024-04-25T01:55:15.905482+0000","last_became_active":"2024-04-25T01:52:44.105670+0000","last_became_peered":"2024-04-25T01:52:44.105670+0000","last_unstale":"2024-04-25T01:55:15.905482+0000","last_undegraded":"2024-04-25T01:55:15.905482+0000","last_fullsized":"2024-04-25T01:55:15.905482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:00.377724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905232+0000","last_change":"2024-04-25T01:52:44.109707+0000","last_active":"2024-04-25T01:55:15.905232+0000","last_peered":"2024-04-25T01:55:15.905232+0000","last_clean":"2024-04-25T01:55:15.905232+0000","last_became_active":"2024-04-25T01:52:44.108550+0000","last_became_peered":"2024-04-25T01:52:44.108550+0000","last_unstale":"2024-04-25T01:55:15.905232+0000","last_undegraded":"2024-04-25T01:55:15.905232+0000","last_fullsized":"2024-04-25T01:55:15.905232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:46.296789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576000+0000","last_change":"2024-04-25T01:52:44.104504+0000","last_active":"2024-04-25T01:55:15.576000+0000","last_peered":"2024-04-25T01:55:15.576000+0000","last_clean":"2024-04-25T01:55:15.576000+0000","last_became_active":"2024-04-25T01:52:44.104180+0000","last_became_peered":"2024-04-25T01:52:44.104180+0000","last_unstale":"2024-04-25T01:55:15.576000+0000","last_undegraded":"2024-04-25T01:55:15.576000+0000","last_fullsized":"2024-04-25T01:55:15.576000+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:17.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905168+0000","last_change":"2024-04-25T01:52:44.107550+0000","last_active":"2024-04-25T01:55:15.905168+0000","last_peered":"2024-04-25T01:55:15.905168+0000","last_clean":"2024-04-25T01:55:15.905168+0000","last_became_active":"2024-04-25T01:52:44.107195+0000","last_became_peered":"2024-04-25T01:52:44.107195+0000","last_unstale":"2024-04-25T01:55:15.905168+0000","last_undegraded":"2024-04-25T01:55:15.905168+0000","last_fullsized":"2024-04-25T01:55:15.905168+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:59:42.220575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.585058+0000","last_change":"2024-04-25T01:52:44.101165+0000","last_active":"2024-04-25T01:55:15.585058+0000","last_peered":"2024-04-25T01:55:15.585058+0000","last_clean":"2024-04-25T01:55:15.585058+0000","last_became_active":"2024-04-25T01:52:44.100386+0000","last_became_peered":"2024-04-25T01:52:44.100386+0000","last_unstale":"2024-04-25T01:55:15.585058+0000","last_undegraded":"2024-04-25T01:55:15.585058+0000","last_fullsized":"2024-04-25T01:55:15.585058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:30.922707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904739+0000","last_change":"2024-04-25T01:52:44.116694+0000","last_active":"2024-04-25T01:55:15.904739+0000","last_peered":"2024-04-25T01:55:15.904739+0000","last_clean":"2024-04-25T01:55:15.904739+0000","last_became_active":"2024-04-25T01:52:44.116326+0000","last_became_peered":"2024-04-25T01:52:44.116326+0000","last_unstale":"2024-04-25T01:55:15.904739+0000","last_undegraded":"2024-04-25T01:55:15.904739+0000","last_fullsized":"2024-04-25T01:55:15.904739+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:16.329899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584165+0000","last_change":"2024-04-25T01:52:43.072358+0000","last_active":"2024-04-25T01:55:15.584165+0000","last_peered":"2024-04-25T01:55:15.584165+0000","last_clean":"2024-04-25T01:55:15.584165+0000","last_became_active":"2024-04-25T01:52:43.071815+0000","last_became_peered":"2024-04-25T01:52:43.071815+0000","last_unstale":"2024-04-25T01:55:15.584165+0000","last_undegraded":"2024-04-25T01:55:15.584165+0000","last_fullsized":"2024-04-25T01:55:15.584165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:15.295744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.585355+0000","last_change":"2024-04-25T01:52:44.094509+0000","last_active":"2024-04-25T01:55:15.585355+0000","last_peered":"2024-04-25T01:55:15.585355+0000","last_clean":"2024-04-25T01:55:15.585355+0000","last_became_active":"2024-04-25T01:52:44.093955+0000","last_became_peered":"2024-04-25T01:52:44.093955+0000","last_unstale":"2024-04-25T01:55:15.585355+0000","last_undegraded":"2024-04-25T01:55:15.585355+0000","last_fullsized":"2024-04-25T01:55:15.585355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:37.701835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.585603+0000","last_change":"2024-04-25T01:52:43.094435+0000","last_active":"2024-04-25T01:55:15.585603+0000","last_peered":"2024-04-25T01:55:15.585603+0000","last_clean":"2024-04-25T01:55:15.585603+0000","last_became_active":"2024-04-25T01:52:43.094018+0000","last_became_peered":"2024-04-25T01:52:43.094018+0000","last_unstale":"2024-04-25T01:55:15.585603+0000","last_undegraded":"2024-04-25T01:55:15.585603+0000","last_fullsized":"2024-04-25T01:55:15.585603+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:36.781014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906512+0000","last_change":"2024-04-25T01:52:44.104347+0000","last_active":"2024-04-25T01:55:15.906512+0000","last_peered":"2024-04-25T01:55:15.906512+0000","last_clean":"2024-04-25T01:55:15.906512+0000","last_became_active":"2024-04-25T01:52:44.103803+0000","last_became_peered":"2024-04-25T01:52:44.103803+0000","last_unstale":"2024-04-25T01:55:15.906512+0000","last_undegraded":"2024-04-25T01:55:15.906512+0000","last_fullsized":"2024-04-25T01:55:15.906512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:01.784893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702759+0000","last_change":"2024-04-25T01:52:43.070902+0000","last_active":"2024-04-25T01:55:15.702759+0000","last_peered":"2024-04-25T01:55:15.702759+0000","last_clean":"2024-04-25T01:55:15.702759+0000","last_became_active":"2024-04-25T01:52:43.070313+0000","last_became_peered":"2024-04-25T01:52:43.070313+0000","last_unstale":"2024-04-25T01:55:15.702759+0000","last_undegraded":"2024-04-25T01:55:15.702759+0000","last_fullsized":"2024-04-25T01:55:15.702759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:10.677320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576406+0000","last_change":"2024-04-25T01:52:44.093293+0000","last_active":"2024-04-25T01:55:15.576406+0000","last_peered":"2024-04-25T01:55:15.576406+0000","last_clean":"2024-04-25T01:55:15.576406+0000","last_became_active":"2024-04-25T01:52:44.092932+0000","last_became_peered":"2024-04-25T01:52:44.092932+0000","last_unstale":"2024-04-25T01:55:15.576406+0000","last_undegraded":"2024-04-25T01:55:15.576406+0000","last_fullsized":"2024-04-25T01:55:15.576406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:58:37.705594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702640+0000","last_change":"2024-04-25T01:52:43.069743+0000","last_active":"2024-04-25T01:55:15.702640+0000","last_peered":"2024-04-25T01:55:15.702640+0000","last_clean":"2024-04-25T01:55:15.702640+0000","last_became_active":"2024-04-25T01:52:43.068947+0000","last_became_peered":"2024-04-25T01:52:43.068947+0000","last_unstale":"2024-04-25T01:55:15.702640+0000","last_undegraded":"2024-04-25T01:55:15.702640+0000","last_fullsized":"2024-04-25T01:55:15.702640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:02:52.582176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584113+0000","last_change":"2024-04-25T01:53:43.865168+0000","last_active":"2024-04-25T01:55:15.584113+0000","last_peered":"2024-04-25T01:55:15.584113+0000","last_clean":"2024-04-25T01:55:15.584113+0000","last_became_active":"2024-04-25T01:53:43.864055+0000","last_became_peered":"2024-04-25T01:53:43.864055+0000","last_unstale":"2024-04-25T01:55:15.584113+0000","last_undegraded":"2024-04-25T01:55:15.584113+0000","last_fullsized":"2024-04-25T01:55:15.584113+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:01.707773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"90'9","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.337563+0000","last_change":"2024-04-25T01:52:43.068450+0000","last_active":"2024-04-25T01:55:16.337563+0000","last_peered":"2024-04-25T01:55:16.337563+0000","last_clean":"2024-04-25T01:55:16.337563+0000","last_became_active":"2024-04-25T01:52:43.067881+0000","last_became_peered":"2024-04-25T01:52:43.067881+0000","last_unstale":"2024-04-25T01:55:16.337563+0000","last_undegraded":"2024-04-25T01:55:16.337563+0000","last_fullsized":"2024-04-25T01:55:16.337563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:12.739321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1936,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":153,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702491+0000","last_change":"2024-04-25T01:52:44.101743+0000","last_active":"2024-04-25T01:55:15.702491+0000","last_peered":"2024-04-25T01:55:15.702491+0000","last_clean":"2024-04-25T01:55:15.702491+0000","last_became_active":"2024-04-25T01:52:44.101351+0000","last_became_peered":"2024-04-25T01:52:44.101351+0000","last_unstale":"2024-04-25T01:55:15.702491+0000","last_undegraded":"2024-04-25T01:55:15.702491+0000","last_fullsized":"2024-04-25T01:55:15.702491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:54:24.104819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.717869+0000","last_change":"2024-04-25T01:52:43.075880+0000","last_active":"2024-04-25T01:55:15.717869+0000","last_peered":"2024-04-25T01:55:15.717869+0000","last_clean":"2024-04-25T01:55:15.717869+0000","last_became_active":"2024-04-25T01:52:43.075522+0000","last_became_peered":"2024-04-25T01:52:43.075522+0000","last_unstale":"2024-04-25T01:55:15.717869+0000","last_undegraded":"2024-04-25T01:55:15.717869+0000","last_fullsized":"2024-04-25T01:55:15.717869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:26.218774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714269+0000","last_change":"2024-04-25T01:52:44.116772+0000","last_active":"2024-04-25T01:55:15.714269+0000","last_peered":"2024-04-25T01:55:15.714269+0000","last_clean":"2024-04-25T01:55:15.714269+0000","last_became_active":"2024-04-25T01:52:44.116408+0000","last_became_peered":"2024-04-25T01:52:44.116408+0000","last_unstale":"2024-04-25T01:55:15.714269+0000","last_undegraded":"2024-04-25T01:55:15.714269+0000","last_fullsized":"2024-04-25T01:55:15.714269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:21.988425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575458+0000","last_change":"2024-04-25T01:52:43.076964+0000","last_active":"2024-04-25T01:55:15.575458+0000","last_peered":"2024-04-25T01:55:15.575458+0000","last_clean":"2024-04-25T01:55:15.575458+0000","last_became_active":"2024-04-25T01:52:43.073915+0000","last_became_peered":"2024-04-25T01:52:43.073915+0000","last_unstale":"2024-04-25T01:55:15.575458+0000","last_undegraded":"2024-04-25T01:55:15.575458+0000","last_fullsized":"2024-04-25T01:55:15.575458+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:03.321920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583105+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:55:15.583105+0000","last_peered":"2024-04-25T01:55:15.583105+0000","last_clean":"2024-04-25T01:55:15.583105+0000","last_became_active":"2024-04-25T01:52:44.088198+0000","last_became_peered":"2024-04-25T01:52:44.088198+0000","last_unstale":"2024-04-25T01:55:15.583105+0000","last_undegraded":"2024-04-25T01:55:15.583105+0000","last_fullsized":"2024-04-25T01:55:15.583105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:30.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714027+0000","last_change":"2024-04-25T01:52:43.069853+0000","last_active":"2024-04-25T01:55:15.714027+0000","last_peered":"2024-04-25T01:55:15.714027+0000","last_clean":"2024-04-25T01:55:15.714027+0000","last_became_active":"2024-04-25T01:52:43.069526+0000","last_became_peered":"2024-04-25T01:52:43.069526+0000","last_unstale":"2024-04-25T01:55:15.714027+0000","last_undegraded":"2024-04-25T01:55:15.714027+0000","last_fullsized":"2024-04-25T01:55:15.714027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:35:03.272610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583451+0000","last_change":"2024-04-25T01:52:44.114296+0000","last_active":"2024-04-25T01:55:15.583451+0000","last_peered":"2024-04-25T01:55:15.583451+0000","last_clean":"2024-04-25T01:55:15.583451+0000","last_became_active":"2024-04-25T01:52:44.113958+0000","last_became_peered":"2024-04-25T01:52:44.113958+0000","last_unstale":"2024-04-25T01:55:15.583451+0000","last_undegraded":"2024-04-25T01:55:15.583451+0000","last_fullsized":"2024-04-25T01:55:15.583451+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:45.068080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.336627+0000","last_change":"2024-04-25T01:52:43.071388+0000","last_active":"2024-04-25T01:55:16.336627+0000","last_peered":"2024-04-25T01:55:16.336627+0000","last_clean":"2024-04-25T01:55:16.336627+0000","last_became_active":"2024-04-25T01:52:43.071005+0000","last_became_peered":"2024-04-25T01:52:43.071005+0000","last_unstale":"2024-04-25T01:55:16.336627+0000","last_undegraded":"2024-04-25T01:55:16.336627+0000","last_fullsized":"2024-04-25T01:55:16.336627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:56:59.580458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576398+0000","last_change":"2024-04-25T01:52:44.106106+0000","last_active":"2024-04-25T01:55:15.576398+0000","last_peered":"2024-04-25T01:55:15.576398+0000","last_clean":"2024-04-25T01:55:15.576398+0000","last_became_active":"2024-04-25T01:52:44.105760+0000","last_became_peered":"2024-04-25T01:52:44.105760+0000","last_unstale":"2024-04-25T01:55:15.576398+0000","last_undegraded":"2024-04-25T01:55:15.576398+0000","last_fullsized":"2024-04-25T01:55:15.576398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:24.626753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918950+0000","last_change":"2024-04-25T01:52:43.068926+0000","last_active":"2024-04-25T01:55:15.918950+0000","last_peered":"2024-04-25T01:55:15.918950+0000","last_clean":"2024-04-25T01:55:15.918950+0000","last_became_active":"2024-04-25T01:52:43.067356+0000","last_became_peered":"2024-04-25T01:52:43.067356+0000","last_unstale":"2024-04-25T01:55:15.918950+0000","last_undegraded":"2024-04-25T01:55:15.918950+0000","last_fullsized":"2024-04-25T01:55:15.918950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:19.134633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906572+0000","last_change":"2024-04-25T01:52:44.108299+0000","last_active":"2024-04-25T01:55:15.906572+0000","last_peered":"2024-04-25T01:55:15.906572+0000","last_clean":"2024-04-25T01:55:15.906572+0000","last_became_active":"2024-04-25T01:52:44.107573+0000","last_became_peered":"2024-04-25T01:52:44.107573+0000","last_unstale":"2024-04-25T01:55:15.906572+0000","last_undegraded":"2024-04-25T01:55:15.906572+0000","last_fullsized":"2024-04-25T01:55:15.906572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.804579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714366+0000","last_change":"2024-04-25T01:52:43.079263+0000","last_active":"2024-04-25T01:55:15.714366+0000","last_peered":"2024-04-25T01:55:15.714366+0000","last_clean":"2024-04-25T01:55:15.714366+0000","last_became_active":"2024-04-25T01:52:43.078459+0000","last_became_peered":"2024-04-25T01:52:43.078459+0000","last_unstale":"2024-04-25T01:55:15.714366+0000","last_undegraded":"2024-04-25T01:55:15.714366+0000","last_fullsized":"2024-04-25T01:55:15.714366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:59.349897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576541+0000","last_change":"2024-04-25T01:52:44.088044+0000","last_active":"2024-04-25T01:55:15.576541+0000","last_peered":"2024-04-25T01:55:15.576541+0000","last_clean":"2024-04-25T01:55:15.576541+0000","last_became_active":"2024-04-25T01:52:44.087682+0000","last_became_peered":"2024-04-25T01:52:44.087682+0000","last_unstale":"2024-04-25T01:55:15.576541+0000","last_undegraded":"2024-04-25T01:55:15.576541+0000","last_fullsized":"2024-04-25T01:55:15.576541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:08.933923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":380,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.009718+0000","last_change":"2024-04-25T01:53:43.869516+0000","last_active":"2024-04-25T01:55:16.009718+0000","last_peered":"2024-04-25T01:55:16.009718+0000","last_clean":"2024-04-25T01:55:16.009718+0000","last_became_active":"2024-04-25T01:53:43.861718+0000","last_became_peered":"2024-04-25T01:53:43.861718+0000","last_unstale":"2024-04-25T01:55:16.009718+0000","last_undegraded":"2024-04-25T01:55:16.009718+0000","last_fullsized":"2024-04-25T01:55:16.009718+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:13:01.820264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67760,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":297,"num_read_kb":215,"num_write":36,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702638+0000","last_change":"2024-04-25T01:52:44.091547+0000","last_active":"2024-04-25T01:55:15.702638+0000","last_peered":"2024-04-25T01:55:15.702638+0000","last_clean":"2024-04-25T01:55:15.702638+0000","last_became_active":"2024-04-25T01:52:44.091167+0000","last_became_peered":"2024-04-25T01:52:44.091167+0000","last_unstale":"2024-04-25T01:55:15.702638+0000","last_undegraded":"2024-04-25T01:55:15.702638+0000","last_fullsized":"2024-04-25T01:55:15.702638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:24.559694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904856+0000","last_change":"2024-04-25T01:52:43.088385+0000","last_active":"2024-04-25T01:55:15.904856+0000","last_peered":"2024-04-25T01:55:15.904856+0000","last_clean":"2024-04-25T01:55:15.904856+0000","last_became_active":"2024-04-25T01:52:43.087982+0000","last_became_peered":"2024-04-25T01:52:43.087982+0000","last_unstale":"2024-04-25T01:55:15.904856+0000","last_undegraded":"2024-04-25T01:55:15.904856+0000","last_fullsized":"2024-04-25T01:55:15.904856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:28.944427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575264+0000","last_change":"2024-04-25T01:52:44.099420+0000","last_active":"2024-04-25T01:55:15.575264+0000","last_peered":"2024-04-25T01:55:15.575264+0000","last_clean":"2024-04-25T01:55:15.575264+0000","last_became_active":"2024-04-25T01:52:44.099045+0000","last_became_peered":"2024-04-25T01:52:44.099045+0000","last_unstale":"2024-04-25T01:55:15.575264+0000","last_undegraded":"2024-04-25T01:55:15.575264+0000","last_fullsized":"2024-04-25T01:55:15.575264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:28.377091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.713969+0000","last_change":"2024-04-25T01:52:43.086252+0000","last_active":"2024-04-25T01:55:15.713969+0000","last_peered":"2024-04-25T01:55:15.713969+0000","last_clean":"2024-04-25T01:55:15.713969+0000","last_became_active":"2024-04-25T01:52:43.085892+0000","last_became_peered":"2024-04-25T01:52:43.085892+0000","last_unstale":"2024-04-25T01:55:15.713969+0000","last_undegraded":"2024-04-25T01:55:15.713969+0000","last_fullsized":"2024-04-25T01:55:15.713969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:25:26.395705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918824+0000","last_change":"2024-04-25T01:52:44.097163+0000","last_active":"2024-04-25T01:55:15.918824+0000","last_peered":"2024-04-25T01:55:15.918824+0000","last_clean":"2024-04-25T01:55:15.918824+0000","last_became_active":"2024-04-25T01:52:44.096758+0000","last_became_peered":"2024-04-25T01:52:44.096758+0000","last_unstale":"2024-04-25T01:55:15.918824+0000","last_undegraded":"2024-04-25T01:55:15.918824+0000","last_fullsized":"2024-04-25T01:55:15.918824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:22:47.394855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.359181+0000","last_change":"2024-04-25T01:52:43.069024+0000","last_active":"2024-04-25T01:55:16.359181+0000","last_peered":"2024-04-25T01:55:16.359181+0000","last_clean":"2024-04-25T01:55:16.359181+0000","last_became_active":"2024-04-25T01:52:43.068058+0000","last_became_peered":"2024-04-25T01:52:43.068058+0000","last_unstale":"2024-04-25T01:55:16.359181+0000","last_undegraded":"2024-04-25T01:55:16.359181+0000","last_fullsized":"2024-04-25T01:55:16.359181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:43.126131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918257+0000","last_change":"2024-04-25T01:52:44.091829+0000","last_active":"2024-04-25T01:55:15.918257+0000","last_peered":"2024-04-25T01:55:15.918257+0000","last_clean":"2024-04-25T01:55:15.918257+0000","last_became_active":"2024-04-25T01:52:44.091528+0000","last_became_peered":"2024-04-25T01:52:44.091528+0000","last_unstale":"2024-04-25T01:55:15.918257+0000","last_undegraded":"2024-04-25T01:55:15.918257+0000","last_fullsized":"2024-04-25T01:55:15.918257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:39.585296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904263+0000","last_change":"2024-04-25T01:53:43.888956+0000","last_active":"2024-04-25T01:55:15.904263+0000","last_peered":"2024-04-25T01:55:15.904263+0000","last_clean":"2024-04-25T01:55:15.904263+0000","last_became_active":"2024-04-25T01:53:43.870789+0000","last_became_peered":"2024-04-25T01:53:43.870789+0000","last_unstale":"2024-04-25T01:55:15.904263+0000","last_undegraded":"2024-04-25T01:55:15.904263+0000","last_fullsized":"2024-04-25T01:55:15.904263+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:08:48.790006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714034+0000","last_change":"2024-04-25T01:52:44.094696+0000","last_active":"2024-04-25T01:55:15.714034+0000","last_peered":"2024-04-25T01:55:15.714034+0000","last_clean":"2024-04-25T01:55:15.714034+0000","last_became_active":"2024-04-25T01:52:44.094371+0000","last_became_peered":"2024-04-25T01:52:44.094371+0000","last_unstale":"2024-04-25T01:55:15.714034+0000","last_undegraded":"2024-04-25T01:55:15.714034+0000","last_fullsized":"2024-04-25T01:55:15.714034+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:20.035674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702786+0000","last_change":"2024-04-25T01:52:43.071275+0000","last_active":"2024-04-25T01:55:15.702786+0000","last_peered":"2024-04-25T01:55:15.702786+0000","last_clean":"2024-04-25T01:55:15.702786+0000","last_became_active":"2024-04-25T01:52:43.070958+0000","last_became_peered":"2024-04-25T01:52:43.070958+0000","last_unstale":"2024-04-25T01:55:15.702786+0000","last_undegraded":"2024-04-25T01:55:15.702786+0000","last_fullsized":"2024-04-25T01:55:15.702786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:23:55.284943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904719+0000","last_change":"2024-04-25T01:52:44.092742+0000","last_active":"2024-04-25T01:55:15.904719+0000","last_peered":"2024-04-25T01:55:15.904719+0000","last_clean":"2024-04-25T01:55:15.904719+0000","last_became_active":"2024-04-25T01:52:44.092345+0000","last_became_peered":"2024-04-25T01:52:44.092345+0000","last_unstale":"2024-04-25T01:55:15.904719+0000","last_undegraded":"2024-04-25T01:55:15.904719+0000","last_fullsized":"2024-04-25T01:55:15.904719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:14.325814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584193+0000","last_change":"2024-04-25T01:52:43.090057+0000","last_active":"2024-04-25T01:55:15.584193+0000","last_peered":"2024-04-25T01:55:15.584193+0000","last_clean":"2024-04-25T01:55:15.584193+0000","last_became_active":"2024-04-25T01:52:43.089280+0000","last_became_peered":"2024-04-25T01:52:43.089280+0000","last_unstale":"2024-04-25T01:55:15.584193+0000","last_undegraded":"2024-04-25T01:55:15.584193+0000","last_fullsized":"2024-04-25T01:55:15.584193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:41.048908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.718275+0000","last_change":"2024-04-25T01:52:44.097651+0000","last_active":"2024-04-25T01:55:15.718275+0000","last_peered":"2024-04-25T01:55:15.718275+0000","last_clean":"2024-04-25T01:55:15.718275+0000","last_became_active":"2024-04-25T01:52:44.097254+0000","last_became_peered":"2024-04-25T01:52:44.097254+0000","last_unstale":"2024-04-25T01:55:15.718275+0000","last_undegraded":"2024-04-25T01:55:15.718275+0000","last_fullsized":"2024-04-25T01:55:15.718275+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:15.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918409+0000","last_change":"2024-04-25T01:52:43.076925+0000","last_active":"2024-04-25T01:55:15.918409+0000","last_peered":"2024-04-25T01:55:15.918409+0000","last_clean":"2024-04-25T01:55:15.918409+0000","last_became_active":"2024-04-25T01:52:43.076514+0000","last_became_peered":"2024-04-25T01:52:43.076514+0000","last_unstale":"2024-04-25T01:55:15.918409+0000","last_undegraded":"2024-04-25T01:55:15.918409+0000","last_fullsized":"2024-04-25T01:55:15.918409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:31:17.305379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.718045+0000","last_change":"2024-04-25T01:52:44.103731+0000","last_active":"2024-04-25T01:55:15.718045+0000","last_peered":"2024-04-25T01:55:15.718045+0000","last_clean":"2024-04-25T01:55:15.718045+0000","last_became_active":"2024-04-25T01:52:44.103173+0000","last_became_peered":"2024-04-25T01:52:44.103173+0000","last_unstale":"2024-04-25T01:55:15.718045+0000","last_undegraded":"2024-04-25T01:55:15.718045+0000","last_fullsized":"2024-04-25T01:55:15.718045+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:43.617579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714132+0000","last_change":"2024-04-25T01:52:44.090847+0000","last_active":"2024-04-25T01:55:15.714132+0000","last_peered":"2024-04-25T01:55:15.714132+0000","last_clean":"2024-04-25T01:55:15.714132+0000","last_became_active":"2024-04-25T01:52:44.090506+0000","last_became_peered":"2024-04-25T01:52:44.090506+0000","last_unstale":"2024-04-25T01:55:15.714132+0000","last_undegraded":"2024-04-25T01:55:15.714132+0000","last_fullsized":"2024-04-25T01:55:15.714132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:24.539726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.922129+0000","last_change":"2024-04-25T01:52:43.090469+0000","last_active":"2024-04-25T01:55:15.922129+0000","last_peered":"2024-04-25T01:55:15.922129+0000","last_clean":"2024-04-25T01:55:15.922129+0000","last_became_active":"2024-04-25T01:52:43.090159+0000","last_became_peered":"2024-04-25T01:52:43.090159+0000","last_unstale":"2024-04-25T01:55:15.922129+0000","last_undegraded":"2024-04-25T01:55:15.922129+0000","last_fullsized":"2024-04-25T01:55:15.922129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:42:45.532017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584403+0000","last_change":"2024-04-25T01:52:44.093527+0000","last_active":"2024-04-25T01:55:15.584403+0000","last_peered":"2024-04-25T01:55:15.584403+0000","last_clean":"2024-04-25T01:55:15.584403+0000","last_became_active":"2024-04-25T01:52:44.092729+0000","last_became_peered":"2024-04-25T01:52:44.092729+0000","last_unstale":"2024-04-25T01:55:15.584403+0000","last_undegraded":"2024-04-25T01:55:15.584403+0000","last_fullsized":"2024-04-25T01:55:15.584403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:32:50.668352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904349+0000","last_change":"2024-04-25T01:52:43.075323+0000","last_active":"2024-04-25T01:55:15.904349+0000","last_peered":"2024-04-25T01:55:15.904349+0000","last_clean":"2024-04-25T01:55:15.904349+0000","last_became_active":"2024-04-25T01:52:43.074876+0000","last_became_peered":"2024-04-25T01:52:43.074876+0000","last_unstale":"2024-04-25T01:55:15.904349+0000","last_undegraded":"2024-04-25T01:55:15.904349+0000","last_fullsized":"2024-04-25T01:55:15.904349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:47.472920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918263+0000","last_change":"2024-04-25T01:51:20.173285+0000","last_active":"2024-04-25T01:55:15.918263+0000","last_peered":"2024-04-25T01:55:15.918263+0000","last_clean":"2024-04-25T01:55:15.918263+0000","last_became_active":"2024-04-25T01:51:20.172811+0000","last_became_peered":"2024-04-25T01:51:20.172811+0000","last_unstale":"2024-04-25T01:55:15.918263+0000","last_undegraded":"2024-04-25T01:55:15.918263+0000","last_fullsized":"2024-04-25T01:55:15.918263+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_clean_scrub_stamp":"2024-04-25T01:47:31.712321+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:33.708082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.574950+0000","last_change":"2024-04-25T01:53:43.874632+0000","last_active":"2024-04-25T01:55:15.574950+0000","last_peered":"2024-04-25T01:55:15.574950+0000","last_clean":"2024-04-25T01:55:15.574950+0000","last_became_active":"2024-04-25T01:53:43.873732+0000","last_became_peered":"2024-04-25T01:53:43.873732+0000","last_unstale":"2024-04-25T01:55:15.574950+0000","last_undegraded":"2024-04-25T01:55:15.574950+0000","last_fullsized":"2024-04-25T01:55:15.574950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:27.874789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918402+0000","last_change":"2024-04-25T01:52:43.072565+0000","last_active":"2024-04-25T01:55:15.918402+0000","last_peered":"2024-04-25T01:55:15.918402+0000","last_clean":"2024-04-25T01:55:15.918402+0000","last_became_active":"2024-04-25T01:52:43.072193+0000","last_became_peered":"2024-04-25T01:52:43.072193+0000","last_unstale":"2024-04-25T01:55:15.918402+0000","last_undegraded":"2024-04-25T01:55:15.918402+0000","last_fullsized":"2024-04-25T01:55:15.918402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:07.261859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904537+0000","last_change":"2024-04-25T01:53:43.878499+0000","last_active":"2024-04-25T01:55:15.904537+0000","last_peered":"2024-04-25T01:55:15.904537+0000","last_clean":"2024-04-25T01:55:15.904537+0000","last_became_active":"2024-04-25T01:53:43.870876+0000","last_became_peered":"2024-04-25T01:53:43.870876+0000","last_unstale":"2024-04-25T01:55:15.904537+0000","last_undegraded":"2024-04-25T01:55:15.904537+0000","last_fullsized":"2024-04-25T01:55:15.904537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:08.809127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575785+0000","last_change":"2024-04-25T01:52:44.104892+0000","last_active":"2024-04-25T01:55:15.575785+0000","last_peered":"2024-04-25T01:55:15.575785+0000","last_clean":"2024-04-25T01:55:15.575785+0000","last_became_active":"2024-04-25T01:52:44.103802+0000","last_became_peered":"2024-04-25T01:52:44.103802+0000","last_unstale":"2024-04-25T01:55:15.575785+0000","last_undegraded":"2024-04-25T01:55:15.575785+0000","last_fullsized":"2024-04-25T01:55:15.575785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:16.445968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'44","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.862454+0000","last_change":"2024-04-25T01:52:43.087143+0000","last_active":"2024-04-25T01:55:15.862454+0000","last_peered":"2024-04-25T01:55:15.862454+0000","last_clean":"2024-04-25T01:55:15.862454+0000","last_became_active":"2024-04-25T01:52:43.085987+0000","last_became_peered":"2024-04-25T01:52:43.085987+0000","last_unstale":"2024-04-25T01:55:15.862454+0000","last_undegraded":"2024-04-25T01:55:15.862454+0000","last_fullsized":"2024-04-25T01:55:15.862454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:17.936671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584485+0000","last_change":"2024-04-25T01:52:44.092342+0000","last_active":"2024-04-25T01:55:15.584485+0000","last_peered":"2024-04-25T01:55:15.584485+0000","last_clean":"2024-04-25T01:55:15.584485+0000","last_became_active":"2024-04-25T01:52:44.091991+0000","last_became_peered":"2024-04-25T01:52:44.091991+0000","last_unstale":"2024-04-25T01:55:15.584485+0000","last_undegraded":"2024-04-25T01:55:15.584485+0000","last_fullsized":"2024-04-25T01:55:15.584485+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:22.147284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904395+0000","last_change":"2024-04-25T01:52:43.077972+0000","last_active":"2024-04-25T01:55:15.904395+0000","last_peered":"2024-04-25T01:55:15.904395+0000","last_clean":"2024-04-25T01:55:15.904395+0000","last_became_active":"2024-04-25T01:52:43.077655+0000","last_became_peered":"2024-04-25T01:52:43.077655+0000","last_unstale":"2024-04-25T01:55:15.904395+0000","last_undegraded":"2024-04-25T01:55:15.904395+0000","last_fullsized":"2024-04-25T01:55:15.904395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:24.820910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905887+0000","last_change":"2024-04-25T01:52:44.119560+0000","last_active":"2024-04-25T01:55:15.905887+0000","last_peered":"2024-04-25T01:55:15.905887+0000","last_clean":"2024-04-25T01:55:15.905887+0000","last_became_active":"2024-04-25T01:52:44.119106+0000","last_became_peered":"2024-04-25T01:52:44.119106+0000","last_unstale":"2024-04-25T01:55:15.905887+0000","last_undegraded":"2024-04-25T01:55:15.905887+0000","last_fullsized":"2024-04-25T01:55:15.905887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:50.702393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576106+0000","last_change":"2024-04-25T01:52:43.075783+0000","last_active":"2024-04-25T01:55:15.576106+0000","last_peered":"2024-04-25T01:55:15.576106+0000","last_clean":"2024-04-25T01:55:15.576106+0000","last_became_active":"2024-04-25T01:52:43.075462+0000","last_became_peered":"2024-04-25T01:52:43.075462+0000","last_unstale":"2024-04-25T01:55:15.576106+0000","last_undegraded":"2024-04-25T01:55:15.576106+0000","last_fullsized":"2024-04-25T01:55:15.576106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:07.077047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702282+0000","last_change":"2024-04-25T01:52:44.090695+0000","last_active":"2024-04-25T01:55:15.702282+0000","last_peered":"2024-04-25T01:55:15.702282+0000","last_clean":"2024-04-25T01:55:15.702282+0000","last_became_active":"2024-04-25T01:52:44.090203+0000","last_became_peered":"2024-04-25T01:52:44.090203+0000","last_unstale":"2024-04-25T01:55:15.702282+0000","last_undegraded":"2024-04-25T01:55:15.702282+0000","last_fullsized":"2024-04-25T01:55:15.702282+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:58.104569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583701+0000","last_change":"2024-04-25T01:52:43.086057+0000","last_active":"2024-04-25T01:55:15.583701+0000","last_peered":"2024-04-25T01:55:15.583701+0000","last_clean":"2024-04-25T01:55:15.583701+0000","last_became_active":"2024-04-25T01:52:43.085475+0000","last_became_peered":"2024-04-25T01:52:43.085475+0000","last_unstale":"2024-04-25T01:55:15.583701+0000","last_undegraded":"2024-04-25T01:55:15.583701+0000","last_fullsized":"2024-04-25T01:55:15.583701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:42.237414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904383+0000","last_change":"2024-04-25T01:52:44.106128+0000","last_active":"2024-04-25T01:55:15.904383+0000","last_peered":"2024-04-25T01:55:15.904383+0000","last_clean":"2024-04-25T01:55:15.904383+0000","last_became_active":"2024-04-25T01:52:44.105805+0000","last_became_peered":"2024-04-25T01:52:44.105805+0000","last_unstale":"2024-04-25T01:55:15.904383+0000","last_undegraded":"2024-04-25T01:55:15.904383+0000","last_fullsized":"2024-04-25T01:55:15.904383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:41.901070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904922+0000","last_change":"2024-04-25T01:52:43.088347+0000","last_active":"2024-04-25T01:55:15.904922+0000","last_peered":"2024-04-25T01:55:15.904922+0000","last_clean":"2024-04-25T01:55:15.904922+0000","last_became_active":"2024-04-25T01:52:43.087932+0000","last_became_peered":"2024-04-25T01:52:43.087932+0000","last_unstale":"2024-04-25T01:55:15.904922+0000","last_undegraded":"2024-04-25T01:55:15.904922+0000","last_fullsized":"2024-04-25T01:55:15.904922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:26.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584766+0000","last_change":"2024-04-25T01:52:44.108686+0000","last_active":"2024-04-25T01:55:15.584766+0000","last_peered":"2024-04-25T01:55:15.584766+0000","last_clean":"2024-04-25T01:55:15.584766+0000","last_became_active":"2024-04-25T01:52:44.107732+0000","last_became_peered":"2024-04-25T01:52:44.107732+0000","last_unstale":"2024-04-25T01:55:15.584766+0000","last_undegraded":"2024-04-25T01:55:15.584766+0000","last_fullsized":"2024-04-25T01:55:15.584766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:14.820979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702353+0000","last_change":"2024-04-25T01:52:43.069300+0000","last_active":"2024-04-25T01:55:15.702353+0000","last_peered":"2024-04-25T01:55:15.702353+0000","last_clean":"2024-04-25T01:55:15.702353+0000","last_became_active":"2024-04-25T01:52:43.068644+0000","last_became_peered":"2024-04-25T01:52:43.068644+0000","last_unstale":"2024-04-25T01:55:15.702353+0000","last_undegraded":"2024-04-25T01:55:15.702353+0000","last_fullsized":"2024-04-25T01:55:15.702353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:04.897638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918494+0000","last_change":"2024-04-25T01:52:44.100467+0000","last_active":"2024-04-25T01:55:15.918494+0000","last_peered":"2024-04-25T01:55:15.918494+0000","last_clean":"2024-04-25T01:55:15.918494+0000","last_became_active":"2024-04-25T01:52:44.099958+0000","last_became_peered":"2024-04-25T01:52:44.099958+0000","last_unstale":"2024-04-25T01:55:15.918494+0000","last_undegraded":"2024-04-25T01:55:15.918494+0000","last_fullsized":"2024-04-25T01:55:15.918494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:29.919785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904238+0000","last_change":"2024-04-25T01:53:43.875865+0000","last_active":"2024-04-25T01:55:15.904238+0000","last_peered":"2024-04-25T01:55:15.904238+0000","last_clean":"2024-04-25T01:55:15.904238+0000","last_became_active":"2024-04-25T01:53:43.871525+0000","last_became_peered":"2024-04-25T01:53:43.871525+0000","last_unstale":"2024-04-25T01:55:15.904238+0000","last_undegraded":"2024-04-25T01:55:15.904238+0000","last_fullsized":"2024-04-25T01:55:15.904238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:25.179902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918951+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:55:15.918951+0000","last_peered":"2024-04-25T01:55:15.918951+0000","last_clean":"2024-04-25T01:55:15.918951+0000","last_became_active":"2024-04-25T01:52:44.087911+0000","last_became_peered":"2024-04-25T01:52:44.087911+0000","last_unstale":"2024-04-25T01:55:15.918951+0000","last_undegraded":"2024-04-25T01:55:15.918951+0000","last_fullsized":"2024-04-25T01:55:15.918951+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:29:38.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.859264+0000","last_change":"2024-04-25T01:52:43.086083+0000","last_active":"2024-04-25T01:55:15.859264+0000","last_peered":"2024-04-25T01:55:15.859264+0000","last_clean":"2024-04-25T01:55:15.859264+0000","last_became_active":"2024-04-25T01:52:43.085613+0000","last_became_peered":"2024-04-25T01:52:43.085613+0000","last_unstale":"2024-04-25T01:55:15.859264+0000","last_undegraded":"2024-04-25T01:55:15.859264+0000","last_fullsized":"2024-04-25T01:55:15.859264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:23.851812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918421+0000","last_change":"2024-04-25T01:52:44.106435+0000","last_active":"2024-04-25T01:55:15.918421+0000","last_peered":"2024-04-25T01:55:15.918421+0000","last_clean":"2024-04-25T01:55:15.918421+0000","last_became_active":"2024-04-25T01:52:44.106074+0000","last_became_peered":"2024-04-25T01:52:44.106074+0000","last_unstale":"2024-04-25T01:55:15.918421+0000","last_undegraded":"2024-04-25T01:55:15.918421+0000","last_fullsized":"2024-04-25T01:55:15.918421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:59.286266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575223+0000","last_change":"2024-04-25T01:52:43.088902+0000","last_active":"2024-04-25T01:55:15.575223+0000","last_peered":"2024-04-25T01:55:15.575223+0000","last_clean":"2024-04-25T01:55:15.575223+0000","last_became_active":"2024-04-25T01:52:43.088289+0000","last_became_peered":"2024-04-25T01:52:43.088289+0000","last_unstale":"2024-04-25T01:55:15.575223+0000","last_undegraded":"2024-04-25T01:55:15.575223+0000","last_fullsized":"2024-04-25T01:55:15.575223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:37:44.562369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918975+0000","last_change":"2024-04-25T01:52:44.094185+0000","last_active":"2024-04-25T01:55:15.918975+0000","last_peered":"2024-04-25T01:55:15.918975+0000","last_clean":"2024-04-25T01:55:15.918975+0000","last_became_active":"2024-04-25T01:52:44.093753+0000","last_became_peered":"2024-04-25T01:52:44.093753+0000","last_unstale":"2024-04-25T01:55:15.918975+0000","last_undegraded":"2024-04-25T01:55:15.918975+0000","last_fullsized":"2024-04-25T01:55:15.918975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:06.089670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904437+0000","last_change":"2024-04-25T01:52:44.093895+0000","last_active":"2024-04-25T01:55:15.904437+0000","last_peered":"2024-04-25T01:55:15.904437+0000","last_clean":"2024-04-25T01:55:15.904437+0000","last_became_active":"2024-04-25T01:52:44.093490+0000","last_became_peered":"2024-04-25T01:52:44.093490+0000","last_unstale":"2024-04-25T01:55:15.904437+0000","last_undegraded":"2024-04-25T01:55:15.904437+0000","last_fullsized":"2024-04-25T01:55:15.904437+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:53.848979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904530+0000","last_change":"2024-04-25T01:52:44.092500+0000","last_active":"2024-04-25T01:55:15.904530+0000","last_peered":"2024-04-25T01:55:15.904530+0000","last_clean":"2024-04-25T01:55:15.904530+0000","last_became_active":"2024-04-25T01:52:44.092044+0000","last_became_peered":"2024-04-25T01:52:44.092044+0000","last_unstale":"2024-04-25T01:55:15.904530+0000","last_undegraded":"2024-04-25T01:55:15.904530+0000","last_fullsized":"2024-04-25T01:55:15.904530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:28.279676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904716+0000","last_change":"2024-04-25T01:52:44.116668+0000","last_active":"2024-04-25T01:55:15.904716+0000","last_peered":"2024-04-25T01:55:15.904716+0000","last_clean":"2024-04-25T01:55:15.904716+0000","last_became_active":"2024-04-25T01:52:44.116295+0000","last_became_peered":"2024-04-25T01:52:44.116295+0000","last_unstale":"2024-04-25T01:55:15.904716+0000","last_undegraded":"2024-04-25T01:55:15.904716+0000","last_fullsized":"2024-04-25T01:55:15.904716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:49.109403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905629+0000","last_change":"2024-04-25T01:52:44.109950+0000","last_active":"2024-04-25T01:55:15.905629+0000","last_peered":"2024-04-25T01:55:15.905629+0000","last_clean":"2024-04-25T01:55:15.905629+0000","last_became_active":"2024-04-25T01:52:44.109636+0000","last_became_peered":"2024-04-25T01:52:44.109636+0000","last_unstale":"2024-04-25T01:55:15.905629+0000","last_undegraded":"2024-04-25T01:55:15.905629+0000","last_fullsized":"2024-04-25T01:55:15.905629+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:02:57.610067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575687+0000","last_change":"2024-04-25T01:52:44.105237+0000","last_active":"2024-04-25T01:55:15.575687+0000","last_peered":"2024-04-25T01:55:15.575687+0000","last_clean":"2024-04-25T01:55:15.575687+0000","last_became_active":"2024-04-25T01:52:44.104531+0000","last_became_peered":"2024-04-25T01:52:44.104531+0000","last_unstale":"2024-04-25T01:55:15.575687+0000","last_undegraded":"2024-04-25T01:55:15.575687+0000","last_fullsized":"2024-04-25T01:55:15.575687+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.149119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904755+0000","last_change":"2024-04-25T01:52:44.095655+0000","last_active":"2024-04-25T01:55:15.904755+0000","last_peered":"2024-04-25T01:55:15.904755+0000","last_clean":"2024-04-25T01:55:15.904755+0000","last_became_active":"2024-04-25T01:52:44.095300+0000","last_became_peered":"2024-04-25T01:52:44.095300+0000","last_unstale":"2024-04-25T01:55:15.904755+0000","last_undegraded":"2024-04-25T01:55:15.904755+0000","last_fullsized":"2024-04-25T01:55:15.904755+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:51.240170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904995+0000","last_change":"2024-04-25T01:52:44.087382+0000","last_active":"2024-04-25T01:55:15.904995+0000","last_peered":"2024-04-25T01:55:15.904995+0000","last_clean":"2024-04-25T01:55:15.904995+0000","last_became_active":"2024-04-25T01:52:44.086815+0000","last_became_peered":"2024-04-25T01:52:44.086815+0000","last_unstale":"2024-04-25T01:55:15.904995+0000","last_undegraded":"2024-04-25T01:55:15.904995+0000","last_fullsized":"2024-04-25T01:55:15.904995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:52:13.682819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584351+0000","last_change":"2024-04-25T01:53:43.871148+0000","last_active":"2024-04-25T01:55:15.584351+0000","last_peered":"2024-04-25T01:55:15.584351+0000","last_clean":"2024-04-25T01:55:15.584351+0000","last_became_active":"2024-04-25T01:53:43.870490+0000","last_became_peered":"2024-04-25T01:53:43.870490+0000","last_unstale":"2024-04-25T01:55:15.584351+0000","last_undegraded":"2024-04-25T01:55:15.584351+0000","last_fullsized":"2024-04-25T01:55:15.584351+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:41.790442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.713960+0000","last_change":"2024-04-25T01:52:44.114767+0000","last_active":"2024-04-25T01:55:15.713960+0000","last_peered":"2024-04-25T01:55:15.713960+0000","last_clean":"2024-04-25T01:55:15.713960+0000","last_became_active":"2024-04-25T01:52:44.114483+0000","last_became_peered":"2024-04-25T01:52:44.114483+0000","last_unstale":"2024-04-25T01:55:15.713960+0000","last_undegraded":"2024-04-25T01:55:15.713960+0000","last_fullsized":"2024-04-25T01:55:15.713960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:14.439287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583527+0000","last_change":"2024-04-25T01:52:44.094424+0000","last_active":"2024-04-25T01:55:15.583527+0000","last_peered":"2024-04-25T01:55:15.583527+0000","last_clean":"2024-04-25T01:55:15.583527+0000","last_became_active":"2024-04-25T01:52:44.093074+0000","last_became_peered":"2024-04-25T01:52:44.093074+0000","last_unstale":"2024-04-25T01:55:15.583527+0000","last_undegraded":"2024-04-25T01:55:15.583527+0000","last_fullsized":"2024-04-25T01:55:15.583527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:20.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576299+0000","last_change":"2024-04-25T01:52:44.113465+0000","last_active":"2024-04-25T01:55:15.576299+0000","last_peered":"2024-04-25T01:55:15.576299+0000","last_clean":"2024-04-25T01:55:15.576299+0000","last_became_active":"2024-04-25T01:52:44.113168+0000","last_became_peered":"2024-04-25T01:52:44.113168+0000","last_unstale":"2024-04-25T01:55:15.576299+0000","last_undegraded":"2024-04-25T01:55:15.576299+0000","last_fullsized":"2024-04-25T01:55:15.576299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:42:47.758996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575462+0000","last_change":"2024-04-25T01:52:44.104763+0000","last_active":"2024-04-25T01:55:15.575462+0000","last_peered":"2024-04-25T01:55:15.575462+0000","last_clean":"2024-04-25T01:55:15.575462+0000","last_became_active":"2024-04-25T01:52:44.104377+0000","last_became_peered":"2024-04-25T01:52:44.104377+0000","last_unstale":"2024-04-25T01:55:15.575462+0000","last_undegraded":"2024-04-25T01:55:15.575462+0000","last_fullsized":"2024-04-25T01:55:15.575462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:13.647020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904326+0000","last_change":"2024-04-25T01:53:43.864792+0000","last_active":"2024-04-25T01:55:15.904326+0000","last_peered":"2024-04-25T01:55:15.904326+0000","last_clean":"2024-04-25T01:55:15.904326+0000","last_became_active":"2024-04-25T01:53:43.864432+0000","last_became_peered":"2024-04-25T01:53:43.864432+0000","last_unstale":"2024-04-25T01:55:15.904326+0000","last_undegraded":"2024-04-25T01:55:15.904326+0000","last_fullsized":"2024-04-25T01:55:15.904326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:28.149564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598282,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1000,"num_read_kb":609,"num_write":287,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":61762,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318317,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":471947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579953,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41328,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703824,"statfs":{"total":95995035648,"available":95952715776,"internally_reserved":0,"allocated":1814528,"data_stored":1263436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743480,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1384448,"data_stored":865265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005117,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32328,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712824,"statfs":{"total":95995035648,"available":95961931776,"internally_reserved":0,"allocated":1175552,"data_stored":668443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266755,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1585152,"data_stored":1065604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430279,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":471438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":532836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":463782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":991232,"data_stored":473313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117080,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":995328,"data_stored":471749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855443,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":540999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691917,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1196106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":69768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:55:20.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:55:20.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2946896041' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:20.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:19 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1115765941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:20.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2946896041' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:20.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:19 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1115765941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:20.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.303+0000 7ff149200640 1 -- 172.21.15.102:0/3807501547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1441037e0 msgr2=0x7ff144103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.303+0000 7ff149200640 1 --2- 172.21.15.102:0/3807501547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1441037e0 0x7ff144103c00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff130009900 tx=0x7ff13002a410 comp rx=0 tx=0).stop 2024-04-25T01:55:20.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.303+0000 7ff149200640 1 -- 172.21.15.102:0/3807501547 shutdown_connections 2024-04-25T01:55:20.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.303+0000 7ff149200640 1 --2- 172.21.15.102:0/3807501547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff144105390 0x7ff14410bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.304+0000 7ff149200640 1 --2- 172.21.15.102:0/3807501547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1441049d0 0x7ff144104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.304+0000 7ff149200640 1 --2- 172.21.15.102:0/3807501547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1441037e0 0x7ff144103c00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.304+0000 7ff149200640 1 -- 172.21.15.102:0/3807501547 >> 172.21.15.102:0/3807501547 conn(0x7ff1440fed50 msgr2=0x7ff144101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:20.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.304+0000 7ff149200640 1 -- 172.21.15.102:0/3807501547 shutdown_connections 2024-04-25T01:55:20.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.304+0000 7ff149200640 1 -- 172.21.15.102:0/3807501547 wait complete. 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff149200640 1 Processor -- start 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff149200640 1 -- start start 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff149200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1441037e0 0x7ff144199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff149200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1441049d0 0x7ff14419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff149200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff144105390 0x7ff1441a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff149200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410f480 con 0x7ff144105390 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff149200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410f300 con 0x7ff1441037e0 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff149200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410f600 con 0x7ff1441049d0 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff143400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1441037e0 0x7ff144199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff143e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff144105390 0x7ff1441a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff143400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1441037e0 0x7ff144199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33156/0 (socket says 172.21.15.102:33156) 2024-04-25T01:55:20.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff143400640 1 -- 172.21.15.102:0/288149091 learned_addr learned my addr 172.21.15.102:0/288149091 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.305+0000 7ff142a00640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1441049d0 0x7ff14419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff143400640 1 -- 172.21.15.102:0/288149091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1441049d0 msgr2=0x7ff14419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff143400640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1441049d0 0x7ff14419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff143400640 1 -- 172.21.15.102:0/288149091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff144105390 msgr2=0x7ff1441a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff143400640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff144105390 0x7ff1441a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff143400640 1 -- 172.21.15.102:0/288149091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1441a1710 con 0x7ff1441037e0 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff143400640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1441037e0 0x7ff144199af0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff130031440 tx=0x7ff130032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff13be00640 1 -- 172.21.15.102:0/288149091 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13003f030 con 0x7ff1441037e0 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1441a19a0 con 0x7ff1441037e0 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff13be00640 1 -- 172.21.15.102:0/288149091 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff130003ce0 con 0x7ff1441037e0 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.306+0000 7ff13be00640 1 -- 172.21.15.102:0/288149091 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130035070 con 0x7ff1441037e0 2024-04-25T01:55:20.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.307+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1441a1ee0 con 0x7ff1441037e0 2024-04-25T01:55:20.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.308+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff14410ab00 con 0x7ff1441037e0 2024-04-25T01:55:20.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.310+0000 7ff13be00640 1 -- 172.21.15.102:0/288149091 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 23) ==== 137954+0+0 (secure 0 0 0) 0x7ff130042030 con 0x7ff1441037e0 2024-04-25T01:55:20.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.311+0000 7ff13be00640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1280a2d80 0x7ff1280a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:20.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.311+0000 7ff13be00640 1 -- 172.21.15.102:0/288149091 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff1300ef860 con 0x7ff1441037e0 2024-04-25T01:55:20.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.311+0000 7ff142a00640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1280a2d80 0x7ff1280a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:20.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.312+0000 7ff142a00640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1280a2d80 0x7ff1280a5240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff14419b000 tx=0x7ff134005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:20.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.315+0000 7ff13be00640 1 -- 172.21.15.102:0/288149091 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff1300bbe00 con 0x7ff1441037e0 2024-04-25T01:55:20.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.529+0000 7ff13be00640 1 -- 172.21.15.102:0/288149091 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7ff13003b460 con 0x7ff1441037e0 2024-04-25T01:55:20.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.531+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 --> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff144061b80 con 0x7ff1280a2d80 2024-04-25T01:55:20.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.540+0000 7ff13be00640 1 -- 172.21.15.102:0/288149091 <== mgr.25214 v2:172.21.15.102:6800/855543702 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257771 (secure 0 0 0) 0x7ff144061b80 con 0x7ff1280a2d80 2024-04-25T01:55:20.545 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:20.546 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-25T01:55:20.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1280a2d80 msgr2=0x7ff1280a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1280a2d80 0x7ff1280a5240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff14419b000 tx=0x7ff134005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:20.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1441037e0 msgr2=0x7ff144199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1441037e0 0x7ff144199af0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff130031440 tx=0x7ff130032ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 shutdown_connections 2024-04-25T01:55:20.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1280a2d80 0x7ff1280a5240 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff144105390 0x7ff1441a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1441049d0 0x7ff14419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 --2- 172.21.15.102:0/288149091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1441037e0 0x7ff144199af0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 >> 172.21.15.102:0/288149091 conn(0x7ff1440fed50 msgr2=0x7ff144107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:20.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 shutdown_connections 2024-04-25T01:55:20.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.544+0000 7ff149200640 1 -- 172.21.15.102:0/288149091 wait complete. 2024-04-25T01:55:20.563 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T01:55:19.596440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188884,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1136,"num_read_kb":728,"num_write":470,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1242498,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384436,"kb_used_data":14388,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557388,"statfs":{"total":1151940427776,"available":1151546765312,"internally_reserved":0,"allocated":14733312,"data_stored":8484918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":378713403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.984677"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904984+0000","last_change":"2024-04-25T01:52:44.102143+0000","last_active":"2024-04-25T01:55:15.904984+0000","last_peered":"2024-04-25T01:55:15.904984+0000","last_clean":"2024-04-25T01:55:15.904984+0000","last_became_active":"2024-04-25T01:52:44.101810+0000","last_became_peered":"2024-04-25T01:52:44.101810+0000","last_unstale":"2024-04-25T01:55:15.904984+0000","last_undegraded":"2024-04-25T01:55:15.904984+0000","last_fullsized":"2024-04-25T01:55:15.904984+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:02.356294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906327+0000","last_change":"2024-04-25T01:52:44.103074+0000","last_active":"2024-04-25T01:55:15.906327+0000","last_peered":"2024-04-25T01:55:15.906327+0000","last_clean":"2024-04-25T01:55:15.906327+0000","last_became_active":"2024-04-25T01:52:44.102103+0000","last_became_peered":"2024-04-25T01:52:44.102103+0000","last_unstale":"2024-04-25T01:55:15.906327+0000","last_undegraded":"2024-04-25T01:55:15.906327+0000","last_fullsized":"2024-04-25T01:55:15.906327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:36:43.016756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906006+0000","last_change":"2024-04-25T01:52:44.101467+0000","last_active":"2024-04-25T01:55:15.906006+0000","last_peered":"2024-04-25T01:55:15.906006+0000","last_clean":"2024-04-25T01:55:15.906006+0000","last_became_active":"2024-04-25T01:52:44.100911+0000","last_became_peered":"2024-04-25T01:52:44.100911+0000","last_unstale":"2024-04-25T01:55:15.906006+0000","last_undegraded":"2024-04-25T01:55:15.906006+0000","last_fullsized":"2024-04-25T01:55:15.906006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:58.533213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904287+0000","last_change":"2024-04-25T01:52:44.088592+0000","last_active":"2024-04-25T01:55:15.904287+0000","last_peered":"2024-04-25T01:55:15.904287+0000","last_clean":"2024-04-25T01:55:15.904287+0000","last_became_active":"2024-04-25T01:52:44.087651+0000","last_became_peered":"2024-04-25T01:52:44.087651+0000","last_unstale":"2024-04-25T01:55:15.904287+0000","last_undegraded":"2024-04-25T01:55:15.904287+0000","last_fullsized":"2024-04-25T01:55:15.904287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:52.240818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702409+0000","last_change":"2024-04-25T01:52:44.095033+0000","last_active":"2024-04-25T01:55:15.702409+0000","last_peered":"2024-04-25T01:55:15.702409+0000","last_clean":"2024-04-25T01:55:15.702409+0000","last_became_active":"2024-04-25T01:52:44.094719+0000","last_became_peered":"2024-04-25T01:52:44.094719+0000","last_unstale":"2024-04-25T01:55:15.702409+0000","last_undegraded":"2024-04-25T01:55:15.702409+0000","last_fullsized":"2024-04-25T01:55:15.702409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:52.422544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.717941+0000","last_change":"2024-04-25T01:52:44.102574+0000","last_active":"2024-04-25T01:55:15.717941+0000","last_peered":"2024-04-25T01:55:15.717941+0000","last_clean":"2024-04-25T01:55:15.717941+0000","last_became_active":"2024-04-25T01:52:44.102158+0000","last_became_peered":"2024-04-25T01:52:44.102158+0000","last_unstale":"2024-04-25T01:55:15.717941+0000","last_undegraded":"2024-04-25T01:55:15.717941+0000","last_fullsized":"2024-04-25T01:55:15.717941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:20.931392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918311+0000","last_change":"2024-04-25T01:52:44.107662+0000","last_active":"2024-04-25T01:55:15.918311+0000","last_peered":"2024-04-25T01:55:15.918311+0000","last_clean":"2024-04-25T01:55:15.918311+0000","last_became_active":"2024-04-25T01:52:44.107304+0000","last_became_peered":"2024-04-25T01:52:44.107304+0000","last_unstale":"2024-04-25T01:55:15.918311+0000","last_undegraded":"2024-04-25T01:55:15.918311+0000","last_fullsized":"2024-04-25T01:55:15.918311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:42:54.636950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583336+0000","last_change":"2024-04-25T01:52:44.100273+0000","last_active":"2024-04-25T01:55:15.583336+0000","last_peered":"2024-04-25T01:55:15.583336+0000","last_clean":"2024-04-25T01:55:15.583336+0000","last_became_active":"2024-04-25T01:52:44.098223+0000","last_became_peered":"2024-04-25T01:52:44.098223+0000","last_unstale":"2024-04-25T01:55:15.583336+0000","last_undegraded":"2024-04-25T01:55:15.583336+0000","last_fullsized":"2024-04-25T01:55:15.583336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:05.003265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576770+0000","last_change":"2024-04-25T01:52:44.089658+0000","last_active":"2024-04-25T01:55:15.576770+0000","last_peered":"2024-04-25T01:55:15.576770+0000","last_clean":"2024-04-25T01:55:15.576770+0000","last_became_active":"2024-04-25T01:52:44.089232+0000","last_became_peered":"2024-04-25T01:52:44.089232+0000","last_unstale":"2024-04-25T01:55:15.576770+0000","last_undegraded":"2024-04-25T01:55:15.576770+0000","last_fullsized":"2024-04-25T01:55:15.576770+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:54.385782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.718285+0000","last_change":"2024-04-25T01:52:44.086660+0000","last_active":"2024-04-25T01:55:15.718285+0000","last_peered":"2024-04-25T01:55:15.718285+0000","last_clean":"2024-04-25T01:55:15.718285+0000","last_became_active":"2024-04-25T01:52:44.086306+0000","last_became_peered":"2024-04-25T01:52:44.086306+0000","last_unstale":"2024-04-25T01:55:15.718285+0000","last_undegraded":"2024-04-25T01:55:15.718285+0000","last_fullsized":"2024-04-25T01:55:15.718285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:33:20.726616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.919001+0000","last_change":"2024-04-25T01:52:44.090239+0000","last_active":"2024-04-25T01:55:15.919001+0000","last_peered":"2024-04-25T01:55:15.919001+0000","last_clean":"2024-04-25T01:55:15.919001+0000","last_became_active":"2024-04-25T01:52:44.089736+0000","last_became_peered":"2024-04-25T01:52:44.089736+0000","last_unstale":"2024-04-25T01:55:15.919001+0000","last_undegraded":"2024-04-25T01:55:15.919001+0000","last_fullsized":"2024-04-25T01:55:15.919001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:22.954011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904432+0000","last_change":"2024-04-25T01:52:44.088878+0000","last_active":"2024-04-25T01:55:15.904432+0000","last_peered":"2024-04-25T01:55:15.904432+0000","last_clean":"2024-04-25T01:55:15.904432+0000","last_became_active":"2024-04-25T01:52:44.088250+0000","last_became_peered":"2024-04-25T01:52:44.088250+0000","last_unstale":"2024-04-25T01:55:15.904432+0000","last_undegraded":"2024-04-25T01:55:15.904432+0000","last_fullsized":"2024-04-25T01:55:15.904432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:15.490131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575344+0000","last_change":"2024-04-25T01:52:44.094675+0000","last_active":"2024-04-25T01:55:15.575344+0000","last_peered":"2024-04-25T01:55:15.575344+0000","last_clean":"2024-04-25T01:55:15.575344+0000","last_became_active":"2024-04-25T01:52:44.094219+0000","last_became_peered":"2024-04-25T01:52:44.094219+0000","last_unstale":"2024-04-25T01:55:15.575344+0000","last_undegraded":"2024-04-25T01:55:15.575344+0000","last_fullsized":"2024-04-25T01:55:15.575344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:26.160156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905482+0000","last_change":"2024-04-25T01:52:44.111131+0000","last_active":"2024-04-25T01:55:15.905482+0000","last_peered":"2024-04-25T01:55:15.905482+0000","last_clean":"2024-04-25T01:55:15.905482+0000","last_became_active":"2024-04-25T01:52:44.105670+0000","last_became_peered":"2024-04-25T01:52:44.105670+0000","last_unstale":"2024-04-25T01:55:15.905482+0000","last_undegraded":"2024-04-25T01:55:15.905482+0000","last_fullsized":"2024-04-25T01:55:15.905482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:00.377724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905232+0000","last_change":"2024-04-25T01:52:44.109707+0000","last_active":"2024-04-25T01:55:15.905232+0000","last_peered":"2024-04-25T01:55:15.905232+0000","last_clean":"2024-04-25T01:55:15.905232+0000","last_became_active":"2024-04-25T01:52:44.108550+0000","last_became_peered":"2024-04-25T01:52:44.108550+0000","last_unstale":"2024-04-25T01:55:15.905232+0000","last_undegraded":"2024-04-25T01:55:15.905232+0000","last_fullsized":"2024-04-25T01:55:15.905232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:46.296789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576000+0000","last_change":"2024-04-25T01:52:44.104504+0000","last_active":"2024-04-25T01:55:15.576000+0000","last_peered":"2024-04-25T01:55:15.576000+0000","last_clean":"2024-04-25T01:55:15.576000+0000","last_became_active":"2024-04-25T01:52:44.104180+0000","last_became_peered":"2024-04-25T01:52:44.104180+0000","last_unstale":"2024-04-25T01:55:15.576000+0000","last_undegraded":"2024-04-25T01:55:15.576000+0000","last_fullsized":"2024-04-25T01:55:15.576000+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:17.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905168+0000","last_change":"2024-04-25T01:52:44.107550+0000","last_active":"2024-04-25T01:55:15.905168+0000","last_peered":"2024-04-25T01:55:15.905168+0000","last_clean":"2024-04-25T01:55:15.905168+0000","last_became_active":"2024-04-25T01:52:44.107195+0000","last_became_peered":"2024-04-25T01:52:44.107195+0000","last_unstale":"2024-04-25T01:55:15.905168+0000","last_undegraded":"2024-04-25T01:55:15.905168+0000","last_fullsized":"2024-04-25T01:55:15.905168+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:59:42.220575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.585058+0000","last_change":"2024-04-25T01:52:44.101165+0000","last_active":"2024-04-25T01:55:15.585058+0000","last_peered":"2024-04-25T01:55:15.585058+0000","last_clean":"2024-04-25T01:55:15.585058+0000","last_became_active":"2024-04-25T01:52:44.100386+0000","last_became_peered":"2024-04-25T01:52:44.100386+0000","last_unstale":"2024-04-25T01:55:15.585058+0000","last_undegraded":"2024-04-25T01:55:15.585058+0000","last_fullsized":"2024-04-25T01:55:15.585058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:30.922707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904739+0000","last_change":"2024-04-25T01:52:44.116694+0000","last_active":"2024-04-25T01:55:15.904739+0000","last_peered":"2024-04-25T01:55:15.904739+0000","last_clean":"2024-04-25T01:55:15.904739+0000","last_became_active":"2024-04-25T01:52:44.116326+0000","last_became_peered":"2024-04-25T01:52:44.116326+0000","last_unstale":"2024-04-25T01:55:15.904739+0000","last_undegraded":"2024-04-25T01:55:15.904739+0000","last_fullsized":"2024-04-25T01:55:15.904739+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:16.329899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584165+0000","last_change":"2024-04-25T01:52:43.072358+0000","last_active":"2024-04-25T01:55:15.584165+0000","last_peered":"2024-04-25T01:55:15.584165+0000","last_clean":"2024-04-25T01:55:15.584165+0000","last_became_active":"2024-04-25T01:52:43.071815+0000","last_became_peered":"2024-04-25T01:52:43.071815+0000","last_unstale":"2024-04-25T01:55:15.584165+0000","last_undegraded":"2024-04-25T01:55:15.584165+0000","last_fullsized":"2024-04-25T01:55:15.584165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:15.295744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.585355+0000","last_change":"2024-04-25T01:52:44.094509+0000","last_active":"2024-04-25T01:55:15.585355+0000","last_peered":"2024-04-25T01:55:15.585355+0000","last_clean":"2024-04-25T01:55:15.585355+0000","last_became_active":"2024-04-25T01:52:44.093955+0000","last_became_peered":"2024-04-25T01:52:44.093955+0000","last_unstale":"2024-04-25T01:55:15.585355+0000","last_undegraded":"2024-04-25T01:55:15.585355+0000","last_fullsized":"2024-04-25T01:55:15.585355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:37.701835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.585603+0000","last_change":"2024-04-25T01:52:43.094435+0000","last_active":"2024-04-25T01:55:15.585603+0000","last_peered":"2024-04-25T01:55:15.585603+0000","last_clean":"2024-04-25T01:55:15.585603+0000","last_became_active":"2024-04-25T01:52:43.094018+0000","last_became_peered":"2024-04-25T01:52:43.094018+0000","last_unstale":"2024-04-25T01:55:15.585603+0000","last_undegraded":"2024-04-25T01:55:15.585603+0000","last_fullsized":"2024-04-25T01:55:15.585603+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:36.781014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906512+0000","last_change":"2024-04-25T01:52:44.104347+0000","last_active":"2024-04-25T01:55:15.906512+0000","last_peered":"2024-04-25T01:55:15.906512+0000","last_clean":"2024-04-25T01:55:15.906512+0000","last_became_active":"2024-04-25T01:52:44.103803+0000","last_became_peered":"2024-04-25T01:52:44.103803+0000","last_unstale":"2024-04-25T01:55:15.906512+0000","last_undegraded":"2024-04-25T01:55:15.906512+0000","last_fullsized":"2024-04-25T01:55:15.906512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:01.784893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702759+0000","last_change":"2024-04-25T01:52:43.070902+0000","last_active":"2024-04-25T01:55:15.702759+0000","last_peered":"2024-04-25T01:55:15.702759+0000","last_clean":"2024-04-25T01:55:15.702759+0000","last_became_active":"2024-04-25T01:52:43.070313+0000","last_became_peered":"2024-04-25T01:52:43.070313+0000","last_unstale":"2024-04-25T01:55:15.702759+0000","last_undegraded":"2024-04-25T01:55:15.702759+0000","last_fullsized":"2024-04-25T01:55:15.702759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:10.677320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576406+0000","last_change":"2024-04-25T01:52:44.093293+0000","last_active":"2024-04-25T01:55:15.576406+0000","last_peered":"2024-04-25T01:55:15.576406+0000","last_clean":"2024-04-25T01:55:15.576406+0000","last_became_active":"2024-04-25T01:52:44.092932+0000","last_became_peered":"2024-04-25T01:52:44.092932+0000","last_unstale":"2024-04-25T01:55:15.576406+0000","last_undegraded":"2024-04-25T01:55:15.576406+0000","last_fullsized":"2024-04-25T01:55:15.576406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:58:37.705594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702640+0000","last_change":"2024-04-25T01:52:43.069743+0000","last_active":"2024-04-25T01:55:15.702640+0000","last_peered":"2024-04-25T01:55:15.702640+0000","last_clean":"2024-04-25T01:55:15.702640+0000","last_became_active":"2024-04-25T01:52:43.068947+0000","last_became_peered":"2024-04-25T01:52:43.068947+0000","last_unstale":"2024-04-25T01:55:15.702640+0000","last_undegraded":"2024-04-25T01:55:15.702640+0000","last_fullsized":"2024-04-25T01:55:15.702640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:02:52.582176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584113+0000","last_change":"2024-04-25T01:53:43.865168+0000","last_active":"2024-04-25T01:55:15.584113+0000","last_peered":"2024-04-25T01:55:15.584113+0000","last_clean":"2024-04-25T01:55:15.584113+0000","last_became_active":"2024-04-25T01:53:43.864055+0000","last_became_peered":"2024-04-25T01:53:43.864055+0000","last_unstale":"2024-04-25T01:55:15.584113+0000","last_undegraded":"2024-04-25T01:55:15.584113+0000","last_fullsized":"2024-04-25T01:55:15.584113+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:01.707773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"90'9","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.337563+0000","last_change":"2024-04-25T01:52:43.068450+0000","last_active":"2024-04-25T01:55:16.337563+0000","last_peered":"2024-04-25T01:55:16.337563+0000","last_clean":"2024-04-25T01:55:16.337563+0000","last_became_active":"2024-04-25T01:52:43.067881+0000","last_became_peered":"2024-04-25T01:52:43.067881+0000","last_unstale":"2024-04-25T01:55:16.337563+0000","last_undegraded":"2024-04-25T01:55:16.337563+0000","last_fullsized":"2024-04-25T01:55:16.337563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:12.739321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1936,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":153,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702491+0000","last_change":"2024-04-25T01:52:44.101743+0000","last_active":"2024-04-25T01:55:15.702491+0000","last_peered":"2024-04-25T01:55:15.702491+0000","last_clean":"2024-04-25T01:55:15.702491+0000","last_became_active":"2024-04-25T01:52:44.101351+0000","last_became_peered":"2024-04-25T01:52:44.101351+0000","last_unstale":"2024-04-25T01:55:15.702491+0000","last_undegraded":"2024-04-25T01:55:15.702491+0000","last_fullsized":"2024-04-25T01:55:15.702491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:54:24.104819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.717869+0000","last_change":"2024-04-25T01:52:43.075880+0000","last_active":"2024-04-25T01:55:15.717869+0000","last_peered":"2024-04-25T01:55:15.717869+0000","last_clean":"2024-04-25T01:55:15.717869+0000","last_became_active":"2024-04-25T01:52:43.075522+0000","last_became_peered":"2024-04-25T01:52:43.075522+0000","last_unstale":"2024-04-25T01:55:15.717869+0000","last_undegraded":"2024-04-25T01:55:15.717869+0000","last_fullsized":"2024-04-25T01:55:15.717869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:26.218774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714269+0000","last_change":"2024-04-25T01:52:44.116772+0000","last_active":"2024-04-25T01:55:15.714269+0000","last_peered":"2024-04-25T01:55:15.714269+0000","last_clean":"2024-04-25T01:55:15.714269+0000","last_became_active":"2024-04-25T01:52:44.116408+0000","last_became_peered":"2024-04-25T01:52:44.116408+0000","last_unstale":"2024-04-25T01:55:15.714269+0000","last_undegraded":"2024-04-25T01:55:15.714269+0000","last_fullsized":"2024-04-25T01:55:15.714269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:21.988425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575458+0000","last_change":"2024-04-25T01:52:43.076964+0000","last_active":"2024-04-25T01:55:15.575458+0000","last_peered":"2024-04-25T01:55:15.575458+0000","last_clean":"2024-04-25T01:55:15.575458+0000","last_became_active":"2024-04-25T01:52:43.073915+0000","last_became_peered":"2024-04-25T01:52:43.073915+0000","last_unstale":"2024-04-25T01:55:15.575458+0000","last_undegraded":"2024-04-25T01:55:15.575458+0000","last_fullsized":"2024-04-25T01:55:15.575458+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:03.321920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583105+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:55:15.583105+0000","last_peered":"2024-04-25T01:55:15.583105+0000","last_clean":"2024-04-25T01:55:15.583105+0000","last_became_active":"2024-04-25T01:52:44.088198+0000","last_became_peered":"2024-04-25T01:52:44.088198+0000","last_unstale":"2024-04-25T01:55:15.583105+0000","last_undegraded":"2024-04-25T01:55:15.583105+0000","last_fullsized":"2024-04-25T01:55:15.583105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:30.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714027+0000","last_change":"2024-04-25T01:52:43.069853+0000","last_active":"2024-04-25T01:55:15.714027+0000","last_peered":"2024-04-25T01:55:15.714027+0000","last_clean":"2024-04-25T01:55:15.714027+0000","last_became_active":"2024-04-25T01:52:43.069526+0000","last_became_peered":"2024-04-25T01:52:43.069526+0000","last_unstale":"2024-04-25T01:55:15.714027+0000","last_undegraded":"2024-04-25T01:55:15.714027+0000","last_fullsized":"2024-04-25T01:55:15.714027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:35:03.272610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583451+0000","last_change":"2024-04-25T01:52:44.114296+0000","last_active":"2024-04-25T01:55:15.583451+0000","last_peered":"2024-04-25T01:55:15.583451+0000","last_clean":"2024-04-25T01:55:15.583451+0000","last_became_active":"2024-04-25T01:52:44.113958+0000","last_became_peered":"2024-04-25T01:52:44.113958+0000","last_unstale":"2024-04-25T01:55:15.583451+0000","last_undegraded":"2024-04-25T01:55:15.583451+0000","last_fullsized":"2024-04-25T01:55:15.583451+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:45.068080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.336627+0000","last_change":"2024-04-25T01:52:43.071388+0000","last_active":"2024-04-25T01:55:16.336627+0000","last_peered":"2024-04-25T01:55:16.336627+0000","last_clean":"2024-04-25T01:55:16.336627+0000","last_became_active":"2024-04-25T01:52:43.071005+0000","last_became_peered":"2024-04-25T01:52:43.071005+0000","last_unstale":"2024-04-25T01:55:16.336627+0000","last_undegraded":"2024-04-25T01:55:16.336627+0000","last_fullsized":"2024-04-25T01:55:16.336627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:56:59.580458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576398+0000","last_change":"2024-04-25T01:52:44.106106+0000","last_active":"2024-04-25T01:55:15.576398+0000","last_peered":"2024-04-25T01:55:15.576398+0000","last_clean":"2024-04-25T01:55:15.576398+0000","last_became_active":"2024-04-25T01:52:44.105760+0000","last_became_peered":"2024-04-25T01:52:44.105760+0000","last_unstale":"2024-04-25T01:55:15.576398+0000","last_undegraded":"2024-04-25T01:55:15.576398+0000","last_fullsized":"2024-04-25T01:55:15.576398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:24.626753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918950+0000","last_change":"2024-04-25T01:52:43.068926+0000","last_active":"2024-04-25T01:55:15.918950+0000","last_peered":"2024-04-25T01:55:15.918950+0000","last_clean":"2024-04-25T01:55:15.918950+0000","last_became_active":"2024-04-25T01:52:43.067356+0000","last_became_peered":"2024-04-25T01:52:43.067356+0000","last_unstale":"2024-04-25T01:55:15.918950+0000","last_undegraded":"2024-04-25T01:55:15.918950+0000","last_fullsized":"2024-04-25T01:55:15.918950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:19.134633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906572+0000","last_change":"2024-04-25T01:52:44.108299+0000","last_active":"2024-04-25T01:55:15.906572+0000","last_peered":"2024-04-25T01:55:15.906572+0000","last_clean":"2024-04-25T01:55:15.906572+0000","last_became_active":"2024-04-25T01:52:44.107573+0000","last_became_peered":"2024-04-25T01:52:44.107573+0000","last_unstale":"2024-04-25T01:55:15.906572+0000","last_undegraded":"2024-04-25T01:55:15.906572+0000","last_fullsized":"2024-04-25T01:55:15.906572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.804579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714366+0000","last_change":"2024-04-25T01:52:43.079263+0000","last_active":"2024-04-25T01:55:15.714366+0000","last_peered":"2024-04-25T01:55:15.714366+0000","last_clean":"2024-04-25T01:55:15.714366+0000","last_became_active":"2024-04-25T01:52:43.078459+0000","last_became_peered":"2024-04-25T01:52:43.078459+0000","last_unstale":"2024-04-25T01:55:15.714366+0000","last_undegraded":"2024-04-25T01:55:15.714366+0000","last_fullsized":"2024-04-25T01:55:15.714366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:59.349897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576541+0000","last_change":"2024-04-25T01:52:44.088044+0000","last_active":"2024-04-25T01:55:15.576541+0000","last_peered":"2024-04-25T01:55:15.576541+0000","last_clean":"2024-04-25T01:55:15.576541+0000","last_became_active":"2024-04-25T01:52:44.087682+0000","last_became_peered":"2024-04-25T01:52:44.087682+0000","last_unstale":"2024-04-25T01:55:15.576541+0000","last_undegraded":"2024-04-25T01:55:15.576541+0000","last_fullsized":"2024-04-25T01:55:15.576541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:08.933923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":380,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.009718+0000","last_change":"2024-04-25T01:53:43.869516+0000","last_active":"2024-04-25T01:55:16.009718+0000","last_peered":"2024-04-25T01:55:16.009718+0000","last_clean":"2024-04-25T01:55:16.009718+0000","last_became_active":"2024-04-25T01:53:43.861718+0000","last_became_peered":"2024-04-25T01:53:43.861718+0000","last_unstale":"2024-04-25T01:55:16.009718+0000","last_undegraded":"2024-04-25T01:55:16.009718+0000","last_fullsized":"2024-04-25T01:55:16.009718+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:13:01.820264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67760,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":297,"num_read_kb":215,"num_write":36,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702638+0000","last_change":"2024-04-25T01:52:44.091547+0000","last_active":"2024-04-25T01:55:15.702638+0000","last_peered":"2024-04-25T01:55:15.702638+0000","last_clean":"2024-04-25T01:55:15.702638+0000","last_became_active":"2024-04-25T01:52:44.091167+0000","last_became_peered":"2024-04-25T01:52:44.091167+0000","last_unstale":"2024-04-25T01:55:15.702638+0000","last_undegraded":"2024-04-25T01:55:15.702638+0000","last_fullsized":"2024-04-25T01:55:15.702638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:24.559694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904856+0000","last_change":"2024-04-25T01:52:43.088385+0000","last_active":"2024-04-25T01:55:15.904856+0000","last_peered":"2024-04-25T01:55:15.904856+0000","last_clean":"2024-04-25T01:55:15.904856+0000","last_became_active":"2024-04-25T01:52:43.087982+0000","last_became_peered":"2024-04-25T01:52:43.087982+0000","last_unstale":"2024-04-25T01:55:15.904856+0000","last_undegraded":"2024-04-25T01:55:15.904856+0000","last_fullsized":"2024-04-25T01:55:15.904856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:28.944427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575264+0000","last_change":"2024-04-25T01:52:44.099420+0000","last_active":"2024-04-25T01:55:15.575264+0000","last_peered":"2024-04-25T01:55:15.575264+0000","last_clean":"2024-04-25T01:55:15.575264+0000","last_became_active":"2024-04-25T01:52:44.099045+0000","last_became_peered":"2024-04-25T01:52:44.099045+0000","last_unstale":"2024-04-25T01:55:15.575264+0000","last_undegraded":"2024-04-25T01:55:15.575264+0000","last_fullsized":"2024-04-25T01:55:15.575264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:28.377091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.713969+0000","last_change":"2024-04-25T01:52:43.086252+0000","last_active":"2024-04-25T01:55:15.713969+0000","last_peered":"2024-04-25T01:55:15.713969+0000","last_clean":"2024-04-25T01:55:15.713969+0000","last_became_active":"2024-04-25T01:52:43.085892+0000","last_became_peered":"2024-04-25T01:52:43.085892+0000","last_unstale":"2024-04-25T01:55:15.713969+0000","last_undegraded":"2024-04-25T01:55:15.713969+0000","last_fullsized":"2024-04-25T01:55:15.713969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:25:26.395705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918824+0000","last_change":"2024-04-25T01:52:44.097163+0000","last_active":"2024-04-25T01:55:15.918824+0000","last_peered":"2024-04-25T01:55:15.918824+0000","last_clean":"2024-04-25T01:55:15.918824+0000","last_became_active":"2024-04-25T01:52:44.096758+0000","last_became_peered":"2024-04-25T01:52:44.096758+0000","last_unstale":"2024-04-25T01:55:15.918824+0000","last_undegraded":"2024-04-25T01:55:15.918824+0000","last_fullsized":"2024-04-25T01:55:15.918824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:22:47.394855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.359181+0000","last_change":"2024-04-25T01:52:43.069024+0000","last_active":"2024-04-25T01:55:16.359181+0000","last_peered":"2024-04-25T01:55:16.359181+0000","last_clean":"2024-04-25T01:55:16.359181+0000","last_became_active":"2024-04-25T01:52:43.068058+0000","last_became_peered":"2024-04-25T01:52:43.068058+0000","last_unstale":"2024-04-25T01:55:16.359181+0000","last_undegraded":"2024-04-25T01:55:16.359181+0000","last_fullsized":"2024-04-25T01:55:16.359181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:43.126131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918257+0000","last_change":"2024-04-25T01:52:44.091829+0000","last_active":"2024-04-25T01:55:15.918257+0000","last_peered":"2024-04-25T01:55:15.918257+0000","last_clean":"2024-04-25T01:55:15.918257+0000","last_became_active":"2024-04-25T01:52:44.091528+0000","last_became_peered":"2024-04-25T01:52:44.091528+0000","last_unstale":"2024-04-25T01:55:15.918257+0000","last_undegraded":"2024-04-25T01:55:15.918257+0000","last_fullsized":"2024-04-25T01:55:15.918257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:39.585296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904263+0000","last_change":"2024-04-25T01:53:43.888956+0000","last_active":"2024-04-25T01:55:15.904263+0000","last_peered":"2024-04-25T01:55:15.904263+0000","last_clean":"2024-04-25T01:55:15.904263+0000","last_became_active":"2024-04-25T01:53:43.870789+0000","last_became_peered":"2024-04-25T01:53:43.870789+0000","last_unstale":"2024-04-25T01:55:15.904263+0000","last_undegraded":"2024-04-25T01:55:15.904263+0000","last_fullsized":"2024-04-25T01:55:15.904263+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:08:48.790006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714034+0000","last_change":"2024-04-25T01:52:44.094696+0000","last_active":"2024-04-25T01:55:15.714034+0000","last_peered":"2024-04-25T01:55:15.714034+0000","last_clean":"2024-04-25T01:55:15.714034+0000","last_became_active":"2024-04-25T01:52:44.094371+0000","last_became_peered":"2024-04-25T01:52:44.094371+0000","last_unstale":"2024-04-25T01:55:15.714034+0000","last_undegraded":"2024-04-25T01:55:15.714034+0000","last_fullsized":"2024-04-25T01:55:15.714034+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:20.035674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702786+0000","last_change":"2024-04-25T01:52:43.071275+0000","last_active":"2024-04-25T01:55:15.702786+0000","last_peered":"2024-04-25T01:55:15.702786+0000","last_clean":"2024-04-25T01:55:15.702786+0000","last_became_active":"2024-04-25T01:52:43.070958+0000","last_became_peered":"2024-04-25T01:52:43.070958+0000","last_unstale":"2024-04-25T01:55:15.702786+0000","last_undegraded":"2024-04-25T01:55:15.702786+0000","last_fullsized":"2024-04-25T01:55:15.702786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:23:55.284943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904719+0000","last_change":"2024-04-25T01:52:44.092742+0000","last_active":"2024-04-25T01:55:15.904719+0000","last_peered":"2024-04-25T01:55:15.904719+0000","last_clean":"2024-04-25T01:55:15.904719+0000","last_became_active":"2024-04-25T01:52:44.092345+0000","last_became_peered":"2024-04-25T01:52:44.092345+0000","last_unstale":"2024-04-25T01:55:15.904719+0000","last_undegraded":"2024-04-25T01:55:15.904719+0000","last_fullsized":"2024-04-25T01:55:15.904719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:14.325814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584193+0000","last_change":"2024-04-25T01:52:43.090057+0000","last_active":"2024-04-25T01:55:15.584193+0000","last_peered":"2024-04-25T01:55:15.584193+0000","last_clean":"2024-04-25T01:55:15.584193+0000","last_became_active":"2024-04-25T01:52:43.089280+0000","last_became_peered":"2024-04-25T01:52:43.089280+0000","last_unstale":"2024-04-25T01:55:15.584193+0000","last_undegraded":"2024-04-25T01:55:15.584193+0000","last_fullsized":"2024-04-25T01:55:15.584193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:41.048908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.718275+0000","last_change":"2024-04-25T01:52:44.097651+0000","last_active":"2024-04-25T01:55:15.718275+0000","last_peered":"2024-04-25T01:55:15.718275+0000","last_clean":"2024-04-25T01:55:15.718275+0000","last_became_active":"2024-04-25T01:52:44.097254+0000","last_became_peered":"2024-04-25T01:52:44.097254+0000","last_unstale":"2024-04-25T01:55:15.718275+0000","last_undegraded":"2024-04-25T01:55:15.718275+0000","last_fullsized":"2024-04-25T01:55:15.718275+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:15.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918409+0000","last_change":"2024-04-25T01:52:43.076925+0000","last_active":"2024-04-25T01:55:15.918409+0000","last_peered":"2024-04-25T01:55:15.918409+0000","last_clean":"2024-04-25T01:55:15.918409+0000","last_became_active":"2024-04-25T01:52:43.076514+0000","last_became_peered":"2024-04-25T01:52:43.076514+0000","last_unstale":"2024-04-25T01:55:15.918409+0000","last_undegraded":"2024-04-25T01:55:15.918409+0000","last_fullsized":"2024-04-25T01:55:15.918409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:31:17.305379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.718045+0000","last_change":"2024-04-25T01:52:44.103731+0000","last_active":"2024-04-25T01:55:15.718045+0000","last_peered":"2024-04-25T01:55:15.718045+0000","last_clean":"2024-04-25T01:55:15.718045+0000","last_became_active":"2024-04-25T01:52:44.103173+0000","last_became_peered":"2024-04-25T01:52:44.103173+0000","last_unstale":"2024-04-25T01:55:15.718045+0000","last_undegraded":"2024-04-25T01:55:15.718045+0000","last_fullsized":"2024-04-25T01:55:15.718045+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:43.617579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714132+0000","last_change":"2024-04-25T01:52:44.090847+0000","last_active":"2024-04-25T01:55:15.714132+0000","last_peered":"2024-04-25T01:55:15.714132+0000","last_clean":"2024-04-25T01:55:15.714132+0000","last_became_active":"2024-04-25T01:52:44.090506+0000","last_became_peered":"2024-04-25T01:52:44.090506+0000","last_unstale":"2024-04-25T01:55:15.714132+0000","last_undegraded":"2024-04-25T01:55:15.714132+0000","last_fullsized":"2024-04-25T01:55:15.714132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:24.539726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.922129+0000","last_change":"2024-04-25T01:52:43.090469+0000","last_active":"2024-04-25T01:55:15.922129+0000","last_peered":"2024-04-25T01:55:15.922129+0000","last_clean":"2024-04-25T01:55:15.922129+0000","last_became_active":"2024-04-25T01:52:43.090159+0000","last_became_peered":"2024-04-25T01:52:43.090159+0000","last_unstale":"2024-04-25T01:55:15.922129+0000","last_undegraded":"2024-04-25T01:55:15.922129+0000","last_fullsized":"2024-04-25T01:55:15.922129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:42:45.532017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584403+0000","last_change":"2024-04-25T01:52:44.093527+0000","last_active":"2024-04-25T01:55:15.584403+0000","last_peered":"2024-04-25T01:55:15.584403+0000","last_clean":"2024-04-25T01:55:15.584403+0000","last_became_active":"2024-04-25T01:52:44.092729+0000","last_became_peered":"2024-04-25T01:52:44.092729+0000","last_unstale":"2024-04-25T01:55:15.584403+0000","last_undegraded":"2024-04-25T01:55:15.584403+0000","last_fullsized":"2024-04-25T01:55:15.584403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:32:50.668352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904349+0000","last_change":"2024-04-25T01:52:43.075323+0000","last_active":"2024-04-25T01:55:15.904349+0000","last_peered":"2024-04-25T01:55:15.904349+0000","last_clean":"2024-04-25T01:55:15.904349+0000","last_became_active":"2024-04-25T01:52:43.074876+0000","last_became_peered":"2024-04-25T01:52:43.074876+0000","last_unstale":"2024-04-25T01:55:15.904349+0000","last_undegraded":"2024-04-25T01:55:15.904349+0000","last_fullsized":"2024-04-25T01:55:15.904349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:47.472920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918263+0000","last_change":"2024-04-25T01:51:20.173285+0000","last_active":"2024-04-25T01:55:15.918263+0000","last_peered":"2024-04-25T01:55:15.918263+0000","last_clean":"2024-04-25T01:55:15.918263+0000","last_became_active":"2024-04-25T01:51:20.172811+0000","last_became_peered":"2024-04-25T01:51:20.172811+0000","last_unstale":"2024-04-25T01:55:15.918263+0000","last_undegraded":"2024-04-25T01:55:15.918263+0000","last_fullsized":"2024-04-25T01:55:15.918263+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_clean_scrub_stamp":"2024-04-25T01:47:31.712321+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:33.708082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.574950+0000","last_change":"2024-04-25T01:53:43.874632+0000","last_active":"2024-04-25T01:55:15.574950+0000","last_peered":"2024-04-25T01:55:15.574950+0000","last_clean":"2024-04-25T01:55:15.574950+0000","last_became_active":"2024-04-25T01:53:43.873732+0000","last_became_peered":"2024-04-25T01:53:43.873732+0000","last_unstale":"2024-04-25T01:55:15.574950+0000","last_undegraded":"2024-04-25T01:55:15.574950+0000","last_fullsized":"2024-04-25T01:55:15.574950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:27.874789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918402+0000","last_change":"2024-04-25T01:52:43.072565+0000","last_active":"2024-04-25T01:55:15.918402+0000","last_peered":"2024-04-25T01:55:15.918402+0000","last_clean":"2024-04-25T01:55:15.918402+0000","last_became_active":"2024-04-25T01:52:43.072193+0000","last_became_peered":"2024-04-25T01:52:43.072193+0000","last_unstale":"2024-04-25T01:55:15.918402+0000","last_undegraded":"2024-04-25T01:55:15.918402+0000","last_fullsized":"2024-04-25T01:55:15.918402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:07.261859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904537+0000","last_change":"2024-04-25T01:53:43.878499+0000","last_active":"2024-04-25T01:55:15.904537+0000","last_peered":"2024-04-25T01:55:15.904537+0000","last_clean":"2024-04-25T01:55:15.904537+0000","last_became_active":"2024-04-25T01:53:43.870876+0000","last_became_peered":"2024-04-25T01:53:43.870876+0000","last_unstale":"2024-04-25T01:55:15.904537+0000","last_undegraded":"2024-04-25T01:55:15.904537+0000","last_fullsized":"2024-04-25T01:55:15.904537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:08.809127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575785+0000","last_change":"2024-04-25T01:52:44.104892+0000","last_active":"2024-04-25T01:55:15.575785+0000","last_peered":"2024-04-25T01:55:15.575785+0000","last_clean":"2024-04-25T01:55:15.575785+0000","last_became_active":"2024-04-25T01:52:44.103802+0000","last_became_peered":"2024-04-25T01:52:44.103802+0000","last_unstale":"2024-04-25T01:55:15.575785+0000","last_undegraded":"2024-04-25T01:55:15.575785+0000","last_fullsized":"2024-04-25T01:55:15.575785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:16.445968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'44","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.862454+0000","last_change":"2024-04-25T01:52:43.087143+0000","last_active":"2024-04-25T01:55:15.862454+0000","last_peered":"2024-04-25T01:55:15.862454+0000","last_clean":"2024-04-25T01:55:15.862454+0000","last_became_active":"2024-04-25T01:52:43.085987+0000","last_became_peered":"2024-04-25T01:52:43.085987+0000","last_unstale":"2024-04-25T01:55:15.862454+0000","last_undegraded":"2024-04-25T01:55:15.862454+0000","last_fullsized":"2024-04-25T01:55:15.862454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:17.936671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584485+0000","last_change":"2024-04-25T01:52:44.092342+0000","last_active":"2024-04-25T01:55:15.584485+0000","last_peered":"2024-04-25T01:55:15.584485+0000","last_clean":"2024-04-25T01:55:15.584485+0000","last_became_active":"2024-04-25T01:52:44.091991+0000","last_became_peered":"2024-04-25T01:52:44.091991+0000","last_unstale":"2024-04-25T01:55:15.584485+0000","last_undegraded":"2024-04-25T01:55:15.584485+0000","last_fullsized":"2024-04-25T01:55:15.584485+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:22.147284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904395+0000","last_change":"2024-04-25T01:52:43.077972+0000","last_active":"2024-04-25T01:55:15.904395+0000","last_peered":"2024-04-25T01:55:15.904395+0000","last_clean":"2024-04-25T01:55:15.904395+0000","last_became_active":"2024-04-25T01:52:43.077655+0000","last_became_peered":"2024-04-25T01:52:43.077655+0000","last_unstale":"2024-04-25T01:55:15.904395+0000","last_undegraded":"2024-04-25T01:55:15.904395+0000","last_fullsized":"2024-04-25T01:55:15.904395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:24.820910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905887+0000","last_change":"2024-04-25T01:52:44.119560+0000","last_active":"2024-04-25T01:55:15.905887+0000","last_peered":"2024-04-25T01:55:15.905887+0000","last_clean":"2024-04-25T01:55:15.905887+0000","last_became_active":"2024-04-25T01:52:44.119106+0000","last_became_peered":"2024-04-25T01:52:44.119106+0000","last_unstale":"2024-04-25T01:55:15.905887+0000","last_undegraded":"2024-04-25T01:55:15.905887+0000","last_fullsized":"2024-04-25T01:55:15.905887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:50.702393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576106+0000","last_change":"2024-04-25T01:52:43.075783+0000","last_active":"2024-04-25T01:55:15.576106+0000","last_peered":"2024-04-25T01:55:15.576106+0000","last_clean":"2024-04-25T01:55:15.576106+0000","last_became_active":"2024-04-25T01:52:43.075462+0000","last_became_peered":"2024-04-25T01:52:43.075462+0000","last_unstale":"2024-04-25T01:55:15.576106+0000","last_undegraded":"2024-04-25T01:55:15.576106+0000","last_fullsized":"2024-04-25T01:55:15.576106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:07.077047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702282+0000","last_change":"2024-04-25T01:52:44.090695+0000","last_active":"2024-04-25T01:55:15.702282+0000","last_peered":"2024-04-25T01:55:15.702282+0000","last_clean":"2024-04-25T01:55:15.702282+0000","last_became_active":"2024-04-25T01:52:44.090203+0000","last_became_peered":"2024-04-25T01:52:44.090203+0000","last_unstale":"2024-04-25T01:55:15.702282+0000","last_undegraded":"2024-04-25T01:55:15.702282+0000","last_fullsized":"2024-04-25T01:55:15.702282+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:58.104569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583701+0000","last_change":"2024-04-25T01:52:43.086057+0000","last_active":"2024-04-25T01:55:15.583701+0000","last_peered":"2024-04-25T01:55:15.583701+0000","last_clean":"2024-04-25T01:55:15.583701+0000","last_became_active":"2024-04-25T01:52:43.085475+0000","last_became_peered":"2024-04-25T01:52:43.085475+0000","last_unstale":"2024-04-25T01:55:15.583701+0000","last_undegraded":"2024-04-25T01:55:15.583701+0000","last_fullsized":"2024-04-25T01:55:15.583701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:42.237414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904383+0000","last_change":"2024-04-25T01:52:44.106128+0000","last_active":"2024-04-25T01:55:15.904383+0000","last_peered":"2024-04-25T01:55:15.904383+0000","last_clean":"2024-04-25T01:55:15.904383+0000","last_became_active":"2024-04-25T01:52:44.105805+0000","last_became_peered":"2024-04-25T01:52:44.105805+0000","last_unstale":"2024-04-25T01:55:15.904383+0000","last_undegraded":"2024-04-25T01:55:15.904383+0000","last_fullsized":"2024-04-25T01:55:15.904383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:41.901070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904922+0000","last_change":"2024-04-25T01:52:43.088347+0000","last_active":"2024-04-25T01:55:15.904922+0000","last_peered":"2024-04-25T01:55:15.904922+0000","last_clean":"2024-04-25T01:55:15.904922+0000","last_became_active":"2024-04-25T01:52:43.087932+0000","last_became_peered":"2024-04-25T01:52:43.087932+0000","last_unstale":"2024-04-25T01:55:15.904922+0000","last_undegraded":"2024-04-25T01:55:15.904922+0000","last_fullsized":"2024-04-25T01:55:15.904922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:26.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584766+0000","last_change":"2024-04-25T01:52:44.108686+0000","last_active":"2024-04-25T01:55:15.584766+0000","last_peered":"2024-04-25T01:55:15.584766+0000","last_clean":"2024-04-25T01:55:15.584766+0000","last_became_active":"2024-04-25T01:52:44.107732+0000","last_became_peered":"2024-04-25T01:52:44.107732+0000","last_unstale":"2024-04-25T01:55:15.584766+0000","last_undegraded":"2024-04-25T01:55:15.584766+0000","last_fullsized":"2024-04-25T01:55:15.584766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:14.820979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702353+0000","last_change":"2024-04-25T01:52:43.069300+0000","last_active":"2024-04-25T01:55:15.702353+0000","last_peered":"2024-04-25T01:55:15.702353+0000","last_clean":"2024-04-25T01:55:15.702353+0000","last_became_active":"2024-04-25T01:52:43.068644+0000","last_became_peered":"2024-04-25T01:52:43.068644+0000","last_unstale":"2024-04-25T01:55:15.702353+0000","last_undegraded":"2024-04-25T01:55:15.702353+0000","last_fullsized":"2024-04-25T01:55:15.702353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:04.897638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918494+0000","last_change":"2024-04-25T01:52:44.100467+0000","last_active":"2024-04-25T01:55:15.918494+0000","last_peered":"2024-04-25T01:55:15.918494+0000","last_clean":"2024-04-25T01:55:15.918494+0000","last_became_active":"2024-04-25T01:52:44.099958+0000","last_became_peered":"2024-04-25T01:52:44.099958+0000","last_unstale":"2024-04-25T01:55:15.918494+0000","last_undegraded":"2024-04-25T01:55:15.918494+0000","last_fullsized":"2024-04-25T01:55:15.918494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:29.919785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904238+0000","last_change":"2024-04-25T01:53:43.875865+0000","last_active":"2024-04-25T01:55:15.904238+0000","last_peered":"2024-04-25T01:55:15.904238+0000","last_clean":"2024-04-25T01:55:15.904238+0000","last_became_active":"2024-04-25T01:53:43.871525+0000","last_became_peered":"2024-04-25T01:53:43.871525+0000","last_unstale":"2024-04-25T01:55:15.904238+0000","last_undegraded":"2024-04-25T01:55:15.904238+0000","last_fullsized":"2024-04-25T01:55:15.904238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:25.179902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918951+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:55:15.918951+0000","last_peered":"2024-04-25T01:55:15.918951+0000","last_clean":"2024-04-25T01:55:15.918951+0000","last_became_active":"2024-04-25T01:52:44.087911+0000","last_became_peered":"2024-04-25T01:52:44.087911+0000","last_unstale":"2024-04-25T01:55:15.918951+0000","last_undegraded":"2024-04-25T01:55:15.918951+0000","last_fullsized":"2024-04-25T01:55:15.918951+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:29:38.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.859264+0000","last_change":"2024-04-25T01:52:43.086083+0000","last_active":"2024-04-25T01:55:15.859264+0000","last_peered":"2024-04-25T01:55:15.859264+0000","last_clean":"2024-04-25T01:55:15.859264+0000","last_became_active":"2024-04-25T01:52:43.085613+0000","last_became_peered":"2024-04-25T01:52:43.085613+0000","last_unstale":"2024-04-25T01:55:15.859264+0000","last_undegraded":"2024-04-25T01:55:15.859264+0000","last_fullsized":"2024-04-25T01:55:15.859264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:23.851812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918421+0000","last_change":"2024-04-25T01:52:44.106435+0000","last_active":"2024-04-25T01:55:15.918421+0000","last_peered":"2024-04-25T01:55:15.918421+0000","last_clean":"2024-04-25T01:55:15.918421+0000","last_became_active":"2024-04-25T01:52:44.106074+0000","last_became_peered":"2024-04-25T01:52:44.106074+0000","last_unstale":"2024-04-25T01:55:15.918421+0000","last_undegraded":"2024-04-25T01:55:15.918421+0000","last_fullsized":"2024-04-25T01:55:15.918421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:59.286266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575223+0000","last_change":"2024-04-25T01:52:43.088902+0000","last_active":"2024-04-25T01:55:15.575223+0000","last_peered":"2024-04-25T01:55:15.575223+0000","last_clean":"2024-04-25T01:55:15.575223+0000","last_became_active":"2024-04-25T01:52:43.088289+0000","last_became_peered":"2024-04-25T01:52:43.088289+0000","last_unstale":"2024-04-25T01:55:15.575223+0000","last_undegraded":"2024-04-25T01:55:15.575223+0000","last_fullsized":"2024-04-25T01:55:15.575223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:37:44.562369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918975+0000","last_change":"2024-04-25T01:52:44.094185+0000","last_active":"2024-04-25T01:55:15.918975+0000","last_peered":"2024-04-25T01:55:15.918975+0000","last_clean":"2024-04-25T01:55:15.918975+0000","last_became_active":"2024-04-25T01:52:44.093753+0000","last_became_peered":"2024-04-25T01:52:44.093753+0000","last_unstale":"2024-04-25T01:55:15.918975+0000","last_undegraded":"2024-04-25T01:55:15.918975+0000","last_fullsized":"2024-04-25T01:55:15.918975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:06.089670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904437+0000","last_change":"2024-04-25T01:52:44.093895+0000","last_active":"2024-04-25T01:55:15.904437+0000","last_peered":"2024-04-25T01:55:15.904437+0000","last_clean":"2024-04-25T01:55:15.904437+0000","last_became_active":"2024-04-25T01:52:44.093490+0000","last_became_peered":"2024-04-25T01:52:44.093490+0000","last_unstale":"2024-04-25T01:55:15.904437+0000","last_undegraded":"2024-04-25T01:55:15.904437+0000","last_fullsized":"2024-04-25T01:55:15.904437+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:53.848979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904530+0000","last_change":"2024-04-25T01:52:44.092500+0000","last_active":"2024-04-25T01:55:15.904530+0000","last_peered":"2024-04-25T01:55:15.904530+0000","last_clean":"2024-04-25T01:55:15.904530+0000","last_became_active":"2024-04-25T01:52:44.092044+0000","last_became_peered":"2024-04-25T01:52:44.092044+0000","last_unstale":"2024-04-25T01:55:15.904530+0000","last_undegraded":"2024-04-25T01:55:15.904530+0000","last_fullsized":"2024-04-25T01:55:15.904530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:28.279676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904716+0000","last_change":"2024-04-25T01:52:44.116668+0000","last_active":"2024-04-25T01:55:15.904716+0000","last_peered":"2024-04-25T01:55:15.904716+0000","last_clean":"2024-04-25T01:55:15.904716+0000","last_became_active":"2024-04-25T01:52:44.116295+0000","last_became_peered":"2024-04-25T01:52:44.116295+0000","last_unstale":"2024-04-25T01:55:15.904716+0000","last_undegraded":"2024-04-25T01:55:15.904716+0000","last_fullsized":"2024-04-25T01:55:15.904716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:49.109403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905629+0000","last_change":"2024-04-25T01:52:44.109950+0000","last_active":"2024-04-25T01:55:15.905629+0000","last_peered":"2024-04-25T01:55:15.905629+0000","last_clean":"2024-04-25T01:55:15.905629+0000","last_became_active":"2024-04-25T01:52:44.109636+0000","last_became_peered":"2024-04-25T01:52:44.109636+0000","last_unstale":"2024-04-25T01:55:15.905629+0000","last_undegraded":"2024-04-25T01:55:15.905629+0000","last_fullsized":"2024-04-25T01:55:15.905629+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:02:57.610067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575687+0000","last_change":"2024-04-25T01:52:44.105237+0000","last_active":"2024-04-25T01:55:15.575687+0000","last_peered":"2024-04-25T01:55:15.575687+0000","last_clean":"2024-04-25T01:55:15.575687+0000","last_became_active":"2024-04-25T01:52:44.104531+0000","last_became_peered":"2024-04-25T01:52:44.104531+0000","last_unstale":"2024-04-25T01:55:15.575687+0000","last_undegraded":"2024-04-25T01:55:15.575687+0000","last_fullsized":"2024-04-25T01:55:15.575687+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.149119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904755+0000","last_change":"2024-04-25T01:52:44.095655+0000","last_active":"2024-04-25T01:55:15.904755+0000","last_peered":"2024-04-25T01:55:15.904755+0000","last_clean":"2024-04-25T01:55:15.904755+0000","last_became_active":"2024-04-25T01:52:44.095300+0000","last_became_peered":"2024-04-25T01:52:44.095300+0000","last_unstale":"2024-04-25T01:55:15.904755+0000","last_undegraded":"2024-04-25T01:55:15.904755+0000","last_fullsized":"2024-04-25T01:55:15.904755+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:51.240170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904995+0000","last_change":"2024-04-25T01:52:44.087382+0000","last_active":"2024-04-25T01:55:15.904995+0000","last_peered":"2024-04-25T01:55:15.904995+0000","last_clean":"2024-04-25T01:55:15.904995+0000","last_became_active":"2024-04-25T01:52:44.086815+0000","last_became_peered":"2024-04-25T01:52:44.086815+0000","last_unstale":"2024-04-25T01:55:15.904995+0000","last_undegraded":"2024-04-25T01:55:15.904995+0000","last_fullsized":"2024-04-25T01:55:15.904995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:52:13.682819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584351+0000","last_change":"2024-04-25T01:53:43.871148+0000","last_active":"2024-04-25T01:55:15.584351+0000","last_peered":"2024-04-25T01:55:15.584351+0000","last_clean":"2024-04-25T01:55:15.584351+0000","last_became_active":"2024-04-25T01:53:43.870490+0000","last_became_peered":"2024-04-25T01:53:43.870490+0000","last_unstale":"2024-04-25T01:55:15.584351+0000","last_undegraded":"2024-04-25T01:55:15.584351+0000","last_fullsized":"2024-04-25T01:55:15.584351+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:41.790442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.713960+0000","last_change":"2024-04-25T01:52:44.114767+0000","last_active":"2024-04-25T01:55:15.713960+0000","last_peered":"2024-04-25T01:55:15.713960+0000","last_clean":"2024-04-25T01:55:15.713960+0000","last_became_active":"2024-04-25T01:52:44.114483+0000","last_became_peered":"2024-04-25T01:52:44.114483+0000","last_unstale":"2024-04-25T01:55:15.713960+0000","last_undegraded":"2024-04-25T01:55:15.713960+0000","last_fullsized":"2024-04-25T01:55:15.713960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:14.439287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583527+0000","last_change":"2024-04-25T01:52:44.094424+0000","last_active":"2024-04-25T01:55:15.583527+0000","last_peered":"2024-04-25T01:55:15.583527+0000","last_clean":"2024-04-25T01:55:15.583527+0000","last_became_active":"2024-04-25T01:52:44.093074+0000","last_became_peered":"2024-04-25T01:52:44.093074+0000","last_unstale":"2024-04-25T01:55:15.583527+0000","last_undegraded":"2024-04-25T01:55:15.583527+0000","last_fullsized":"2024-04-25T01:55:15.583527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:20.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576299+0000","last_change":"2024-04-25T01:52:44.113465+0000","last_active":"2024-04-25T01:55:15.576299+0000","last_peered":"2024-04-25T01:55:15.576299+0000","last_clean":"2024-04-25T01:55:15.576299+0000","last_became_active":"2024-04-25T01:52:44.113168+0000","last_became_peered":"2024-04-25T01:52:44.113168+0000","last_unstale":"2024-04-25T01:55:15.576299+0000","last_undegraded":"2024-04-25T01:55:15.576299+0000","last_fullsized":"2024-04-25T01:55:15.576299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:42:47.758996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575462+0000","last_change":"2024-04-25T01:52:44.104763+0000","last_active":"2024-04-25T01:55:15.575462+0000","last_peered":"2024-04-25T01:55:15.575462+0000","last_clean":"2024-04-25T01:55:15.575462+0000","last_became_active":"2024-04-25T01:52:44.104377+0000","last_became_peered":"2024-04-25T01:52:44.104377+0000","last_unstale":"2024-04-25T01:55:15.575462+0000","last_undegraded":"2024-04-25T01:55:15.575462+0000","last_fullsized":"2024-04-25T01:55:15.575462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:13.647020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904326+0000","last_change":"2024-04-25T01:53:43.864792+0000","last_active":"2024-04-25T01:55:15.904326+0000","last_peered":"2024-04-25T01:55:15.904326+0000","last_clean":"2024-04-25T01:55:15.904326+0000","last_became_active":"2024-04-25T01:53:43.864432+0000","last_became_peered":"2024-04-25T01:53:43.864432+0000","last_unstale":"2024-04-25T01:55:15.904326+0000","last_undegraded":"2024-04-25T01:55:15.904326+0000","last_fullsized":"2024-04-25T01:55:15.904326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:28.149564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598282,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1000,"num_read_kb":609,"num_write":287,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":61762,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318317,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":471947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579953,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41328,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703824,"statfs":{"total":95995035648,"available":95952715776,"internally_reserved":0,"allocated":1814528,"data_stored":1263436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743480,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1384448,"data_stored":865265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005117,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32328,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712824,"statfs":{"total":95995035648,"available":95961931776,"internally_reserved":0,"allocated":1175552,"data_stored":668443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266755,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1585152,"data_stored":1065604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430279,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":471438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":532836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":463782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":991232,"data_stored":473313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117080,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":995328,"data_stored":471749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855443,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":540999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691917,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1196106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":69768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:55:20.567 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-25T01:55:20.568 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T01:55:20.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.790+0000 7f54f2a00640 1 -- 172.21.15.102:0/2663903575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 msgr2=0x7f54ec101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.790+0000 7f54f2a00640 1 --2- 172.21.15.102:0/2663903575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 0x7f54ec101a00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f54e0009900 tx=0x7f54e002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.790+0000 7f54f2a00640 1 -- 172.21.15.102:0/2663903575 shutdown_connections 2024-04-25T01:55:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.790+0000 7f54f2a00640 1 --2- 172.21.15.102:0/2663903575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54ec103190 0x7f54ec109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.790+0000 7f54f2a00640 1 --2- 172.21.15.102:0/2663903575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54ec1027d0 0x7f54ec102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.790+0000 7f54f2a00640 1 --2- 172.21.15.102:0/2663903575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 0x7f54ec101a00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.790+0000 7f54f2a00640 1 -- 172.21.15.102:0/2663903575 >> 172.21.15.102:0/2663903575 conn(0x7f54ec0fcb50 msgr2=0x7f54ec0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.790+0000 7f54f2a00640 1 -- 172.21.15.102:0/2663903575 shutdown_connections 2024-04-25T01:55:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.790+0000 7f54f2a00640 1 -- 172.21.15.102:0/2663903575 wait complete. 2024-04-25T01:55:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.791+0000 7f54f2a00640 1 Processor -- start 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.791+0000 7f54f2a00640 1 -- start start 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.791+0000 7f54f2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 0x7f54ec197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54ec1027d0 0x7f54ec197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54ec103190 0x7f54ec19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f2a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f54ec10d250 con 0x7f54ec1015e0 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f2a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f54ec10d0d0 con 0x7f54ec103190 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f2a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f54ec10d3d0 con 0x7f54ec1027d0 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 0x7f54ec197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 0x7f54ec197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33964/0 (socket says 172.21.15.102:33964) 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f1600640 1 -- 172.21.15.102:0/4283565688 learned_addr learned my addr 172.21.15.102:0/4283565688 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f2000640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54ec103190 0x7f54ec19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f1600640 1 -- 172.21.15.102:0/4283565688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54ec1027d0 msgr2=0x7f54ec197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f1600640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54ec1027d0 0x7f54ec197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f1600640 1 -- 172.21.15.102:0/4283565688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54ec103190 msgr2=0x7f54ec19eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:20.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f1600640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54ec103190 0x7f54ec19eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:20.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.792+0000 7f54f1600640 1 -- 172.21.15.102:0/4283565688 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54ec19f540 con 0x7f54ec1015e0 2024-04-25T01:55:20.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.793+0000 7f54f1600640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 0x7f54ec197920 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f54e0031610 tx=0x7f54e0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:20.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.793+0000 7f54ea000640 1 -- 172.21.15.102:0/4283565688 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e003f030 con 0x7f54ec1015e0 2024-04-25T01:55:20.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.793+0000 7f54ea000640 1 -- 172.21.15.102:0/4283565688 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54e0003ad0 con 0x7f54ec1015e0 2024-04-25T01:55:20.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.793+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54ec19f7d0 con 0x7f54ec1015e0 2024-04-25T01:55:20.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.793+0000 7f54ea000640 1 -- 172.21.15.102:0/4283565688 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e0035070 con 0x7f54ec1015e0 2024-04-25T01:55:20.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.793+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54ec19fd10 con 0x7f54ec1015e0 2024-04-25T01:55:20.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.794+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54cc0052c0 con 0x7f54ec1015e0 2024-04-25T01:55:20.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.797+0000 7f54ea000640 1 -- 172.21.15.102:0/4283565688 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f54e0042030 con 0x7f54ec1015e0 2024-04-25T01:55:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.798+0000 7f54ea000640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f54d40a2e10 0x7f54d40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.798+0000 7f54ea000640 1 -- 172.21.15.102:0/4283565688 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f54e00ef960 con 0x7f54ec1015e0 2024-04-25T01:55:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.798+0000 7f54f0c00640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f54d40a2e10 0x7f54d40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:20.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.799+0000 7f54f0c00640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f54d40a2e10 0x7f54d40a52d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f54d8005d00 tx=0x7f54d8005c50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:20.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.804+0000 7f54ea000640 1 -- 172.21.15.102:0/4283565688 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f54e00bc3e0 con 0x7f54ec1015e0 2024-04-25T01:55:20.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:20.995+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 --> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f54cc000bf0 con 0x7f54d40a2e10 2024-04-25T01:55:21.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.004+0000 7f54ea000640 1 -- 172.21.15.102:0/4283565688 <== mgr.25214 v2:172.21.15.102:6800/855543702 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257771 (secure 0 0 0) 0x7f54cc000bf0 con 0x7f54d40a2e10 2024-04-25T01:55:21.007 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:21.011 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-04-25T01:55:21.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f54d40a2e10 msgr2=0x7f54d40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f54d40a2e10 0x7f54d40a52d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f54d8005d00 tx=0x7f54d8005c50 comp rx=0 tx=0).stop 2024-04-25T01:55:21.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 msgr2=0x7f54ec197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 0x7f54ec197920 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f54e0031610 tx=0x7f54e0004270 comp rx=0 tx=0).stop 2024-04-25T01:55:21.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 shutdown_connections 2024-04-25T01:55:21.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f54d40a2e10 0x7f54d40a52d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54ec103190 0x7f54ec19eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54ec1027d0 0x7f54ec197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 --2- 172.21.15.102:0/4283565688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54ec1015e0 0x7f54ec197920 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 >> 172.21.15.102:0/4283565688 conn(0x7f54ec0fcb50 msgr2=0x7f54ec105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:21.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 shutdown_connections 2024-04-25T01:55:21.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.010+0000 7f54f2a00640 1 -- 172.21.15.102:0/4283565688 wait complete. 2024-04-25T01:55:21.029 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T01:55:19.596440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188884,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1136,"num_read_kb":728,"num_write":470,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1242498,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":384436,"kb_used_data":14388,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557388,"statfs":{"total":1151940427776,"available":1151546765312,"internally_reserved":0,"allocated":14733312,"data_stored":8484918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":378713403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.984677"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904984+0000","last_change":"2024-04-25T01:52:44.102143+0000","last_active":"2024-04-25T01:55:15.904984+0000","last_peered":"2024-04-25T01:55:15.904984+0000","last_clean":"2024-04-25T01:55:15.904984+0000","last_became_active":"2024-04-25T01:52:44.101810+0000","last_became_peered":"2024-04-25T01:52:44.101810+0000","last_unstale":"2024-04-25T01:55:15.904984+0000","last_undegraded":"2024-04-25T01:55:15.904984+0000","last_fullsized":"2024-04-25T01:55:15.904984+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:02.356294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906327+0000","last_change":"2024-04-25T01:52:44.103074+0000","last_active":"2024-04-25T01:55:15.906327+0000","last_peered":"2024-04-25T01:55:15.906327+0000","last_clean":"2024-04-25T01:55:15.906327+0000","last_became_active":"2024-04-25T01:52:44.102103+0000","last_became_peered":"2024-04-25T01:52:44.102103+0000","last_unstale":"2024-04-25T01:55:15.906327+0000","last_undegraded":"2024-04-25T01:55:15.906327+0000","last_fullsized":"2024-04-25T01:55:15.906327+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:36:43.016756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906006+0000","last_change":"2024-04-25T01:52:44.101467+0000","last_active":"2024-04-25T01:55:15.906006+0000","last_peered":"2024-04-25T01:55:15.906006+0000","last_clean":"2024-04-25T01:55:15.906006+0000","last_became_active":"2024-04-25T01:52:44.100911+0000","last_became_peered":"2024-04-25T01:52:44.100911+0000","last_unstale":"2024-04-25T01:55:15.906006+0000","last_undegraded":"2024-04-25T01:55:15.906006+0000","last_fullsized":"2024-04-25T01:55:15.906006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:21:58.533213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904287+0000","last_change":"2024-04-25T01:52:44.088592+0000","last_active":"2024-04-25T01:55:15.904287+0000","last_peered":"2024-04-25T01:55:15.904287+0000","last_clean":"2024-04-25T01:55:15.904287+0000","last_became_active":"2024-04-25T01:52:44.087651+0000","last_became_peered":"2024-04-25T01:52:44.087651+0000","last_unstale":"2024-04-25T01:55:15.904287+0000","last_undegraded":"2024-04-25T01:55:15.904287+0000","last_fullsized":"2024-04-25T01:55:15.904287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:01:52.240818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702409+0000","last_change":"2024-04-25T01:52:44.095033+0000","last_active":"2024-04-25T01:55:15.702409+0000","last_peered":"2024-04-25T01:55:15.702409+0000","last_clean":"2024-04-25T01:55:15.702409+0000","last_became_active":"2024-04-25T01:52:44.094719+0000","last_became_peered":"2024-04-25T01:52:44.094719+0000","last_unstale":"2024-04-25T01:55:15.702409+0000","last_undegraded":"2024-04-25T01:55:15.702409+0000","last_fullsized":"2024-04-25T01:55:15.702409+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:52.422544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.717941+0000","last_change":"2024-04-25T01:52:44.102574+0000","last_active":"2024-04-25T01:55:15.717941+0000","last_peered":"2024-04-25T01:55:15.717941+0000","last_clean":"2024-04-25T01:55:15.717941+0000","last_became_active":"2024-04-25T01:52:44.102158+0000","last_became_peered":"2024-04-25T01:52:44.102158+0000","last_unstale":"2024-04-25T01:55:15.717941+0000","last_undegraded":"2024-04-25T01:55:15.717941+0000","last_fullsized":"2024-04-25T01:55:15.717941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:34:20.931392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918311+0000","last_change":"2024-04-25T01:52:44.107662+0000","last_active":"2024-04-25T01:55:15.918311+0000","last_peered":"2024-04-25T01:55:15.918311+0000","last_clean":"2024-04-25T01:55:15.918311+0000","last_became_active":"2024-04-25T01:52:44.107304+0000","last_became_peered":"2024-04-25T01:52:44.107304+0000","last_unstale":"2024-04-25T01:55:15.918311+0000","last_undegraded":"2024-04-25T01:55:15.918311+0000","last_fullsized":"2024-04-25T01:55:15.918311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:42:54.636950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583336+0000","last_change":"2024-04-25T01:52:44.100273+0000","last_active":"2024-04-25T01:55:15.583336+0000","last_peered":"2024-04-25T01:55:15.583336+0000","last_clean":"2024-04-25T01:55:15.583336+0000","last_became_active":"2024-04-25T01:52:44.098223+0000","last_became_peered":"2024-04-25T01:52:44.098223+0000","last_unstale":"2024-04-25T01:55:15.583336+0000","last_undegraded":"2024-04-25T01:55:15.583336+0000","last_fullsized":"2024-04-25T01:55:15.583336+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:05.003265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576770+0000","last_change":"2024-04-25T01:52:44.089658+0000","last_active":"2024-04-25T01:55:15.576770+0000","last_peered":"2024-04-25T01:55:15.576770+0000","last_clean":"2024-04-25T01:55:15.576770+0000","last_became_active":"2024-04-25T01:52:44.089232+0000","last_became_peered":"2024-04-25T01:52:44.089232+0000","last_unstale":"2024-04-25T01:55:15.576770+0000","last_undegraded":"2024-04-25T01:55:15.576770+0000","last_fullsized":"2024-04-25T01:55:15.576770+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:54.385782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.718285+0000","last_change":"2024-04-25T01:52:44.086660+0000","last_active":"2024-04-25T01:55:15.718285+0000","last_peered":"2024-04-25T01:55:15.718285+0000","last_clean":"2024-04-25T01:55:15.718285+0000","last_became_active":"2024-04-25T01:52:44.086306+0000","last_became_peered":"2024-04-25T01:52:44.086306+0000","last_unstale":"2024-04-25T01:55:15.718285+0000","last_undegraded":"2024-04-25T01:55:15.718285+0000","last_fullsized":"2024-04-25T01:55:15.718285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:33:20.726616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.919001+0000","last_change":"2024-04-25T01:52:44.090239+0000","last_active":"2024-04-25T01:55:15.919001+0000","last_peered":"2024-04-25T01:55:15.919001+0000","last_clean":"2024-04-25T01:55:15.919001+0000","last_became_active":"2024-04-25T01:52:44.089736+0000","last_became_peered":"2024-04-25T01:52:44.089736+0000","last_unstale":"2024-04-25T01:55:15.919001+0000","last_undegraded":"2024-04-25T01:55:15.919001+0000","last_fullsized":"2024-04-25T01:55:15.919001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:31:22.954011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904432+0000","last_change":"2024-04-25T01:52:44.088878+0000","last_active":"2024-04-25T01:55:15.904432+0000","last_peered":"2024-04-25T01:55:15.904432+0000","last_clean":"2024-04-25T01:55:15.904432+0000","last_became_active":"2024-04-25T01:52:44.088250+0000","last_became_peered":"2024-04-25T01:52:44.088250+0000","last_unstale":"2024-04-25T01:55:15.904432+0000","last_undegraded":"2024-04-25T01:55:15.904432+0000","last_fullsized":"2024-04-25T01:55:15.904432+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:58:15.490131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575344+0000","last_change":"2024-04-25T01:52:44.094675+0000","last_active":"2024-04-25T01:55:15.575344+0000","last_peered":"2024-04-25T01:55:15.575344+0000","last_clean":"2024-04-25T01:55:15.575344+0000","last_became_active":"2024-04-25T01:52:44.094219+0000","last_became_peered":"2024-04-25T01:52:44.094219+0000","last_unstale":"2024-04-25T01:55:15.575344+0000","last_undegraded":"2024-04-25T01:55:15.575344+0000","last_fullsized":"2024-04-25T01:55:15.575344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:51:26.160156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905482+0000","last_change":"2024-04-25T01:52:44.111131+0000","last_active":"2024-04-25T01:55:15.905482+0000","last_peered":"2024-04-25T01:55:15.905482+0000","last_clean":"2024-04-25T01:55:15.905482+0000","last_became_active":"2024-04-25T01:52:44.105670+0000","last_became_peered":"2024-04-25T01:52:44.105670+0000","last_unstale":"2024-04-25T01:55:15.905482+0000","last_undegraded":"2024-04-25T01:55:15.905482+0000","last_fullsized":"2024-04-25T01:55:15.905482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:00.377724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905232+0000","last_change":"2024-04-25T01:52:44.109707+0000","last_active":"2024-04-25T01:55:15.905232+0000","last_peered":"2024-04-25T01:55:15.905232+0000","last_clean":"2024-04-25T01:55:15.905232+0000","last_became_active":"2024-04-25T01:52:44.108550+0000","last_became_peered":"2024-04-25T01:52:44.108550+0000","last_unstale":"2024-04-25T01:55:15.905232+0000","last_undegraded":"2024-04-25T01:55:15.905232+0000","last_fullsized":"2024-04-25T01:55:15.905232+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:10:46.296789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576000+0000","last_change":"2024-04-25T01:52:44.104504+0000","last_active":"2024-04-25T01:55:15.576000+0000","last_peered":"2024-04-25T01:55:15.576000+0000","last_clean":"2024-04-25T01:55:15.576000+0000","last_became_active":"2024-04-25T01:52:44.104180+0000","last_became_peered":"2024-04-25T01:52:44.104180+0000","last_unstale":"2024-04-25T01:55:15.576000+0000","last_undegraded":"2024-04-25T01:55:15.576000+0000","last_fullsized":"2024-04-25T01:55:15.576000+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:36:17.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905168+0000","last_change":"2024-04-25T01:52:44.107550+0000","last_active":"2024-04-25T01:55:15.905168+0000","last_peered":"2024-04-25T01:55:15.905168+0000","last_clean":"2024-04-25T01:55:15.905168+0000","last_became_active":"2024-04-25T01:52:44.107195+0000","last_became_peered":"2024-04-25T01:52:44.107195+0000","last_unstale":"2024-04-25T01:55:15.905168+0000","last_undegraded":"2024-04-25T01:55:15.905168+0000","last_fullsized":"2024-04-25T01:55:15.905168+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:59:42.220575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.585058+0000","last_change":"2024-04-25T01:52:44.101165+0000","last_active":"2024-04-25T01:55:15.585058+0000","last_peered":"2024-04-25T01:55:15.585058+0000","last_clean":"2024-04-25T01:55:15.585058+0000","last_became_active":"2024-04-25T01:52:44.100386+0000","last_became_peered":"2024-04-25T01:52:44.100386+0000","last_unstale":"2024-04-25T01:55:15.585058+0000","last_undegraded":"2024-04-25T01:55:15.585058+0000","last_fullsized":"2024-04-25T01:55:15.585058+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:56:30.922707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904739+0000","last_change":"2024-04-25T01:52:44.116694+0000","last_active":"2024-04-25T01:55:15.904739+0000","last_peered":"2024-04-25T01:55:15.904739+0000","last_clean":"2024-04-25T01:55:15.904739+0000","last_became_active":"2024-04-25T01:52:44.116326+0000","last_became_peered":"2024-04-25T01:52:44.116326+0000","last_unstale":"2024-04-25T01:55:15.904739+0000","last_undegraded":"2024-04-25T01:55:15.904739+0000","last_fullsized":"2024-04-25T01:55:15.904739+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:16.329899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584165+0000","last_change":"2024-04-25T01:52:43.072358+0000","last_active":"2024-04-25T01:55:15.584165+0000","last_peered":"2024-04-25T01:55:15.584165+0000","last_clean":"2024-04-25T01:55:15.584165+0000","last_became_active":"2024-04-25T01:52:43.071815+0000","last_became_peered":"2024-04-25T01:52:43.071815+0000","last_unstale":"2024-04-25T01:55:15.584165+0000","last_undegraded":"2024-04-25T01:55:15.584165+0000","last_fullsized":"2024-04-25T01:55:15.584165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:04:15.295744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.585355+0000","last_change":"2024-04-25T01:52:44.094509+0000","last_active":"2024-04-25T01:55:15.585355+0000","last_peered":"2024-04-25T01:55:15.585355+0000","last_clean":"2024-04-25T01:55:15.585355+0000","last_became_active":"2024-04-25T01:52:44.093955+0000","last_became_peered":"2024-04-25T01:52:44.093955+0000","last_unstale":"2024-04-25T01:55:15.585355+0000","last_undegraded":"2024-04-25T01:55:15.585355+0000","last_fullsized":"2024-04-25T01:55:15.585355+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:37.701835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.585603+0000","last_change":"2024-04-25T01:52:43.094435+0000","last_active":"2024-04-25T01:55:15.585603+0000","last_peered":"2024-04-25T01:55:15.585603+0000","last_clean":"2024-04-25T01:55:15.585603+0000","last_became_active":"2024-04-25T01:52:43.094018+0000","last_became_peered":"2024-04-25T01:52:43.094018+0000","last_unstale":"2024-04-25T01:55:15.585603+0000","last_undegraded":"2024-04-25T01:55:15.585603+0000","last_fullsized":"2024-04-25T01:55:15.585603+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:40:36.781014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906512+0000","last_change":"2024-04-25T01:52:44.104347+0000","last_active":"2024-04-25T01:55:15.906512+0000","last_peered":"2024-04-25T01:55:15.906512+0000","last_clean":"2024-04-25T01:55:15.906512+0000","last_became_active":"2024-04-25T01:52:44.103803+0000","last_became_peered":"2024-04-25T01:52:44.103803+0000","last_unstale":"2024-04-25T01:55:15.906512+0000","last_undegraded":"2024-04-25T01:55:15.906512+0000","last_fullsized":"2024-04-25T01:55:15.906512+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:13:01.784893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702759+0000","last_change":"2024-04-25T01:52:43.070902+0000","last_active":"2024-04-25T01:55:15.702759+0000","last_peered":"2024-04-25T01:55:15.702759+0000","last_clean":"2024-04-25T01:55:15.702759+0000","last_became_active":"2024-04-25T01:52:43.070313+0000","last_became_peered":"2024-04-25T01:52:43.070313+0000","last_unstale":"2024-04-25T01:55:15.702759+0000","last_undegraded":"2024-04-25T01:55:15.702759+0000","last_fullsized":"2024-04-25T01:55:15.702759+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:47:10.677320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576406+0000","last_change":"2024-04-25T01:52:44.093293+0000","last_active":"2024-04-25T01:55:15.576406+0000","last_peered":"2024-04-25T01:55:15.576406+0000","last_clean":"2024-04-25T01:55:15.576406+0000","last_became_active":"2024-04-25T01:52:44.092932+0000","last_became_peered":"2024-04-25T01:52:44.092932+0000","last_unstale":"2024-04-25T01:55:15.576406+0000","last_undegraded":"2024-04-25T01:55:15.576406+0000","last_fullsized":"2024-04-25T01:55:15.576406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:58:37.705594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702640+0000","last_change":"2024-04-25T01:52:43.069743+0000","last_active":"2024-04-25T01:55:15.702640+0000","last_peered":"2024-04-25T01:55:15.702640+0000","last_clean":"2024-04-25T01:55:15.702640+0000","last_became_active":"2024-04-25T01:52:43.068947+0000","last_became_peered":"2024-04-25T01:52:43.068947+0000","last_unstale":"2024-04-25T01:55:15.702640+0000","last_undegraded":"2024-04-25T01:55:15.702640+0000","last_fullsized":"2024-04-25T01:55:15.702640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:02:52.582176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584113+0000","last_change":"2024-04-25T01:53:43.865168+0000","last_active":"2024-04-25T01:55:15.584113+0000","last_peered":"2024-04-25T01:55:15.584113+0000","last_clean":"2024-04-25T01:55:15.584113+0000","last_became_active":"2024-04-25T01:53:43.864055+0000","last_became_peered":"2024-04-25T01:53:43.864055+0000","last_unstale":"2024-04-25T01:55:15.584113+0000","last_undegraded":"2024-04-25T01:55:15.584113+0000","last_fullsized":"2024-04-25T01:55:15.584113+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:12:01.707773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"90'9","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.337563+0000","last_change":"2024-04-25T01:52:43.068450+0000","last_active":"2024-04-25T01:55:16.337563+0000","last_peered":"2024-04-25T01:55:16.337563+0000","last_clean":"2024-04-25T01:55:16.337563+0000","last_became_active":"2024-04-25T01:52:43.067881+0000","last_became_peered":"2024-04-25T01:52:43.067881+0000","last_unstale":"2024-04-25T01:55:16.337563+0000","last_undegraded":"2024-04-25T01:55:16.337563+0000","last_fullsized":"2024-04-25T01:55:16.337563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:37:12.739321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1936,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":153,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702491+0000","last_change":"2024-04-25T01:52:44.101743+0000","last_active":"2024-04-25T01:55:15.702491+0000","last_peered":"2024-04-25T01:55:15.702491+0000","last_clean":"2024-04-25T01:55:15.702491+0000","last_became_active":"2024-04-25T01:52:44.101351+0000","last_became_peered":"2024-04-25T01:52:44.101351+0000","last_unstale":"2024-04-25T01:55:15.702491+0000","last_undegraded":"2024-04-25T01:55:15.702491+0000","last_fullsized":"2024-04-25T01:55:15.702491+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:54:24.104819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.717869+0000","last_change":"2024-04-25T01:52:43.075880+0000","last_active":"2024-04-25T01:55:15.717869+0000","last_peered":"2024-04-25T01:55:15.717869+0000","last_clean":"2024-04-25T01:55:15.717869+0000","last_became_active":"2024-04-25T01:52:43.075522+0000","last_became_peered":"2024-04-25T01:52:43.075522+0000","last_unstale":"2024-04-25T01:55:15.717869+0000","last_undegraded":"2024-04-25T01:55:15.717869+0000","last_fullsized":"2024-04-25T01:55:15.717869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:19:26.218774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714269+0000","last_change":"2024-04-25T01:52:44.116772+0000","last_active":"2024-04-25T01:55:15.714269+0000","last_peered":"2024-04-25T01:55:15.714269+0000","last_clean":"2024-04-25T01:55:15.714269+0000","last_became_active":"2024-04-25T01:52:44.116408+0000","last_became_peered":"2024-04-25T01:52:44.116408+0000","last_unstale":"2024-04-25T01:55:15.714269+0000","last_undegraded":"2024-04-25T01:55:15.714269+0000","last_fullsized":"2024-04-25T01:55:15.714269+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:30:21.988425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575458+0000","last_change":"2024-04-25T01:52:43.076964+0000","last_active":"2024-04-25T01:55:15.575458+0000","last_peered":"2024-04-25T01:55:15.575458+0000","last_clean":"2024-04-25T01:55:15.575458+0000","last_became_active":"2024-04-25T01:52:43.073915+0000","last_became_peered":"2024-04-25T01:52:43.073915+0000","last_unstale":"2024-04-25T01:55:15.575458+0000","last_undegraded":"2024-04-25T01:55:15.575458+0000","last_fullsized":"2024-04-25T01:55:15.575458+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:03.321920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583105+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:55:15.583105+0000","last_peered":"2024-04-25T01:55:15.583105+0000","last_clean":"2024-04-25T01:55:15.583105+0000","last_became_active":"2024-04-25T01:52:44.088198+0000","last_became_peered":"2024-04-25T01:52:44.088198+0000","last_unstale":"2024-04-25T01:55:15.583105+0000","last_undegraded":"2024-04-25T01:55:15.583105+0000","last_fullsized":"2024-04-25T01:55:15.583105+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:52:30.171857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714027+0000","last_change":"2024-04-25T01:52:43.069853+0000","last_active":"2024-04-25T01:55:15.714027+0000","last_peered":"2024-04-25T01:55:15.714027+0000","last_clean":"2024-04-25T01:55:15.714027+0000","last_became_active":"2024-04-25T01:52:43.069526+0000","last_became_peered":"2024-04-25T01:52:43.069526+0000","last_unstale":"2024-04-25T01:55:15.714027+0000","last_undegraded":"2024-04-25T01:55:15.714027+0000","last_fullsized":"2024-04-25T01:55:15.714027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:35:03.272610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583451+0000","last_change":"2024-04-25T01:52:44.114296+0000","last_active":"2024-04-25T01:55:15.583451+0000","last_peered":"2024-04-25T01:55:15.583451+0000","last_clean":"2024-04-25T01:55:15.583451+0000","last_became_active":"2024-04-25T01:52:44.113958+0000","last_became_peered":"2024-04-25T01:52:44.113958+0000","last_unstale":"2024-04-25T01:55:15.583451+0000","last_undegraded":"2024-04-25T01:55:15.583451+0000","last_fullsized":"2024-04-25T01:55:15.583451+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:45.068080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.336627+0000","last_change":"2024-04-25T01:52:43.071388+0000","last_active":"2024-04-25T01:55:16.336627+0000","last_peered":"2024-04-25T01:55:16.336627+0000","last_clean":"2024-04-25T01:55:16.336627+0000","last_became_active":"2024-04-25T01:52:43.071005+0000","last_became_peered":"2024-04-25T01:52:43.071005+0000","last_unstale":"2024-04-25T01:55:16.336627+0000","last_undegraded":"2024-04-25T01:55:16.336627+0000","last_fullsized":"2024-04-25T01:55:16.336627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:56:59.580458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576398+0000","last_change":"2024-04-25T01:52:44.106106+0000","last_active":"2024-04-25T01:55:15.576398+0000","last_peered":"2024-04-25T01:55:15.576398+0000","last_clean":"2024-04-25T01:55:15.576398+0000","last_became_active":"2024-04-25T01:52:44.105760+0000","last_became_peered":"2024-04-25T01:52:44.105760+0000","last_unstale":"2024-04-25T01:55:15.576398+0000","last_undegraded":"2024-04-25T01:55:15.576398+0000","last_fullsized":"2024-04-25T01:55:15.576398+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:19:24.626753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918950+0000","last_change":"2024-04-25T01:52:43.068926+0000","last_active":"2024-04-25T01:55:15.918950+0000","last_peered":"2024-04-25T01:55:15.918950+0000","last_clean":"2024-04-25T01:55:15.918950+0000","last_became_active":"2024-04-25T01:52:43.067356+0000","last_became_peered":"2024-04-25T01:52:43.067356+0000","last_unstale":"2024-04-25T01:55:15.918950+0000","last_undegraded":"2024-04-25T01:55:15.918950+0000","last_fullsized":"2024-04-25T01:55:15.918950+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:00:19.134633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.906572+0000","last_change":"2024-04-25T01:52:44.108299+0000","last_active":"2024-04-25T01:55:15.906572+0000","last_peered":"2024-04-25T01:55:15.906572+0000","last_clean":"2024-04-25T01:55:15.906572+0000","last_became_active":"2024-04-25T01:52:44.107573+0000","last_became_peered":"2024-04-25T01:52:44.107573+0000","last_unstale":"2024-04-25T01:55:15.906572+0000","last_undegraded":"2024-04-25T01:55:15.906572+0000","last_fullsized":"2024-04-25T01:55:15.906572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:46:05.804579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714366+0000","last_change":"2024-04-25T01:52:43.079263+0000","last_active":"2024-04-25T01:55:15.714366+0000","last_peered":"2024-04-25T01:55:15.714366+0000","last_clean":"2024-04-25T01:55:15.714366+0000","last_became_active":"2024-04-25T01:52:43.078459+0000","last_became_peered":"2024-04-25T01:52:43.078459+0000","last_unstale":"2024-04-25T01:55:15.714366+0000","last_undegraded":"2024-04-25T01:55:15.714366+0000","last_fullsized":"2024-04-25T01:55:15.714366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:36:59.349897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576541+0000","last_change":"2024-04-25T01:52:44.088044+0000","last_active":"2024-04-25T01:55:15.576541+0000","last_peered":"2024-04-25T01:55:15.576541+0000","last_clean":"2024-04-25T01:55:15.576541+0000","last_became_active":"2024-04-25T01:52:44.087682+0000","last_became_peered":"2024-04-25T01:52:44.087682+0000","last_unstale":"2024-04-25T01:55:15.576541+0000","last_undegraded":"2024-04-25T01:55:15.576541+0000","last_fullsized":"2024-04-25T01:55:15.576541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:12:08.933923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":380,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.009718+0000","last_change":"2024-04-25T01:53:43.869516+0000","last_active":"2024-04-25T01:55:16.009718+0000","last_peered":"2024-04-25T01:55:16.009718+0000","last_clean":"2024-04-25T01:55:16.009718+0000","last_became_active":"2024-04-25T01:53:43.861718+0000","last_became_peered":"2024-04-25T01:53:43.861718+0000","last_unstale":"2024-04-25T01:55:16.009718+0000","last_undegraded":"2024-04-25T01:55:16.009718+0000","last_fullsized":"2024-04-25T01:55:16.009718+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:13:01.820264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67760,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":297,"num_read_kb":215,"num_write":36,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":61046,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702638+0000","last_change":"2024-04-25T01:52:44.091547+0000","last_active":"2024-04-25T01:55:15.702638+0000","last_peered":"2024-04-25T01:55:15.702638+0000","last_clean":"2024-04-25T01:55:15.702638+0000","last_became_active":"2024-04-25T01:52:44.091167+0000","last_became_peered":"2024-04-25T01:52:44.091167+0000","last_unstale":"2024-04-25T01:55:15.702638+0000","last_undegraded":"2024-04-25T01:55:15.702638+0000","last_fullsized":"2024-04-25T01:55:15.702638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:24.559694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904856+0000","last_change":"2024-04-25T01:52:43.088385+0000","last_active":"2024-04-25T01:55:15.904856+0000","last_peered":"2024-04-25T01:55:15.904856+0000","last_clean":"2024-04-25T01:55:15.904856+0000","last_became_active":"2024-04-25T01:52:43.087982+0000","last_became_peered":"2024-04-25T01:52:43.087982+0000","last_unstale":"2024-04-25T01:55:15.904856+0000","last_undegraded":"2024-04-25T01:55:15.904856+0000","last_fullsized":"2024-04-25T01:55:15.904856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:28.944427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575264+0000","last_change":"2024-04-25T01:52:44.099420+0000","last_active":"2024-04-25T01:55:15.575264+0000","last_peered":"2024-04-25T01:55:15.575264+0000","last_clean":"2024-04-25T01:55:15.575264+0000","last_became_active":"2024-04-25T01:52:44.099045+0000","last_became_peered":"2024-04-25T01:52:44.099045+0000","last_unstale":"2024-04-25T01:55:15.575264+0000","last_undegraded":"2024-04-25T01:55:15.575264+0000","last_fullsized":"2024-04-25T01:55:15.575264+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:50:28.377091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.713969+0000","last_change":"2024-04-25T01:52:43.086252+0000","last_active":"2024-04-25T01:55:15.713969+0000","last_peered":"2024-04-25T01:55:15.713969+0000","last_clean":"2024-04-25T01:55:15.713969+0000","last_became_active":"2024-04-25T01:52:43.085892+0000","last_became_peered":"2024-04-25T01:52:43.085892+0000","last_unstale":"2024-04-25T01:55:15.713969+0000","last_undegraded":"2024-04-25T01:55:15.713969+0000","last_fullsized":"2024-04-25T01:55:15.713969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:25:26.395705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918824+0000","last_change":"2024-04-25T01:52:44.097163+0000","last_active":"2024-04-25T01:55:15.918824+0000","last_peered":"2024-04-25T01:55:15.918824+0000","last_clean":"2024-04-25T01:55:15.918824+0000","last_became_active":"2024-04-25T01:52:44.096758+0000","last_became_peered":"2024-04-25T01:52:44.096758+0000","last_unstale":"2024-04-25T01:55:15.918824+0000","last_undegraded":"2024-04-25T01:55:15.918824+0000","last_fullsized":"2024-04-25T01:55:15.918824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:22:47.394855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:16.359181+0000","last_change":"2024-04-25T01:52:43.069024+0000","last_active":"2024-04-25T01:55:16.359181+0000","last_peered":"2024-04-25T01:55:16.359181+0000","last_clean":"2024-04-25T01:55:16.359181+0000","last_became_active":"2024-04-25T01:52:43.068058+0000","last_became_peered":"2024-04-25T01:52:43.068058+0000","last_unstale":"2024-04-25T01:55:16.359181+0000","last_undegraded":"2024-04-25T01:55:16.359181+0000","last_fullsized":"2024-04-25T01:55:16.359181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:43.126131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1370,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918257+0000","last_change":"2024-04-25T01:52:44.091829+0000","last_active":"2024-04-25T01:55:15.918257+0000","last_peered":"2024-04-25T01:55:15.918257+0000","last_clean":"2024-04-25T01:55:15.918257+0000","last_became_active":"2024-04-25T01:52:44.091528+0000","last_became_peered":"2024-04-25T01:52:44.091528+0000","last_unstale":"2024-04-25T01:55:15.918257+0000","last_undegraded":"2024-04-25T01:55:15.918257+0000","last_fullsized":"2024-04-25T01:55:15.918257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:39.585296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904263+0000","last_change":"2024-04-25T01:53:43.888956+0000","last_active":"2024-04-25T01:55:15.904263+0000","last_peered":"2024-04-25T01:55:15.904263+0000","last_clean":"2024-04-25T01:55:15.904263+0000","last_became_active":"2024-04-25T01:53:43.870789+0000","last_became_peered":"2024-04-25T01:53:43.870789+0000","last_unstale":"2024-04-25T01:55:15.904263+0000","last_undegraded":"2024-04-25T01:55:15.904263+0000","last_fullsized":"2024-04-25T01:55:15.904263+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:08:48.790006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714034+0000","last_change":"2024-04-25T01:52:44.094696+0000","last_active":"2024-04-25T01:55:15.714034+0000","last_peered":"2024-04-25T01:55:15.714034+0000","last_clean":"2024-04-25T01:55:15.714034+0000","last_became_active":"2024-04-25T01:52:44.094371+0000","last_became_peered":"2024-04-25T01:52:44.094371+0000","last_unstale":"2024-04-25T01:55:15.714034+0000","last_undegraded":"2024-04-25T01:55:15.714034+0000","last_fullsized":"2024-04-25T01:55:15.714034+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:42:20.035674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702786+0000","last_change":"2024-04-25T01:52:43.071275+0000","last_active":"2024-04-25T01:55:15.702786+0000","last_peered":"2024-04-25T01:55:15.702786+0000","last_clean":"2024-04-25T01:55:15.702786+0000","last_became_active":"2024-04-25T01:52:43.070958+0000","last_became_peered":"2024-04-25T01:52:43.070958+0000","last_unstale":"2024-04-25T01:55:15.702786+0000","last_undegraded":"2024-04-25T01:55:15.702786+0000","last_fullsized":"2024-04-25T01:55:15.702786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:23:55.284943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904719+0000","last_change":"2024-04-25T01:52:44.092742+0000","last_active":"2024-04-25T01:55:15.904719+0000","last_peered":"2024-04-25T01:55:15.904719+0000","last_clean":"2024-04-25T01:55:15.904719+0000","last_became_active":"2024-04-25T01:52:44.092345+0000","last_became_peered":"2024-04-25T01:52:44.092345+0000","last_unstale":"2024-04-25T01:55:15.904719+0000","last_undegraded":"2024-04-25T01:55:15.904719+0000","last_fullsized":"2024-04-25T01:55:15.904719+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:14.325814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584193+0000","last_change":"2024-04-25T01:52:43.090057+0000","last_active":"2024-04-25T01:55:15.584193+0000","last_peered":"2024-04-25T01:55:15.584193+0000","last_clean":"2024-04-25T01:55:15.584193+0000","last_became_active":"2024-04-25T01:52:43.089280+0000","last_became_peered":"2024-04-25T01:52:43.089280+0000","last_unstale":"2024-04-25T01:55:15.584193+0000","last_undegraded":"2024-04-25T01:55:15.584193+0000","last_fullsized":"2024-04-25T01:55:15.584193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:41.048908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.718275+0000","last_change":"2024-04-25T01:52:44.097651+0000","last_active":"2024-04-25T01:55:15.718275+0000","last_peered":"2024-04-25T01:55:15.718275+0000","last_clean":"2024-04-25T01:55:15.718275+0000","last_became_active":"2024-04-25T01:52:44.097254+0000","last_became_peered":"2024-04-25T01:52:44.097254+0000","last_unstale":"2024-04-25T01:55:15.718275+0000","last_undegraded":"2024-04-25T01:55:15.718275+0000","last_fullsized":"2024-04-25T01:55:15.718275+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:15.169426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918409+0000","last_change":"2024-04-25T01:52:43.076925+0000","last_active":"2024-04-25T01:55:15.918409+0000","last_peered":"2024-04-25T01:55:15.918409+0000","last_clean":"2024-04-25T01:55:15.918409+0000","last_became_active":"2024-04-25T01:52:43.076514+0000","last_became_peered":"2024-04-25T01:52:43.076514+0000","last_unstale":"2024-04-25T01:55:15.918409+0000","last_undegraded":"2024-04-25T01:55:15.918409+0000","last_fullsized":"2024-04-25T01:55:15.918409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:31:17.305379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.718045+0000","last_change":"2024-04-25T01:52:44.103731+0000","last_active":"2024-04-25T01:55:15.718045+0000","last_peered":"2024-04-25T01:55:15.718045+0000","last_clean":"2024-04-25T01:55:15.718045+0000","last_became_active":"2024-04-25T01:52:44.103173+0000","last_became_peered":"2024-04-25T01:52:44.103173+0000","last_unstale":"2024-04-25T01:55:15.718045+0000","last_undegraded":"2024-04-25T01:55:15.718045+0000","last_fullsized":"2024-04-25T01:55:15.718045+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:43.617579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.714132+0000","last_change":"2024-04-25T01:52:44.090847+0000","last_active":"2024-04-25T01:55:15.714132+0000","last_peered":"2024-04-25T01:55:15.714132+0000","last_clean":"2024-04-25T01:55:15.714132+0000","last_became_active":"2024-04-25T01:52:44.090506+0000","last_became_peered":"2024-04-25T01:52:44.090506+0000","last_unstale":"2024-04-25T01:55:15.714132+0000","last_undegraded":"2024-04-25T01:55:15.714132+0000","last_fullsized":"2024-04-25T01:55:15.714132+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:24.539726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.922129+0000","last_change":"2024-04-25T01:52:43.090469+0000","last_active":"2024-04-25T01:55:15.922129+0000","last_peered":"2024-04-25T01:55:15.922129+0000","last_clean":"2024-04-25T01:55:15.922129+0000","last_became_active":"2024-04-25T01:52:43.090159+0000","last_became_peered":"2024-04-25T01:52:43.090159+0000","last_unstale":"2024-04-25T01:55:15.922129+0000","last_undegraded":"2024-04-25T01:55:15.922129+0000","last_fullsized":"2024-04-25T01:55:15.922129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:42:45.532017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584403+0000","last_change":"2024-04-25T01:52:44.093527+0000","last_active":"2024-04-25T01:55:15.584403+0000","last_peered":"2024-04-25T01:55:15.584403+0000","last_clean":"2024-04-25T01:55:15.584403+0000","last_became_active":"2024-04-25T01:52:44.092729+0000","last_became_peered":"2024-04-25T01:52:44.092729+0000","last_unstale":"2024-04-25T01:55:15.584403+0000","last_undegraded":"2024-04-25T01:55:15.584403+0000","last_fullsized":"2024-04-25T01:55:15.584403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:32:50.668352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904349+0000","last_change":"2024-04-25T01:52:43.075323+0000","last_active":"2024-04-25T01:55:15.904349+0000","last_peered":"2024-04-25T01:55:15.904349+0000","last_clean":"2024-04-25T01:55:15.904349+0000","last_became_active":"2024-04-25T01:52:43.074876+0000","last_became_peered":"2024-04-25T01:52:43.074876+0000","last_unstale":"2024-04-25T01:55:15.904349+0000","last_undegraded":"2024-04-25T01:55:15.904349+0000","last_fullsized":"2024-04-25T01:55:15.904349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:34:47.472920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":357,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918263+0000","last_change":"2024-04-25T01:51:20.173285+0000","last_active":"2024-04-25T01:55:15.918263+0000","last_peered":"2024-04-25T01:55:15.918263+0000","last_clean":"2024-04-25T01:55:15.918263+0000","last_became_active":"2024-04-25T01:51:20.172811+0000","last_became_peered":"2024-04-25T01:51:20.172811+0000","last_unstale":"2024-04-25T01:55:15.918263+0000","last_undegraded":"2024-04-25T01:55:15.918263+0000","last_fullsized":"2024-04-25T01:55:15.918263+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:47:31.712321+0000","last_clean_scrub_stamp":"2024-04-25T01:47:31.712321+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:16:33.708082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.574950+0000","last_change":"2024-04-25T01:53:43.874632+0000","last_active":"2024-04-25T01:55:15.574950+0000","last_peered":"2024-04-25T01:55:15.574950+0000","last_clean":"2024-04-25T01:55:15.574950+0000","last_became_active":"2024-04-25T01:53:43.873732+0000","last_became_peered":"2024-04-25T01:53:43.873732+0000","last_unstale":"2024-04-25T01:55:15.574950+0000","last_undegraded":"2024-04-25T01:55:15.574950+0000","last_fullsized":"2024-04-25T01:55:15.574950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:22:27.874789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918402+0000","last_change":"2024-04-25T01:52:43.072565+0000","last_active":"2024-04-25T01:55:15.918402+0000","last_peered":"2024-04-25T01:55:15.918402+0000","last_clean":"2024-04-25T01:55:15.918402+0000","last_became_active":"2024-04-25T01:52:43.072193+0000","last_became_peered":"2024-04-25T01:52:43.072193+0000","last_unstale":"2024-04-25T01:55:15.918402+0000","last_undegraded":"2024-04-25T01:55:15.918402+0000","last_fullsized":"2024-04-25T01:55:15.918402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:28:07.261859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904537+0000","last_change":"2024-04-25T01:53:43.878499+0000","last_active":"2024-04-25T01:55:15.904537+0000","last_peered":"2024-04-25T01:55:15.904537+0000","last_clean":"2024-04-25T01:55:15.904537+0000","last_became_active":"2024-04-25T01:53:43.870876+0000","last_became_peered":"2024-04-25T01:53:43.870876+0000","last_unstale":"2024-04-25T01:55:15.904537+0000","last_undegraded":"2024-04-25T01:55:15.904537+0000","last_fullsized":"2024-04-25T01:55:15.904537+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:33:08.809127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575785+0000","last_change":"2024-04-25T01:52:44.104892+0000","last_active":"2024-04-25T01:55:15.575785+0000","last_peered":"2024-04-25T01:55:15.575785+0000","last_clean":"2024-04-25T01:55:15.575785+0000","last_became_active":"2024-04-25T01:52:44.103802+0000","last_became_peered":"2024-04-25T01:52:44.103802+0000","last_unstale":"2024-04-25T01:55:15.575785+0000","last_undegraded":"2024-04-25T01:55:15.575785+0000","last_fullsized":"2024-04-25T01:55:15.575785+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:16.445968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'44","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.862454+0000","last_change":"2024-04-25T01:52:43.087143+0000","last_active":"2024-04-25T01:55:15.862454+0000","last_peered":"2024-04-25T01:55:15.862454+0000","last_clean":"2024-04-25T01:55:15.862454+0000","last_became_active":"2024-04-25T01:52:43.085987+0000","last_became_peered":"2024-04-25T01:52:43.085987+0000","last_unstale":"2024-04-25T01:55:15.862454+0000","last_undegraded":"2024-04-25T01:55:15.862454+0000","last_fullsized":"2024-04-25T01:55:15.862454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:25:17.936671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":78,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584485+0000","last_change":"2024-04-25T01:52:44.092342+0000","last_active":"2024-04-25T01:55:15.584485+0000","last_peered":"2024-04-25T01:55:15.584485+0000","last_clean":"2024-04-25T01:55:15.584485+0000","last_became_active":"2024-04-25T01:52:44.091991+0000","last_became_peered":"2024-04-25T01:52:44.091991+0000","last_unstale":"2024-04-25T01:55:15.584485+0000","last_undegraded":"2024-04-25T01:55:15.584485+0000","last_fullsized":"2024-04-25T01:55:15.584485+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:38:22.147284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904395+0000","last_change":"2024-04-25T01:52:43.077972+0000","last_active":"2024-04-25T01:55:15.904395+0000","last_peered":"2024-04-25T01:55:15.904395+0000","last_clean":"2024-04-25T01:55:15.904395+0000","last_became_active":"2024-04-25T01:52:43.077655+0000","last_became_peered":"2024-04-25T01:52:43.077655+0000","last_unstale":"2024-04-25T01:55:15.904395+0000","last_undegraded":"2024-04-25T01:55:15.904395+0000","last_fullsized":"2024-04-25T01:55:15.904395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:24.820910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905887+0000","last_change":"2024-04-25T01:52:44.119560+0000","last_active":"2024-04-25T01:55:15.905887+0000","last_peered":"2024-04-25T01:55:15.905887+0000","last_clean":"2024-04-25T01:55:15.905887+0000","last_became_active":"2024-04-25T01:52:44.119106+0000","last_became_peered":"2024-04-25T01:52:44.119106+0000","last_unstale":"2024-04-25T01:55:15.905887+0000","last_undegraded":"2024-04-25T01:55:15.905887+0000","last_fullsized":"2024-04-25T01:55:15.905887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:26:50.702393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576106+0000","last_change":"2024-04-25T01:52:43.075783+0000","last_active":"2024-04-25T01:55:15.576106+0000","last_peered":"2024-04-25T01:55:15.576106+0000","last_clean":"2024-04-25T01:55:15.576106+0000","last_became_active":"2024-04-25T01:52:43.075462+0000","last_became_peered":"2024-04-25T01:52:43.075462+0000","last_unstale":"2024-04-25T01:55:15.576106+0000","last_undegraded":"2024-04-25T01:55:15.576106+0000","last_fullsized":"2024-04-25T01:55:15.576106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:37:07.077047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702282+0000","last_change":"2024-04-25T01:52:44.090695+0000","last_active":"2024-04-25T01:55:15.702282+0000","last_peered":"2024-04-25T01:55:15.702282+0000","last_clean":"2024-04-25T01:55:15.702282+0000","last_became_active":"2024-04-25T01:52:44.090203+0000","last_became_peered":"2024-04-25T01:52:44.090203+0000","last_unstale":"2024-04-25T01:55:15.702282+0000","last_undegraded":"2024-04-25T01:55:15.702282+0000","last_fullsized":"2024-04-25T01:55:15.702282+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:04:58.104569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583701+0000","last_change":"2024-04-25T01:52:43.086057+0000","last_active":"2024-04-25T01:55:15.583701+0000","last_peered":"2024-04-25T01:55:15.583701+0000","last_clean":"2024-04-25T01:55:15.583701+0000","last_became_active":"2024-04-25T01:52:43.085475+0000","last_became_peered":"2024-04-25T01:52:43.085475+0000","last_unstale":"2024-04-25T01:55:15.583701+0000","last_undegraded":"2024-04-25T01:55:15.583701+0000","last_fullsized":"2024-04-25T01:55:15.583701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:37:42.237414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904383+0000","last_change":"2024-04-25T01:52:44.106128+0000","last_active":"2024-04-25T01:55:15.904383+0000","last_peered":"2024-04-25T01:55:15.904383+0000","last_clean":"2024-04-25T01:55:15.904383+0000","last_became_active":"2024-04-25T01:52:44.105805+0000","last_became_peered":"2024-04-25T01:52:44.105805+0000","last_unstale":"2024-04-25T01:55:15.904383+0000","last_undegraded":"2024-04-25T01:55:15.904383+0000","last_fullsized":"2024-04-25T01:55:15.904383+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:41.901070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904922+0000","last_change":"2024-04-25T01:52:43.088347+0000","last_active":"2024-04-25T01:55:15.904922+0000","last_peered":"2024-04-25T01:55:15.904922+0000","last_clean":"2024-04-25T01:55:15.904922+0000","last_became_active":"2024-04-25T01:52:43.087932+0000","last_became_peered":"2024-04-25T01:52:43.087932+0000","last_unstale":"2024-04-25T01:55:15.904922+0000","last_undegraded":"2024-04-25T01:55:15.904922+0000","last_fullsized":"2024-04-25T01:55:15.904922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:56:26.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584766+0000","last_change":"2024-04-25T01:52:44.108686+0000","last_active":"2024-04-25T01:55:15.584766+0000","last_peered":"2024-04-25T01:55:15.584766+0000","last_clean":"2024-04-25T01:55:15.584766+0000","last_became_active":"2024-04-25T01:52:44.107732+0000","last_became_peered":"2024-04-25T01:52:44.107732+0000","last_unstale":"2024-04-25T01:55:15.584766+0000","last_undegraded":"2024-04-25T01:55:15.584766+0000","last_fullsized":"2024-04-25T01:55:15.584766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:14.820979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.702353+0000","last_change":"2024-04-25T01:52:43.069300+0000","last_active":"2024-04-25T01:55:15.702353+0000","last_peered":"2024-04-25T01:55:15.702353+0000","last_clean":"2024-04-25T01:55:15.702353+0000","last_became_active":"2024-04-25T01:52:43.068644+0000","last_became_peered":"2024-04-25T01:52:43.068644+0000","last_unstale":"2024-04-25T01:55:15.702353+0000","last_undegraded":"2024-04-25T01:55:15.702353+0000","last_fullsized":"2024-04-25T01:55:15.702353+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:19:04.897638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918494+0000","last_change":"2024-04-25T01:52:44.100467+0000","last_active":"2024-04-25T01:55:15.918494+0000","last_peered":"2024-04-25T01:55:15.918494+0000","last_clean":"2024-04-25T01:55:15.918494+0000","last_became_active":"2024-04-25T01:52:44.099958+0000","last_became_peered":"2024-04-25T01:52:44.099958+0000","last_unstale":"2024-04-25T01:55:15.918494+0000","last_undegraded":"2024-04-25T01:55:15.918494+0000","last_fullsized":"2024-04-25T01:55:15.918494+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:29.919785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904238+0000","last_change":"2024-04-25T01:53:43.875865+0000","last_active":"2024-04-25T01:55:15.904238+0000","last_peered":"2024-04-25T01:55:15.904238+0000","last_clean":"2024-04-25T01:55:15.904238+0000","last_became_active":"2024-04-25T01:53:43.871525+0000","last_became_peered":"2024-04-25T01:53:43.871525+0000","last_unstale":"2024-04-25T01:55:15.904238+0000","last_undegraded":"2024-04-25T01:55:15.904238+0000","last_fullsized":"2024-04-25T01:55:15.904238+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:25.179902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918951+0000","last_change":"2024-04-25T01:52:44.088892+0000","last_active":"2024-04-25T01:55:15.918951+0000","last_peered":"2024-04-25T01:55:15.918951+0000","last_clean":"2024-04-25T01:55:15.918951+0000","last_became_active":"2024-04-25T01:52:44.087911+0000","last_became_peered":"2024-04-25T01:52:44.087911+0000","last_unstale":"2024-04-25T01:55:15.918951+0000","last_undegraded":"2024-04-25T01:55:15.918951+0000","last_fullsized":"2024-04-25T01:55:15.918951+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:29:38.063772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'39","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.859264+0000","last_change":"2024-04-25T01:52:43.086083+0000","last_active":"2024-04-25T01:55:15.859264+0000","last_peered":"2024-04-25T01:55:15.859264+0000","last_clean":"2024-04-25T01:55:15.859264+0000","last_became_active":"2024-04-25T01:52:43.085613+0000","last_became_peered":"2024-04-25T01:52:43.085613+0000","last_unstale":"2024-04-25T01:55:15.859264+0000","last_undegraded":"2024-04-25T01:55:15.859264+0000","last_fullsized":"2024-04-25T01:55:15.859264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:23.851812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918421+0000","last_change":"2024-04-25T01:52:44.106435+0000","last_active":"2024-04-25T01:55:15.918421+0000","last_peered":"2024-04-25T01:55:15.918421+0000","last_clean":"2024-04-25T01:55:15.918421+0000","last_became_active":"2024-04-25T01:52:44.106074+0000","last_became_peered":"2024-04-25T01:52:44.106074+0000","last_unstale":"2024-04-25T01:55:15.918421+0000","last_undegraded":"2024-04-25T01:55:15.918421+0000","last_fullsized":"2024-04-25T01:55:15.918421+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:46:59.286266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575223+0000","last_change":"2024-04-25T01:52:43.088902+0000","last_active":"2024-04-25T01:55:15.575223+0000","last_peered":"2024-04-25T01:55:15.575223+0000","last_clean":"2024-04-25T01:55:15.575223+0000","last_became_active":"2024-04-25T01:52:43.088289+0000","last_became_peered":"2024-04-25T01:52:43.088289+0000","last_unstale":"2024-04-25T01:55:15.575223+0000","last_undegraded":"2024-04-25T01:55:15.575223+0000","last_fullsized":"2024-04-25T01:55:15.575223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:42.039075+0000","last_clean_scrub_stamp":"2024-04-25T01:52:42.039075+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:37:44.562369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.918975+0000","last_change":"2024-04-25T01:52:44.094185+0000","last_active":"2024-04-25T01:55:15.918975+0000","last_peered":"2024-04-25T01:55:15.918975+0000","last_clean":"2024-04-25T01:55:15.918975+0000","last_became_active":"2024-04-25T01:52:44.093753+0000","last_became_peered":"2024-04-25T01:52:44.093753+0000","last_unstale":"2024-04-25T01:55:15.918975+0000","last_undegraded":"2024-04-25T01:55:15.918975+0000","last_fullsized":"2024-04-25T01:55:15.918975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:33:06.089670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904437+0000","last_change":"2024-04-25T01:52:44.093895+0000","last_active":"2024-04-25T01:55:15.904437+0000","last_peered":"2024-04-25T01:55:15.904437+0000","last_clean":"2024-04-25T01:55:15.904437+0000","last_became_active":"2024-04-25T01:52:44.093490+0000","last_became_peered":"2024-04-25T01:52:44.093490+0000","last_unstale":"2024-04-25T01:55:15.904437+0000","last_undegraded":"2024-04-25T01:55:15.904437+0000","last_fullsized":"2024-04-25T01:55:15.904437+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:56:53.848979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904530+0000","last_change":"2024-04-25T01:52:44.092500+0000","last_active":"2024-04-25T01:55:15.904530+0000","last_peered":"2024-04-25T01:55:15.904530+0000","last_clean":"2024-04-25T01:55:15.904530+0000","last_became_active":"2024-04-25T01:52:44.092044+0000","last_became_peered":"2024-04-25T01:52:44.092044+0000","last_unstale":"2024-04-25T01:55:15.904530+0000","last_undegraded":"2024-04-25T01:55:15.904530+0000","last_fullsized":"2024-04-25T01:55:15.904530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:28.279676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904716+0000","last_change":"2024-04-25T01:52:44.116668+0000","last_active":"2024-04-25T01:55:15.904716+0000","last_peered":"2024-04-25T01:55:15.904716+0000","last_clean":"2024-04-25T01:55:15.904716+0000","last_became_active":"2024-04-25T01:52:44.116295+0000","last_became_peered":"2024-04-25T01:52:44.116295+0000","last_unstale":"2024-04-25T01:55:15.904716+0000","last_undegraded":"2024-04-25T01:55:15.904716+0000","last_fullsized":"2024-04-25T01:55:15.904716+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:56:49.109403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.905629+0000","last_change":"2024-04-25T01:52:44.109950+0000","last_active":"2024-04-25T01:55:15.905629+0000","last_peered":"2024-04-25T01:55:15.905629+0000","last_clean":"2024-04-25T01:55:15.905629+0000","last_became_active":"2024-04-25T01:52:44.109636+0000","last_became_peered":"2024-04-25T01:52:44.109636+0000","last_unstale":"2024-04-25T01:55:15.905629+0000","last_undegraded":"2024-04-25T01:55:15.905629+0000","last_fullsized":"2024-04-25T01:55:15.905629+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:02:57.610067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575687+0000","last_change":"2024-04-25T01:52:44.105237+0000","last_active":"2024-04-25T01:55:15.575687+0000","last_peered":"2024-04-25T01:55:15.575687+0000","last_clean":"2024-04-25T01:55:15.575687+0000","last_became_active":"2024-04-25T01:52:44.104531+0000","last_became_peered":"2024-04-25T01:52:44.104531+0000","last_unstale":"2024-04-25T01:55:15.575687+0000","last_undegraded":"2024-04-25T01:55:15.575687+0000","last_fullsized":"2024-04-25T01:55:15.575687+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.149119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904755+0000","last_change":"2024-04-25T01:52:44.095655+0000","last_active":"2024-04-25T01:55:15.904755+0000","last_peered":"2024-04-25T01:55:15.904755+0000","last_clean":"2024-04-25T01:55:15.904755+0000","last_became_active":"2024-04-25T01:52:44.095300+0000","last_became_peered":"2024-04-25T01:52:44.095300+0000","last_unstale":"2024-04-25T01:55:15.904755+0000","last_undegraded":"2024-04-25T01:55:15.904755+0000","last_fullsized":"2024-04-25T01:55:15.904755+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:23:51.240170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904995+0000","last_change":"2024-04-25T01:52:44.087382+0000","last_active":"2024-04-25T01:55:15.904995+0000","last_peered":"2024-04-25T01:55:15.904995+0000","last_clean":"2024-04-25T01:55:15.904995+0000","last_became_active":"2024-04-25T01:52:44.086815+0000","last_became_peered":"2024-04-25T01:52:44.086815+0000","last_unstale":"2024-04-25T01:55:15.904995+0000","last_undegraded":"2024-04-25T01:55:15.904995+0000","last_fullsized":"2024-04-25T01:55:15.904995+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:52:13.682819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.584351+0000","last_change":"2024-04-25T01:53:43.871148+0000","last_active":"2024-04-25T01:55:15.584351+0000","last_peered":"2024-04-25T01:55:15.584351+0000","last_clean":"2024-04-25T01:55:15.584351+0000","last_became_active":"2024-04-25T01:53:43.870490+0000","last_became_peered":"2024-04-25T01:53:43.870490+0000","last_unstale":"2024-04-25T01:55:15.584351+0000","last_undegraded":"2024-04-25T01:55:15.584351+0000","last_fullsized":"2024-04-25T01:55:15.584351+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:43:41.790442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.713960+0000","last_change":"2024-04-25T01:52:44.114767+0000","last_active":"2024-04-25T01:55:15.713960+0000","last_peered":"2024-04-25T01:55:15.713960+0000","last_clean":"2024-04-25T01:55:15.713960+0000","last_became_active":"2024-04-25T01:52:44.114483+0000","last_became_peered":"2024-04-25T01:52:44.114483+0000","last_unstale":"2024-04-25T01:55:15.713960+0000","last_undegraded":"2024-04-25T01:55:15.713960+0000","last_fullsized":"2024-04-25T01:55:15.713960+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:14.439287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.583527+0000","last_change":"2024-04-25T01:52:44.094424+0000","last_active":"2024-04-25T01:55:15.583527+0000","last_peered":"2024-04-25T01:55:15.583527+0000","last_clean":"2024-04-25T01:55:15.583527+0000","last_became_active":"2024-04-25T01:52:44.093074+0000","last_became_peered":"2024-04-25T01:52:44.093074+0000","last_unstale":"2024-04-25T01:55:15.583527+0000","last_undegraded":"2024-04-25T01:55:15.583527+0000","last_fullsized":"2024-04-25T01:55:15.583527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:49:20.411376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.576299+0000","last_change":"2024-04-25T01:52:44.113465+0000","last_active":"2024-04-25T01:55:15.576299+0000","last_peered":"2024-04-25T01:55:15.576299+0000","last_clean":"2024-04-25T01:55:15.576299+0000","last_became_active":"2024-04-25T01:52:44.113168+0000","last_became_peered":"2024-04-25T01:52:44.113168+0000","last_unstale":"2024-04-25T01:55:15.576299+0000","last_undegraded":"2024-04-25T01:55:15.576299+0000","last_fullsized":"2024-04-25T01:55:15.576299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:42:47.758996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.575462+0000","last_change":"2024-04-25T01:52:44.104763+0000","last_active":"2024-04-25T01:55:15.575462+0000","last_peered":"2024-04-25T01:55:15.575462+0000","last_clean":"2024-04-25T01:55:15.575462+0000","last_became_active":"2024-04-25T01:52:44.104377+0000","last_became_peered":"2024-04-25T01:52:44.104377+0000","last_unstale":"2024-04-25T01:55:15.575462+0000","last_undegraded":"2024-04-25T01:55:15.575462+0000","last_fullsized":"2024-04-25T01:55:15.575462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:35:13.647020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-25T01:55:15.904326+0000","last_change":"2024-04-25T01:53:43.864792+0000","last_active":"2024-04-25T01:55:15.904326+0000","last_peered":"2024-04-25T01:55:15.904326+0000","last_clean":"2024-04-25T01:55:15.904326+0000","last_became_active":"2024-04-25T01:53:43.864432+0000","last_became_peered":"2024-04-25T01:53:43.864432+0000","last_unstale":"2024-04-25T01:55:15.904326+0000","last_undegraded":"2024-04-25T01:55:15.904326+0000","last_fullsized":"2024-04-25T01:55:15.904326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T01:52:43.051099+0000","last_clean_scrub_stamp":"2024-04-25T01:52:43.051099+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:28.149564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598282,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1000,"num_read_kb":609,"num_write":287,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":61762,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1794846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318317,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":471947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579953,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41328,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703824,"statfs":{"total":95995035648,"available":95952715776,"internally_reserved":0,"allocated":1814528,"data_stored":1263436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":40434113},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743480,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28372,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716780,"statfs":{"total":95995035648,"available":95965982720,"internally_reserved":0,"allocated":1384448,"data_stored":865265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005117,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32328,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712824,"statfs":{"total":95995035648,"available":95961931776,"internally_reserved":0,"allocated":1175552,"data_stored":668443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266755,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1585152,"data_stored":1065604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430279,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":471438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":532836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":983040,"data_stored":463782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":991232,"data_stored":473313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117080,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":995328,"data_stored":471749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855443,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":540999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691917,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1196106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":67760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":69768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":134390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T01:55:21.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:55:21.132 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:20 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-25T01:55:21.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:20 smithi102 ceph-mon[27796]: Unable to set osd_memory_target on smithi110 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T01:55:21.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:20 smithi102 ceph-mon[27796]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:55:21.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:20 smithi102 ceph-mon[27796]: from='client.25331 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:55:21.133 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:20 smithi102 ceph-mon[27796]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T01:55:21.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:20 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-25T01:55:21.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:20 smithi110 ceph-mon[32848]: Unable to set osd_memory_target on smithi110 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T01:55:21.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:20 smithi110 ceph-mon[32848]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:55:21.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:20 smithi110 ceph-mon[32848]: from='client.25331 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:55:21.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:20 smithi110 ceph-mon[32848]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T01:55:21.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:20 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-25T01:55:21.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:20 smithi193 ceph-mon[32981]: Unable to set osd_memory_target on smithi110 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T01:55:21.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:20 smithi193 ceph-mon[32981]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T01:55:21.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:20 smithi193 ceph-mon[32981]: from='client.25331 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:55:21.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:20 smithi193 ceph-mon[32981]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-25T01:55:21.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.257+0000 7fbc66600640 1 -- 172.21.15.102:0/2060573745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc601072a0 msgr2=0x7fbc601096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.257+0000 7fbc66600640 1 --2- 172.21.15.102:0/2060573745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc601072a0 0x7fbc601096f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009be0 tx=0x7fbc5802a6c0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.257+0000 7fbc66600640 1 -- 172.21.15.102:0/2060573745 shutdown_connections 2024-04-25T01:55:21.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.257+0000 7fbc66600640 1 --2- 172.21.15.102:0/2060573745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc601072a0 0x7fbc601096f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.257+0000 7fbc66600640 1 --2- 172.21.15.102:0/2060573745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc60104970 0x7fbc60106d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.257+0000 7fbc66600640 1 --2- 172.21.15.102:0/2060573745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc60102040 0x7fbc60104430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.257+0000 7fbc66600640 1 -- 172.21.15.102:0/2060573745 >> 172.21.15.102:0/2060573745 conn(0x7fbc600fba10 msgr2=0x7fbc600fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:21.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.257+0000 7fbc66600640 1 -- 172.21.15.102:0/2060573745 shutdown_connections 2024-04-25T01:55:21.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.258+0000 7fbc66600640 1 -- 172.21.15.102:0/2060573745 wait complete. 2024-04-25T01:55:21.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc66600640 1 Processor -- start 2024-04-25T01:55:21.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc66600640 1 -- start start 2024-04-25T01:55:21.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc66600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc60102040 0x7fbc60199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:21.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc66600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc60104970 0x7fbc6019a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:21.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc66600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601072a0 0x7fbc601a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:21.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc66600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6010d2b0 con 0x7fbc60102040 2024-04-25T01:55:21.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc66600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6010d130 con 0x7fbc601072a0 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc66600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6010d430 con 0x7fbc60104970 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc65200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc60102040 0x7fbc60199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc65200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc60102040 0x7fbc60199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33974/0 (socket says 172.21.15.102:33974) 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.259+0000 7fbc65200640 1 -- 172.21.15.102:0/446921379 learned_addr learned my addr 172.21.15.102:0/446921379 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc65c00640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601072a0 0x7fbc601a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc5fe00640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc60104970 0x7fbc6019a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc65200640 1 -- 172.21.15.102:0/446921379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc60104970 msgr2=0x7fbc6019a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc65200640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc60104970 0x7fbc6019a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc65200640 1 -- 172.21.15.102:0/446921379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601072a0 msgr2=0x7fbc601a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc65200640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601072a0 0x7fbc601a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc65200640 1 -- 172.21.15.102:0/446921379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc601a1740 con 0x7fbc60102040 2024-04-25T01:55:21.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc65200640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc60102040 0x7fbc60199b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400b4a0 tx=0x7fbc5400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:21.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc5d600640 1 -- 172.21.15.102:0/446921379 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54014070 con 0x7fbc60102040 2024-04-25T01:55:21.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc5d600640 1 -- 172.21.15.102:0/446921379 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc54010af0 con 0x7fbc60102040 2024-04-25T01:55:21.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc601a1a30 con 0x7fbc60102040 2024-04-25T01:55:21.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.260+0000 7fbc5d600640 1 -- 172.21.15.102:0/446921379 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc54017d10 con 0x7fbc60102040 2024-04-25T01:55:21.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.261+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc601a1fb0 con 0x7fbc60102040 2024-04-25T01:55:21.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.262+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc60108d90 con 0x7fbc60102040 2024-04-25T01:55:21.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.264+0000 7fbc5d600640 1 -- 172.21.15.102:0/446921379 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fbc54017070 con 0x7fbc60102040 2024-04-25T01:55:21.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.265+0000 7fbc5d600640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbc480a2d90 0x7fbc480a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:21.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.265+0000 7fbc5d600640 1 -- 172.21.15.102:0/446921379 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fbc540cecf0 con 0x7fbc60102040 2024-04-25T01:55:21.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.265+0000 7fbc5fe00640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbc480a2d90 0x7fbc480a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:21.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.266+0000 7fbc5fe00640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbc480a2d90 0x7fbc480a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc6019b030 tx=0x7fbc4c009260 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:21.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.272+0000 7fbc5d600640 1 -- 172.21.15.102:0/446921379 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbc5409b880 con 0x7fbc60102040 2024-04-25T01:55:21.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.480+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc60062340 con 0x7fbc60102040 2024-04-25T01:55:21.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.487+0000 7fbc5d600640 1 -- 172.21.15.102:0/446921379 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23064 (secure 0 0 0) 0x7fbc5409b180 con 0x7fbc60102040 2024-04-25T01:55:21.490 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:21.490 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":90,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:55:15.567792+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6801/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/459072323":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/2875896366":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1835499087":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/511834001":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:55:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.489+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbc480a2d90 msgr2=0x7fbc480a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.489+0000 7fbc66600640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbc480a2d90 0x7fbc480a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc6019b030 tx=0x7fbc4c009260 comp rx=0 tx=0).stop 2024-04-25T01:55:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.489+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc60102040 msgr2=0x7fbc60199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.489+0000 7fbc66600640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc60102040 0x7fbc60199b20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400b4a0 tx=0x7fbc5400b960 comp rx=0 tx=0).stop 2024-04-25T01:55:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.490+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 shutdown_connections 2024-04-25T01:55:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.490+0000 7fbc66600640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fbc480a2d90 0x7fbc480a5250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.490+0000 7fbc66600640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc601072a0 0x7fbc601a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.490+0000 7fbc66600640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc60104970 0x7fbc6019a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.490+0000 7fbc66600640 1 --2- 172.21.15.102:0/446921379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc60102040 0x7fbc60199b20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.490+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 >> 172.21.15.102:0/446921379 conn(0x7fbc600fba10 msgr2=0x7fbc60101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.490+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 shutdown_connections 2024-04-25T01:55:21.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.490+0000 7fbc66600640 1 -- 172.21.15.102:0/446921379 wait complete. 2024-04-25T01:55:21.511 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:41.585137+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:43.022666+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T01:55:21.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T01:55:21.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.686+0000 7fde5d200640 1 -- 172.21.15.102:0/2141690722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde58104a70 msgr2=0x7fde58104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.686+0000 7fde5d200640 1 --2- 172.21.15.102:0/2141690722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde58104a70 0x7fde58104ef0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fde40009900 tx=0x7fde4002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.687+0000 7fde5d200640 1 -- 172.21.15.102:0/2141690722 shutdown_connections 2024-04-25T01:55:21.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.687+0000 7fde5d200640 1 --2- 172.21.15.102:0/2141690722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde58105430 0x7fde5810bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.687+0000 7fde5d200640 1 --2- 172.21.15.102:0/2141690722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde58104a70 0x7fde58104ef0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.687+0000 7fde5d200640 1 --2- 172.21.15.102:0/2141690722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde58103880 0x7fde58103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.687+0000 7fde5d200640 1 -- 172.21.15.102:0/2141690722 >> 172.21.15.102:0/2141690722 conn(0x7fde580fedf0 msgr2=0x7fde58101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:21.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.687+0000 7fde5d200640 1 -- 172.21.15.102:0/2141690722 shutdown_connections 2024-04-25T01:55:21.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.687+0000 7fde5d200640 1 -- 172.21.15.102:0/2141690722 wait complete. 2024-04-25T01:55:21.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde5d200640 1 Processor -- start 2024-04-25T01:55:21.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde5d200640 1 -- start start 2024-04-25T01:55:21.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde5d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde58103880 0x7fde58199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:21.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde5d200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde58104a70 0x7fde5819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde5d200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde58105430 0x7fde581a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde5d200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde5810f530 con 0x7fde58103880 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde5d200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fde5810f3b0 con 0x7fde58104a70 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde5d200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde5810f6b0 con 0x7fde58105430 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde57400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde58103880 0x7fde58199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde4ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde58104a70 0x7fde5819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde57400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde58103880 0x7fde58199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33996/0 (socket says 172.21.15.102:33996) 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde4ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde58104a70 0x7fde5819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33218/0 (socket says 172.21.15.102:33218) 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.688+0000 7fde57400640 1 -- 172.21.15.102:0/3757422259 learned_addr learned my addr 172.21.15.102:0/3757422259 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.689+0000 7fde57e00640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde58105430 0x7fde581a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.689+0000 7fde4ea00640 1 -- 172.21.15.102:0/3757422259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde58105430 msgr2=0x7fde581a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.689+0000 7fde4ea00640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde58105430 0x7fde581a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.689+0000 7fde4ea00640 1 -- 172.21.15.102:0/3757422259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde58103880 msgr2=0x7fde58199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.689+0000 7fde4ea00640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde58103880 0x7fde58199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.689+0000 7fde4ea00640 1 -- 172.21.15.102:0/3757422259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde581a1800 con 0x7fde58104a70 2024-04-25T01:55:21.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.689+0000 7fde4ea00640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde58104a70 0x7fde5819a120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fde4002a9e0 tx=0x7fde40032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:21.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.690+0000 7fde54c00640 1 -- 172.21.15.102:0/3757422259 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4003f030 con 0x7fde58104a70 2024-04-25T01:55:21.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.690+0000 7fde54c00640 1 -- 172.21.15.102:0/3757422259 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde40003ce0 con 0x7fde58104a70 2024-04-25T01:55:21.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.690+0000 7fde54c00640 1 -- 172.21.15.102:0/3757422259 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde40035070 con 0x7fde58104a70 2024-04-25T01:55:21.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.690+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde581a1a90 con 0x7fde58104a70 2024-04-25T01:55:21.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.690+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde581a1f00 con 0x7fde58104a70 2024-04-25T01:55:21.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.691+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde5810ab40 con 0x7fde58104a70 2024-04-25T01:55:21.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.693+0000 7fde54c00640 1 -- 172.21.15.102:0/3757422259 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fde40042030 con 0x7fde58104a70 2024-04-25T01:55:21.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.694+0000 7fde54c00640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde3c0a2e10 0x7fde3c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:21.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.694+0000 7fde54c00640 1 -- 172.21.15.102:0/3757422259 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fde400efbd0 con 0x7fde58104a70 2024-04-25T01:55:21.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.695+0000 7fde57400640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde3c0a2e10 0x7fde3c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:21.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.695+0000 7fde57400640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde3c0a2e10 0x7fde3c0a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde480058c0 tx=0x7fde48005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:21.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.698+0000 7fde54c00640 1 -- 172.21.15.102:0/3757422259 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fde400bc000 con 0x7fde58104a70 2024-04-25T01:55:21.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.903+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fde5802d3c0 con 0x7fde58104a70 2024-04-25T01:55:21.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.904+0000 7fde54c00640 1 -- 172.21.15.102:0/3757422259 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fde400c0300 con 0x7fde58104a70 2024-04-25T01:55:21.906 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T01:55:21.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.906+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde3c0a2e10 msgr2=0x7fde3c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.906+0000 7fde5d200640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde3c0a2e10 0x7fde3c0a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde480058c0 tx=0x7fde48005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:21.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.906+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde58104a70 msgr2=0x7fde5819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:21.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.906+0000 7fde5d200640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde58104a70 0x7fde5819a120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fde4002a9e0 tx=0x7fde40032ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.907+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 shutdown_connections 2024-04-25T01:55:21.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.907+0000 7fde5d200640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde3c0a2e10 0x7fde3c0a52d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.907+0000 7fde5d200640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde58105430 0x7fde581a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.907+0000 7fde5d200640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde58104a70 0x7fde5819a120 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.907+0000 7fde5d200640 1 --2- 172.21.15.102:0/3757422259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde58103880 0x7fde58199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:21.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.907+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 >> 172.21.15.102:0/3757422259 conn(0x7fde580fedf0 msgr2=0x7fde58107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:21.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.907+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 shutdown_connections 2024-04-25T01:55:21.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:21.907+0000 7fde5d200640 1 -- 172.21.15.102:0/3757422259 wait complete. 2024-04-25T01:55:21.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T01:55:22.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.102+0000 7f574fa00640 1 -- 172.21.15.102:0/1760872947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 msgr2=0x7f57480ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.102+0000 7f574fa00640 1 --2- 172.21.15.102:0/1760872947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 0x7f57480ffe70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f573c009900 tx=0x7f573c02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.102+0000 7f574fa00640 1 -- 172.21.15.102:0/1760872947 shutdown_connections 2024-04-25T01:55:22.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.102+0000 7f574fa00640 1 --2- 172.21.15.102:0/1760872947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5748101170 0x7f57481019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.102+0000 7f574fa00640 1 --2- 172.21.15.102:0/1760872947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57481007b0 0x7f5748100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.102+0000 7f574fa00640 1 --2- 172.21.15.102:0/1760872947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 0x7f57480ffe70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.102+0000 7f574fa00640 1 -- 172.21.15.102:0/1760872947 >> 172.21.15.102:0/1760872947 conn(0x7f57480fb380 msgr2=0x7f57480fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.102+0000 7f574fa00640 1 -- 172.21.15.102:0/1760872947 shutdown_connections 2024-04-25T01:55:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.102+0000 7f574fa00640 1 -- 172.21.15.102:0/1760872947 wait complete. 2024-04-25T01:55:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.103+0000 7f574fa00640 1 Processor -- start 2024-04-25T01:55:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.103+0000 7f574fa00640 1 -- start start 2024-04-25T01:55:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.103+0000 7f574fa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 0x7f5748199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.103+0000 7f574fa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57481007b0 0x7f574819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.103+0000 7f574fa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5748101170 0x7f57481a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.103+0000 7f574fa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f574810f680 con 0x7f57481007b0 2024-04-25T01:55:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.103+0000 7f574fa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f574810f500 con 0x7f5748101170 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.103+0000 7f574fa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f574810f800 con 0x7f57480ffa50 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f5747e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57481007b0 0x7f574819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f574d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 0x7f5748199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f5747e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57481007b0 0x7f574819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34016/0 (socket says 172.21.15.102:34016) 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f574d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 0x7f5748199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35792/0 (socket says 172.21.15.102:35792) 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f5747e00640 1 -- 172.21.15.102:0/2519460914 learned_addr learned my addr 172.21.15.102:0/2519460914 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f574da00640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5748101170 0x7f57481a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f574d000640 1 -- 172.21.15.102:0/2519460914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5748101170 msgr2=0x7f57481a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f574d000640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5748101170 0x7f57481a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f574d000640 1 -- 172.21.15.102:0/2519460914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57481007b0 msgr2=0x7f574819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f574d000640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57481007b0 0x7f574819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f574d000640 1 -- 172.21.15.102:0/2519460914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57481a18d0 con 0x7f57480ffa50 2024-04-25T01:55:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.104+0000 7f574d000640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 0x7f5748199c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f573c02a910 tx=0x7f573c032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.105+0000 7f5745600640 1 -- 172.21.15.102:0/2519460914 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c03f030 con 0x7f57480ffa50 2024-04-25T01:55:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.105+0000 7f5745600640 1 -- 172.21.15.102:0/2519460914 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f573c003ce0 con 0x7f57480ffa50 2024-04-25T01:55:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.105+0000 7f5745600640 1 -- 172.21.15.102:0/2519460914 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c035070 con 0x7f57480ffa50 2024-04-25T01:55:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.105+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57481a1b60 con 0x7f57480ffa50 2024-04-25T01:55:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.105+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57481a1fd0 con 0x7f57480ffa50 2024-04-25T01:55:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.106+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f574810b560 con 0x7f57480ffa50 2024-04-25T01:55:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.108+0000 7f5745600640 1 -- 172.21.15.102:0/2519460914 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f573c042030 con 0x7f57480ffa50 2024-04-25T01:55:22.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.109+0000 7f5745600640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57300a2d90 0x7f57300a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.109+0000 7f5745600640 1 -- 172.21.15.102:0/2519460914 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f573c0ef9d0 con 0x7f57480ffa50 2024-04-25T01:55:22.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.109+0000 7f5747e00640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57300a2d90 0x7f57300a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.110+0000 7f5747e00640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57300a2d90 0x7f57300a5250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f57340095f0 tx=0x7f5734005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.113+0000 7f5745600640 1 -- 172.21.15.102:0/2519460914 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f573c0bbe00 con 0x7f57480ffa50 2024-04-25T01:55:22.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='client.25337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:55:22.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:55:22.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:22.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:22.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:55:22.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:55:22.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:21 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/446921379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:22.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='client.25337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:55:22.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:55:22.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:55:22.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:21 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/446921379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:22.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='client.25337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:55:22.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T01:55:22.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:22.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:22.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T01:55:22.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:55:22.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:55:22.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:21 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/446921379' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.317+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5748062370 con 0x7f57480ffa50 2024-04-25T01:55:22.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.317+0000 7f5745600640 1 -- 172.21.15.102:0/2519460914 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f573c0c0100 con 0x7f57480ffa50 2024-04-25T01:55:22.320 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-25T01:55:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.319+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57300a2d90 msgr2=0x7f57300a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.319+0000 7f574fa00640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57300a2d90 0x7f57300a5250 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f57340095f0 tx=0x7f5734005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.319+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 msgr2=0x7f5748199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.319+0000 7f574fa00640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 0x7f5748199c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f573c02a910 tx=0x7f573c032ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.320+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 shutdown_connections 2024-04-25T01:55:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.320+0000 7f574fa00640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57300a2d90 0x7f57300a5250 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.320+0000 7f574fa00640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5748101170 0x7f57481a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.320+0000 7f574fa00640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f57481007b0 0x7f574819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.320+0000 7f574fa00640 1 --2- 172.21.15.102:0/2519460914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57480ffa50 0x7f5748199c90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.320+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 >> 172.21.15.102:0/2519460914 conn(0x7f57480fb380 msgr2=0x7f57480fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.320+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 shutdown_connections 2024-04-25T01:55:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.320+0000 7f574fa00640 1 -- 172.21.15.102:0/2519460914 wait complete. 2024-04-25T01:55:22.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T01:55:22.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.507+0000 7f6dabc00640 1 -- 172.21.15.102:0/995924785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6da4076100 msgr2=0x7f6da4076520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.507+0000 7f6dabc00640 1 --2- 172.21.15.102:0/995924785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6da4076100 0x7f6da4076520 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009900 tx=0x7f6d9802a410 comp rx=0 tx=0).stop 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.507+0000 7f6dabc00640 1 -- 172.21.15.102:0/995924785 shutdown_connections 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.507+0000 7f6dabc00640 1 --2- 172.21.15.102:0/995924785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4077ce0 0x7f6da4078580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.507+0000 7f6dabc00640 1 --2- 172.21.15.102:0/995924785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da4077340 0x7f6da40777a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.507+0000 7f6dabc00640 1 --2- 172.21.15.102:0/995924785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6da4076100 0x7f6da4076520 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.507+0000 7f6dabc00640 1 -- 172.21.15.102:0/995924785 >> 172.21.15.102:0/995924785 conn(0x7f6da40fef30 msgr2=0x7f6da4101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.507+0000 7f6dabc00640 1 -- 172.21.15.102:0/995924785 shutdown_connections 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.508+0000 7f6dabc00640 1 -- 172.21.15.102:0/995924785 wait complete. 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.508+0000 7f6dabc00640 1 Processor -- start 2024-04-25T01:55:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6dabc00640 1 -- start start 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6dabc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4076100 0x7f6da419e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6dabc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da4077340 0x7f6da419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6dabc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6da4077ce0 0x7f6da41a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6dabc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4113a90 con 0x7f6da4077340 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6dabc00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4113910 con 0x7f6da4076100 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6dabc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4113c10 con 0x7f6da4077ce0 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6da9200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4076100 0x7f6da419e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6da3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da4077340 0x7f6da419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6da9200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4076100 0x7f6da419e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33236/0 (socket says 172.21.15.102:33236) 2024-04-25T01:55:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6da9200640 1 -- 172.21.15.102:0/1113067116 learned_addr learned my addr 172.21.15.102:0/1113067116 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6da9c00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6da4077ce0 0x7f6da41a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6da9200640 1 -- 172.21.15.102:0/1113067116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6da4077ce0 msgr2=0x7f6da41a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6da9200640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6da4077ce0 0x7f6da41a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.509+0000 7f6da9200640 1 -- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da4077340 msgr2=0x7f6da419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.510+0000 7f6da9200640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da4077340 0x7f6da419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.510+0000 7f6da9200640 1 -- 172.21.15.102:0/1113067116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da41a5d00 con 0x7f6da4076100 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.510+0000 7f6da3e00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da4077340 0x7f6da419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.510+0000 7f6da9200640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4076100 0x7f6da419e0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6d98031440 tx=0x7f6d98032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.510+0000 7f6da1600640 1 -- 172.21.15.102:0/1113067116 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9803f030 con 0x7f6da4076100 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.510+0000 7f6da1600640 1 -- 172.21.15.102:0/1113067116 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d98003ce0 con 0x7f6da4076100 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.510+0000 7f6da1600640 1 -- 172.21.15.102:0/1113067116 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d98035070 con 0x7f6da4076100 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.510+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da41a5f90 con 0x7f6da4076100 2024-04-25T01:55:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.510+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da41a63a0 con 0x7f6da4076100 2024-04-25T01:55:22.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.511+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da4110330 con 0x7f6da4076100 2024-04-25T01:55:22.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.514+0000 7f6da1600640 1 -- 172.21.15.102:0/1113067116 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f6d98042030 con 0x7f6da4076100 2024-04-25T01:55:22.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.515+0000 7f6da1600640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6d8c0a2e10 0x7f6d8c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.516+0000 7f6da1600640 1 -- 172.21.15.102:0/1113067116 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6d980efa30 con 0x7f6da4076100 2024-04-25T01:55:22.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.516+0000 7f6da3e00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6d8c0a2e10 0x7f6d8c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.516+0000 7f6da3e00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6d8c0a2e10 0x7f6d8c0a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6da419f5f0 tx=0x7f6d90005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:22.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.519+0000 7f6da1600640 1 -- 172.21.15.102:0/1113067116 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6d980bbe60 con 0x7f6da4076100 2024-04-25T01:55:22.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.723+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6da402d3c0 con 0x7f6da4076100 2024-04-25T01:55:22.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.723+0000 7f6da1600640 1 -- 172.21.15.102:0/1113067116 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f6d980c0160 con 0x7f6da4076100 2024-04-25T01:55:22.725 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-25T01:55:22.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6d8c0a2e10 msgr2=0x7f6d8c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6d8c0a2e10 0x7f6d8c0a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6da419f5f0 tx=0x7f6d90005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:22.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4076100 msgr2=0x7f6da419e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4076100 0x7f6da419e0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6d98031440 tx=0x7f6d98032ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 shutdown_connections 2024-04-25T01:55:22.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6d8c0a2e10 0x7f6d8c0a52d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6da4077ce0 0x7f6da41a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da4077340 0x7f6da419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 --2- 172.21.15.102:0/1113067116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6da4076100 0x7f6da419e0e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 >> 172.21.15.102:0/1113067116 conn(0x7f6da40fef30 msgr2=0x7f6da4100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:22.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 shutdown_connections 2024-04-25T01:55:22.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.726+0000 7f6dabc00640 1 -- 172.21.15.102:0/1113067116 wait complete. 2024-04-25T01:55:22.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:55:22.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.920+0000 7fe492800640 1 -- 172.21.15.102:0/37116175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c102960 msgr2=0x7fe48c104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.920+0000 7fe492800640 1 --2- 172.21.15.102:0/37116175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c102960 0x7fe48c104dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe478009900 tx=0x7fe47802a410 comp rx=0 tx=0).stop 2024-04-25T01:55:22.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.920+0000 7fe492800640 1 -- 172.21.15.102:0/37116175 shutdown_connections 2024-04-25T01:55:22.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.920+0000 7fe492800640 1 --2- 172.21.15.102:0/37116175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48c105310 0x7fe48c105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.920+0000 7fe492800640 1 --2- 172.21.15.102:0/37116175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c102960 0x7fe48c104dd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.920+0000 7fe492800640 1 --2- 172.21.15.102:0/37116175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c0ffff0 0x7fe48c102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.920+0000 7fe492800640 1 -- 172.21.15.102:0/37116175 >> 172.21.15.102:0/37116175 conn(0x7fe48c0fbbf0 msgr2=0x7fe48c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:22.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.920+0000 7fe492800640 1 -- 172.21.15.102:0/37116175 shutdown_connections 2024-04-25T01:55:22.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.920+0000 7fe492800640 1 -- 172.21.15.102:0/37116175 wait complete. 2024-04-25T01:55:22.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.921+0000 7fe492800640 1 Processor -- start 2024-04-25T01:55:22.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.921+0000 7fe492800640 1 -- start start 2024-04-25T01:55:22.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe492800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48c0ffff0 0x7fe48c19e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe492800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c102960 0x7fe48c19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe492800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c105310 0x7fe48c1a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe492800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c113a40 con 0x7fe48c105310 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe492800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c1138c0 con 0x7fe48c102960 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe492800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c113bc0 con 0x7fe48c0ffff0 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c102960 0x7fe48c19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c102960 0x7fe48c19e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33176/0 (socket says 172.21.15.102:33176) 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48aa00640 1 -- 172.21.15.102:0/4193009392 learned_addr learned my addr 172.21.15.102:0/4193009392 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48be00640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c105310 0x7fe48c1a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48b400640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48c0ffff0 0x7fe48c19e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48aa00640 1 -- 172.21.15.102:0/4193009392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48c0ffff0 msgr2=0x7fe48c19e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48aa00640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48c0ffff0 0x7fe48c19e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48aa00640 1 -- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c105310 msgr2=0x7fe48c1a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48aa00640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c105310 0x7fe48c1a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:22.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.922+0000 7fe48aa00640 1 -- 172.21.15.102:0/4193009392 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe48c1a5c90 con 0x7fe48c102960 2024-04-25T01:55:22.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.923+0000 7fe48aa00640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c102960 0x7fe48c19e600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe478031440 tx=0x7fe478032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:22.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.923+0000 7fe483e00640 1 -- 172.21.15.102:0/4193009392 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47803f030 con 0x7fe48c102960 2024-04-25T01:55:22.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.923+0000 7fe483e00640 1 -- 172.21.15.102:0/4193009392 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe478003ce0 con 0x7fe48c102960 2024-04-25T01:55:22.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.923+0000 7fe483e00640 1 -- 172.21.15.102:0/4193009392 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe478035070 con 0x7fe48c102960 2024-04-25T01:55:22.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.923+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe48c1a5f20 con 0x7fe48c102960 2024-04-25T01:55:22.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.923+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe48c1a6330 con 0x7fe48c102960 2024-04-25T01:55:22.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.924+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe48c0fd6e0 con 0x7fe48c102960 2024-04-25T01:55:22.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.926+0000 7fe483e00640 1 -- 172.21.15.102:0/4193009392 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fe478042030 con 0x7fe48c102960 2024-04-25T01:55:22.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.928+0000 7fe483e00640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4700a2d90 0x7fe4700a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.928+0000 7fe483e00640 1 -- 172.21.15.102:0/4193009392 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe4780efe10 con 0x7fe48c102960 2024-04-25T01:55:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.928+0000 7fe48b400640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4700a2d90 0x7fe4700a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.929+0000 7fe48b400640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4700a2d90 0x7fe4700a5250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009bd0 tx=0x7fe47c005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:22.932+0000 7fe483e00640 1 -- 172.21.15.102:0/4193009392 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe4780f3030 con 0x7fe48c102960 2024-04-25T01:55:23.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi102 to 176.4M 2024-04-25T01:55:23.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Unable to set osd_memory_target on smithi102 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-25T01:55:23.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-25T01:55:23.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3757422259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2519460914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:22 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1113067116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:23.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.190+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe48c02d170 con 0x7fe48c102960 2024-04-25T01:55:23.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.191+0000 7fe483e00640 1 -- 172.21.15.102:0/4193009392 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11896 (secure 0 0 0) 0x7fe4780bc240 con 0x7fe48c102960 2024-04-25T01:55:23.193 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:23.193 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:55:15.481210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587,24533,24541]},"id":1}]} 2024-04-25T01:55:23.193 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 23 2024-04-25T01:55:23.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4700a2d90 msgr2=0x7fe4700a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4700a2d90 0x7fe4700a5250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009bd0 tx=0x7fe47c005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:23.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c102960 msgr2=0x7fe48c19e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c102960 0x7fe48c19e600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe478031440 tx=0x7fe478032ce0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 shutdown_connections 2024-04-25T01:55:23.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4700a2d90 0x7fe4700a5250 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe48c105310 0x7fe48c1a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe48c102960 0x7fe48c19e600 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 --2- 172.21.15.102:0/4193009392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48c0ffff0 0x7fe48c19e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 >> 172.21.15.102:0/4193009392 conn(0x7fe48c0fbbf0 msgr2=0x7fe48c100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.193+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 shutdown_connections 2024-04-25T01:55:23.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.194+0000 7fe492800640 1 -- 172.21.15.102:0/4193009392 wait complete. 2024-04-25T01:55:23.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T01:55:23.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi102 to 176.4M 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Unable to set osd_memory_target on smithi102 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3757422259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:55:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:55:23.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:55:23.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2519460914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:23.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:22 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1113067116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:23.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi102 to 176.4M 2024-04-25T01:55:23.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Unable to set osd_memory_target on smithi102 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-25T01:55:23.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi102:/etc/ceph/ceph.conf 2024-04-25T01:55:23.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi110:/etc/ceph/ceph.conf 2024-04-25T01:55:23.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi193:/etc/ceph/ceph.conf 2024-04-25T01:55:23.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 0 B/s wr, 26 op/s 2024-04-25T01:55:23.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3757422259' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: Updating smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2519460914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:22 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1113067116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T01:55:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.388+0000 7f000c000640 1 -- 172.21.15.102:0/1507309774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0004109920 msgr2=0x7f0004109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.388+0000 7f000c000640 1 --2- 172.21.15.102:0/1507309774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0004109920 0x7f0004109d00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efff8009930 tx=0x7efff802a420 comp rx=0 tx=0).stop 2024-04-25T01:55:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.388+0000 7f000c000640 1 -- 172.21.15.102:0/1507309774 shutdown_connections 2024-04-25T01:55:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.388+0000 7f000c000640 1 --2- 172.21.15.102:0/1507309774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00041042d0 0x7f00041107e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.388+0000 7f000c000640 1 --2- 172.21.15.102:0/1507309774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0004103910 0x7f0004103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.388+0000 7f000c000640 1 --2- 172.21.15.102:0/1507309774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0004109920 0x7f0004109d00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.388+0000 7f000c000640 1 -- 172.21.15.102:0/1507309774 >> 172.21.15.102:0/1507309774 conn(0x7f00040ff440 msgr2=0x7f0004101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.388+0000 7f000c000640 1 -- 172.21.15.102:0/1507309774 shutdown_connections 2024-04-25T01:55:23.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.388+0000 7f000c000640 1 -- 172.21.15.102:0/1507309774 wait complete. 2024-04-25T01:55:23.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.389+0000 7f000c000640 1 Processor -- start 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.389+0000 7f000c000640 1 -- start start 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000c000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0004103910 0x7f000419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000c000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00041042d0 0x7f000419a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000c000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0004109920 0x7f000419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000c000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0004113d30 con 0x7f0004109920 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000c000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0004113bb0 con 0x7f00041042d0 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000c000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0004113eb0 con 0x7f0004103910 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f0008c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00041042d0 0x7f000419a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0004109920 0x7f000419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f0009600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0004103910 0x7f000419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f0008c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00041042d0 0x7f000419a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33196/0 (socket says 172.21.15.102:33196) 2024-04-25T01:55:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f0008c00640 1 -- 172.21.15.102:0/3114810136 learned_addr learned my addr 172.21.15.102:0/3114810136 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000a000640 1 -- 172.21.15.102:0/3114810136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0004103910 msgr2=0x7f000419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000a000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0004103910 0x7f000419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000a000640 1 -- 172.21.15.102:0/3114810136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00041042d0 msgr2=0x7f000419a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000a000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00041042d0 0x7f000419a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f000a000640 1 -- 172.21.15.102:0/3114810136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f000419f280 con 0x7f0004109920 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f0009600640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0004103910 0x7f000419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.390+0000 7f0008c00640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00041042d0 0x7f000419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.391+0000 7f000a000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0004109920 0x7f000419eb50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efffc00e940 tx=0x7efffc00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.391+0000 7f0002000640 1 -- 172.21.15.102:0/3114810136 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc004d20 con 0x7f0004109920 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.391+0000 7f0002000640 1 -- 172.21.15.102:0/3114810136 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efffc005020 con 0x7f0004109920 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.391+0000 7f0002000640 1 -- 172.21.15.102:0/3114810136 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc0116b0 con 0x7f0004109920 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.391+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f000419f570 con 0x7f0004109920 2024-04-25T01:55:23.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.391+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00041a6e00 con 0x7f0004109920 2024-04-25T01:55:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.393+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00040689b0 con 0x7f0004109920 2024-04-25T01:55:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.394+0000 7f0002000640 1 -- 172.21.15.102:0/3114810136 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7efffc0054d0 con 0x7f0004109920 2024-04-25T01:55:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.396+0000 7f0002000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7effec0a2e10 0x7effec0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.396+0000 7f0002000640 1 -- 172.21.15.102:0/3114810136 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7efffc0cf870 con 0x7f0004109920 2024-04-25T01:55:23.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.396+0000 7f0009600640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7effec0a2e10 0x7effec0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.397+0000 7f0009600640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7effec0a2e10 0x7effec0a52d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efff8009900 tx=0x7efff80058d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.401+0000 7f0002000640 1 -- 172.21.15.102:0/3114810136 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efffc016030 con 0x7f0004109920 2024-04-25T01:55:23.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.607+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f000402d170 con 0x7f0004109920 2024-04-25T01:55:23.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.608+0000 7f0002000640 1 -- 172.21.15.102:0/3114810136 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+23064 (secure 0 0 0) 0x7efffc09bca0 con 0x7f0004109920 2024-04-25T01:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":90,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T01:55:15.567792+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6801/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/459072323":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/2875896366":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1835499087":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/511834001":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T01:55:23.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7effec0a2e10 msgr2=0x7effec0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7effec0a2e10 0x7effec0a52d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efff8009900 tx=0x7efff80058d0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0004109920 msgr2=0x7f000419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0004109920 0x7f000419eb50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efffc00e940 tx=0x7efffc00ee00 comp rx=0 tx=0).stop 2024-04-25T01:55:23.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 shutdown_connections 2024-04-25T01:55:23.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7effec0a2e10 0x7effec0a52d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0004109920 0x7f000419eb50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00041042d0 0x7f000419a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 --2- 172.21.15.102:0/3114810136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0004103910 0x7f000419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 >> 172.21.15.102:0/3114810136 conn(0x7f00040ff440 msgr2=0x7f00040ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 shutdown_connections 2024-04-25T01:55:23.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.611+0000 7f000c000640 1 -- 172.21.15.102:0/3114810136 wait complete. 2024-04-25T01:55:23.631 DEBUG:tasks.fwd_scrub:Yielding 2024-04-25T01:55:23.631 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:55:23.642 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:55:23.642 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T01:55:23.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-25T01:55:23.643 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-25T01:55:23.643 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:55:23.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:23.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:55:23.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.817+0000 7feccaa00640 1 -- 172.21.15.102:0/1708046791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc4109780 msgr2=0x7fecc4109b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.817+0000 7feccaa00640 1 --2- 172.21.15.102:0/1708046791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc4109780 0x7fecc4109b60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009930 tx=0x7fecb802a420 comp rx=0 tx=0).stop 2024-04-25T01:55:23.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.817+0000 7feccaa00640 1 -- 172.21.15.102:0/1708046791 shutdown_connections 2024-04-25T01:55:23.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.817+0000 7feccaa00640 1 --2- 172.21.15.102:0/1708046791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc4104130 0x7fecc4110640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.817+0000 7feccaa00640 1 --2- 172.21.15.102:0/1708046791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc4103770 0x7fecc4103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.817+0000 7feccaa00640 1 --2- 172.21.15.102:0/1708046791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc4109780 0x7fecc4109b60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.817+0000 7feccaa00640 1 -- 172.21.15.102:0/1708046791 >> 172.21.15.102:0/1708046791 conn(0x7fecc40ff220 msgr2=0x7fecc4101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.817+0000 7feccaa00640 1 -- 172.21.15.102:0/1708046791 shutdown_connections 2024-04-25T01:55:23.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.818+0000 7feccaa00640 1 -- 172.21.15.102:0/1708046791 wait complete. 2024-04-25T01:55:23.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.819+0000 7feccaa00640 1 Processor -- start 2024-04-25T01:55:23.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.819+0000 7feccaa00640 1 -- start start 2024-04-25T01:55:23.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7feccaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc4103770 0x7fecc419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7feccaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc4104130 0x7fecc419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7feccaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc4109780 0x7fecc419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7feccaa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fecc4113ba0 con 0x7fecc4104130 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7feccaa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fecc4113a20 con 0x7fecc4103770 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7feccaa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fecc4113d20 con 0x7fecc4109780 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7fecc9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc4103770 0x7fecc419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7fecc9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc4103770 0x7fecc419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33220/0 (socket says 172.21.15.102:33220) 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7fecc9600640 1 -- 172.21.15.102:0/114483111 learned_addr learned my addr 172.21.15.102:0/114483111 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7fecc8c00640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc4104130 0x7fecc419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.820+0000 7fecca000640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc4109780 0x7fecc419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.821+0000 7fecc9600640 1 -- 172.21.15.102:0/114483111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc4109780 msgr2=0x7fecc419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.821+0000 7fecc9600640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc4109780 0x7fecc419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.821+0000 7fecc9600640 1 -- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc4104130 msgr2=0x7fecc419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.821+0000 7fecc9600640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc4104130 0x7fecc419a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.821+0000 7fecc9600640 1 -- 172.21.15.102:0/114483111 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc419f080 con 0x7fecc4103770 2024-04-25T01:55:23.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.821+0000 7fecc9600640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc4103770 0x7fecc419a0e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fecb802a940 tx=0x7fecb8003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.822+0000 7fecc2000640 1 -- 172.21.15.102:0/114483111 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb803f070 con 0x7fecc4103770 2024-04-25T01:55:23.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.822+0000 7fecc2000640 1 -- 172.21.15.102:0/114483111 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb8031c90 con 0x7fecc4103770 2024-04-25T01:55:23.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.822+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc419f310 con 0x7fecc4103770 2024-04-25T01:55:23.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.822+0000 7fecc2000640 1 -- 172.21.15.102:0/114483111 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8035030 con 0x7fecc4103770 2024-04-25T01:55:23.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.822+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc41a6c00 con 0x7fecc4103770 2024-04-25T01:55:23.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.824+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecc404fd50 con 0x7fecc4103770 2024-04-25T01:55:23.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.826+0000 7fecc2000640 1 -- 172.21.15.102:0/114483111 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fecb8042070 con 0x7fecc4103770 2024-04-25T01:55:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7f108fe00640 1 -- 172.21.15.102:0/358933124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1090109800 msgr2=0x7f1090109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7f108fe00640 1 --2- 172.21.15.102:0/358933124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1090109800 0x7f1090109be0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1080009930 tx=0x7f108002a420 comp rx=0 tx=0).stop 2024-04-25T01:55:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7f108fe00640 1 -- 172.21.15.102:0/358933124 shutdown_connections 2024-04-25T01:55:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7f108fe00640 1 --2- 172.21.15.102:0/358933124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10901041b0 0x7f10901106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7f108fe00640 1 --2- 172.21.15.102:0/358933124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10901037f0 0x7f1090103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7f108fe00640 1 --2- 172.21.15.102:0/358933124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1090109800 0x7f1090109be0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7f108fe00640 1 -- 172.21.15.102:0/358933124 >> 172.21.15.102:0/358933124 conn(0x7f10900ff280 msgr2=0x7f10901016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7f108fe00640 1 -- 172.21.15.102:0/358933124 shutdown_connections 2024-04-25T01:55:23.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7f108fe00640 1 -- 172.21.15.102:0/358933124 wait complete. 2024-04-25T01:55:23.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7fecc2000640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.827+0000 7fecc2000640 1 -- 172.21.15.102:0/114483111 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fecb80ef900 con 0x7fecc4103770 2024-04-25T01:55:23.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7fecc8c00640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7f108fe00640 1 Processor -- start 2024-04-25T01:55:23.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7f108fe00640 1 -- start start 2024-04-25T01:55:23.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7fecc8c00640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fecc41018a0 tx=0x7fecb000b390 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7f108fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10901037f0 0x7f1090110560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7f108fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10901041b0 0x7f109010d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7f108fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090109800 0x7f109010db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7f108fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1090113c20 con 0x7f1090109800 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7f108fe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1090113aa0 con 0x7f10901041b0 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.828+0000 7f108fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1090113da0 con 0x7f10901037f0 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10901037f0 0x7f1090110560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090109800 0x7f109010db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10901037f0 0x7f1090110560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55966/0 (socket says 172.21.15.102:55966) 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108ea00640 1 -- 172.21.15.102:0/2637142788 learned_addr learned my addr 172.21.15.102:0/2637142788 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108e000640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10901041b0 0x7f109010d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108ea00640 1 -- 172.21.15.102:0/2637142788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10901041b0 msgr2=0x7f109010d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108ea00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10901041b0 0x7f109010d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108ea00640 1 -- 172.21.15.102:0/2637142788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090109800 msgr2=0x7f109010db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108ea00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090109800 0x7f109010db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.829+0000 7f108ea00640 1 -- 172.21.15.102:0/2637142788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109010e400 con 0x7f10901037f0 2024-04-25T01:55:23.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.830+0000 7f108ea00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10901037f0 0x7f1090110560 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f108002a940 tx=0x7f1080003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.830+0000 7f1094a00640 1 -- 172.21.15.102:0/2637142788 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108003f070 con 0x7f10901037f0 2024-04-25T01:55:23.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.830+0000 7f108fe00640 1 -- 172.21.15.102:0/2637142788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10901a01c0 con 0x7f10901037f0 2024-04-25T01:55:23.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.830+0000 7f1094a00640 1 -- 172.21.15.102:0/2637142788 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1080031c90 con 0x7f10901037f0 2024-04-25T01:55:23.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.830+0000 7f1094a00640 1 -- 172.21.15.102:0/2637142788 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080035030 con 0x7f10901037f0 2024-04-25T01:55:23.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.830+0000 7f108fe00640 1 -- 172.21.15.102:0/2637142788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10901a0700 con 0x7f10901037f0 2024-04-25T01:55:23.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.832+0000 7fecc2000640 1 -- 172.21.15.102:0/114483111 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fecb80bbd30 con 0x7fecc4103770 2024-04-25T01:55:23.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.833+0000 7f1094a00640 1 -- 172.21.15.102:0/2637142788 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f1080042070 con 0x7f10901037f0 2024-04-25T01:55:23.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.833+0000 7f1098c2a740 1 Processor -- start 2024-04-25T01:55:23.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.834+0000 7f1098c2a740 1 -- start start 2024-04-25T01:55:23.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5d7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5e19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f108f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5d7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f108f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5d7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53072/0 (socket says 172.21.15.102:53072) 2024-04-25T01:55:23.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 0x55975f5e4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f108f400640 1 -- 172.21.15.102:0/1006036585 learned_addr learned my addr 172.21.15.102:0/1006036585 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:23.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f1098c2a740 1 -- 172.21.15.102:0/1006036585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55975f535f40 con 0x55975f5d6f20 2024-04-25T01:55:23.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f1098c2a740 1 -- 172.21.15.102:0/1006036585 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55975f53b190 con 0x55975f5d7840 2024-04-25T01:55:23.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f1098c2a740 1 -- 172.21.15.102:0/1006036585 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55975f4e3a10 con 0x55975f5e1f10 2024-04-25T01:55:23.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.835+0000 7f108ea00640 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5e19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f108e000640 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 0x55975f5e4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f1094a00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f10740a2e10 0x7f10740a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f108f400640 1 -- 172.21.15.102:0/1006036585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 msgr2=0x55975f5e4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:23.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f108f400640 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 0x55975f5e4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f108f400640 1 -- 172.21.15.102:0/1006036585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 msgr2=0x55975f5e19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:23.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f108f400640 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5e19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f108f400640 1 -- 172.21.15.102:0/1006036585 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55975f3177e0 con 0x55975f5d6f20 2024-04-25T01:55:23.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f108e000640 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 0x55975f5e4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:23.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f1094a00640 1 -- 172.21.15.102:0/2637142788 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f10800efc40 con 0x7f10901037f0 2024-04-25T01:55:23.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.836+0000 7f108f400640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f10740a2e10 0x7f10740a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.837+0000 7f108f400640 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5d7300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1088007e60 tx=0x7f108800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e331d2202af0d86e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.838+0000 7f108f400640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f10740a2e10 0x7f10740a52d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1088007fd0 tx=0x7f108800a000 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.838+0000 7f107be00640 1 -- 172.21.15.102:0/1006036585 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088010760 con 0x55975f5d6f20 2024-04-25T01:55:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.838+0000 7f107be00640 1 -- 172.21.15.102:0/1006036585 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f108801a8f0 con 0x55975f5d6f20 2024-04-25T01:55:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.838+0000 7f107be00640 1 -- 172.21.15.102:0/1006036585 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088024a40 con 0x55975f5d6f20 2024-04-25T01:55:23.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.839+0000 7f1098c2a740 1 -- 172.21.15.102:0/1006036585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 msgr2=0x55975f5d7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.839+0000 7f1098c2a740 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5d7300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1088007e60 tx=0x7f108800f790 comp rx=0 tx=0).stop 2024-04-25T01:55:23.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.839+0000 7f1098c2a740 1 -- 172.21.15.102:0/1006036585 shutdown_connections 2024-04-25T01:55:23.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.839+0000 7f1098c2a740 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 0x55975f5e4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.839+0000 7f1098c2a740 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5e19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.839+0000 7f1098c2a740 1 --2- 172.21.15.102:0/1006036585 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5d7300 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.839+0000 7f1098c2a740 1 -- 172.21.15.102:0/1006036585 >> 172.21.15.102:0/1006036585 conn(0x55975f5d3240 msgr2=0x55975f5d3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.839+0000 7f1098c2a740 1 -- 172.21.15.102:0/1006036585 shutdown_connections 2024-04-25T01:55:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.839+0000 7f1098c2a740 1 -- 172.21.15.102:0/1006036585 wait complete. 2024-04-25T01:55:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.840+0000 7f1098c2a740 1 -- start start 2024-04-25T01:55:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.840+0000 7f1098c2a740 1 Processor -- start 2024-04-25T01:55:23.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.841+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5def50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.841+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5df490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.841+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5e1f10 0x55975f5df9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.841+0000 7f108ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5def50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.842+0000 7f108ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5def50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55982/0 (socket says 172.21.15.102:55982) 2024-04-25T01:55:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.842+0000 7f108ea00640 1 -- 172.21.15.102:0/3039299259 learned_addr learned my addr 172.21.15.102:0/3039299259 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.841+0000 7f108e000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5df490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.842+0000 7f108f400640 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5e1f10 0x55975f5df9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.842+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55975f4f6e70 con 0x55975f5e1f10 2024-04-25T01:55:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.842+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55975f537150 con 0x55975f5d7840 2024-04-25T01:55:23.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.842+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55975f5e4da0 con 0x55975f5d6f20 2024-04-25T01:55:23.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.842+0000 7f1098c2a740 10 client.0 _ll_register_callbacks cb 0x55975f5321f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.843+0000 7f108ea00640 1 -- 172.21.15.102:0/3039299259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 msgr2=0x55975f5df490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.843+0000 7f108ea00640 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5df490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.843+0000 7f108ea00640 1 -- 172.21.15.102:0/3039299259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5e1f10 msgr2=0x55975f5df9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.843+0000 7f108ea00640 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5e1f10 0x55975f5df9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.843+0000 7f108ea00640 1 -- 172.21.15.102:0/3039299259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55975f377f70 con 0x55975f5d6f20 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.843+0000 7f108e000640 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5df490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.843+0000 7f108f400640 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5e1f10 0x55975f5df9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.843+0000 7f108ea00640 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5def50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1080000c00 tx=0x7f1080005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4a9961080d298c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.843+0000 7f1078c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.844+0000 7f1078c00640 1 -- 172.21.15.102:0/3039299259 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080049070 con 0x55975f5d6f20 2024-04-25T01:55:23.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.844+0000 7f1078c00640 1 -- 172.21.15.102:0/3039299259 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f108003bcc0 con 0x55975f5d6f20 2024-04-25T01:55:23.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.844+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f108801a8f0 con 0x55975f5d6f20 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.844+0000 7f1078c00640 1 -- 172.21.15.102:0/3039299259 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f10800bbd70 con 0x55975f5d6f20 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.844+0000 7f1098c2a740 10 client.25384 fetch_fsmap learned FSMap version 23 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.844+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55975f20f170 con 0x55975f5d6f20 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.844+0000 7f1078c00640 1 -- 172.21.15.102:0/3039299259 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f108801a8f0 con 0x55975f5d6f20 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.845+0000 7f1078c00640 1 -- 172.21.15.102:0/3039299259 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f10800ed550 con 0x55975f5d6f20 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.845+0000 7f1098c2a740 10 client.25384 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.845+0000 7f1098c2a740 10 client.25384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.845+0000 7f1098c2a740 20 client.25384 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.845+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55975f5f22b0 0x55975f5f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.845+0000 7f1098c2a740 4 client.25384 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.845+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55975f5486d0 con 0x55975f5f22b0 2024-04-25T01:55:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.845+0000 7f108e000640 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55975f5f22b0 0x55975f5f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.846+0000 7f108e000640 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55975f5f22b0 0x55975f5f46a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c916a3b6b6551502 server_cookie=56eae2b551e9c8b8 in_seq=0 out_seq=0 2024-04-25T01:55:23.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.846+0000 7f1078c00640 10 client.25384 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.846+0000 7f1078c00640 1 -- 172.21.15.102:0/3039299259 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55975f5486d0 con 0x55975f5f22b0 2024-04-25T01:55:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.846+0000 7f1078c00640 10 client.25384 handle_command_reply: tid=0 2024-04-25T01:55:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.846+0000 7f1098c2a740 1 client.25384 shutdown 2024-04-25T01:55:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 msgr2=0x55975f5def50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5def50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1080000c00 tx=0x7f1080005e50 comp rx=0 tx=0).stop 2024-04-25T01:55:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 shutdown_connections 2024-04-25T01:55:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55975f5f22b0 0x55975f5f46a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5e1f10 0x55975f5df9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5df490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3039299259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5def50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1078c00640 0 client.25384 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 >> 172.21.15.102:0/3039299259 conn(0x55975f5d3240 msgr2=0x55975f5d9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 shutdown_connections 2024-04-25T01:55:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 1 -- 172.21.15.102:0/3039299259 wait complete. 2024-04-25T01:55:23.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.847+0000 7f1098c2a740 20 client.25384 trim_cache size 0 max 16384 2024-04-25T01:55:23.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f1098c2a740 1 Processor -- start 2024-04-25T01:55:23.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f1098c2a740 1 -- start start 2024-04-25T01:55:23.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5edb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d7840 0x55975f5df7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5e1f10 0x55975f5f9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f1098c2a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f10901be840 con 0x55975f5d7840 2024-04-25T01:55:23.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f1098c2a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f10901bc110 con 0x55975f5e1f10 2024-04-25T01:55:23.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f1098c2a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f10901b9f60 con 0x55975f5d6f20 2024-04-25T01:55:23.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f108e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d7840 0x55975f5df7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f108e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d7840 0x55975f5df7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53088/0 (socket says 172.21.15.102:53088) 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f108e000640 1 -- 172.21.15.102:0/3766977492 learned_addr learned my addr 172.21.15.102:0/3766977492 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f108ea00640 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5edb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.866+0000 7f108f400640 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5e1f10 0x55975f5f9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f108e000640 1 -- 172.21.15.102:0/3766977492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 msgr2=0x55975f5edb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f108e000640 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5edb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f108e000640 1 -- 172.21.15.102:0/3766977492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5e1f10 msgr2=0x55975f5f9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f108e000640 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5e1f10 0x55975f5f9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f108e000640 1 -- 172.21.15.102:0/3766977492 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55975f5dd4a0 con 0x55975f5d7840 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f108f400640 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5e1f10 0x55975f5f9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f108e000640 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d7840 0x55975f5df7f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55975f5d43d0 tx=0x7f107c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58a4e0c9fedac1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f107b400640 1 -- 172.21.15.102:0/3766977492 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c012110 con 0x55975f5d7840 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f107b400640 1 -- 172.21.15.102:0/3766977492 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f107c027070 con 0x55975f5d7840 2024-04-25T01:55:23.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.867+0000 7f107b400640 1 -- 172.21.15.102:0/3766977492 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c003a10 con 0x55975f5d7840 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.868+0000 7f1098c2a740 1 -- 172.21.15.102:0/3766977492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d7840 msgr2=0x55975f5df7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.868+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d7840 0x55975f5df7f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x55975f5d43d0 tx=0x7f107c005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.868+0000 7f1098c2a740 1 -- 172.21.15.102:0/3766977492 shutdown_connections 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.868+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5e1f10 0x55975f5f9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.868+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d7840 0x55975f5df7f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.868+0000 7f1098c2a740 1 --2- 172.21.15.102:0/3766977492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5d6f20 0x55975f5edb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.868+0000 7f1098c2a740 1 -- 172.21.15.102:0/3766977492 >> 172.21.15.102:0/3766977492 conn(0x55975f5d3240 msgr2=0x55975f5d8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.868+0000 7f1098c2a740 1 -- 172.21.15.102:0/3766977492 shutdown_connections 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.868+0000 7f1098c2a740 1 -- 172.21.15.102:0/3766977492 wait complete. 2024-04-25T01:55:23.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.869+0000 7f1098c2a740 1 -- start start 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.869+0000 7f1098c2a740 1 Processor -- start 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.869+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5dbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.869+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5dc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.869+0000 7f1098c2a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 0x55975f5dc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.869+0000 7f1098c2a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f10901ba5d0 con 0x55975f5d6f20 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.869+0000 7f1098c2a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f10901bba40 con 0x55975f5d7840 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.869+0000 7f1098c2a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f10901baca0 con 0x55975f5e1f10 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.869+0000 7f1098c2a740 10 client.0 _ll_register_callbacks cb 0x7f10800ed130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5dbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5dbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53096/0 (socket says 172.21.15.102:53096) 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108ea00640 1 -- 172.21.15.102:0/2313465076 learned_addr learned my addr 172.21.15.102:0/2313465076 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108e000640 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108f400640 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 0x55975f5dc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108ea00640 1 -- 172.21.15.102:0/2313465076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 msgr2=0x55975f5dc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108ea00640 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 0x55975f5dc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108ea00640 1 -- 172.21.15.102:0/2313465076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 msgr2=0x55975f5dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108ea00640 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108ea00640 1 -- 172.21.15.102:0/2313465076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55975f657a60 con 0x55975f5d6f20 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.870+0000 7f108e000640 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.871+0000 7f108ea00640 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5dbee0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f10800dc0b0 tx=0x7f108003bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1eb8582d385cb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.871+0000 7f1078c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.871+0000 7f1078c00640 1 -- 172.21.15.102:0/2313465076 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080049030 con 0x55975f5d6f20 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.871+0000 7f1078c00640 1 -- 172.21.15.102:0/2313465076 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f10800ed550 con 0x55975f5d6f20 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.871+0000 7f1098c2a740 1 -- 172.21.15.102:0/2313465076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f107c027070 con 0x55975f5d6f20 2024-04-25T01:55:23.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.871+0000 7f1078c00640 1 -- 172.21.15.102:0/2313465076 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f10800bbb90 con 0x55975f5d6f20 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.871+0000 7f1098c2a740 10 client.15312 fetch_fsmap learned FSMap version 23 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.871+0000 7f1098c2a740 1 -- 172.21.15.102:0/2313465076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55975f658da0 con 0x55975f5d6f20 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.871+0000 7f1078c00640 1 -- 172.21.15.102:0/2313465076 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f107c027070 con 0x55975f5d6f20 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.872+0000 7f1078c00640 1 -- 172.21.15.102:0/2313465076 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f10800d9030 con 0x55975f5d6f20 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.872+0000 7f1098c2a740 10 client.15312 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.872+0000 7f1098c2a740 10 client.15312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.872+0000 7f1098c2a740 20 client.15312 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.872+0000 7f1098c2a740 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55975f659380 0x55975f65b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.872+0000 7f1098c2a740 4 client.15312 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.872+0000 7f1098c2a740 1 -- 172.21.15.102:0/2313465076 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55975f542a20 con 0x55975f659380 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.872+0000 7f108e000640 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55975f659380 0x55975f65b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.873+0000 7f108e000640 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55975f659380 0x55975f65b770 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61292862a591bfe1 server_cookie=6046dac1966deb6f in_seq=0 out_seq=0 2024-04-25T01:55:23.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.873+0000 7f1078c00640 10 client.15312 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:23.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1078c00640 1 -- 172.21.15.102:0/2313465076 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55975f542a20 con 0x55975f659380 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1078c00640 10 client.15312 handle_command_reply: tid=0 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 client.15312 shutdown 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 -- 172.21.15.102:0/2313465076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 msgr2=0x55975f5dbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5dbee0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f10800dc0b0 tx=0x7f108003bef0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 -- 172.21.15.102:0/2313465076 shutdown_connections 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55975f659380 0x55975f65b770 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55975f5e1f10 0x55975f5dc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55975f5d7840 0x55975f5dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 --2- 172.21.15.102:0/2313465076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55975f5d6f20 0x55975f5dbee0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 -- 172.21.15.102:0/2313465076 >> 172.21.15.102:0/2313465076 conn(0x55975f5d3240 msgr2=0x55975f5e1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1078c00640 0 client.15312 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.874+0000 7f1098c2a740 1 -- 172.21.15.102:0/2313465076 shutdown_connections 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f1098c2a740 1 -- 172.21.15.102:0/2313465076 wait complete. 2024-04-25T01:55:23.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f1098c2a740 20 client.15312 trim_cache size 0 max 16384 2024-04-25T01:55:23.880 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:55:23.880 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T01:55:23.880 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "19994714-dfdc-4300-87bb-df084dcb4d07", 2024-04-25T01:55:23.880 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T01:55:23.880 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 -- 172.21.15.102:0/2637142788 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f10740a2e10 msgr2=0x7f10740a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f10740a2e10 0x7f10740a52d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1088007fd0 tx=0x7f108800a000 comp rx=0 tx=0).stop 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 -- 172.21.15.102:0/2637142788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10901037f0 msgr2=0x7f1090110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10901037f0 0x7f1090110560 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f108002a940 tx=0x7f1080003b90 comp rx=0 tx=0).stop 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 -- 172.21.15.102:0/2637142788 shutdown_connections 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f10740a2e10 0x7f10740a52d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1090109800 0x7f109010db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10901041b0 0x7f109010d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 --2- 172.21.15.102:0/2637142788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10901037f0 0x7f1090110560 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.875+0000 7f108fe00640 1 -- 172.21.15.102:0/2637142788 >> 172.21.15.102:0/2637142788 conn(0x7f10900ff280 msgr2=0x7f10900ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.876+0000 7f108fe00640 1 -- 172.21.15.102:0/2637142788 shutdown_connections 2024-04-25T01:55:23.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:23.876+0000 7f108fe00640 1 -- 172.21.15.102:0/2637142788 wait complete. 2024-04-25T01:55:23.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:23 smithi102 ceph-mon[27796]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:55:23.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:23 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/4193009392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:23.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:23 smithi102 ceph-mon[27796]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:55:23.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:23 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3114810136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:23.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:23 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:23 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.892 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:23 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:23.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:55:23.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:55:24.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.065+0000 7faf4be00640 1 -- 172.21.15.102:0/2691081721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf44069d40 msgr2=0x7faf4406a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.065+0000 7faf4be00640 1 --2- 172.21.15.102:0/2691081721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf44069d40 0x7faf4406a1c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faf30009930 tx=0x7faf3002a420 comp rx=0 tx=0).stop 2024-04-25T01:55:24.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.066+0000 7faf4be00640 1 -- 172.21.15.102:0/2691081721 shutdown_connections 2024-04-25T01:55:24.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.066+0000 7faf4be00640 1 --2- 172.21.15.102:0/2691081721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf4406a700 0x7faf4410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.066+0000 7faf4be00640 1 --2- 172.21.15.102:0/2691081721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf44069d40 0x7faf4406a1c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.066+0000 7faf4be00640 1 --2- 172.21.15.102:0/2691081721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf44107720 0x7faf44107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.066+0000 7faf4be00640 1 -- 172.21.15.102:0/2691081721 >> 172.21.15.102:0/2691081721 conn(0x7faf44076440 msgr2=0x7faf44076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.066+0000 7faf4be00640 1 -- 172.21.15.102:0/2691081721 shutdown_connections 2024-04-25T01:55:24.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.066+0000 7faf4be00640 1 -- 172.21.15.102:0/2691081721 wait complete. 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.067+0000 7faf4be00640 1 Processor -- start 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.067+0000 7faf4be00640 1 -- start start 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.067+0000 7faf4be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf44069d40 0x7faf4419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.067+0000 7faf4be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf4406a700 0x7faf4419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.067+0000 7faf4be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf44107720 0x7faf4419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.067+0000 7faf4be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faf44111b30 con 0x7faf4406a700 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.067+0000 7faf4be00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf441119b0 con 0x7faf44069d40 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.067+0000 7faf4be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faf44111cb0 con 0x7faf44107720 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.067+0000 7faf48a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf4406a700 0x7faf4419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf49e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf44107720 0x7faf4419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf48a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf4406a700 0x7faf4419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53104/0 (socket says 172.21.15.102:53104) 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf48a00640 1 -- 172.21.15.102:0/3350358525 learned_addr learned my addr 172.21.15.102:0/3350358525 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf49400640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf44069d40 0x7faf4419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf49e00640 1 -- 172.21.15.102:0/3350358525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf44069d40 msgr2=0x7faf4419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf49e00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf44069d40 0x7faf4419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf49e00640 1 -- 172.21.15.102:0/3350358525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf4406a700 msgr2=0x7faf4419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf49e00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf4406a700 0x7faf4419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf49e00640 1 -- 172.21.15.102:0/3350358525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4419f240 con 0x7faf44107720 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.068+0000 7faf49e00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf44107720 0x7faf4419eb10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faf3c00da10 tx=0x7faf3c00ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.069+0000 7faf42000640 1 -- 172.21.15.102:0/3350358525 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c00bd70 con 0x7faf44107720 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.069+0000 7faf4be00640 1 -- 172.21.15.102:0/3350358525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf4419f530 con 0x7faf44107720 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.069+0000 7faf42000640 1 -- 172.21.15.102:0/3350358525 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf3c003e50 con 0x7faf44107720 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.069+0000 7faf42000640 1 -- 172.21.15.102:0/3350358525 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c0106a0 con 0x7faf44107720 2024-04-25T01:55:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.069+0000 7faf4be00640 1 -- 172.21.15.102:0/3350358525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf441a6dc0 con 0x7faf44107720 2024-04-25T01:55:24.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.071+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 --> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fecc4061b80 con 0x7fecac0a2e00 2024-04-25T01:55:24.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.072+0000 7faf4edbb740 1 Processor -- start 2024-04-25T01:55:24.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.073+0000 7faf42000640 1 -- 172.21.15.102:0/3350358525 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7faf3c003970 con 0x7faf44107720 2024-04-25T01:55:24.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.073+0000 7faf4edbb740 1 -- start start 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.073+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a453e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.073+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a4fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.073+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 0x5618a7a523e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.073+0000 7faf4edbb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5618a79a9560 con 0x5618a7a4fff0 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.073+0000 7faf4edbb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5618a7951930 con 0x5618a7a45000 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.073+0000 7faf4edbb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5618a79a5520 con 0x5618a7a45920 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf49400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a4fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf49400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a4fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56008/0 (socket says 172.21.15.102:56008) 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf49400640 1 -- 172.21.15.102:0/2253043789 learned_addr learned my addr 172.21.15.102:0/2253043789 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf48a00640 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 0x5618a7a523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf48a00640 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf49400640 1 -- 172.21.15.102:0/2253043789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 msgr2=0x5618a7a453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:24.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf49400640 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf49400640 1 -- 172.21.15.102:0/2253043789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 msgr2=0x5618a7a523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf49400640 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 0x5618a7a523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf49400640 1 -- 172.21.15.102:0/2253043789 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618a77877d0 con 0x5618a7a45920 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf48a00640 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf49400640 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a4fab0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf38009c70 tx=0x7faf38011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f84f122a3d2ad0a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf42000640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf2c0a2d90 0x7faf2c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.074+0000 7faf36600640 1 -- 172.21.15.102:0/2253043789 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf38004230 con 0x5618a7a45920 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.075+0000 7faf42000640 1 -- 172.21.15.102:0/3350358525 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7faf3c0cf970 con 0x7faf44107720 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.075+0000 7faf36600640 1 -- 172.21.15.102:0/2253043789 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf380043d0 con 0x5618a7a45920 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.075+0000 7faf36600640 1 -- 172.21.15.102:0/2253043789 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf38019800 con 0x5618a7a45920 2024-04-25T01:55:24.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.075+0000 7faf48a00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf2c0a2d90 0x7faf2c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf4edbb740 1 -- 172.21.15.102:0/2253043789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 msgr2=0x5618a7a4fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf4edbb740 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a4fab0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf38009c70 tx=0x7faf38011810 comp rx=0 tx=0).stop 2024-04-25T01:55:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf48a00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf2c0a2d90 0x7faf2c0a5250 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faf4410e070 tx=0x7faf3002a420 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf4edbb740 1 -- 172.21.15.102:0/2253043789 shutdown_connections 2024-04-25T01:55:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf4edbb740 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 0x5618a7a523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf4edbb740 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a4fab0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf4edbb740 1 --2- 172.21.15.102:0/2253043789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf4edbb740 1 -- 172.21.15.102:0/2253043789 >> 172.21.15.102:0/2253043789 conn(0x5618a7a41320 msgr2=0x5618a7a41700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf4edbb740 1 -- 172.21.15.102:0/2253043789 shutdown_connections 2024-04-25T01:55:24.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.076+0000 7faf4edbb740 1 -- 172.21.15.102:0/2253043789 wait complete. 2024-04-25T01:55:24.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.077+0000 7faf4edbb740 1 -- start start 2024-04-25T01:55:24.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.077+0000 7faf4edbb740 1 Processor -- start 2024-04-25T01:55:24.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.077+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45000 0x5618a7a4cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45920 0x5618a7a4d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a4d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf4edbb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5618a7a52de0 con 0x5618a7a45920 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf4edbb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5618a7964d90 con 0x5618a7a4fff0 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf4edbb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5618a7a52f60 con 0x5618a7a45000 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf4edbb740 10 client.0 _ll_register_callbacks cb 0x5618a7964780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf48a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45920 0x5618a7a4d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf49400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45000 0x5618a7a4cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf48a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45920 0x5618a7a4d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53112/0 (socket says 172.21.15.102:53112) 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf48a00640 1 -- 172.21.15.102:0/1279154686 learned_addr learned my addr 172.21.15.102:0/1279154686 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf49e00640 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a4d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf49400640 1 -- 172.21.15.102:0/1279154686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 msgr2=0x5618a7a4d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf49400640 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a4d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf49400640 1 -- 172.21.15.102:0/1279154686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45920 msgr2=0x5618a7a4d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf49400640 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45920 0x5618a7a4d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf49400640 1 -- 172.21.15.102:0/1279154686 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618a76bd3f0 con 0x5618a7a45000 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.078+0000 7faf49e00640 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a4d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.079+0000 7faf49400640 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45000 0x5618a7a4cf30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faf38009c70 tx=0x7faf380052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6c1ad94efd234b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.079+0000 7faf2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.079+0000 7faf2aa00640 1 -- 172.21.15.102:0/1279154686 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf38027070 con 0x5618a7a45000 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.079+0000 7faf2aa00640 1 -- 172.21.15.102:0/1279154686 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7faf380037f0 con 0x5618a7a45000 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.079+0000 7faf4edbb740 1 -- 172.21.15.102:0/1279154686 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf380043d0 con 0x5618a7a45000 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.079+0000 7faf2aa00640 1 -- 172.21.15.102:0/1279154686 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7faf38033050 con 0x5618a7a45000 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.079+0000 7faf4edbb740 10 client.25399 fetch_fsmap learned FSMap version 23 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.079+0000 7faf4edbb740 1 -- 172.21.15.102:0/1279154686 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5618a767d0f0 con 0x5618a7a45000 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.080+0000 7faf2aa00640 1 -- 172.21.15.102:0/1279154686 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf380043d0 con 0x5618a7a45000 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.080+0000 7faf2aa00640 1 -- 172.21.15.102:0/1279154686 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7faf38004ba0 con 0x5618a7a45000 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.080+0000 7faf4edbb740 10 client.25399 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.080+0000 7faf4edbb740 10 client.25399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.080+0000 7faf4edbb740 20 client.25399 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.080+0000 7faf4edbb740 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5618a7a689f0 0x5618a7a6ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.080+0000 7faf4edbb740 4 client.25399 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:24.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.080+0000 7faf4edbb740 1 -- 172.21.15.102:0/1279154686 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618a79bcd90 con 0x5618a7a689f0 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.081+0000 7faf48a00640 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5618a7a689f0 0x5618a7a6ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.081+0000 7faf48a00640 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5618a7a689f0 0x5618a7a6ade0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c27850cf4824a0cd server_cookie=a1e66b81b5a10b31 in_seq=0 out_seq=0 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.081+0000 7faf2aa00640 10 client.25399 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.082+0000 7faf2aa00640 1 -- 172.21.15.102:0/1279154686 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5618a79bcd90 con 0x5618a7a689f0 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.082+0000 7faf2aa00640 10 client.25399 handle_command_reply: tid=0 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.082+0000 7faf4edbb740 1 client.25399 shutdown 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 -- 172.21.15.102:0/1279154686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45000 msgr2=0x5618a7a4cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45000 0x5618a7a4cf30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faf38009c70 tx=0x7faf380052f0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 -- 172.21.15.102:0/1279154686 shutdown_connections 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5618a7a689f0 0x5618a7a6ade0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a4d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45920 0x5618a7a4d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 --2- 172.21.15.102:0/1279154686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45000 0x5618a7a4cf30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 -- 172.21.15.102:0/1279154686 >> 172.21.15.102:0/1279154686 conn(0x5618a7a41320 msgr2=0x5618a7a47120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf2aa00640 0 client.25399 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 -- 172.21.15.102:0/1279154686 shutdown_connections 2024-04-25T01:55:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 1 -- 172.21.15.102:0/1279154686 wait complete. 2024-04-25T01:55:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.083+0000 7faf4edbb740 20 client.25399 trim_cache size 0 max 16384 2024-04-25T01:55:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf4edbb740 1 Processor -- start 2024-04-25T01:55:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf4edbb740 1 -- start start 2024-04-25T01:55:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45000 0x5618a7a64530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a40300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a70610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf4edbb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faf441adc30 con 0x5618a7a45000 2024-04-25T01:55:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf4edbb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf441ab500 con 0x5618a7a4fff0 2024-04-25T01:55:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf4edbb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faf441a9390 con 0x5618a7a45920 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf49400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45000 0x5618a7a64530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf49e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a70610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf49e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a70610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33336/0 (socket says 172.21.15.102:33336) 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf49e00640 1 -- 172.21.15.102:0/2927190528 learned_addr learned my addr 172.21.15.102:0/2927190528 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf49400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45000 0x5618a7a64530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53126/0 (socket says 172.21.15.102:53126) 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.109+0000 7faf48a00640 1 --2- 172.21.15.102:0/2927190528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a40300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.110+0000 7faf49e00640 1 -- 172.21.15.102:0/2927190528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 msgr2=0x5618a7a40300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.110+0000 7faf49e00640 1 --2- 172.21.15.102:0/2927190528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a40300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.110+0000 7faf49e00640 1 -- 172.21.15.102:0/2927190528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45000 msgr2=0x5618a7a64530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:24.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.110+0000 7faf49e00640 1 --2- 172.21.15.102:0/2927190528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45000 0x5618a7a64530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.110+0000 7faf49e00640 1 -- 172.21.15.102:0/2927190528 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618a7a45750 con 0x5618a7a4fff0 2024-04-25T01:55:24.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.110+0000 7faf49e00640 1 --2- 172.21.15.102:0/2927190528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a70610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0981b0 tx=0x7faf3c097ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64bec612be82686e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.111+0000 7faf35c00640 1 -- 172.21.15.102:0/2927190528 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c0a0100 con 0x5618a7a4fff0 2024-04-25T01:55:24.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.111+0000 7faf35c00640 1 -- 172.21.15.102:0/2927190528 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf3c0a49c0 con 0x5618a7a4fff0 2024-04-25T01:55:24.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.111+0000 7faf35c00640 1 -- 172.21.15.102:0/2927190528 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3c0cd2e0 con 0x5618a7a4fff0 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.112+0000 7faf4edbb740 1 -- 172.21.15.102:0/2927190528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 msgr2=0x5618a7a70610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.112+0000 7faf4edbb740 1 --2- 172.21.15.102:0/2927190528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a70610 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0981b0 tx=0x7faf3c097ff0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.112+0000 7faf4edbb740 1 -- 172.21.15.102:0/2927190528 shutdown_connections 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.112+0000 7faf4edbb740 1 --2- 172.21.15.102:0/2927190528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a4fff0 0x5618a7a70610 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.112+0000 7faf4edbb740 1 --2- 172.21.15.102:0/2927190528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a40300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.112+0000 7faf4edbb740 1 --2- 172.21.15.102:0/2927190528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a45000 0x5618a7a64530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.112+0000 7faf4edbb740 1 -- 172.21.15.102:0/2927190528 >> 172.21.15.102:0/2927190528 conn(0x5618a7a41320 msgr2=0x5618a7a3fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.112+0000 7faf4edbb740 1 -- 172.21.15.102:0/2927190528 shutdown_connections 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.112+0000 7faf4edbb740 1 -- 172.21.15.102:0/2927190528 wait complete. 2024-04-25T01:55:24.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.113+0000 7faf4edbb740 1 -- start start 2024-04-25T01:55:24.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.113+0000 7faf4edbb740 1 Processor -- start 2024-04-25T01:55:24.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.113+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a60de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.113+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a49880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.113+0000 7faf4edbb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 0x5618a7a49dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.113+0000 7faf4edbb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faf441a99e0 con 0x5618a7a4fff0 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.113+0000 7faf4edbb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf441aae30 con 0x5618a7a45000 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.113+0000 7faf4edbb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faf441aa090 con 0x5618a7a45920 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.113+0000 7faf4edbb740 10 client.0 _ll_register_callbacks cb 0x7faf38004280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf48a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a49880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf49400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a60de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf48a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a49880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56020/0 (socket says 172.21.15.102:56020) 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf48a00640 1 -- 172.21.15.102:0/4273751047 learned_addr learned my addr 172.21.15.102:0/4273751047 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf49e00640 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 0x5618a7a49dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf49400640 1 -- 172.21.15.102:0/4273751047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 msgr2=0x5618a7a49880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf49400640 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a49880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf49400640 1 -- 172.21.15.102:0/4273751047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 msgr2=0x5618a7a49dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:24.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf49400640 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 0x5618a7a49dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf49400640 1 -- 172.21.15.102:0/4273751047 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618a7a4a5a0 con 0x5618a7a45000 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf48a00640 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a49880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf49400640 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a60de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf38001eb0 tx=0x7faf38005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d873531994df8062 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.114+0000 7faf2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.115+0000 7faf2aa00640 1 -- 172.21.15.102:0/4273751047 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf38027070 con 0x5618a7a45000 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.115+0000 7faf2aa00640 1 -- 172.21.15.102:0/4273751047 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7faf380097a0 con 0x5618a7a45000 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.115+0000 7faf4edbb740 1 -- 172.21.15.102:0/4273751047 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf3c0a49c0 con 0x5618a7a45000 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.115+0000 7faf2aa00640 1 -- 172.21.15.102:0/4273751047 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7faf38019bb0 con 0x5618a7a45000 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.115+0000 7faf4edbb740 10 client.25415 fetch_fsmap learned FSMap version 23 2024-04-25T01:55:24.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.115+0000 7faf4edbb740 1 -- 172.21.15.102:0/4273751047 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5618a7acede0 con 0x5618a7a45000 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.116+0000 7faf2aa00640 1 -- 172.21.15.102:0/4273751047 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf3c0a49c0 con 0x5618a7a45000 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.116+0000 7faf2aa00640 1 -- 172.21.15.102:0/4273751047 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7faf38004b60 con 0x5618a7a45000 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.116+0000 7faf4edbb740 10 client.25415 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.116+0000 7faf4edbb740 10 client.25415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.116+0000 7faf4edbb740 20 client.25415 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.116+0000 7faf4edbb740 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5618a7acf550 0x5618a7ad1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.116+0000 7faf4edbb740 4 client.25415 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.116+0000 7faf4edbb740 1 -- 172.21.15.102:0/4273751047 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5618a79b6aa0 con 0x5618a7acf550 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.117+0000 7faf48a00640 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5618a7acf550 0x5618a7ad1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.117+0000 7faf2aa00640 10 client.25415 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.117+0000 7faf48a00640 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5618a7acf550 0x5618a7ad1940 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f83a414e10459b4d server_cookie=f35278d509afb84b in_seq=0 out_seq=0 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf2aa00640 1 -- 172.21.15.102:0/4273751047 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5618a79b6aa0 con 0x5618a7acf550 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf2aa00640 10 client.25415 handle_command_reply: tid=0 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 client.25415 shutdown 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 -- 172.21.15.102:0/4273751047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 msgr2=0x5618a7a60de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a60de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf38001eb0 tx=0x7faf38005d30 comp rx=0 tx=0).stop 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 -- 172.21.15.102:0/4273751047 shutdown_connections 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5618a7acf550 0x5618a7ad1940 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5618a7a4fff0 0x5618a7a49dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5618a7a45920 0x5618a7a49880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf2aa00640 0 client.25415 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 --2- 172.21.15.102:0/4273751047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5618a7a45000 0x5618a7a60de0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 -- 172.21.15.102:0/4273751047 >> 172.21.15.102:0/4273751047 conn(0x5618a7a41320 msgr2=0x5618a7a40480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 -- 172.21.15.102:0/4273751047 shutdown_connections 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.118+0000 7faf4edbb740 1 -- 172.21.15.102:0/4273751047 wait complete. 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.119+0000 7faf4edbb740 20 client.25415 trim_cache size 0 max 16384 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stdout: "19994714-dfdc-4300-87bb-df084dcb4d07": { 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "19994714-dfdc-4300-87bb-df084dcb4d07", 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T01:55:24.123 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:55:24.124 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:55:24.124 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:55:24.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.119+0000 7faf4be00640 1 -- 172.21.15.102:0/3350358525 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf2c0a2d90 msgr2=0x7faf2c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.119+0000 7faf4be00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf2c0a2d90 0x7faf2c0a5250 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faf4410e070 tx=0x7faf3002a420 comp rx=0 tx=0).stop 2024-04-25T01:55:24.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.119+0000 7faf4be00640 1 -- 172.21.15.102:0/3350358525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf44107720 msgr2=0x7faf4419eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.119+0000 7faf4be00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf44107720 0x7faf4419eb10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faf3c00da10 tx=0x7faf3c00ded0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.120+0000 7faf4be00640 1 -- 172.21.15.102:0/3350358525 shutdown_connections 2024-04-25T01:55:24.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.120+0000 7faf4be00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf2c0a2d90 0x7faf2c0a5250 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.120+0000 7faf4be00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf44107720 0x7faf4419eb10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.120+0000 7faf4be00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf4406a700 0x7faf4419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.120+0000 7faf4be00640 1 --2- 172.21.15.102:0/3350358525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf44069d40 0x7faf4419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.120+0000 7faf4be00640 1 -- 172.21.15.102:0/3350358525 >> 172.21.15.102:0/3350358525 conn(0x7faf44076440 msgr2=0x7faf44101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.120+0000 7faf4be00640 1 -- 172.21.15.102:0/3350358525 shutdown_connections 2024-04-25T01:55:24.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.120+0000 7faf4be00640 1 -- 172.21.15.102:0/3350358525 wait complete. 2024-04-25T01:55:24.141 INFO:tasks.cephfs.filesystem:scrub status for tag:19994714-dfdc-4300-87bb-df084dcb4d07 - {'path': '/', 'tag': '19994714-dfdc-4300-87bb-df084dcb4d07', 'options': 'recursive,force'} 2024-04-25T01:55:24.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.156+0000 7fecc2000640 1 -- 172.21.15.102:0/114483111 <== mgr.25214 v2:172.21.15.102:6800/855543702 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fecc4061b80 con 0x7fecac0a2e00 2024-04-25T01:55:24.159 INFO:teuthology.orchestra.run.smithi102.stdout:Schedule set for path / 2024-04-25T01:55:24.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 msgr2=0x7fecac0a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fecc41018a0 tx=0x7fecb000b390 comp rx=0 tx=0).stop 2024-04-25T01:55:24.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc4103770 msgr2=0x7fecc419a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc4103770 0x7fecc419a0e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fecb802a940 tx=0x7fecb8003b90 comp rx=0 tx=0).stop 2024-04-25T01:55:24.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 shutdown_connections 2024-04-25T01:55:24.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc4109780 0x7fecc419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc4104130 0x7fecc419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 --2- 172.21.15.102:0/114483111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc4103770 0x7fecc419a0e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 >> 172.21.15.102:0/114483111 conn(0x7fecc40ff220 msgr2=0x7fecc40ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 shutdown_connections 2024-04-25T01:55:24.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.159+0000 7feccaa00640 1 -- 172.21.15.102:0/114483111 wait complete. 2024-04-25T01:55:24.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-25T01:55:24.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:23 smithi110 ceph-mon[32848]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:55:24.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:23 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/4193009392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:24.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:23 smithi110 ceph-mon[32848]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:55:24.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:23 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3114810136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:24.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:23 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:24.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:23 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:24.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:23 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:24.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:23 smithi193 ceph-mon[32981]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-04-25T01:55:24.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:23 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/4193009392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:24.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:23 smithi193 ceph-mon[32981]: Updating smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T01:55:24.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:23 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3114810136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T01:55:24.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:23 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:24.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:23 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:24.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:23 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:24.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.391+0000 7f5642400640 1 -- 172.21.15.102:0/950416481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f563c103780 msgr2=0x7f563c103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.391+0000 7f5642400640 1 --2- 172.21.15.102:0/950416481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f563c103780 0x7f563c103c00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5628009930 tx=0x7f562802a440 comp rx=0 tx=0).stop 2024-04-25T01:55:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.391+0000 7f5642400640 1 -- 172.21.15.102:0/950416481 shutdown_connections 2024-04-25T01:55:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.391+0000 7f5642400640 1 --2- 172.21.15.102:0/950416481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f563c104140 0x7f563c110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.391+0000 7f5642400640 1 --2- 172.21.15.102:0/950416481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f563c103780 0x7f563c103c00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.391+0000 7f5642400640 1 --2- 172.21.15.102:0/950416481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f563c109790 0x7f563c109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.391+0000 7f5642400640 1 -- 172.21.15.102:0/950416481 >> 172.21.15.102:0/950416481 conn(0x7f563c0ff250 msgr2=0x7f563c101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.391+0000 7f5642400640 1 -- 172.21.15.102:0/950416481 shutdown_connections 2024-04-25T01:55:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.391+0000 7f5642400640 1 -- 172.21.15.102:0/950416481 wait complete. 2024-04-25T01:55:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.392+0000 7f5642400640 1 Processor -- start 2024-04-25T01:55:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.392+0000 7f5642400640 1 -- start start 2024-04-25T01:55:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f5642400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f563c103780 0x7f563c19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f5642400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f563c104140 0x7f563c19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f5642400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f563c109790 0x7f563c19e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f5642400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f563c113ba0 con 0x7f563c104140 2024-04-25T01:55:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f5642400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f563c113a20 con 0x7f563c103780 2024-04-25T01:55:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f5642400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f563c113d20 con 0x7f563c109790 2024-04-25T01:55:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f563be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f563c104140 0x7f563c19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f5641000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f563c103780 0x7f563c19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f5641a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f563c109790 0x7f563c19e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f563be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f563c104140 0x7f563c19a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53144/0 (socket says 172.21.15.102:53144) 2024-04-25T01:55:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.393+0000 7f563be00640 1 -- 172.21.15.102:0/2268252594 learned_addr learned my addr 172.21.15.102:0/2268252594 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5641000640 1 -- 172.21.15.102:0/2268252594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f563c109790 msgr2=0x7f563c19e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5641000640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f563c109790 0x7f563c19e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5641000640 1 -- 172.21.15.102:0/2268252594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f563c104140 msgr2=0x7f563c19a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5641000640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f563c104140 0x7f563c19a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5641000640 1 -- 172.21.15.102:0/2268252594 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f563c19f070 con 0x7f563c103780 2024-04-25T01:55:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5641000640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f563c103780 0x7f563c19a080 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f563000e9f0 tx=0x7f563000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5639600640 1 -- 172.21.15.102:0/2268252594 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563000ccc0 con 0x7f563c103780 2024-04-25T01:55:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f563c19f360 con 0x7f563c103780 2024-04-25T01:55:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5639600640 1 -- 172.21.15.102:0/2268252594 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5630010910 con 0x7f563c103780 2024-04-25T01:55:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5639600640 1 -- 172.21.15.102:0/2268252594 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5630018960 con 0x7f563c103780 2024-04-25T01:55:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.394+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f563c1a6bf0 con 0x7f563c103780 2024-04-25T01:55:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.395+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f563c04fd50 con 0x7f563c103780 2024-04-25T01:55:24.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.397+0000 7f5639600640 1 -- 172.21.15.102:0/2268252594 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f5630010ac0 con 0x7f563c103780 2024-04-25T01:55:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.398+0000 7f5639600640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56240a2d80 0x7f56240a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.399+0000 7f5639600640 1 -- 172.21.15.102:0/2268252594 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f56300cf930 con 0x7f563c103780 2024-04-25T01:55:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.399+0000 7f563be00640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56240a2d80 0x7f56240a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.399+0000 7f563be00640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56240a2d80 0x7f56240a5240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f563c101830 tx=0x7f5628009560 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.403+0000 7f5639600640 1 -- 172.21.15.102:0/2268252594 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f563009bd60 con 0x7f563c103780 2024-04-25T01:55:24.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.616+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 --> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f563c061b80 con 0x7f56240a2d80 2024-04-25T01:55:24.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.702+0000 7f5639600640 1 -- 172.21.15.102:0/2268252594 <== mgr.25214 v2:172.21.15.102:6800/855543702 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f563c061b80 con 0x7f56240a2d80 2024-04-25T01:55:24.705 INFO:teuthology.orchestra.run.smithi102.stdout:Retention added to path / 2024-04-25T01:55:24.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56240a2d80 msgr2=0x7f56240a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56240a2d80 0x7f56240a5240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f563c101830 tx=0x7f5628009560 comp rx=0 tx=0).stop 2024-04-25T01:55:24.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f563c103780 msgr2=0x7f563c19a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f563c103780 0x7f563c19a080 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f563000e9f0 tx=0x7f563000eeb0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 shutdown_connections 2024-04-25T01:55:24.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56240a2d80 0x7f56240a5240 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f563c109790 0x7f563c19e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f563c104140 0x7f563c19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 --2- 172.21.15.102:0/2268252594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f563c103780 0x7f563c19a080 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 >> 172.21.15.102:0/2268252594 conn(0x7f563c0ff250 msgr2=0x7f563c0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 shutdown_connections 2024-04-25T01:55:24.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.705+0000 7f5642400640 1 -- 172.21.15.102:0/2268252594 wait complete. 2024-04-25T01:55:24.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T01:55:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.905+0000 7fd990a00640 1 -- 172.21.15.102:0/1382371185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd98c075c70 msgr2=0x7fd98c076050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.905+0000 7fd990a00640 1 --2- 172.21.15.102:0/1382371185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd98c075c70 0x7fd98c076050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd97c009930 tx=0x7fd97c02a440 comp rx=0 tx=0).stop 2024-04-25T01:55:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.905+0000 7fd990a00640 1 -- 172.21.15.102:0/1382371185 shutdown_connections 2024-04-25T01:55:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.905+0000 7fd990a00640 1 --2- 172.21.15.102:0/1382371185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd98c1125c0 0x7fd98c1149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.905+0000 7fd990a00640 1 --2- 172.21.15.102:0/1382371185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd98c076590 0x7fd98c112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.905+0000 7fd990a00640 1 --2- 172.21.15.102:0/1382371185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd98c075c70 0x7fd98c076050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.905+0000 7fd990a00640 1 -- 172.21.15.102:0/1382371185 >> 172.21.15.102:0/1382371185 conn(0x7fd98c0ff430 msgr2=0x7fd98c101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.905+0000 7fd990a00640 1 -- 172.21.15.102:0/1382371185 shutdown_connections 2024-04-25T01:55:24.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.905+0000 7fd990a00640 1 -- 172.21.15.102:0/1382371185 wait complete. 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.906+0000 7fd990a00640 1 Processor -- start 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd990a00640 1 -- start start 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd990a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd98c075c70 0x7fd98c19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd990a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd98c076590 0x7fd98c19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd990a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd98c1125c0 0x7fd98c1a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd990a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c118160 con 0x7fd98c1125c0 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd990a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c117fe0 con 0x7fd98c076590 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd990a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c1182e0 con 0x7fd98c075c70 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd989e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd98c075c70 0x7fd98c19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd98a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd98c1125c0 0x7fd98c1a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd989e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd98c075c70 0x7fd98c19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56060/0 (socket says 172.21.15.102:56060) 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd989e00640 1 -- 172.21.15.102:0/244884595 learned_addr learned my addr 172.21.15.102:0/244884595 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.907+0000 7fd981400640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd98c076590 0x7fd98c19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.908+0000 7fd98a800640 1 -- 172.21.15.102:0/244884595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd98c075c70 msgr2=0x7fd98c19e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.908+0000 7fd98a800640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd98c075c70 0x7fd98c19e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.908+0000 7fd98a800640 1 -- 172.21.15.102:0/244884595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd98c076590 msgr2=0x7fd98c19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.908+0000 7fd98a800640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd98c076590 0x7fd98c19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.908+0000 7fd98a800640 1 -- 172.21.15.102:0/244884595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c1a3590 con 0x7fd98c1125c0 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.908+0000 7fd981400640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd98c076590 0x7fd98c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.908+0000 7fd98a800640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd98c1125c0 0x7fd98c1a2e60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd984005fd0 tx=0x7fd98400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.909+0000 7fd983400640 1 -- 172.21.15.102:0/244884595 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984004d20 con 0x7fd98c1125c0 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.909+0000 7fd983400640 1 -- 172.21.15.102:0/244884595 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd984005020 con 0x7fd98c1125c0 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.909+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c1a3880 con 0x7fd98c1125c0 2024-04-25T01:55:24.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.909+0000 7fd983400640 1 -- 172.21.15.102:0/244884595 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9840116d0 con 0x7fd98c1125c0 2024-04-25T01:55:24.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.909+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c1ab110 con 0x7fd98c1125c0 2024-04-25T01:55:24.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.910+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98c04f500 con 0x7fd98c1125c0 2024-04-25T01:55:24.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.912+0000 7fd983400640 1 -- 172.21.15.102:0/244884595 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fd984005540 con 0x7fd98c1125c0 2024-04-25T01:55:24.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.913+0000 7fd983400640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd9700a2e40 0x7fd9700a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:24.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.913+0000 7fd983400640 1 -- 172.21.15.102:0/244884595 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd9840cf630 con 0x7fd98c1125c0 2024-04-25T01:55:24.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.913+0000 7fd989e00640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd9700a2e40 0x7fd9700a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:24.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.914+0000 7fd989e00640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd9700a2e40 0x7fd9700a5300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0096c0 tx=0x7fd97c005b50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:24.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:24.917+0000 7fd983400640 1 -- 172.21.15.102:0/244884595 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd984016030 con 0x7fd98c1125c0 2024-04-25T01:55:25.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.140+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 --> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fd98c061b80 con 0x7fd9700a2e40 2024-04-25T01:55:25.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.154+0000 7fd983400640 1 -- 172.21.15.102:0/244884595 <== mgr.25214 v2:172.21.15.102:6800/855543702 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fd98c061b80 con 0x7fd9700a2e40 2024-04-25T01:55:25.157 INFO:teuthology.orchestra.run.smithi102.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T01:55:24", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-25T01:55:25.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd9700a2e40 msgr2=0x7fd9700a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:25.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd9700a2e40 0x7fd9700a5300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0096c0 tx=0x7fd97c005b50 comp rx=0 tx=0).stop 2024-04-25T01:55:25.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd98c1125c0 msgr2=0x7fd98c1a2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:25.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd98c1125c0 0x7fd98c1a2e60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd984005fd0 tx=0x7fd98400edd0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 shutdown_connections 2024-04-25T01:55:25.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd9700a2e40 0x7fd9700a5300 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd98c1125c0 0x7fd98c1a2e60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd98c076590 0x7fd98c19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 --2- 172.21.15.102:0/244884595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd98c075c70 0x7fd98c19e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 >> 172.21.15.102:0/244884595 conn(0x7fd98c0ff430 msgr2=0x7fd98c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:25.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 shutdown_connections 2024-04-25T01:55:25.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.157+0000 7fd990a00640 1 -- 172.21.15.102:0/244884595 wait complete. 2024-04-25T01:55:25.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 0 B/s wr, 21 op/s 2024-04-25T01:55:25.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T01:55:25.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:55:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:55:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: from='client.25385 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:55:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:55:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:24 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:25.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T01:55:25.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 0 B/s wr, 21 op/s 2024-04-25T01:55:25.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: from='client.25385 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:55:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:24 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:25.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 0 B/s wr, 21 op/s 2024-04-25T01:55:25.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T01:55:25.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:25.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:55:25.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:25.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:55:25.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: from='client.25385 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:25.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:25.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:25.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:55:25.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:55:25.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:24 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:55:25.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.367+0000 7fc1d7c00640 1 -- 172.21.15.102:0/2368337002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1d0102840 msgr2=0x7fc1d00ffc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:25.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.367+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/2368337002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1d0102840 0x7fc1d00ffc40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0009900 tx=0x7fc1c002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.367+0000 7fc1d7c00640 1 -- 172.21.15.102:0/2368337002 shutdown_connections 2024-04-25T01:55:25.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.367+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/2368337002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d010e3c0 0x7fc1d01107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.367+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/2368337002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1d0100180 0x7fc1d01005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.367+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/2368337002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1d0102840 0x7fc1d00ffc40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.367+0000 7fc1d7c00640 1 -- 172.21.15.102:0/2368337002 >> 172.21.15.102:0/2368337002 conn(0x7fc1d0076bc0 msgr2=0x7fc1d00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:25.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.367+0000 7fc1d7c00640 1 -- 172.21.15.102:0/2368337002 shutdown_connections 2024-04-25T01:55:25.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.367+0000 7fc1d7c00640 1 -- 172.21.15.102:0/2368337002 wait complete. 2024-04-25T01:55:25.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.368+0000 7fc1d7c00640 1 Processor -- start 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.368+0000 7fc1d7c00640 1 -- start start 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d7c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1d0100180 0x7fc1d019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d7c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1d0102840 0x7fc1d019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d7c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d7c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0113f70 con 0x7fc1d0102840 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d7c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0113df0 con 0x7fc1d0100180 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d7c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d01140f0 con 0x7fc1d010e3c0 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1c7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1d0102840 0x7fc1d019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d5c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d5c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56078/0 (socket says 172.21.15.102:56078) 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d5c00640 1 -- 172.21.15.102:0/3760296743 learned_addr learned my addr 172.21.15.102:0/3760296743 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:25.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1c7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1d0102840 0x7fc1d019a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53170/0 (socket says 172.21.15.102:53170) 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d5200640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1d0100180 0x7fc1d019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d5c00640 1 -- 172.21.15.102:0/3760296743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1d0100180 msgr2=0x7fc1d019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d5c00640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1d0100180 0x7fc1d019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d5c00640 1 -- 172.21.15.102:0/3760296743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1d0102840 msgr2=0x7fc1d019a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d5c00640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1d0102840 0x7fc1d019a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.369+0000 7fc1d5c00640 1 -- 172.21.15.102:0/3760296743 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d019f360 con 0x7fc1d010e3c0 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.370+0000 7fc1d5c00640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ec30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800eb10 tx=0x7fc1c800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.370+0000 7fc1ce000640 1 -- 172.21.15.102:0/3760296743 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8004b20 con 0x7fc1d010e3c0 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.370+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d019f650 con 0x7fc1d010e3c0 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.370+0000 7fc1ce000640 1 -- 172.21.15.102:0/3760296743 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c8004cc0 con 0x7fc1d010e3c0 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.370+0000 7fc1ce000640 1 -- 172.21.15.102:0/3760296743 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8010760 con 0x7fc1d010e3c0 2024-04-25T01:55:25.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.370+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d01a6ee0 con 0x7fc1d010e3c0 2024-04-25T01:55:25.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.371+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1d004f500 con 0x7fc1d010e3c0 2024-04-25T01:55:25.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.373+0000 7fc1ce000640 1 -- 172.21.15.102:0/3760296743 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fc1c8005020 con 0x7fc1d010e3c0 2024-04-25T01:55:25.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.375+0000 7fc1ce000640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc1b40a2ec0 0x7fc1b40a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:25.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.375+0000 7fc1ce000640 1 -- 172.21.15.102:0/3760296743 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc1c80cfbf0 con 0x7fc1d010e3c0 2024-04-25T01:55:25.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.375+0000 7fc1d5200640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc1b40a2ec0 0x7fc1b40a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:25.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.376+0000 7fc1d5200640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc1b40a2ec0 0x7fc1b40a5380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0000c00 tx=0x7fc1c0005b50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:25.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.379+0000 7fc1ce000640 1 -- 172.21.15.102:0/3760296743 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc1c809c670 con 0x7fc1d010e3c0 2024-04-25T01:55:25.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.602+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 --> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fc1d010feb0 con 0x7fc1b40a2ec0 2024-04-25T01:55:25.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.614+0000 7fc1ce000640 1 -- 172.21.15.102:0/3760296743 <== mgr.25214 v2:172.21.15.102:6800/855543702 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fc1d010feb0 con 0x7fc1b40a2ec0 2024-04-25T01:55:25.616 INFO:teuthology.orchestra.run.smithi102.stdout:/ 1m 6m3h 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.617+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc1b40a2ec0 msgr2=0x7fc1b40a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.617+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc1b40a2ec0 0x7fc1b40a5380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0000c00 tx=0x7fc1c0005b50 comp rx=0 tx=0).stop 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.617+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d010e3c0 msgr2=0x7fc1d019ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.617+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ec30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800eb10 tx=0x7fc1c800c2d0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.617+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 shutdown_connections 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.617+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc1b40a2ec0 0x7fc1b40a5380 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.617+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1d010e3c0 0x7fc1d019ec30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.617+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1d0102840 0x7fc1d019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.617+0000 7fc1d7c00640 1 --2- 172.21.15.102:0/3760296743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1d0100180 0x7fc1d019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.618+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 >> 172.21.15.102:0/3760296743 conn(0x7fc1d0076bc0 msgr2=0x7fc1d010f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.618+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 shutdown_connections 2024-04-25T01:55:25.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:25.618+0000 7fc1d7c00640 1 -- 172.21.15.102:0/3760296743 wait complete. 2024-04-25T01:55:25.640 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-25T01:55:25.717 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-25T01:55:25.728 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-25T01:55:25.738 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T01:55:25.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-25T01:55:25.744 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T01:55:25.753 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T01:55:25.753 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T01:55:25.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-04-25T01:55:25.842 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:25.880 INFO:teuthology.orchestra.run.smithi102.stdout:Filesystem Size Used Avail Use% Mounted on 2024-04-25T01:55:25.880 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6789,172.21.15.110:6789,172.21.15.193:6789:/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76 340G 0 340G 0% /tmp/cephfs 2024-04-25T01:55:25.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-04-25T01:55:25.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:25 smithi102 ceph-mon[27796]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:25.924 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:25 smithi102 ceph-mon[27796]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:26.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-04-25T01:55:26.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-04-25T01:55:26.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:25 smithi110 ceph-mon[32848]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:26.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:25 smithi110 ceph-mon[32848]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:26.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:25 smithi193 ceph-mon[32981]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:26.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:25 smithi193 ceph-mon[32981]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:27.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:26 smithi102 ceph-mon[27796]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-25T01:55:27.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:26 smithi102 ceph-mon[27796]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:27.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:26 smithi110 ceph-mon[32848]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-25T01:55:27.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:26 smithi110 ceph-mon[32848]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:27.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:26 smithi193 ceph-mon[32981]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-25T01:55:27.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:26 smithi193 ceph-mon[32981]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T01:55:28.281 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-04-25T01:55:28.390 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-04-25T01:55:28.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-04-25T01:55:28.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-25T01:55:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/: 2024-04-25T01:55:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T01:55:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 .. 2024-04-25T01:55:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 data 2024-04-25T01:55:28.622 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data: 2024-04-25T01:55:28.622 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:28.623 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 .. 2024-04-25T01:55:28.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-04-25T01:55:28.768 INFO:teuthology.orchestra.run.smithi102.stdout:[Service] 2024-04-25T01:55:28.768 INFO:teuthology.orchestra.run.smithi102.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-04-25T01:55:28.768 INFO:teuthology.orchestra.run.smithi102.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-04-25T01:55:28.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-04-25T01:55:29.020 INFO:teuthology.orchestra.run.smithi102.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-04-25T01:55:29.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:28 smithi102 ceph-mon[27796]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-25T01:55:29.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:28 smithi110 ceph-mon[32848]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-25T01:55:29.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:28 smithi193 ceph-mon[32981]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-04-25T01:55:31.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:30 smithi102 ceph-mon[27796]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-25T01:55:31.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:30 smithi110 ceph-mon[32848]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-25T01:55:31.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:30 smithi193 ceph-mon[32981]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-25T01:55:33.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:32 smithi102 ceph-mon[27796]: pgmap v11: 97 pgs: 97 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 115 KiB/s wr, 51 op/s 2024-04-25T01:55:33.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:32 smithi110 ceph-mon[32848]: pgmap v11: 97 pgs: 97 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 115 KiB/s wr, 51 op/s 2024-04-25T01:55:33.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:32 smithi193 ceph-mon[32981]: pgmap v11: 97 pgs: 97 active+clean; 1.7 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 115 KiB/s wr, 51 op/s 2024-04-25T01:55:35.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:34 smithi102 ceph-mon[27796]: pgmap v12: 97 pgs: 97 active+clean; 21 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.7 MiB/s wr, 48 op/s 2024-04-25T01:55:35.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:34 smithi110 ceph-mon[32848]: pgmap v12: 97 pgs: 97 active+clean; 21 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.7 MiB/s wr, 48 op/s 2024-04-25T01:55:35.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:34 smithi193 ceph-mon[32981]: pgmap v12: 97 pgs: 97 active+clean; 21 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.7 MiB/s wr, 48 op/s 2024-04-25T01:55:37.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:36 smithi102 ceph-mon[27796]: pgmap v13: 97 pgs: 97 active+clean; 21 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 1.7 MiB/s wr, 46 op/s 2024-04-25T01:55:37.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:36 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:55:37.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:36 smithi110 ceph-mon[32848]: pgmap v13: 97 pgs: 97 active+clean; 21 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 1.7 MiB/s wr, 46 op/s 2024-04-25T01:55:37.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:36 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:55:37.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:36 smithi193 ceph-mon[32981]: pgmap v13: 97 pgs: 97 active+clean; 21 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 1.7 MiB/s wr, 46 op/s 2024-04-25T01:55:37.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:36 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:55:38.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:37 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:38.236 INFO:teuthology.orchestra.run.smithi102.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-04-25T01:55:38.238 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:37 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:38.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-25T01:55:38.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:37 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/: 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 .. 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 data 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data: 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout:total 35 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 . 2024-04-25T01:55:38.377 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 .. 2024-04-25T01:55:38.378 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 base 2024-04-25T01:55:38.378 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 25 01:55 global 2024-04-25T01:55:38.378 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 log 2024-04-25T01:55:38.378 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_commit_ts 2024-04-25T01:55:38.378 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_dynshmem 2024-04-25T01:55:38.378 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 25 01:55 pg_hba.conf 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 25 01:55 pg_ident.conf 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 pg_logical 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 pg_multixact 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_notify 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_replslot 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_serial 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_snapshots 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_stat 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_stat_tmp 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 pg_subtrans 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_tblspc 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_twophase 2024-04-25T01:55:38.379 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 PG_VERSION 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 pg_wal 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 pg_xact 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 25 01:55 postgresql.auto.conf 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 25 01:55 postgresql.conf 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/base: 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 . 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 1 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 13493 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 13494 2024-04-25T01:55:38.380 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.381 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-25T01:55:38.381 INFO:teuthology.orchestra.run.smithi102.stdout:total 7777 2024-04-25T01:55:38.381 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 . 2024-04-25T01:55:38.381 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 .. 2024-04-25T01:55:38.381 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 112 2024-04-25T01:55:38.381 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 113 2024-04-25T01:55:38.382 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 1247 2024-04-25T01:55:38.382 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1247_fsm 2024-04-25T01:55:38.382 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1247_vm 2024-04-25T01:55:38.382 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 01:55 1249 2024-04-25T01:55:38.382 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1249_fsm 2024-04-25T01:55:38.382 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1249_vm 2024-04-25T01:55:38.382 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 01:55 1255 2024-04-25T01:55:38.382 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1255_fsm 2024-04-25T01:55:38.382 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1255_vm 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 01:55 1259 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1259_fsm 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1259_vm 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 01:55 13344 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13344_fsm 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13344_vm 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13346 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13348 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13349_fsm 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349_vm 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13351 2024-04-25T01:55:38.383 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13353 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13354_fsm 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354_vm 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13356 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13358 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13359_fsm 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359_vm 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13361 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13363 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1417 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1418 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 174 2024-04-25T01:55:38.384 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 175 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2187 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2224 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2328 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2336 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2337 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2579 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2600 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2600_fsm 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2600_vm 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2601_fsm 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601_vm 2024-04-25T01:55:38.385 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2602 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2602_fsm 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2602_vm 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2603 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2603_fsm 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2603_vm 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2604 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2605 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2605_fsm 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2605_vm 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2606_fsm 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606_vm 2024-04-25T01:55:38.386 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2607 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2607_fsm 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2607_vm 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 01:55 2608 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2608_fsm 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2608_vm 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 01:55 2609 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2609_fsm 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2609_vm 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2610 2024-04-25T01:55:38.387 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2610_fsm 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2610_vm 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2611 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2612_fsm 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612_vm 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2613 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2615_fsm 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615_vm 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2616 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2616_fsm 2024-04-25T01:55:38.388 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2616_vm 2024-04-25T01:55:38.389 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 01:55 2617 2024-04-25T01:55:38.389 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2617_fsm 2024-04-25T01:55:38.389 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2617_vm 2024-04-25T01:55:38.389 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 01:55 2618 2024-04-25T01:55:38.389 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2618_fsm 2024-04-25T01:55:38.389 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2618_vm 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 01:55 2619 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2619_fsm 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2619_vm 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2620 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2650 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2651 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2652 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2653 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2654 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2655 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2656 2024-04-25T01:55:38.390 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2657 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 01:55 2658 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 2659 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2660 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2661 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2662 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2663 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2664 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2665 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2666 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2667 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2668 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2669 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2670 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 01:55 2673 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 01:55 2674 2024-04-25T01:55:38.391 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 01:55 2675 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2678 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2679 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2680 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2681 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2682 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2683 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2684 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2685 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2686 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2687 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2688 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2689 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 01:55 2690 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 01:55 2691 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2692 2024-04-25T01:55:38.392 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2693 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2696 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2699 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2701 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2702 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2703 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2704 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2753 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2753_fsm 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2753_vm 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2754 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2755 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2756 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2757 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2830 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2831 2024-04-25T01:55:38.393 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2832 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2833 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2834 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2835 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2836 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2837 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 01:55 2838 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2838_fsm 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2838_vm 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2839 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840_fsm 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2840_vm 2024-04-25T01:55:38.394 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2841 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2995 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2996 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3079_fsm 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079_vm 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3080 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3081 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3085 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3118 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3119 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 01:55 3164 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3256 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3257 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3258 2024-04-25T01:55:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3350 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3351 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3379 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3380 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3381 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3394 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3394_fsm 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3394_vm 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3395 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3429 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3430 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3431 2024-04-25T01:55:38.396 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3433 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3439 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3440 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3455 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 01:55 3456 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3456_fsm 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3456_vm 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3466 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3467 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3468 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3501 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3502 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3503 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3534 2024-04-25T01:55:38.397 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3541_fsm 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541_vm 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3542 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3574 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3575 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3576 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3596 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3597 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3598 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3599 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3600_fsm 2024-04-25T01:55:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600_vm 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3601_fsm 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601_vm 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3602_fsm 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602_vm 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603_fsm 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3603_vm 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3604 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3605 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3606 2024-04-25T01:55:38.399 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3607 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3608 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 3609 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3712 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3764_fsm 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764_vm 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3766 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3767 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3997 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4143 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4144 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4145 2024-04-25T01:55:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4146 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4147 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4148 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4149 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4150 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4151 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4152 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4153 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4154 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4155 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4156 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4157 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4158 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4159 2024-04-25T01:55:38.401 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4160 2024-04-25T01:55:38.402 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4161 2024-04-25T01:55:38.402 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4162 2024-04-25T01:55:38.402 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4163 2024-04-25T01:55:38.402 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4164 2024-04-25T01:55:38.402 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4165 2024-04-25T01:55:38.403 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4166 2024-04-25T01:55:38.403 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4167 2024-04-25T01:55:38.404 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4168 2024-04-25T01:55:38.404 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4169 2024-04-25T01:55:38.404 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4170 2024-04-25T01:55:38.404 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4171 2024-04-25T01:55:38.404 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4172 2024-04-25T01:55:38.404 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4173 2024-04-25T01:55:38.404 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4174 2024-04-25T01:55:38.404 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 5002 2024-04-25T01:55:38.404 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 548 2024-04-25T01:55:38.405 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 549 2024-04-25T01:55:38.405 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6102 2024-04-25T01:55:38.405 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6104 2024-04-25T01:55:38.405 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6106 2024-04-25T01:55:38.405 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6110 2024-04-25T01:55:38.405 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6111 2024-04-25T01:55:38.405 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6112 2024-04-25T01:55:38.405 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6113 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6117 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 826 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 827 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 828 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 01:55 pg_filenode.map 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 PG_VERSION 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:total 7777 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 . 2024-04-25T01:55:38.406 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 .. 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 112 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 113 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 1247 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1247_fsm 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1247_vm 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 01:55 1249 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1249_fsm 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1249_vm 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 01:55 1255 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1255_fsm 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1255_vm 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 01:55 1259 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1259_fsm 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1259_vm 2024-04-25T01:55:38.407 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 01:55 13344 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13344_fsm 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13344_vm 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13346 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13348 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13349_fsm 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349_vm 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13351 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13353 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13354_fsm 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354_vm 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13356 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13358 2024-04-25T01:55:38.408 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13359_fsm 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359_vm 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13361 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13363 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1417 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1418 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 174 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 175 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2187 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2224 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2328 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2336 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2337 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2579 2024-04-25T01:55:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2600 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2600_fsm 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2600_vm 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2601_fsm 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601_vm 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2602 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2602_fsm 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2602_vm 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2603 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2603_fsm 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2603_vm 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2604 2024-04-25T01:55:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2605 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2605_fsm 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2605_vm 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2606_fsm 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606_vm 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2607 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2607_fsm 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2607_vm 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 01:55 2608 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2608_fsm 2024-04-25T01:55:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2608_vm 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 01:55 2609 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2609_fsm 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2609_vm 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2610 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2610_fsm 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2610_vm 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2611 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2612_fsm 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612_vm 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2613 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615 2024-04-25T01:55:38.412 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2615_fsm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615_vm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2616 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2616_fsm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2616_vm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 01:55 2617 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2617_fsm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2617_vm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 01:55 2618 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2618_fsm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2618_vm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 01:55 2619 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2619_fsm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2619_vm 2024-04-25T01:55:38.413 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2620 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2650 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2651 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2652 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2653 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2654 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2655 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2656 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2657 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 01:55 2658 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 2659 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2660 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2661 2024-04-25T01:55:38.414 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2662 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2663 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2664 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2665 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2666 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2667 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2668 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2669 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2670 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 01:55 2673 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 01:55 2674 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 01:55 2675 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2678 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2679 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2680 2024-04-25T01:55:38.415 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2681 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2682 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2683 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2684 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2685 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2686 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2687 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2688 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2689 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 01:55 2690 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 01:55 2691 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2692 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2693 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2696 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2699 2024-04-25T01:55:38.416 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2701 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2702 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2703 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2704 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2753 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2753_fsm 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2753_vm 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2754 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2755 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2756 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2757 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2830 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2831 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2832 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2833 2024-04-25T01:55:38.417 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2834 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2835 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2836 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2837 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 01:55 2838 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2838_fsm 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2838_vm 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2839 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840_fsm 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2840_vm 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2841 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2995 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2996 2024-04-25T01:55:38.418 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3079_fsm 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079_vm 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3080 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3081 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3085 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3118 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3119 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 01:55 3164 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3256 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3257 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3258 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3350 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3351 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3379 2024-04-25T01:55:38.419 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3380 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3381 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3394 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3394_fsm 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3394_vm 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3395 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3429 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3430 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3431 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3433 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3439 2024-04-25T01:55:38.420 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3440 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3455 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 01:55 3456 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3456_fsm 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3456_vm 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3466 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3467 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3468 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3501 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3502 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3503 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3534 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3541_fsm 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541_vm 2024-04-25T01:55:38.421 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3542 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3574 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3575 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3576 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3596 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3597 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3598 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3599 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3600_fsm 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600_vm 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3601_fsm 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601_vm 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602 2024-04-25T01:55:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3602_fsm 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602_vm 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603_fsm 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3603_vm 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3604 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3605 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3606 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3607 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3608 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 3609 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3712 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764 2024-04-25T01:55:38.423 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3764_fsm 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764_vm 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3766 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3767 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3997 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4143 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4144 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4145 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4146 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4147 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4148 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4149 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4150 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4151 2024-04-25T01:55:38.424 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4152 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4153 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4154 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4155 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4156 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4157 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4158 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4159 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4160 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4161 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4162 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4163 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4164 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4165 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4166 2024-04-25T01:55:38.425 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4167 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4168 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4169 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4170 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4171 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4172 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4173 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4174 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 5002 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 548 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 549 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6102 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6104 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6106 2024-04-25T01:55:38.426 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6110 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6111 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6112 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6113 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6117 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 826 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 827 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 828 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 01:55 pg_filenode.map 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 PG_VERSION 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:total 7777 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 . 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 .. 2024-04-25T01:55:38.427 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 112 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 113 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 1247 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1247_fsm 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1247_vm 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 01:55 1249 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1249_fsm 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1249_vm 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 01:55 1255 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1255_fsm 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1255_vm 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 01:55 1259 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1259_fsm 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1259_vm 2024-04-25T01:55:38.428 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 01:55 13344 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13344_fsm 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13344_vm 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13346 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13348 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13349_fsm 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349_vm 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13351 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13353 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13354_fsm 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354_vm 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13356 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13358 2024-04-25T01:55:38.429 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13359_fsm 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359_vm 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13361 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13363 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1417 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1418 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 174 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 175 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2187 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2224 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2328 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2336 2024-04-25T01:55:38.430 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2337 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2579 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2600 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2600_fsm 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2600_vm 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2601_fsm 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601_vm 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2602 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2602_fsm 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2602_vm 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2603 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2603_fsm 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2603_vm 2024-04-25T01:55:38.431 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2604 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2605 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2605_fsm 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2605_vm 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2606_fsm 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606_vm 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2607 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2607_fsm 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2607_vm 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 01:55 2608 2024-04-25T01:55:38.432 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2608_fsm 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2608_vm 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 01:55 2609 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2609_fsm 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2609_vm 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2610 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2610_fsm 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2610_vm 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2611 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2612_fsm 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612_vm 2024-04-25T01:55:38.433 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2613 2024-04-25T01:55:38.434 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615 2024-04-25T01:55:38.434 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2615_fsm 2024-04-25T01:55:38.434 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615_vm 2024-04-25T01:55:38.434 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2616 2024-04-25T01:55:38.434 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2616_fsm 2024-04-25T01:55:38.434 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2616_vm 2024-04-25T01:55:38.434 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 01:55 2617 2024-04-25T01:55:38.434 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2617_fsm 2024-04-25T01:55:38.434 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2617_vm 2024-04-25T01:55:38.435 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 01:55 2618 2024-04-25T01:55:38.435 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2618_fsm 2024-04-25T01:55:38.435 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2618_vm 2024-04-25T01:55:38.435 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 01:55 2619 2024-04-25T01:55:38.435 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2619_fsm 2024-04-25T01:55:38.435 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2619_vm 2024-04-25T01:55:38.435 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2620 2024-04-25T01:55:38.435 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2650 2024-04-25T01:55:38.435 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2651 2024-04-25T01:55:38.436 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2652 2024-04-25T01:55:38.436 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2653 2024-04-25T01:55:38.436 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2654 2024-04-25T01:55:38.436 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2655 2024-04-25T01:55:38.436 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2656 2024-04-25T01:55:38.436 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2657 2024-04-25T01:55:38.436 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 01:55 2658 2024-04-25T01:55:38.436 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 2659 2024-04-25T01:55:38.436 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2660 2024-04-25T01:55:38.437 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2661 2024-04-25T01:55:38.437 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2662 2024-04-25T01:55:38.437 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2663 2024-04-25T01:55:38.437 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2664 2024-04-25T01:55:38.437 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2665 2024-04-25T01:55:38.437 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2666 2024-04-25T01:55:38.437 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2667 2024-04-25T01:55:38.437 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2668 2024-04-25T01:55:38.437 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2669 2024-04-25T01:55:38.438 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2670 2024-04-25T01:55:38.438 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 01:55 2673 2024-04-25T01:55:38.438 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 01:55 2674 2024-04-25T01:55:38.438 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 01:55 2675 2024-04-25T01:55:38.438 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2678 2024-04-25T01:55:38.438 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2679 2024-04-25T01:55:38.438 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2680 2024-04-25T01:55:38.438 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2681 2024-04-25T01:55:38.438 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2682 2024-04-25T01:55:38.439 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2683 2024-04-25T01:55:38.439 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2684 2024-04-25T01:55:38.439 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2685 2024-04-25T01:55:38.439 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2686 2024-04-25T01:55:38.439 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2687 2024-04-25T01:55:38.439 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2688 2024-04-25T01:55:38.439 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2689 2024-04-25T01:55:38.439 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 01:55 2690 2024-04-25T01:55:38.440 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 01:55 2691 2024-04-25T01:55:38.440 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2692 2024-04-25T01:55:38.440 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2693 2024-04-25T01:55:38.440 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2696 2024-04-25T01:55:38.440 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2699 2024-04-25T01:55:38.440 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2701 2024-04-25T01:55:38.440 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2702 2024-04-25T01:55:38.440 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2703 2024-04-25T01:55:38.440 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2704 2024-04-25T01:55:38.441 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2753 2024-04-25T01:55:38.441 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2753_fsm 2024-04-25T01:55:38.441 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2753_vm 2024-04-25T01:55:38.441 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2754 2024-04-25T01:55:38.441 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2755 2024-04-25T01:55:38.441 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2756 2024-04-25T01:55:38.441 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2757 2024-04-25T01:55:38.441 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2830 2024-04-25T01:55:38.441 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2831 2024-04-25T01:55:38.442 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2832 2024-04-25T01:55:38.442 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2833 2024-04-25T01:55:38.442 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2834 2024-04-25T01:55:38.442 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2835 2024-04-25T01:55:38.442 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2836 2024-04-25T01:55:38.442 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2837 2024-04-25T01:55:38.442 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 01:55 2838 2024-04-25T01:55:38.442 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2838_fsm 2024-04-25T01:55:38.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2838_vm 2024-04-25T01:55:38.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2839 2024-04-25T01:55:38.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840 2024-04-25T01:55:38.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840_fsm 2024-04-25T01:55:38.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2840_vm 2024-04-25T01:55:38.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2841 2024-04-25T01:55:38.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2995 2024-04-25T01:55:38.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2996 2024-04-25T01:55:38.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3079_fsm 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079_vm 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3080 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3081 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3085 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3118 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3119 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 01:55 3164 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3256 2024-04-25T01:55:38.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3257 2024-04-25T01:55:38.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3258 2024-04-25T01:55:38.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3350 2024-04-25T01:55:38.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3351 2024-04-25T01:55:38.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3379 2024-04-25T01:55:38.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3380 2024-04-25T01:55:38.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3381 2024-04-25T01:55:38.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3394 2024-04-25T01:55:38.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3394_fsm 2024-04-25T01:55:38.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3394_vm 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3395 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3429 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3430 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3431 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3433 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3439 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3440 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3455 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 01:55 3456 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3456_fsm 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3456_vm 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3466 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3467 2024-04-25T01:55:38.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3468 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3501 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3502 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3503 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3534 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3541_fsm 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541_vm 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3542 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3574 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3575 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3576 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3596 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3597 2024-04-25T01:55:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3598 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3599 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3600_fsm 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600_vm 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3601_fsm 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601_vm 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3602_fsm 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602_vm 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603_fsm 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3603_vm 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3604 2024-04-25T01:55:38.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3605 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3606 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3607 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3608 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 3609 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3712 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3764_fsm 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764_vm 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3766 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3767 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3997 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4143 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4144 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4145 2024-04-25T01:55:38.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4146 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4147 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4148 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4149 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4150 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4151 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4152 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4153 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4154 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4155 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4156 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4157 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4158 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4159 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4160 2024-04-25T01:55:38.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4161 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4162 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4163 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4164 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4165 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4166 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4167 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4168 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4169 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4170 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4171 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4172 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4173 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4174 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 5002 2024-04-25T01:55:38.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 548 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 549 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6102 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6104 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6106 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6110 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6111 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6112 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6113 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6117 2024-04-25T01:55:38.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 826 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 827 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 828 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 01:55 pg_filenode.map 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 PG_VERSION 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/global: 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:total 537 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Apr 25 01:55 . 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1213 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1213_fsm 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1213_vm 2024-04-25T01:55:38.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1214 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1214_fsm 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1214_vm 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 1232 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 1233 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1260 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1260_fsm 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1260_vm 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1261 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1261_fsm 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1261_vm 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1262 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1262_fsm 2024-04-25T01:55:38.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1262_vm 2024-04-25T01:55:38.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2396 2024-04-25T01:55:38.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2396_fsm 2024-04-25T01:55:38.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2396_vm 2024-04-25T01:55:38.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2397 2024-04-25T01:55:38.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2671 2024-04-25T01:55:38.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2672 2024-04-25T01:55:38.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2676 2024-04-25T01:55:38.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2677 2024-04-25T01:55:38.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2694 2024-04-25T01:55:38.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2695 2024-04-25T01:55:38.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2697 2024-04-25T01:55:38.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2698 2024-04-25T01:55:38.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2846 2024-04-25T01:55:38.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2847 2024-04-25T01:55:38.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2964 2024-04-25T01:55:38.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2965 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2966 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2967 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3592 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3593 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4060 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4061 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4175 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4176 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4177 2024-04-25T01:55:38.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4178 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4181 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4182 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4183 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4184 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4185 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4186 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6000 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6001 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6002 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6100 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6114 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6115 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 pg_control 2024-04-25T01:55:38.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 01:55 pg_filenode.map 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/log: 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:total 1 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 . 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 mappings 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 25 01:55 replorigin_checkpoint 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 snapshots 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 .. 2024-04-25T01:55:38.461 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 .. 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 . 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 members 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 offsets 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.462 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:total 8 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 .. 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 0000 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:total 8 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 .. 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 0000 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.463 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.464 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:total 8 2024-04-25T01:55:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 0000 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.466 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:total 16384 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 . 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 01:55 000000010000000000000001 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 archive_status 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 .. 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:total 8 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T01:55:38.467 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Apr 25 01:55 .. 2024-04-25T01:55:38.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 0000 2024-04-25T01:55:38.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-04-25T01:55:38.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-04-25T01:55:38.900 INFO:teuthology.orchestra.run.smithi102.stderr:dropping old tables... 2024-04-25T01:55:38.901 INFO:teuthology.orchestra.run.smithi102.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-04-25T01:55:38.901 INFO:teuthology.orchestra.run.smithi102.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-04-25T01:55:38.901 INFO:teuthology.orchestra.run.smithi102.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-04-25T01:55:38.901 INFO:teuthology.orchestra.run.smithi102.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-04-25T01:55:38.901 INFO:teuthology.orchestra.run.smithi102.stderr:creating tables... 2024-04-25T01:55:38.925 INFO:teuthology.orchestra.run.smithi102.stderr:generating data (client-side)... 2024-04-25T01:55:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.972+0000 7f3c16000640 1 -- 172.21.15.102:0/2003767247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 msgr2=0x7f3c10109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.972+0000 7f3c16000640 1 --2- 172.21.15.102:0/2003767247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 0x7f3c10109d10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009860 tx=0x7f3c0002a340 comp rx=0 tx=0).stop 2024-04-25T01:55:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.972+0000 7f3c16000640 1 -- 172.21.15.102:0/2003767247 shutdown_connections 2024-04-25T01:55:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.972+0000 7f3c16000640 1 --2- 172.21.15.102:0/2003767247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c101042e0 0x7f3c101107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.972+0000 7f3c16000640 1 --2- 172.21.15.102:0/2003767247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c10103920 0x7f3c10103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.972+0000 7f3c16000640 1 --2- 172.21.15.102:0/2003767247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 0x7f3c10109d10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.972+0000 7f3c16000640 1 -- 172.21.15.102:0/2003767247 >> 172.21.15.102:0/2003767247 conn(0x7f3c100ff430 msgr2=0x7f3c10101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.972+0000 7f3c16000640 1 -- 172.21.15.102:0/2003767247 shutdown_connections 2024-04-25T01:55:38.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.972+0000 7f3c16000640 1 -- 172.21.15.102:0/2003767247 wait complete. 2024-04-25T01:55:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c16000640 1 Processor -- start 2024-04-25T01:55:38.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c16000640 1 -- start start 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c16000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c10103920 0x7f3c1019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c16000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c101042e0 0x7f3c1019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c16000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 0x7f3c1019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c16000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3c10113d40 con 0x7f3c101042e0 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c16000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3c10113bc0 con 0x7f3c10109930 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c16000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3c10113ec0 con 0x7f3c10103920 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c0fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 0x7f3c1019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c0f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c10103920 0x7f3c1019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c0f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c10103920 0x7f3c1019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46218/0 (socket says 172.21.15.102:46218) 2024-04-25T01:55:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.974+0000 7f3c0fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 0x7f3c1019ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:57994/0 (socket says 172.21.15.102:57994) 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0f400640 1 -- 172.21.15.102:0/3259202443 learned_addr learned my addr 172.21.15.102:0/3259202443 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c101042e0 0x7f3c1019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0f400640 1 -- 172.21.15.102:0/3259202443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 msgr2=0x7f3c1019ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0f400640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 0x7f3c1019ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0f400640 1 -- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c101042e0 msgr2=0x7f3c1019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0f400640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c101042e0 0x7f3c1019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0f400640 1 -- 172.21.15.102:0/3259202443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1019f180 con 0x7f3c10103920 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c101042e0 0x7f3c1019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0fe00640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 0x7f3c1019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.975+0000 7f3c0f400640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c10103920 0x7f3c1019a1e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3c00035040 tx=0x7f3c00033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.976+0000 7f3c07e00640 1 -- 172.21.15.102:0/3259202443 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00037040 con 0x7f3c10103920 2024-04-25T01:55:38.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.976+0000 7f3c07e00640 1 -- 172.21.15.102:0/3259202443 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c00040070 con 0x7f3c10103920 2024-04-25T01:55:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.976+0000 7f3c07e00640 1 -- 172.21.15.102:0/3259202443 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c000039f0 con 0x7f3c10103920 2024-04-25T01:55:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.976+0000 7f3c16000640 1 -- 172.21.15.102:0/3259202443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1019f410 con 0x7f3c10103920 2024-04-25T01:55:38.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.976+0000 7f3c16000640 1 -- 172.21.15.102:0/3259202443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c101a6d00 con 0x7f3c10103920 2024-04-25T01:55:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.979+0000 7f3c18fdd740 1 Processor -- start 2024-04-25T01:55:38.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.979+0000 7f3c07e00640 1 -- 172.21.15.102:0/3259202443 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f3c00003bb0 con 0x7f3c10103920 2024-04-25T01:55:38.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.979+0000 7f3c18fdd740 1 -- start start 2024-04-25T01:55:38.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469ee00 0x559a9469f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a946a9fe0 0x559a946ac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559a94602ab0 con 0x559a9469ee00 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559a945ab640 con 0x559a9469f720 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559a945fea70 con 0x559a946a9fe0 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c0fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469ee00 0x559a9469f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c0f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c0fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469ee00 0x559a9469f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42458/0 (socket says 172.21.15.102:42458) 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c0fe00640 1 -- 172.21.15.102:0/2337498688 learned_addr learned my addr 172.21.15.102:0/2337498688 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.980+0000 7f3c0ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a946a9fe0 0x559a946ac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c0ea00640 1 -- 172.21.15.102:0/2337498688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 msgr2=0x559a946a9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/2337498688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946a9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c0ea00640 1 -- 172.21.15.102:0/2337498688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469ee00 msgr2=0x559a9469f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/2337498688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469ee00 0x559a9469f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c0ea00640 1 -- 172.21.15.102:0/2337498688 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a94317210 con 0x559a946a9fe0 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c0f400640 1 --2- 172.21.15.102:0/2337498688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946a9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c07e00640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf40a2ec0 0x7f3bf40a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/2337498688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a946a9fe0 0x559a946ac3d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc005fd0 tx=0x7f3bfc0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a3219ceb5b2cc8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c07e00640 1 -- 172.21.15.102:0/3259202443 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3c000ef580 con 0x7f3c10103920 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3c0fe00640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf40a2ec0 0x7f3bf40a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3bfbe00640 1 -- 172.21.15.102:0/2337498688 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc012450 con 0x559a946a9fe0 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3bfbe00640 1 -- 172.21.15.102:0/2337498688 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bfc012ad0 con 0x559a946a9fe0 2024-04-25T01:55:38.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.981+0000 7f3bfbe00640 1 -- 172.21.15.102:0/2337498688 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc00fb30 con 0x559a946a9fe0 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2337498688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a946a9fe0 msgr2=0x559a946ac3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/2337498688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a946a9fe0 0x559a946ac3d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc005fd0 tx=0x7f3bfc0117b0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c0fe00640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf40a2ec0 0x7f3bf40a5380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009fd0 tx=0x7f3c0800d040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2337498688 shutdown_connections 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/2337498688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a946a9fe0 0x559a946ac3d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/2337498688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946a9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/2337498688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469ee00 0x559a9469f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2337498688 >> 172.21.15.102:0/2337498688 conn(0x559a9469b310 msgr2=0x559a9469b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2337498688 shutdown_connections 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.982+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2337498688 wait complete. 2024-04-25T01:55:38.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.983+0000 7f3c18fdd740 1 -- start start 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.983+0000 7f3c18fdd740 1 Processor -- start 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.983+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469ee00 0x559a946a6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469f720 0x559a946a7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 0x559a946a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559a946acc10 con 0x559a946a9fe0 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559a945be990 con 0x559a9469ee00 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559a946acd90 con 0x559a9469f720 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c18fdd740 10 client.0 _ll_register_callbacks cb 0x559a945c0950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469ee00 0x559a946a6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 0x559a946a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469ee00 0x559a946a6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58010/0 (socket says 172.21.15.102:58010) 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0f400640 1 -- 172.21.15.102:0/21241806 learned_addr learned my addr 172.21.15.102:0/21241806 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469f720 0x559a946a7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0f400640 1 -- 172.21.15.102:0/21241806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469f720 msgr2=0x559a946a7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0f400640 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469f720 0x559a946a7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0f400640 1 -- 172.21.15.102:0/21241806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 msgr2=0x559a946a79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0f400640 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 0x559a946a79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.984+0000 7f3c0f400640 1 -- 172.21.15.102:0/21241806 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a943e3620 con 0x559a9469ee00 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3c0f400640 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469ee00 0x559a946a6f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3c00000c00 tx=0x7f3c0002a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da19923ca13cecd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3bf8c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3bf8c00640 1 -- 172.21.15.102:0/21241806 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c000bb7e0 con 0x559a9469ee00 2024-04-25T01:55:38.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3bf8c00640 1 -- 172.21.15.102:0/21241806 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3c000c43f0 con 0x559a9469ee00 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3c18fdd740 1 -- 172.21.15.102:0/21241806 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bfc012ad0 con 0x559a9469ee00 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3bf8c00640 1 -- 172.21.15.102:0/21241806 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3c000bb370 con 0x559a9469ee00 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3c18fdd740 10 client.25445 fetch_fsmap learned FSMap version 24 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3c18fdd740 1 -- 172.21.15.102:0/21241806 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559a946b8a90 con 0x559a9469ee00 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3bf8c00640 1 -- 172.21.15.102:0/21241806 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bfc012ad0 con 0x559a9469ee00 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.985+0000 7f3bf8c00640 1 -- 172.21.15.102:0/21241806 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7f3c0003a070 con 0x559a9469ee00 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.986+0000 7f3c18fdd740 10 client.25445 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.986+0000 7f3c18fdd740 10 client.25445 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.986+0000 7f3c18fdd740 20 client.25445 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.986+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a946b9f40 0x559a946bc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.986+0000 7f3c18fdd740 4 client.25445 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.986+0000 7f3c18fdd740 1 -- 172.21.15.102:0/21241806 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a9460fff0 con 0x559a946b9f40 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.986+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a946b9f40 0x559a946bc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:38.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.986+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a946b9f40 0x559a946bc330 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23d3a500a89d9bb1 server_cookie=7627c2547ff12b5e in_seq=0 out_seq=0 2024-04-25T01:55:38.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.986+0000 7f3bf8c00640 10 client.25445 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.987+0000 7f3bf8c00640 1 -- 172.21.15.102:0/21241806 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a9460fff0 con 0x559a946b9f40 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.987+0000 7f3bf8c00640 10 client.25445 handle_command_reply: tid=0 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.987+0000 7f3c18fdd740 1 client.25445 shutdown 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.987+0000 7f3c18fdd740 1 -- 172.21.15.102:0/21241806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469ee00 msgr2=0x559a946a6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.987+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469ee00 0x559a946a6f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3c00000c00 tx=0x7f3c0002a340 comp rx=0 tx=0).stop 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3c18fdd740 1 -- 172.21.15.102:0/21241806 shutdown_connections 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a946b9f40 0x559a946bc330 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 0x559a946a79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469f720 0x559a946a7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3bf8c00640 0 client.25445 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/21241806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469ee00 0x559a946a6f30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:38.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3c18fdd740 1 -- 172.21.15.102:0/21241806 >> 172.21.15.102:0/21241806 conn(0x559a9469b310 msgr2=0x559a946a1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:38.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3c18fdd740 1 -- 172.21.15.102:0/21241806 shutdown_connections 2024-04-25T01:55:38.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3c18fdd740 1 -- 172.21.15.102:0/21241806 wait complete. 2024-04-25T01:55:38.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:38.988+0000 7f3c18fdd740 20 client.25445 trim_cache size 0 max 16384 2024-04-25T01:55:39.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.004+0000 7f3c18fdd740 1 Processor -- start 2024-04-25T01:55:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.004+0000 7f3c18fdd740 1 -- start start 2024-04-25T01:55:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.004+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 0x559a946b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.004+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469f720 0x559a9469a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.004+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a946a9fe0 0x559a946a77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.004+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3c101b1e40 con 0x559a9469f720 2024-04-25T01:55:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.004+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3c101af710 con 0x559a946a9fe0 2024-04-25T01:55:39.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.004+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3c101ad560 con 0x559a9469ee00 2024-04-25T01:55:39.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469f720 0x559a9469a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469f720 0x559a9469a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42482/0 (socket says 172.21.15.102:42482) 2024-04-25T01:55:39.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0ea00640 1 -- 172.21.15.102:0/2384954907 learned_addr learned my addr 172.21.15.102:0/2384954907 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:39.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0f400640 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 0x559a946b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0fe00640 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a946a9fe0 0x559a946a77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0f400640 1 -- 172.21.15.102:0/2384954907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a946a9fe0 msgr2=0x559a946a77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0f400640 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a946a9fe0 0x559a946a77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0f400640 1 -- 172.21.15.102:0/2384954907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469f720 msgr2=0x559a9469a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0f400640 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469f720 0x559a9469a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0f400640 1 -- 172.21.15.102:0/2384954907 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a946a2ad0 con 0x559a9469ee00 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469f720 0x559a9469a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0fe00640 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a946a9fe0 0x559a946a77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.005+0000 7f3c0f400640 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 0x559a946b5ad0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3c0002a930 tx=0x7f3c00009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fd3bd884d3267ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3bfb400640 1 -- 172.21.15.102:0/2384954907 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c000bb550 con 0x559a9469ee00 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3bfb400640 1 -- 172.21.15.102:0/2384954907 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c000bc110 con 0x559a9469ee00 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3bfb400640 1 -- 172.21.15.102:0/2384954907 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c000bfdc0 con 0x559a9469ee00 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2384954907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 msgr2=0x559a946b5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 0x559a946b5ad0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3c0002a930 tx=0x7f3c00009590 comp rx=0 tx=0).stop 2024-04-25T01:55:39.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2384954907 shutdown_connections 2024-04-25T01:55:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a946a9fe0 0x559a946a77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a9469f720 0x559a9469a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/2384954907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 0x559a946b5ad0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.006+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2384954907 >> 172.21.15.102:0/2384954907 conn(0x559a9469b310 msgr2=0x559a946a8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.007+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2384954907 shutdown_connections 2024-04-25T01:55:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.007+0000 7f3c18fdd740 1 -- 172.21.15.102:0/2384954907 wait complete. 2024-04-25T01:55:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.007+0000 7f3c18fdd740 1 -- start start 2024-04-25T01:55:39.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.007+0000 7f3c18fdd740 1 Processor -- start 2024-04-25T01:55:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 0x559a946adb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946ae040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c18fdd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 0x559a946ae580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3c101adbd0 con 0x559a946a9fe0 2024-04-25T01:55:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3c101af040 con 0x559a9469f720 2024-04-25T01:55:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c18fdd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3c101ae2a0 con 0x559a9469ee00 2024-04-25T01:55:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c18fdd740 10 client.0 _ll_register_callbacks cb 0x7f3c00009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 0x559a946ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 0x559a946adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58038/0 (socket says 172.21.15.102:58038) 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 0x559a946ae580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42496/0 (socket says 172.21.15.102:42496) 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0ea00640 1 -- 172.21.15.102:0/790933884 learned_addr learned my addr 172.21.15.102:0/790933884 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0ea00640 1 -- 172.21.15.102:0/790933884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 msgr2=0x559a946adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 0x559a946adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0ea00640 1 -- 172.21.15.102:0/790933884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 msgr2=0x559a946ae580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 0x559a946ae580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.008+0000 7f3c0ea00640 1 -- 172.21.15.102:0/790933884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a9471f590 con 0x559a9469f720 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.009+0000 7f3c0ea00640 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946ae040 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc011cb0 tx=0x7f3bfc009e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b501351f708b4076 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.009+0000 7f3bf8c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:55:39.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.009+0000 7f3bf8c00640 1 -- 172.21.15.102:0/790933884 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc02e070 con 0x559a9469f720 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.009+0000 7f3c18fdd740 1 -- 172.21.15.102:0/790933884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c000bc110 con 0x559a9469f720 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.009+0000 7f3bf8c00640 1 -- 172.21.15.102:0/790933884 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3bfc005250 con 0x559a9469f720 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.009+0000 7f3bf8c00640 1 -- 172.21.15.102:0/790933884 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3bfc03a050 con 0x559a9469f720 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.009+0000 7f3c18fdd740 10 client.25451 fetch_fsmap learned FSMap version 24 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.009+0000 7f3c18fdd740 1 -- 172.21.15.102:0/790933884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559a94720920 con 0x559a9469f720 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3bf8c00640 1 -- 172.21.15.102:0/790933884 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c000bc110 con 0x559a9469f720 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3bf8c00640 1 -- 172.21.15.102:0/790933884 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 24) ==== 8342+0+0 (secure 0 0 0) 0x7f3bfc02a2f0 con 0x559a9469f720 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3c18fdd740 10 client.25451 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3c18fdd740 10 client.25451 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3c18fdd740 20 client.25451 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a94720e10 0x559a94723230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3c18fdd740 4 client.25451 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3c18fdd740 1 -- 172.21.15.102:0/790933884 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559a9460a340 con 0x559a94720e10 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3c0f400640 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a94720e10 0x559a94723230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:39.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3c0f400640 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a94720e10 0x559a94723230 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93e4cbfcd487d161 server_cookie=c3be8de017b1fc06 in_seq=0 out_seq=0 2024-04-25T01:55:39.016 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.010+0000 7f3bf8c00640 10 client.25451 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.191+0000 7f3bf8c00640 1 -- 172.21.15.102:0/790933884 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559a9460a340 con 0x559a94720e10 2024-04-25T01:55:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.191+0000 7f3bf8c00640 10 client.25451 handle_command_reply: tid=0 2024-04-25T01:55:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.191+0000 7f3c18fdd740 1 client.25451 shutdown 2024-04-25T01:55:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.191+0000 7f3c18fdd740 1 -- 172.21.15.102:0/790933884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 msgr2=0x559a946ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.191+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946ae040 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc011cb0 tx=0x7f3bfc009e40 comp rx=0 tx=0).stop 2024-04-25T01:55:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3c18fdd740 1 -- 172.21.15.102:0/790933884 shutdown_connections 2024-04-25T01:55:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a94720e10 0x559a94723230 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a946a9fe0 0x559a946ae580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a9469f720 0x559a946ae040 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3bf8c00640 0 client.25451 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3c18fdd740 1 --2- 172.21.15.102:0/790933884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a9469ee00 0x559a946adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3c18fdd740 1 -- 172.21.15.102:0/790933884 >> 172.21.15.102:0/790933884 conn(0x559a9469b310 msgr2=0x559a9469a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3c18fdd740 1 -- 172.21.15.102:0/790933884 shutdown_connections 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3c18fdd740 1 -- 172.21.15.102:0/790933884 wait complete. 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.192+0000 7f3c18fdd740 20 client.25451 trim_cache size 0 max 16384 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:55:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:55:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:55:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:55:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 -- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf40a2ec0 msgr2=0x7f3bf40a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf40a2ec0 0x7f3bf40a5380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009fd0 tx=0x7f3c0800d040 comp rx=0 tx=0).stop 2024-04-25T01:55:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 -- 172.21.15.102:0/3259202443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c10103920 msgr2=0x7f3c1019a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c10103920 0x7f3c1019a1e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3c00035040 tx=0x7f3c00033d90 comp rx=0 tx=0).stop 2024-04-25T01:55:39.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 -- 172.21.15.102:0/3259202443 shutdown_connections 2024-04-25T01:55:39.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf40a2ec0 0x7f3bf40a5380 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c10109930 0x7f3c1019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c101042e0 0x7f3c1019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 --2- 172.21.15.102:0/3259202443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c10103920 0x7f3c1019a1e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:39.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 -- 172.21.15.102:0/3259202443 >> 172.21.15.102:0/3259202443 conn(0x7f3c100ff430 msgr2=0x7f3c100ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:39.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 -- 172.21.15.102:0/3259202443 shutdown_connections 2024-04-25T01:55:39.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:39.193+0000 7f3c16000640 1 -- 172.21.15.102:0/3259202443 wait complete. 2024-04-25T01:55:39.258 INFO:teuthology.orchestra.run.smithi102.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.29 s, remaining 9.02 s) 2024-04-25T01:55:39.431 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:39 smithi102 ceph-mon[27796]: pgmap v14: 97 pgs: 97 active+clean; 26 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.2 MiB/s wr, 66 op/s 2024-04-25T01:55:39.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:39 smithi110 ceph-mon[32848]: pgmap v14: 97 pgs: 97 active+clean; 26 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.2 MiB/s wr, 66 op/s 2024-04-25T01:55:39.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:39 smithi193 ceph-mon[32981]: pgmap v14: 97 pgs: 97 active+clean; 26 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.2 MiB/s wr, 66 op/s 2024-04-25T01:55:39.635 INFO:teuthology.orchestra.run.smithi102.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.67 s, remaining 10.03 s) 2024-04-25T01:55:40.050 INFO:teuthology.orchestra.run.smithi102.stderr:300000 of 3200000 tuples (9%) done (elapsed 1.08 s, remaining 10.47 s) 2024-04-25T01:55:40.156 INFO:teuthology.orchestra.run.smithi102.stderr:400000 of 3200000 tuples (12%) done (elapsed 1.19 s, remaining 8.33 s) 2024-04-25T01:55:40.705 INFO:teuthology.orchestra.run.smithi102.stderr:500000 of 3200000 tuples (15%) done (elapsed 1.74 s, remaining 9.39 s) 2024-04-25T01:55:41.083 INFO:teuthology.orchestra.run.smithi102.stderr:600000 of 3200000 tuples (18%) done (elapsed 2.12 s, remaining 9.17 s) 2024-04-25T01:55:41.184 INFO:teuthology.orchestra.run.smithi102.stderr:700000 of 3200000 tuples (21%) done (elapsed 2.22 s, remaining 7.92 s) 2024-04-25T01:55:41.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:41 smithi102 ceph-mon[27796]: pgmap v15: 97 pgs: 97 active+clean; 26 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 2.1 MiB/s wr, 31 op/s 2024-04-25T01:55:41.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:41 smithi110 ceph-mon[32848]: pgmap v15: 97 pgs: 97 active+clean; 26 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 2.1 MiB/s wr, 31 op/s 2024-04-25T01:55:41.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:41 smithi193 ceph-mon[32981]: pgmap v15: 97 pgs: 97 active+clean; 26 MiB data, 437 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 2.1 MiB/s wr, 31 op/s 2024-04-25T01:55:41.656 INFO:teuthology.orchestra.run.smithi102.stderr:800000 of 3200000 tuples (25%) done (elapsed 2.69 s, remaining 8.07 s) 2024-04-25T01:55:42.003 INFO:teuthology.orchestra.run.smithi102.stderr:900000 of 3200000 tuples (28%) done (elapsed 3.04 s, remaining 7.76 s) 2024-04-25T01:55:42.099 INFO:teuthology.orchestra.run.smithi102.stderr:1000000 of 3200000 tuples (31%) done (elapsed 3.13 s, remaining 6.89 s) 2024-04-25T01:55:42.538 INFO:teuthology.orchestra.run.smithi102.stderr:1100000 of 3200000 tuples (34%) done (elapsed 3.57 s, remaining 6.82 s) 2024-04-25T01:55:42.670 INFO:teuthology.orchestra.run.smithi102.stderr:1200000 of 3200000 tuples (37%) done (elapsed 3.70 s, remaining 6.17 s) 2024-04-25T01:55:43.067 INFO:teuthology.orchestra.run.smithi102.stderr:1300000 of 3200000 tuples (40%) done (elapsed 4.10 s, remaining 5.99 s) 2024-04-25T01:55:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:43 smithi102 ceph-mon[27796]: pgmap v16: 97 pgs: 97 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 7.7 MiB/s wr, 115 op/s 2024-04-25T01:55:43.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:43 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:43.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:43 smithi110 ceph-mon[32848]: pgmap v16: 97 pgs: 97 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 7.7 MiB/s wr, 115 op/s 2024-04-25T01:55:43.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:43 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:1400000 of 3200000 tuples (43%) done (elapsed 4.53 s, remaining 5.83 s) 2024-04-25T01:55:43.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:43 smithi193 ceph-mon[32981]: pgmap v16: 97 pgs: 97 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 7.7 MiB/s wr, 115 op/s 2024-04-25T01:55:43.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:43 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:43.626 INFO:teuthology.orchestra.run.smithi102.stderr:1500000 of 3200000 tuples (46%) done (elapsed 4.66 s, remaining 5.28 s) 2024-04-25T01:55:44.043 INFO:teuthology.orchestra.run.smithi102.stderr:1600000 of 3200000 tuples (50%) done (elapsed 5.08 s, remaining 5.08 s) 2024-04-25T01:55:44.202 INFO:teuthology.orchestra.run.smithi102.stderr:1700000 of 3200000 tuples (53%) done (elapsed 5.24 s, remaining 4.62 s) 2024-04-25T01:55:44.627 INFO:teuthology.orchestra.run.smithi102.stderr:1800000 of 3200000 tuples (56%) done (elapsed 5.66 s, remaining 4.40 s) 2024-04-25T01:55:45.026 INFO:teuthology.orchestra.run.smithi102.stderr:1900000 of 3200000 tuples (59%) done (elapsed 6.06 s, remaining 4.15 s) 2024-04-25T01:55:45.207 INFO:teuthology.orchestra.run.smithi102.stderr:2000000 of 3200000 tuples (62%) done (elapsed 6.24 s, remaining 3.74 s) 2024-04-25T01:55:45.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:45 smithi102 ceph-mon[27796]: pgmap v17: 97 pgs: 97 active+clean; 125 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 17 MiB/s wr, 292 op/s 2024-04-25T01:55:45.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:45 smithi110 ceph-mon[32848]: pgmap v17: 97 pgs: 97 active+clean; 125 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 17 MiB/s wr, 292 op/s 2024-04-25T01:55:45.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:45 smithi193 ceph-mon[32981]: pgmap v17: 97 pgs: 97 active+clean; 125 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 17 MiB/s wr, 292 op/s 2024-04-25T01:55:45.663 INFO:teuthology.orchestra.run.smithi102.stderr:2100000 of 3200000 tuples (65%) done (elapsed 6.70 s, remaining 3.51 s) 2024-04-25T01:55:46.100 INFO:teuthology.orchestra.run.smithi102.stderr:2200000 of 3200000 tuples (68%) done (elapsed 7.13 s, remaining 3.24 s) 2024-04-25T01:55:46.282 INFO:teuthology.orchestra.run.smithi102.stderr:2300000 of 3200000 tuples (71%) done (elapsed 7.32 s, remaining 2.86 s) 2024-04-25T01:55:46.671 INFO:teuthology.orchestra.run.smithi102.stderr:2400000 of 3200000 tuples (75%) done (elapsed 7.70 s, remaining 2.57 s) 2024-04-25T01:55:47.000 INFO:teuthology.orchestra.run.smithi102.stderr:2500000 of 3200000 tuples (78%) done (elapsed 8.03 s, remaining 2.25 s) 2024-04-25T01:55:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2600000 of 3200000 tuples (81%) done (elapsed 8.20 s, remaining 1.89 s) 2024-04-25T01:55:47.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:47 smithi102 ceph-mon[27796]: pgmap v18: 97 pgs: 97 active+clean; 125 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 15 MiB/s wr, 284 op/s 2024-04-25T01:55:47.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:47 smithi110 ceph-mon[32848]: pgmap v18: 97 pgs: 97 active+clean; 125 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 15 MiB/s wr, 284 op/s 2024-04-25T01:55:47.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:47 smithi193 ceph-mon[32981]: pgmap v18: 97 pgs: 97 active+clean; 125 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 15 MiB/s wr, 284 op/s 2024-04-25T01:55:47.507 INFO:teuthology.orchestra.run.smithi102.stderr:2700000 of 3200000 tuples (84%) done (elapsed 8.54 s, remaining 1.58 s) 2024-04-25T01:55:47.876 INFO:teuthology.orchestra.run.smithi102.stderr:2800000 of 3200000 tuples (87%) done (elapsed 8.91 s, remaining 1.27 s) 2024-04-25T01:55:47.969 INFO:teuthology.orchestra.run.smithi102.stderr:2900000 of 3200000 tuples (90%) done (elapsed 9.00 s, remaining 0.93 s) 2024-04-25T01:55:48.385 INFO:teuthology.orchestra.run.smithi102.stderr:3000000 of 3200000 tuples (93%) done (elapsed 9.42 s, remaining 0.63 s) 2024-04-25T01:55:48.731 INFO:teuthology.orchestra.run.smithi102.stderr:3100000 of 3200000 tuples (96%) done (elapsed 9.76 s, remaining 0.31 s) 2024-04-25T01:55:48.847 INFO:teuthology.orchestra.run.smithi102.stderr:3200000 of 3200000 tuples (100%) done (elapsed 9.88 s, remaining 0.00 s) 2024-04-25T01:55:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:vacuuming... 2024-04-25T01:55:49.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:49 smithi102 ceph-mon[27796]: pgmap v19: 97 pgs: 97 active+clean; 285 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 42 MiB/s wr, 305 op/s 2024-04-25T01:55:49.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:49 smithi110 ceph-mon[32848]: pgmap v19: 97 pgs: 97 active+clean; 285 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 42 MiB/s wr, 305 op/s 2024-04-25T01:55:49.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:49 smithi193 ceph-mon[32981]: pgmap v19: 97 pgs: 97 active+clean; 285 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 42 MiB/s wr, 305 op/s 2024-04-25T01:55:50.021 INFO:teuthology.orchestra.run.smithi102.stderr:creating primary keys... 2024-04-25T01:55:51.424 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:51 smithi102 ceph-mon[27796]: pgmap v20: 97 pgs: 97 active+clean; 285 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 42 MiB/s wr, 285 op/s 2024-04-25T01:55:51.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:51 smithi110 ceph-mon[32848]: pgmap v20: 97 pgs: 97 active+clean; 285 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 42 MiB/s wr, 285 op/s 2024-04-25T01:55:51.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:51 smithi193 ceph-mon[32981]: pgmap v20: 97 pgs: 97 active+clean; 285 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 42 MiB/s wr, 285 op/s 2024-04-25T01:55:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.339+0000 7f567ae00640 1 -- 172.21.15.102:0/3420726744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5674074490 msgr2=0x7f567410da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.339+0000 7f567ae00640 1 --2- 172.21.15.102:0/3420726744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5674074490 0x7f567410da20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f565c009900 tx=0x7f565c02a480 comp rx=0 tx=0).stop 2024-04-25T01:55:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.339+0000 7f567ae00640 1 -- 172.21.15.102:0/3420726744 shutdown_connections 2024-04-25T01:55:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.339+0000 7f567ae00640 1 --2- 172.21.15.102:0/3420726744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567410df60 0x7f56741103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.339+0000 7f567ae00640 1 --2- 172.21.15.102:0/3420726744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5674074490 0x7f567410da20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.339+0000 7f567ae00640 1 --2- 172.21.15.102:0/3420726744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5674073b70 0x7f5674073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.339+0000 7f567ae00640 1 -- 172.21.15.102:0/3420726744 >> 172.21.15.102:0/3420726744 conn(0x7f56740fd1f0 msgr2=0x7f56740ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.339+0000 7f567ae00640 1 -- 172.21.15.102:0/3420726744 shutdown_connections 2024-04-25T01:55:52.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.340+0000 7f567ae00640 1 -- 172.21.15.102:0/3420726744 wait complete. 2024-04-25T01:55:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.340+0000 7f567ae00640 1 Processor -- start 2024-04-25T01:55:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f567ae00640 1 -- start start 2024-04-25T01:55:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f567ae00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5674073b70 0x7f567419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f567ae00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5674074490 0x7f567419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f567ae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567410df60 0x7f567419e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f567ae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5674113d50 con 0x7f567410df60 2024-04-25T01:55:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f567ae00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5674113bd0 con 0x7f5674073b70 2024-04-25T01:55:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f567ae00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5674113ed0 con 0x7f5674074490 2024-04-25T01:55:52.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f5678e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567410df60 0x7f567419e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f5673e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5674073b70 0x7f567419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f5673e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5674073b70 0x7f567419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:37542/0 (socket says 172.21.15.102:37542) 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f5678e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567410df60 0x7f567419e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58760/0 (socket says 172.21.15.102:58760) 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f5673e00640 1 -- 172.21.15.102:0/1598807799 learned_addr learned my addr 172.21.15.102:0/1598807799 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.341+0000 7f566be00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5674074490 0x7f567419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.342+0000 7f5678e00640 1 -- 172.21.15.102:0/1598807799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5674074490 msgr2=0x7f567419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.342+0000 7f5678e00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5674074490 0x7f567419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.342+0000 7f5678e00640 1 -- 172.21.15.102:0/1598807799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5674073b70 msgr2=0x7f567419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.342+0000 7f5678e00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5674073b70 0x7f567419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.342+0000 7f5678e00640 1 -- 172.21.15.102:0/1598807799 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f567419f120 con 0x7f567410df60 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.342+0000 7f5673e00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5674073b70 0x7f567419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.342+0000 7f566be00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5674074490 0x7f567419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.342+0000 7f5678e00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567410df60 0x7f567419e9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f566c00cab0 tx=0x7f566c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:52.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.342+0000 7f5671600640 1 -- 172.21.15.102:0/1598807799 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c00ae20 con 0x7f567410df60 2024-04-25T01:55:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.343+0000 7f567ae00640 1 -- 172.21.15.102:0/1598807799 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f567419f410 con 0x7f567410df60 2024-04-25T01:55:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.343+0000 7f5671600640 1 -- 172.21.15.102:0/1598807799 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f566c00f930 con 0x7f567410df60 2024-04-25T01:55:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.343+0000 7f5671600640 1 -- 172.21.15.102:0/1598807799 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c018b30 con 0x7f567410df60 2024-04-25T01:55:52.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.343+0000 7f567ae00640 1 -- 172.21.15.102:0/1598807799 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56741a6ca0 con 0x7f567410df60 2024-04-25T01:55:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.346+0000 7f5671600640 1 -- 172.21.15.102:0/1598807799 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f566c00fae0 con 0x7f567410df60 2024-04-25T01:55:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.346+0000 7f567dd69740 1 Processor -- start 2024-04-25T01:55:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f567dd69740 1 -- start start 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f567dd69740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 0x55d2571ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f567dd69740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f567dd69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f567dd69740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d25712dab0 con 0x55d2571c9e00 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f567dd69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d2570d6640 con 0x55d2571d4fe0 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f567dd69740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d257129a70 con 0x55d2571ca720 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f5673e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f566be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 0x55d2571ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f5673e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:47756/0 (socket says 172.21.15.102:47756) 2024-04-25T01:55:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.347+0000 7f5673e00640 1 -- 172.21.15.102:0/3999294921 learned_addr learned my addr 172.21.15.102:0/3999294921 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f566be00640 1 --2- 172.21.15.102:0/3999294921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5673e00640 1 -- 172.21.15.102:0/3999294921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 msgr2=0x55d2571d73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5673e00640 1 --2- 172.21.15.102:0/3999294921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5671600640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56580a2e40 0x7f56580a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5673e00640 1 -- 172.21.15.102:0/3999294921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 msgr2=0x55d2571ca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5673e00640 1 --2- 172.21.15.102:0/3999294921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 0x55d2571ca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5673e00640 1 -- 172.21.15.102:0/3999294921 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d256e42210 con 0x55d2571ca720 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f566be00640 1 --2- 172.21.15.102:0/3999294921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5671600640 1 -- 172.21.15.102:0/1598807799 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f566c0cfc70 con 0x7f567410df60 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5673e00640 1 --2- 172.21.15.102:0/3999294921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d4aa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5664011650 tx=0x7f5664009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6216d7e00b48c8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5669600640 1 -- 172.21.15.102:0/3999294921 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566401e070 con 0x55d2571ca720 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5669600640 1 -- 172.21.15.102:0/3999294921 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f566400fba0 con 0x55d2571ca720 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.348+0000 7f5669600640 1 -- 172.21.15.102:0/3999294921 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5664019c70 con 0x55d2571ca720 2024-04-25T01:55:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f5678e00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56580a2e40 0x7f56580a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f567dd69740 1 -- 172.21.15.102:0/3999294921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 msgr2=0x55d2571d4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f5678e00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56580a2e40 0x7f56580a5300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f566c0cd330 tx=0x7f566c00a890 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f567dd69740 1 --2- 172.21.15.102:0/3999294921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d4aa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5664011650 tx=0x7f5664009f90 comp rx=0 tx=0).stop 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f567dd69740 1 -- 172.21.15.102:0/3999294921 shutdown_connections 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f567dd69740 1 --2- 172.21.15.102:0/3999294921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f567dd69740 1 --2- 172.21.15.102:0/3999294921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d4aa0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f567dd69740 1 --2- 172.21.15.102:0/3999294921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 0x55d2571ca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f567dd69740 1 -- 172.21.15.102:0/3999294921 >> 172.21.15.102:0/3999294921 conn(0x55d2571c6310 msgr2=0x55d2571c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.349+0000 7f567dd69740 1 -- 172.21.15.102:0/3999294921 shutdown_connections 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.350+0000 7f567dd69740 1 -- 172.21.15.102:0/3999294921 wait complete. 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.350+0000 7f567dd69740 1 -- start start 2024-04-25T01:55:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.350+0000 7f567dd69740 1 Processor -- start 2024-04-25T01:55:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f567dd69740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 0x55d2571d1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f567dd69740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f567dd69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f567dd69740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d2571d7c10 con 0x55d2571c9e00 2024-04-25T01:55:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f567dd69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d2570e9990 con 0x55d2571d4fe0 2024-04-25T01:55:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f567dd69740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d2571d7d90 con 0x55d2571ca720 2024-04-25T01:55:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f567dd69740 10 client.0 _ll_register_callbacks cb 0x55d2570eb950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f566be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 0x55d2571d1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f5673e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f566be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 0x55d2571d1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58782/0 (socket says 172.21.15.102:58782) 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f566be00640 1 -- 172.21.15.102:0/50479642 learned_addr learned my addr 172.21.15.102:0/50479642 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f5673e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:47772/0 (socket says 172.21.15.102:47772) 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f566be00640 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f5673e00640 1 -- 172.21.15.102:0/50479642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 msgr2=0x55d2571d2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f5673e00640 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f5673e00640 1 -- 172.21.15.102:0/50479642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 msgr2=0x55d2571d1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f5673e00640 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 0x55d2571d1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f5673e00640 1 -- 172.21.15.102:0/50479642 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d256f0e620 con 0x55d2571ca720 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.351+0000 7f566be00640 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.352+0000 7f5673e00640 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d24d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5664000c00 tx=0x7f566400fb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f7122461b3e6e32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.352+0000 7f5662000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.352+0000 7f5662000640 1 -- 172.21.15.102:0/50479642 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566401e040 con 0x55d2571ca720 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.352+0000 7f567dd69740 1 -- 172.21.15.102:0/50479642 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f566400fba0 con 0x55d2571ca720 2024-04-25T01:55:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.352+0000 7f5662000640 1 -- 172.21.15.102:0/50479642 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f5664009cc0 con 0x55d2571ca720 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.352+0000 7f5662000640 1 -- 172.21.15.102:0/50479642 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5664030050 con 0x55d2571ca720 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.352+0000 7f567dd69740 10 client.25450 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.352+0000 7f567dd69740 1 -- 172.21.15.102:0/50479642 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d2571e3a70 con 0x55d2571ca720 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.353+0000 7f5662000640 1 -- 172.21.15.102:0/50479642 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f566400fba0 con 0x55d2571ca720 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.353+0000 7f5662000640 1 -- 172.21.15.102:0/50479642 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f5664021070 con 0x55d2571ca720 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.353+0000 7f567dd69740 10 client.25450 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.353+0000 7f567dd69740 10 client.25450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.353+0000 7f567dd69740 20 client.25450 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.353+0000 7f567dd69740 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d2571e4dc0 0x55d2571e71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.353+0000 7f567dd69740 4 client.25450 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.353+0000 7f567dd69740 1 -- 172.21.15.102:0/50479642 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d25713aff0 con 0x55d2571e4dc0 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.353+0000 7f566be00640 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d2571e4dc0 0x55d2571e71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.354+0000 7f566be00640 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d2571e4dc0 0x55d2571e71b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f41499c718ccbb server_cookie=8351043bea97fdf0 in_seq=0 out_seq=0 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.354+0000 7f5662000640 10 client.25450 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.354+0000 7f5662000640 1 -- 172.21.15.102:0/50479642 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d25713aff0 con 0x55d2571e4dc0 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f5662000640 10 client.25450 handle_command_reply: tid=0 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 client.25450 shutdown 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 -- 172.21.15.102:0/50479642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 msgr2=0x55d2571d24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d24d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5664000c00 tx=0x7f566400fb30 comp rx=0 tx=0).stop 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 -- 172.21.15.102:0/50479642 shutdown_connections 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d2571e4dc0 0x55d2571e71b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571d2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571ca720 0x55d2571d24d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 --2- 172.21.15.102:0/50479642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571c9e00 0x55d2571d1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f5662000640 0 client.25450 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 -- 172.21.15.102:0/50479642 >> 172.21.15.102:0/50479642 conn(0x55d2571c6310 msgr2=0x55d2571cc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 -- 172.21.15.102:0/50479642 shutdown_connections 2024-04-25T01:55:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 1 -- 172.21.15.102:0/50479642 wait complete. 2024-04-25T01:55:52.361 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.355+0000 7f567dd69740 20 client.25450 trim_cache size 0 max 16384 2024-04-25T01:55:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.371+0000 7f567dd69740 1 Processor -- start 2024-04-25T01:55:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.371+0000 7f567dd69740 1 -- start start 2024-04-25T01:55:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f567dd69740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571c9e00 0x55d2571e0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f567dd69740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f567dd69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571ec960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f567dd69740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f56741b1e30 con 0x55d2571ca720 2024-04-25T01:55:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f567dd69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f56741af6d0 con 0x55d2571d4fe0 2024-04-25T01:55:52.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f567dd69740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56741ad520 con 0x55d2571c9e00 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f5673e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f566be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571c9e00 0x55d2571e0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f5673e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571d27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58786/0 (socket says 172.21.15.102:58786) 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f5673e00640 1 -- 172.21.15.102:0/815342252 learned_addr learned my addr 172.21.15.102:0/815342252 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.372+0000 7f566be00640 1 --2- 172.21.15.102:0/815342252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571ec960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.373+0000 7f5673e00640 1 -- 172.21.15.102:0/815342252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571c9e00 msgr2=0x55d2571e0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.373+0000 7f5673e00640 1 --2- 172.21.15.102:0/815342252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571c9e00 0x55d2571e0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.373+0000 7f5673e00640 1 -- 172.21.15.102:0/815342252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 msgr2=0x55d2571ec960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.373+0000 7f5673e00640 1 --2- 172.21.15.102:0/815342252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571ec960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.373+0000 7f5673e00640 1 -- 172.21.15.102:0/815342252 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2571d0f40 con 0x55d2571ca720 2024-04-25T01:55:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.373+0000 7f5673e00640 1 --2- 172.21.15.102:0/815342252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571d27e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f56640059f0 tx=0x7f56640114d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02c6890265aab86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.373+0000 7f5668c00640 1 -- 172.21.15.102:0/815342252 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566401e040 con 0x55d2571ca720 2024-04-25T01:55:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.373+0000 7f5668c00640 1 -- 172.21.15.102:0/815342252 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5664027070 con 0x55d2571ca720 2024-04-25T01:55:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.373+0000 7f5668c00640 1 -- 172.21.15.102:0/815342252 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56640199b0 con 0x55d2571ca720 2024-04-25T01:55:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.374+0000 7f567dd69740 1 -- 172.21.15.102:0/815342252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 msgr2=0x55d2571d27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.374+0000 7f567dd69740 1 --2- 172.21.15.102:0/815342252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571d27e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f56640059f0 tx=0x7f56640114d0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.374+0000 7f567dd69740 1 -- 172.21.15.102:0/815342252 shutdown_connections 2024-04-25T01:55:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.374+0000 7f567dd69740 1 --2- 172.21.15.102:0/815342252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571d4fe0 0x55d2571ec960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.374+0000 7f567dd69740 1 --2- 172.21.15.102:0/815342252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571d27e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.374+0000 7f567dd69740 1 --2- 172.21.15.102:0/815342252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571c9e00 0x55d2571e0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.374+0000 7f567dd69740 1 -- 172.21.15.102:0/815342252 >> 172.21.15.102:0/815342252 conn(0x55d2571c6310 msgr2=0x55d2571c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.374+0000 7f567dd69740 1 -- 172.21.15.102:0/815342252 shutdown_connections 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.374+0000 7f567dd69740 1 -- 172.21.15.102:0/815342252 wait complete. 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.375+0000 7f567dd69740 1 -- start start 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.375+0000 7f567dd69740 1 Processor -- start 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.375+0000 7f567dd69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571c9e00 0x55d2571cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.375+0000 7f567dd69740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.375+0000 7f567dd69740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571d4fe0 0x55d2571cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.375+0000 7f567dd69740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f56741adb90 con 0x55d2571ca720 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.375+0000 7f567dd69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f56741af000 con 0x55d2571c9e00 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.375+0000 7f567dd69740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56741ae260 con 0x55d2571d4fe0 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.375+0000 7f567dd69740 10 client.0 _ll_register_callbacks cb 0x7f566401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f5673e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f566be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571d4fe0 0x55d2571cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f5673e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571cf0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58796/0 (socket says 172.21.15.102:58796) 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f5673e00640 1 -- 172.21.15.102:0/523250206 learned_addr learned my addr 172.21.15.102:0/523250206 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f566be00640 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571c9e00 0x55d2571cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f566be00640 1 -- 172.21.15.102:0/523250206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571c9e00 msgr2=0x55d2571cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f566be00640 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571c9e00 0x55d2571cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f566be00640 1 -- 172.21.15.102:0/523250206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 msgr2=0x55d2571cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f566be00640 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571cf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f566be00640 1 -- 172.21.15.102:0/523250206 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2572529e0 con 0x55d2571d4fe0 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f566be00640 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571d4fe0 0x55d2571cf630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55d2571e7900 tx=0x7f566400fa10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0982923890c64ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f5662000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f5662000640 1 -- 172.21.15.102:0/523250206 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c035040 con 0x55d2571d4fe0 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f567dd69740 1 -- 172.21.15.102:0/523250206 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5664027070 con 0x55d2571d4fe0 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.376+0000 7f5662000640 1 -- 172.21.15.102:0/523250206 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f565c04cae0 con 0x55d2571d4fe0 2024-04-25T01:55:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f5662000640 1 -- 172.21.15.102:0/523250206 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f565c004760 con 0x55d2571d4fe0 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f567dd69740 10 client.25456 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f567dd69740 1 -- 172.21.15.102:0/523250206 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d257253c80 con 0x55d2571d4fe0 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f5662000640 1 -- 172.21.15.102:0/523250206 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5664027070 con 0x55d2571d4fe0 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f5662000640 1 -- 172.21.15.102:0/523250206 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f565c0318b0 con 0x55d2571d4fe0 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f567dd69740 10 client.25456 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f567dd69740 10 client.25456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f567dd69740 20 client.25456 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f567dd69740 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d257254370 0x55d257256790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f567dd69740 4 client.25456 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.377+0000 7f567dd69740 1 -- 172.21.15.102:0/523250206 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d257135340 con 0x55d257254370 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.378+0000 7f5673e00640 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d257254370 0x55d257256790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.378+0000 7f5673e00640 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d257254370 0x55d257256790 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdcae1acae249979 server_cookie=ac91fbe0961b3a3 in_seq=0 out_seq=0 2024-04-25T01:55:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.378+0000 7f5662000640 10 client.25456 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.399+0000 7f5662000640 1 -- 172.21.15.102:0/523250206 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d257135340 con 0x55d257254370 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.399+0000 7f5662000640 10 client.25456 handle_command_reply: tid=0 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.399+0000 7f567dd69740 1 client.25456 shutdown 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 -- 172.21.15.102:0/523250206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571d4fe0 msgr2=0x55d2571cf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571d4fe0 0x55d2571cf630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55d2571e7900 tx=0x7f566400fa10 comp rx=0 tx=0).stop 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 -- 172.21.15.102:0/523250206 shutdown_connections 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d257254370 0x55d257256790 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2571d4fe0 0x55d2571cf630 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d2571ca720 0x55d2571cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f5662000640 0 client.25456 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 --2- 172.21.15.102:0/523250206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d2571c9e00 0x55d2571cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 -- 172.21.15.102:0/523250206 >> 172.21.15.102:0/523250206 conn(0x55d2571c6310 msgr2=0x55d2571cc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 -- 172.21.15.102:0/523250206 shutdown_connections 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 1 -- 172.21.15.102:0/523250206 wait complete. 2024-04-25T01:55:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.400+0000 7f567dd69740 20 client.25456 trim_cache size 0 max 16384 2024-04-25T01:55:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:55:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:55:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:55:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:55:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 -- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56580a2e40 msgr2=0x7f56580a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56580a2e40 0x7f56580a5300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f566c0cd330 tx=0x7f566c00a890 comp rx=0 tx=0).stop 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 -- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567410df60 msgr2=0x7f567419e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567410df60 0x7f567419e9f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f566c00cab0 tx=0x7f566c00cf70 comp rx=0 tx=0).stop 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 -- 172.21.15.102:0/1598807799 shutdown_connections 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56580a2e40 0x7f56580a5300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f567410df60 0x7f567419e9f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5674074490 0x7f567419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 --2- 172.21.15.102:0/1598807799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5674073b70 0x7f567419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 -- 172.21.15.102:0/1598807799 >> 172.21.15.102:0/1598807799 conn(0x7f56740fd1f0 msgr2=0x7f56740fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 -- 172.21.15.102:0/1598807799 shutdown_connections 2024-04-25T01:55:52.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:55:52.401+0000 7f567ae00640 1 -- 172.21.15.102:0/1598807799 wait complete. 2024-04-25T01:55:52.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:52 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:52.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:52 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:52.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:52 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:55:52.857 INFO:teuthology.orchestra.run.smithi102.stderr:done in 13.96 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 10.02 s, vacuum 1.07 s, primary keys 2.84 s). 2024-04-25T01:55:52.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-04-25T01:55:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:starting vacuum...end. 2024-04-25T01:55:53.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:53 smithi110 ceph-mon[32848]: pgmap v21: 97 pgs: 97 active+clean; 322 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 47 MiB/s wr, 300 op/s 2024-04-25T01:55:53.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:53 smithi193 ceph-mon[32981]: pgmap v21: 97 pgs: 97 active+clean; 322 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 47 MiB/s wr, 300 op/s 2024-04-25T01:55:53.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:53 smithi102 ceph-mon[27796]: pgmap v21: 97 pgs: 97 active+clean; 322 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 47 MiB/s wr, 300 op/s 2024-04-25T01:55:54.142 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:55:54.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:55:54.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.410+0000 7f6e62800640 1 -- 172.21.15.102:0/2997541078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 msgr2=0x7f6e5c115a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.410+0000 7f6e62800640 1 --2- 172.21.15.102:0/2997541078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 0x7f6e5c115a80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6e540072c0 tx=0x7f6e5402a5a0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.411+0000 7f6e62800640 1 -- 172.21.15.102:0/2997541078 shutdown_connections 2024-04-25T01:55:54.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.411+0000 7f6e62800640 1 --2- 172.21.15.102:0/2997541078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 0x7f6e5c115a80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.411+0000 7f6e62800640 1 --2- 172.21.15.102:0/2997541078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10aa00 0x7f6e5c113150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.411+0000 7f6e62800640 1 --2- 172.21.15.102:0/2997541078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e5c10a0e0 0x7f6e5c10a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.411+0000 7f6e62800640 1 -- 172.21.15.102:0/2997541078 >> 172.21.15.102:0/2997541078 conn(0x7f6e5c06c6c0 msgr2=0x7f6e5c06cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:54.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.412+0000 7f6e62800640 1 -- 172.21.15.102:0/2997541078 shutdown_connections 2024-04-25T01:55:54.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.412+0000 7f6e62800640 1 -- 172.21.15.102:0/2997541078 wait complete. 2024-04-25T01:55:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.415+0000 7f6e62800640 1 Processor -- start 2024-04-25T01:55:54.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.415+0000 7f6e62800640 1 -- start start 2024-04-25T01:55:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.415+0000 7f6e62800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e5c10a0e0 0x7f6e5c19e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.415+0000 7f6e62800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10aa00 0x7f6e5c19ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.415+0000 7f6e62800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 0x7f6e5c1a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.415+0000 7f6e62800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c117fe0 con 0x7f6e5c10a0e0 2024-04-25T01:55:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.415+0000 7f6e62800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c117e60 con 0x7f6e5c113690 2024-04-25T01:55:54.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.415+0000 7f6e62800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c118160 con 0x7f6e5c10aa00 2024-04-25T01:55:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.416+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10aa00 0x7f6e5c19ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.416+0000 7f6e61e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 0x7f6e5c1a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.416+0000 7f6e61e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 0x7f6e5c1a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45024/0 (socket says 172.21.15.102:45024) 2024-04-25T01:55:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.416+0000 7f6e61e00640 1 -- 172.21.15.102:0/3176093965 learned_addr learned my addr 172.21.15.102:0/3176093965 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.417+0000 7f6e61e00640 1 -- 172.21.15.102:0/3176093965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10aa00 msgr2=0x7f6e5c19ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.417+0000 7f6e61e00640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10aa00 0x7f6e5c19ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.417+0000 7f6e61e00640 1 -- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e5c10a0e0 msgr2=0x7f6e5c19e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.417+0000 7f6e61400640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e5c10a0e0 0x7f6e5c19e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.417+0000 7f6e61e00640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e5c10a0e0 0x7f6e5c19e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.417+0000 7f6e61e00640 1 -- 172.21.15.102:0/3176093965 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5c1a34b0 con 0x7f6e5c113690 2024-04-25T01:55:54.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.417+0000 7f6e61400640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e5c10a0e0 0x7f6e5c19e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:54.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.417+0000 7f6e61e00640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 0x7f6e5c1a2d80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6e54006010 tx=0x7f6e5400b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:54.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.418+0000 7f6e5a000640 1 -- 172.21.15.102:0/3176093965 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54036050 con 0x7f6e5c113690 2024-04-25T01:55:54.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.418+0000 7f6e5a000640 1 -- 172.21.15.102:0/3176093965 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e54042070 con 0x7f6e5c113690 2024-04-25T01:55:54.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.418+0000 7f6e5a000640 1 -- 172.21.15.102:0/3176093965 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e54004280 con 0x7f6e5c113690 2024-04-25T01:55:54.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.418+0000 7f6e62800640 1 -- 172.21.15.102:0/3176093965 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e5c1a3740 con 0x7f6e5c113690 2024-04-25T01:55:54.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.418+0000 7f6e62800640 1 -- 172.21.15.102:0/3176093965 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e5c1ab030 con 0x7f6e5c113690 2024-04-25T01:55:54.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.423+0000 7f6e66da2740 1 Processor -- start 2024-04-25T01:55:54.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.424+0000 7f6e5a000640 1 -- 172.21.15.102:0/3176093965 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f6e54004420 con 0x7f6e5c113690 2024-04-25T01:55:54.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.425+0000 7f6e66da2740 1 -- start start 2024-04-25T01:55:54.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.425+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1965c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.425+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.425+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 0x561d3d1a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.425+0000 7f6e66da2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561d3d0fa560 con 0x561d3d196b00 2024-04-25T01:55:54.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.425+0000 7f6e66da2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561d3d0a2930 con 0x561d3d1961e0 2024-04-25T01:55:54.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.425+0000 7f6e66da2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561d3d0f6520 con 0x561d3d1a11d0 2024-04-25T01:55:54.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.426+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.426+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45026/0 (socket says 172.21.15.102:45026) 2024-04-25T01:55:54.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.426+0000 7f6e60a00640 1 -- 172.21.15.102:0/3403771018 learned_addr learned my addr 172.21.15.102:0/3403771018 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:54.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.426+0000 7f6e61400640 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.427+0000 7f6e60a00640 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 0x561d3d1a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.427+0000 7f6e5a000640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6e440a2f20 0x7f6e440a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.427+0000 7f6e5a000640 1 -- 172.21.15.102:0/3176093965 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6e540ef7b0 con 0x7f6e5c113690 2024-04-25T01:55:54.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.428+0000 7f6e60a00640 1 -- 172.21.15.102:0/3403771018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 msgr2=0x561d3d1a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:54.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.428+0000 7f6e61e00640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6e440a2f20 0x7f6e440a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.428+0000 7f6e60a00640 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 0x561d3d1a35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.428+0000 7f6e60a00640 1 -- 172.21.15.102:0/3403771018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 msgr2=0x561d3d1a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:54.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.428+0000 7f6e60a00640 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.428+0000 7f6e60a00640 1 -- 172.21.15.102:0/3403771018 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d3ced87d0 con 0x561d3d1961e0 2024-04-25T01:55:54.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.428+0000 7f6e61400640 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:54.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.429+0000 7f6e61e00640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6e440a2f20 0x7f6e440a53e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6e540eced0 tx=0x7f6e5400d9d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:54.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.430+0000 7f6e60a00640 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1965c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6e4801bf40 tx=0x7f6e48005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce3e32d87202e020 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:54.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.430+0000 7f6e4e600640 1 -- 172.21.15.102:0/3403771018 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4800fbd0 con 0x561d3d1961e0 2024-04-25T01:55:54.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.431+0000 7f6e4e600640 1 -- 172.21.15.102:0/3403771018 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e4800fd70 con 0x561d3d1961e0 2024-04-25T01:55:54.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.431+0000 7f6e4e600640 1 -- 172.21.15.102:0/3403771018 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e480047a0 con 0x561d3d1961e0 2024-04-25T01:55:54.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.432+0000 7f6e66da2740 1 -- 172.21.15.102:0/3403771018 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 msgr2=0x561d3d1965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.432+0000 7f6e66da2740 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1965c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6e4801bf40 tx=0x7f6e48005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:54.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.433+0000 7f6e66da2740 1 -- 172.21.15.102:0/3403771018 shutdown_connections 2024-04-25T01:55:54.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.433+0000 7f6e66da2740 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 0x561d3d1a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.433+0000 7f6e66da2740 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.433+0000 7f6e66da2740 1 --2- 172.21.15.102:0/3403771018 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1965c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.433+0000 7f6e66da2740 1 -- 172.21.15.102:0/3403771018 >> 172.21.15.102:0/3403771018 conn(0x561d3d192500 msgr2=0x561d3d1928e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:54.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.433+0000 7f6e66da2740 1 -- 172.21.15.102:0/3403771018 shutdown_connections 2024-04-25T01:55:54.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.434+0000 7f6e66da2740 1 -- 172.21.15.102:0/3403771018 wait complete. 2024-04-25T01:55:54.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.434+0000 7f6e66da2740 1 -- start start 2024-04-25T01:55:54.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.435+0000 7f6e66da2740 1 Processor -- start 2024-04-25T01:55:54.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 0x561d3d19e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d196b00 0x561d3d19e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d1a11d0 0x561d3d19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e66da2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561d3d1a3f00 con 0x561d3d1a11d0 2024-04-25T01:55:54.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e66da2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561d3d0b5d90 con 0x561d3d196b00 2024-04-25T01:55:54.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e66da2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561d3d1a4080 con 0x561d3d1961e0 2024-04-25T01:55:54.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e66da2740 10 client.0 _ll_register_callbacks cb 0x561d3d0b5780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:54.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e61400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d196b00 0x561d3d19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e61400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d196b00 0x561d3d19e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45042/0 (socket says 172.21.15.102:45042) 2024-04-25T01:55:54.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.437+0000 7f6e61400640 1 -- 172.21.15.102:0/1479967709 learned_addr learned my addr 172.21.15.102:0/1479967709 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.438+0000 7f6e61400640 1 -- 172.21.15.102:0/1479967709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 msgr2=0x561d3d19e100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:55:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.438+0000 7f6e61400640 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 0x561d3d19e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.438+0000 7f6e61400640 1 -- 172.21.15.102:0/1479967709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d1a11d0 msgr2=0x561d3d19eb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:55:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.439+0000 7f6e60a00640 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d1a11d0 0x561d3d19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.440+0000 7f6e61400640 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d1a11d0 0x561d3d19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.440+0000 7f6e61400640 1 -- 172.21.15.102:0/1479967709 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d3ce0e3f0 con 0x561d3d196b00 2024-04-25T01:55:54.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.440+0000 7f6e60a00640 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d1a11d0 0x561d3d19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:54.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.441+0000 7f6e61400640 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d196b00 0x561d3d19e640 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6e50010850 tx=0x7f6e50010d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff10d894d9b6b1d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:54.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.441+0000 7f6e42a00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:55:54.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.442+0000 7f6e42a00640 1 -- 172.21.15.102:0/1479967709 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5001d060 con 0x561d3d196b00 2024-04-25T01:55:54.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.442+0000 7f6e42a00640 1 -- 172.21.15.102:0/1479967709 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6e5000eda0 con 0x561d3d196b00 2024-04-25T01:55:54.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.442+0000 7f6e66da2740 1 -- 172.21.15.102:0/1479967709 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e4800fd70 con 0x561d3d196b00 2024-04-25T01:55:54.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.443+0000 7f6e42a00640 1 -- 172.21.15.102:0/1479967709 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6e50018430 con 0x561d3d196b00 2024-04-25T01:55:54.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.443+0000 7f6e66da2740 10 client.25475 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:54.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.443+0000 7f6e66da2740 1 -- 172.21.15.102:0/1479967709 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561d3cdce0f0 con 0x561d3d196b00 2024-04-25T01:55:54.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.444+0000 7f6e42a00640 1 -- 172.21.15.102:0/1479967709 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e4800fd70 con 0x561d3d196b00 2024-04-25T01:55:54.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.444+0000 7f6e42a00640 1 -- 172.21.15.102:0/1479967709 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f6e5000d5d0 con 0x561d3d196b00 2024-04-25T01:55:54.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.445+0000 7f6e66da2740 10 client.25475 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:54.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.445+0000 7f6e66da2740 10 client.25475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:54.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.445+0000 7f6e66da2740 20 client.25475 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:54.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.445+0000 7f6e66da2740 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561d3d1b1440 0x561d3d1b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.445+0000 7f6e66da2740 4 client.25475 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:54.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.445+0000 7f6e66da2740 1 -- 172.21.15.102:0/1479967709 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d3d10dd90 con 0x561d3d1b1440 2024-04-25T01:55:54.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.445+0000 7f6e60a00640 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561d3d1b1440 0x561d3d1b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.447+0000 7f6e60a00640 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561d3d1b1440 0x561d3d1b3890 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3238d17e17a0bf3c server_cookie=ef23b85de7af2c9b in_seq=0 out_seq=0 2024-04-25T01:55:54.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.448+0000 7f6e42a00640 10 client.25475 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:54.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.448+0000 7f6e42a00640 1 -- 172.21.15.102:0/1479967709 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561d3d10dd90 con 0x561d3d1b1440 2024-04-25T01:55:54.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.448+0000 7f6e42a00640 10 client.25475 handle_command_reply: tid=0 2024-04-25T01:55:54.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.449+0000 7f6e66da2740 1 client.25475 shutdown 2024-04-25T01:55:54.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.450+0000 7f6e66da2740 1 -- 172.21.15.102:0/1479967709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d196b00 msgr2=0x561d3d19e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.450+0000 7f6e66da2740 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d196b00 0x561d3d19e640 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6e50010850 tx=0x7f6e50010d10 comp rx=0 tx=0).stop 2024-04-25T01:55:54.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.450+0000 7f6e66da2740 1 -- 172.21.15.102:0/1479967709 shutdown_connections 2024-04-25T01:55:54.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.450+0000 7f6e66da2740 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561d3d1b1440 0x561d3d1b3890 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.450+0000 7f6e66da2740 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d1a11d0 0x561d3d19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.450+0000 7f6e66da2740 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d196b00 0x561d3d19e640 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.450+0000 7f6e66da2740 1 --2- 172.21.15.102:0/1479967709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 0x561d3d19e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.450+0000 7f6e66da2740 1 -- 172.21.15.102:0/1479967709 >> 172.21.15.102:0/1479967709 conn(0x561d3d192500 msgr2=0x561d3d191800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:54.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.450+0000 7f6e42a00640 0 client.25475 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:54.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.451+0000 7f6e66da2740 1 -- 172.21.15.102:0/1479967709 shutdown_connections 2024-04-25T01:55:54.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.451+0000 7f6e66da2740 1 -- 172.21.15.102:0/1479967709 wait complete. 2024-04-25T01:55:54.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.451+0000 7f6e66da2740 20 client.25475 trim_cache size 0 max 16384 2024-04-25T01:55:54.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.494+0000 7f6e66da2740 1 Processor -- start 2024-04-25T01:55:54.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.496+0000 7f6e66da2740 1 -- start start 2024-04-25T01:55:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.497+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 0x561d3d1ace20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.497+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d198040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.497+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1a11d0 0x561d3d1b91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.497+0000 7f6e66da2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c1b60a0 con 0x561d3d196b00 2024-04-25T01:55:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.497+0000 7f6e66da2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c1b3940 con 0x561d3d1a11d0 2024-04-25T01:55:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.497+0000 7f6e66da2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c1b1790 con 0x561d3d1961e0 2024-04-25T01:55:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.497+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 0x561d3d1ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.497+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 0x561d3d1ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55748/0 (socket says 172.21.15.102:55748) 2024-04-25T01:55:54.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.497+0000 7f6e60a00640 1 -- 172.21.15.102:0/3858474669 learned_addr learned my addr 172.21.15.102:0/3858474669 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.498+0000 7f6e60a00640 1 --2- 172.21.15.102:0/3858474669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1a11d0 0x561d3d1b91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.498+0000 7f6e61400640 1 --2- 172.21.15.102:0/3858474669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.498+0000 7f6e60a00640 1 -- 172.21.15.102:0/3858474669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 msgr2=0x561d3d1ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.498+0000 7f6e60a00640 1 --2- 172.21.15.102:0/3858474669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 0x561d3d1ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.498+0000 7f6e60a00640 1 -- 172.21.15.102:0/3858474669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 msgr2=0x561d3d198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.498+0000 7f6e60a00640 1 --2- 172.21.15.102:0/3858474669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.498+0000 7f6e60a00640 1 -- 172.21.15.102:0/3858474669 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d3d1b0060 con 0x561d3d1a11d0 2024-04-25T01:55:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.498+0000 7f6e60a00640 1 --2- 172.21.15.102:0/3858474669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1a11d0 0x561d3d1b91a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005320 tx=0x7f6e48005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5990008093e3491c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.499+0000 7f6e4dc00640 1 -- 172.21.15.102:0/3858474669 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e480055c0 con 0x561d3d1a11d0 2024-04-25T01:55:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.499+0000 7f6e4dc00640 1 -- 172.21.15.102:0/3858474669 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e4801b4b0 con 0x561d3d1a11d0 2024-04-25T01:55:54.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.499+0000 7f6e4dc00640 1 -- 172.21.15.102:0/3858474669 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e48003dd0 con 0x561d3d1a11d0 2024-04-25T01:55:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.500+0000 7f6e66da2740 1 -- 172.21.15.102:0/3858474669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1a11d0 msgr2=0x561d3d1b91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.500+0000 7f6e66da2740 1 --2- 172.21.15.102:0/3858474669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1a11d0 0x561d3d1b91a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6e48005320 tx=0x7f6e48005f90 comp rx=0 tx=0).stop 2024-04-25T01:55:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.501+0000 7f6e66da2740 1 -- 172.21.15.102:0/3858474669 shutdown_connections 2024-04-25T01:55:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.501+0000 7f6e66da2740 1 --2- 172.21.15.102:0/3858474669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1a11d0 0x561d3d1b91a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.501+0000 7f6e66da2740 1 --2- 172.21.15.102:0/3858474669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.501+0000 7f6e66da2740 1 --2- 172.21.15.102:0/3858474669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1961e0 0x561d3d1ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.501+0000 7f6e66da2740 1 -- 172.21.15.102:0/3858474669 >> 172.21.15.102:0/3858474669 conn(0x561d3d192500 msgr2=0x561d3d197930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.501+0000 7f6e66da2740 1 -- 172.21.15.102:0/3858474669 shutdown_connections 2024-04-25T01:55:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.501+0000 7f6e66da2740 1 -- 172.21.15.102:0/3858474669 wait complete. 2024-04-25T01:55:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.501+0000 7f6e66da2740 1 -- start start 2024-04-25T01:55:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.502+0000 7f6e66da2740 1 Processor -- start 2024-04-25T01:55:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e66da2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 0x561d3d1a5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e66da2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c1b1e00 con 0x561d3d196b00 2024-04-25T01:55:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e66da2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c1b3270 con 0x561d3d1961e0 2024-04-25T01:55:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e66da2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c1b24d0 con 0x561d3d1a11d0 2024-04-25T01:55:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e66da2740 10 client.0 _ll_register_callbacks cb 0x7f6e50010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45062/0 (socket says 172.21.15.102:45062) 2024-04-25T01:55:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e60a00640 1 -- 172.21.15.102:0/824323020 learned_addr learned my addr 172.21.15.102:0/824323020 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e60a00640 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 0x561d3d1a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.503+0000 7f6e61400640 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.504+0000 7f6e60a00640 1 -- 172.21.15.102:0/824323020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 msgr2=0x561d3d1a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.504+0000 7f6e60a00640 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 0x561d3d1a5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.504+0000 7f6e60a00640 1 -- 172.21.15.102:0/824323020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 msgr2=0x561d3d1a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.504+0000 7f6e60a00640 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.504+0000 7f6e60a00640 1 -- 172.21.15.102:0/824323020 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d3d216b40 con 0x561d3d1961e0 2024-04-25T01:55:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.504+0000 7f6e61400640 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.504+0000 7f6e60a00640 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1a4e80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e4801b790 tx=0x7f6e4801b720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b59ad2e038ac9dde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.504+0000 7f6e42a00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:55:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.505+0000 7f6e42a00640 1 -- 172.21.15.102:0/824323020 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4800fbe0 con 0x561d3d1961e0 2024-04-25T01:55:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.505+0000 7f6e66da2740 1 -- 172.21.15.102:0/824323020 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e4801b4b0 con 0x561d3d1961e0 2024-04-25T01:55:54.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.505+0000 7f6e42a00640 1 -- 172.21.15.102:0/824323020 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6e4800fd80 con 0x561d3d1961e0 2024-04-25T01:55:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.506+0000 7f6e42a00640 1 -- 172.21.15.102:0/824323020 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6e480035f0 con 0x561d3d1961e0 2024-04-25T01:55:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.506+0000 7f6e66da2740 10 client.25481 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.506+0000 7f6e66da2740 1 -- 172.21.15.102:0/824323020 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561d3d217e30 con 0x561d3d1961e0 2024-04-25T01:55:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.506+0000 7f6e42a00640 1 -- 172.21.15.102:0/824323020 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e4801b4b0 con 0x561d3d1961e0 2024-04-25T01:55:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.506+0000 7f6e42a00640 1 -- 172.21.15.102:0/824323020 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f6e480334f0 con 0x561d3d1961e0 2024-04-25T01:55:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.507+0000 7f6e66da2740 10 client.25481 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.507+0000 7f6e66da2740 10 client.25481 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.507+0000 7f6e66da2740 20 client.25481 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.507+0000 7f6e66da2740 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561d3d218340 0x561d3d21a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.508+0000 7f6e66da2740 4 client.25481 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:55:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.508+0000 7f6e66da2740 1 -- 172.21.15.102:0/824323020 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561d3d107aa0 con 0x561d3d218340 2024-04-25T01:55:54.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.508+0000 7f6e61400640 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561d3d218340 0x561d3d21a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.509+0000 7f6e61400640 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561d3d218340 0x561d3d21a760 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfb8df04459275e server_cookie=c8fc989c5b1f4fbe in_seq=0 out_seq=0 2024-04-25T01:55:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.509+0000 7f6e42a00640 10 client.25481 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.509+0000 7f6e42a00640 1 -- 172.21.15.102:0/824323020 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561d3d107aa0 con 0x561d3d218340 2024-04-25T01:55:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.509+0000 7f6e42a00640 10 client.25481 handle_command_reply: tid=0 2024-04-25T01:55:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.509+0000 7f6e66da2740 1 client.25481 shutdown 2024-04-25T01:55:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.509+0000 7f6e66da2740 1 -- 172.21.15.102:0/824323020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 msgr2=0x561d3d1a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.509+0000 7f6e66da2740 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1a4e80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e4801b790 tx=0x7f6e4801b720 comp rx=0 tx=0).stop 2024-04-25T01:55:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.509+0000 7f6e66da2740 1 -- 172.21.15.102:0/824323020 shutdown_connections 2024-04-25T01:55:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.510+0000 7f6e66da2740 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561d3d218340 0x561d3d21a760 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.510+0000 7f6e66da2740 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d3d1a11d0 0x561d3d1a5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.510+0000 7f6e66da2740 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561d3d196b00 0x561d3d1a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.510+0000 7f6e66da2740 1 --2- 172.21.15.102:0/824323020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561d3d1961e0 0x561d3d1a4e80 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e4801b790 tx=0x7f6e4801b720 comp rx=0 tx=0).stop 2024-04-25T01:55:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.510+0000 7f6e42a00640 0 client.25481 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.510+0000 7f6e66da2740 1 -- 172.21.15.102:0/824323020 >> 172.21.15.102:0/824323020 conn(0x561d3d192500 msgr2=0x561d3d1981c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.510+0000 7f6e66da2740 1 -- 172.21.15.102:0/824323020 shutdown_connections 2024-04-25T01:55:54.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.510+0000 7f6e66da2740 1 -- 172.21.15.102:0/824323020 wait complete. 2024-04-25T01:55:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.511+0000 7f6e66da2740 20 client.25481 trim_cache size 0 max 16384 2024-04-25T01:55:54.513 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:55:54.513 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T01:55:54.513 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T01:55:54.513 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:55:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 -- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6e440a2f20 msgr2=0x7f6e440a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6e440a2f20 0x7f6e440a53e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6e540eced0 tx=0x7f6e5400d9d0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 -- 172.21.15.102:0/3176093965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 msgr2=0x7f6e5c1a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 0x7f6e5c1a2d80 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6e54006010 tx=0x7f6e5400b800 comp rx=0 tx=0).stop 2024-04-25T01:55:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 -- 172.21.15.102:0/3176093965 shutdown_connections 2024-04-25T01:55:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6e440a2f20 0x7f6e440a53e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e5c113690 0x7f6e5c1a2d80 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10aa00 0x7f6e5c19ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 --2- 172.21.15.102:0/3176093965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e5c10a0e0 0x7f6e5c19e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 -- 172.21.15.102:0/3176093965 >> 172.21.15.102:0/3176093965 conn(0x7f6e5c06c6c0 msgr2=0x7f6e5c070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 -- 172.21.15.102:0/3176093965 shutdown_connections 2024-04-25T01:55:54.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.513+0000 7f6e62800640 1 -- 172.21.15.102:0/3176093965 wait complete. 2024-04-25T01:55:54.542 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:55:54.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:55:54.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.813+0000 7fdeb7600640 1 -- 172.21.15.102:0/3652307213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeb0074430 msgr2=0x7fdeb010da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.813+0000 7fdeb7600640 1 --2- 172.21.15.102:0/3652307213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeb0074430 0x7fdeb010da60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fde9c009900 tx=0x7fde9c02a480 comp rx=0 tx=0).stop 2024-04-25T01:55:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.815+0000 7fdeb7600640 1 -- 172.21.15.102:0/3652307213 shutdown_connections 2024-04-25T01:55:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.815+0000 7fdeb7600640 1 --2- 172.21.15.102:0/3652307213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdeb010dfa0 0x7fdeb01103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.815+0000 7fdeb7600640 1 --2- 172.21.15.102:0/3652307213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeb0074430 0x7fdeb010da60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.815+0000 7fdeb7600640 1 --2- 172.21.15.102:0/3652307213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdeb0073b10 0x7fdeb0073ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.815+0000 7fdeb7600640 1 -- 172.21.15.102:0/3652307213 >> 172.21.15.102:0/3652307213 conn(0x7fdeb0078b40 msgr2=0x7fdeb0078f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.815+0000 7fdeb7600640 1 -- 172.21.15.102:0/3652307213 shutdown_connections 2024-04-25T01:55:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.815+0000 7fdeb7600640 1 -- 172.21.15.102:0/3652307213 wait complete. 2024-04-25T01:55:54.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.816+0000 7fdeb7600640 1 Processor -- start 2024-04-25T01:55:54.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.817+0000 7fdeb7600640 1 -- start start 2024-04-25T01:55:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeb7600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeb0073b10 0x7fdeb019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeb7600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdeb0074430 0x7fdeb019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeb7600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdeb010dfa0 0x7fdeb019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeb7600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb0113d00 con 0x7fdeb010dfa0 2024-04-25T01:55:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeb7600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb0113b80 con 0x7fdeb0074430 2024-04-25T01:55:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeb7600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb0113e80 con 0x7fdeb0073b10 2024-04-25T01:55:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeafe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdeb0074430 0x7fdeb019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeafe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdeb0074430 0x7fdeb019a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45074/0 (socket says 172.21.15.102:45074) 2024-04-25T01:55:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeafe00640 1 -- 172.21.15.102:0/2458593030 learned_addr learned my addr 172.21.15.102:0/2458593030 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeb4c00640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeb0073b10 0x7fdeb019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeb5600640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdeb010dfa0 0x7fdeb019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeafe00640 1 -- 172.21.15.102:0/2458593030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeb0073b10 msgr2=0x7fdeb019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeafe00640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeb0073b10 0x7fdeb019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeafe00640 1 -- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdeb010dfa0 msgr2=0x7fdeb019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeafe00640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdeb010dfa0 0x7fdeb019eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeafe00640 1 -- 172.21.15.102:0/2458593030 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb019f230 con 0x7fdeb0074430 2024-04-25T01:55:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.818+0000 7fdeb5600640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdeb010dfa0 0x7fdeb019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:54.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.819+0000 7fdeafe00640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdeb0074430 0x7fdeb019a780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fde9c006010 tx=0x7fde9c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:54.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.819+0000 7fdead600640 1 -- 172.21.15.102:0/2458593030 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c03f030 con 0x7fdeb0074430 2024-04-25T01:55:54.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.820+0000 7fdead600640 1 -- 172.21.15.102:0/2458593030 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde9c003ce0 con 0x7fdeb0074430 2024-04-25T01:55:54.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.820+0000 7fdead600640 1 -- 172.21.15.102:0/2458593030 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde9c035070 con 0x7fdeb0074430 2024-04-25T01:55:54.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.820+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb019f4c0 con 0x7fdeb0074430 2024-04-25T01:55:54.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.820+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb01015f0 con 0x7fdeb0074430 2024-04-25T01:55:54.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.823+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb00751c0 con 0x7fdeb0074430 2024-04-25T01:55:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.825+0000 7fdead600640 1 -- 172.21.15.102:0/2458593030 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fde9c042030 con 0x7fdeb0074430 2024-04-25T01:55:54.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.829+0000 7fdead600640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde940a2f60 0x7fde940a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:54.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.829+0000 7fdead600640 1 -- 172.21.15.102:0/2458593030 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fde9c0efac0 con 0x7fdeb0074430 2024-04-25T01:55:54.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.830+0000 7fdeb4c00640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde940a2f60 0x7fde940a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:54.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.831+0000 7fdeb4c00640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde940a2f60 0x7fde940a5420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdea8005d00 tx=0x7fdea800b410 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:54.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:54.831+0000 7fdead600640 1 -- 172.21.15.102:0/2458593030 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fde9c049030 con 0x7fdeb0074430 2024-04-25T01:55:55.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.281+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdeb0109120 con 0x7fdeb0074430 2024-04-25T01:55:55.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.282+0000 7fdead600640 1 -- 172.21.15.102:0/2458593030 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11900 (secure 0 0 0) 0x7fde9c0bc540 con 0x7fdeb0074430 2024-04-25T01:55:55.284 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:55:55.285 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:55:50.927772+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587,24533,24541]},"id":1}]} 2024-04-25T01:55:55.285 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 26 2024-04-25T01:55:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.286+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde940a2f60 msgr2=0x7fde940a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.286+0000 7fdeb7600640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde940a2f60 0x7fde940a5420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdea8005d00 tx=0x7fdea800b410 comp rx=0 tx=0).stop 2024-04-25T01:55:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.286+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdeb0074430 msgr2=0x7fdeb019a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.286+0000 7fdeb7600640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdeb0074430 0x7fdeb019a780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fde9c006010 tx=0x7fde9c0042f0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.287+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 shutdown_connections 2024-04-25T01:55:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.287+0000 7fdeb7600640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fde940a2f60 0x7fde940a5420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.287+0000 7fdeb7600640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdeb010dfa0 0x7fdeb019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.287+0000 7fdeb7600640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdeb0074430 0x7fdeb019a780 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.287+0000 7fdeb7600640 1 --2- 172.21.15.102:0/2458593030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdeb0073b10 0x7fdeb019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.287+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 >> 172.21.15.102:0/2458593030 conn(0x7fdeb0078b40 msgr2=0x7fdeb00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.287+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 shutdown_connections 2024-04-25T01:55:55.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.287+0000 7fdeb7600640 1 -- 172.21.15.102:0/2458593030 wait complete. 2024-04-25T01:55:55.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:55:55.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:55 smithi110 ceph-mon[32848]: pgmap v22: 97 pgs: 97 active+clean; 398 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 55 MiB/s wr, 227 op/s 2024-04-25T01:55:55.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:55 smithi193 ceph-mon[32981]: pgmap v22: 97 pgs: 97 active+clean; 398 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 55 MiB/s wr, 227 op/s 2024-04-25T01:55:55.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.618+0000 7fb8b4e00640 1 -- 172.21.15.102:0/1837254683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0109990 msgr2=0x7fb8b0109d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.618+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/1837254683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0109990 0x7fb8b0109d70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009cd0 tx=0x7fb8a802a8f0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.619+0000 7fb8b4e00640 1 -- 172.21.15.102:0/1837254683 shutdown_connections 2024-04-25T01:55:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.619+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/1837254683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b0104340 0x7fb8b0110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.619+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/1837254683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b0103980 0x7fb8b0103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.619+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/1837254683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0109990 0x7fb8b0109d70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.619+0000 7fb8b4e00640 1 -- 172.21.15.102:0/1837254683 >> 172.21.15.102:0/1837254683 conn(0x7fb8b00ff4d0 msgr2=0x7fb8b01018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.619+0000 7fb8b4e00640 1 -- 172.21.15.102:0/1837254683 shutdown_connections 2024-04-25T01:55:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.619+0000 7fb8b4e00640 1 -- 172.21.15.102:0/1837254683 wait complete. 2024-04-25T01:55:55.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.620+0000 7fb8b4e00640 1 Processor -- start 2024-04-25T01:55:55.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.620+0000 7fb8b4e00640 1 -- start start 2024-04-25T01:55:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.621+0000 7fb8b4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b0103980 0x7fb8b019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.621+0000 7fb8b4e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0104340 0x7fb8b019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.621+0000 7fb8b4e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b0109990 0x7fb8b019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.621+0000 7fb8b4e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b0113d70 con 0x7fb8b0103980 2024-04-25T01:55:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.621+0000 7fb8b4e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b0113bf0 con 0x7fb8b0109990 2024-04-25T01:55:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.621+0000 7fb8b4e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b0113ef0 con 0x7fb8b0104340 2024-04-25T01:55:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.621+0000 7fb8ad400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0104340 0x7fb8b019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.621+0000 7fb8ad400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0104340 0x7fb8b019a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55788/0 (socket says 172.21.15.102:55788) 2024-04-25T01:55:55.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.621+0000 7fb8ad400640 1 -- 172.21.15.102:0/3035063086 learned_addr learned my addr 172.21.15.102:0/3035063086 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.622+0000 7fb8ae800640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b0109990 0x7fb8b019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.622+0000 7fb8ade00640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b0103980 0x7fb8b019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.622+0000 7fb8ae800640 1 -- 172.21.15.102:0/3035063086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0104340 msgr2=0x7fb8b019a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.622+0000 7fb8ae800640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0104340 0x7fb8b019a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.622+0000 7fb8ae800640 1 -- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b0103980 msgr2=0x7fb8b019a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.622+0000 7fb8ae800640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b0103980 0x7fb8b019a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.622+0000 7fb8ae800640 1 -- 172.21.15.102:0/3035063086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b019f2b0 con 0x7fb8b0109990 2024-04-25T01:55:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.622+0000 7fb8ad400640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0104340 0x7fb8b019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.623+0000 7fb8ae800640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b0109990 0x7fb8b019eb80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000d680 tx=0x7fb8a000db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.623+0000 7fb8a6a00640 1 -- 172.21.15.102:0/3035063086 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0017070 con 0x7fb8b0109990 2024-04-25T01:55:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.623+0000 7fb8a6a00640 1 -- 172.21.15.102:0/3035063086 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8a0009820 con 0x7fb8b0109990 2024-04-25T01:55:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.623+0000 7fb8a6a00640 1 -- 172.21.15.102:0/3035063086 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0017070 con 0x7fb8b0109990 2024-04-25T01:55:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.623+0000 7fb8b4e00640 1 -- 172.21.15.102:0/3035063086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b019f5a0 con 0x7fb8b0109990 2024-04-25T01:55:55.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.623+0000 7fb8b4e00640 1 -- 172.21.15.102:0/3035063086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b01a6e30 con 0x7fb8b0109990 2024-04-25T01:55:55.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.630+0000 7fb8b7d69740 1 Processor -- start 2024-04-25T01:55:55.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.631+0000 7fb8a6a00640 1 -- 172.21.15.102:0/3035063086 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fb8a00099d0 con 0x7fb8b0109990 2024-04-25T01:55:55.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.631+0000 7fb8b7d69740 1 -- start start 2024-04-25T01:55:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.632+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960aad99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.632+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad9ee0 0x55960aae4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.632+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960aae69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.632+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55960aa3e4e0 con 0x55960aae45b0 2024-04-25T01:55:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.632+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55960a9e6a70 con 0x55960aad95c0 2024-04-25T01:55:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.632+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55960aa3a4a0 con 0x55960aad9ee0 2024-04-25T01:55:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.632+0000 7fb8ade00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad9ee0 0x55960aae4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.633+0000 7fb8ade00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad9ee0 0x55960aae4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55792/0 (socket says 172.21.15.102:55792) 2024-04-25T01:55:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.633+0000 7fb8ade00640 1 -- 172.21.15.102:0/3735965973 learned_addr learned my addr 172.21.15.102:0/3735965973 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.633+0000 7fb8ad400640 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960aae69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.633+0000 7fb8ad400640 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960aad99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.634+0000 7fb8ade00640 1 -- 172.21.15.102:0/3735965973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 msgr2=0x55960aad99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.634+0000 7fb8ade00640 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960aad99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.634+0000 7fb8ade00640 1 -- 172.21.15.102:0/3735965973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 msgr2=0x55960aae69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.634+0000 7fb8ade00640 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960aae69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.634+0000 7fb8ade00640 1 -- 172.21.15.102:0/3735965973 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55960a7523f0 con 0x55960aad9ee0 2024-04-25T01:55:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.634+0000 7fb8ad400640 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960aae69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.634+0000 7fb8a6a00640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb8940a3010 0x7fb8940a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.635+0000 7fb8ade00640 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad9ee0 0x55960aae4070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8000f80 tx=0x7fb8a80098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e82d7d95e46dcfa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.635+0000 7fb8a6a00640 1 -- 172.21.15.102:0/3035063086 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb8a00cea40 con 0x7fb8b0109990 2024-04-25T01:55:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.635+0000 7fb8ad400640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb8940a3010 0x7fb8940a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.635+0000 7fb89b400640 1 -- 172.21.15.102:0/3735965973 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8030d30 con 0x55960aad9ee0 2024-04-25T01:55:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.635+0000 7fb89b400640 1 -- 172.21.15.102:0/3735965973 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8a8036040 con 0x55960aad9ee0 2024-04-25T01:55:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.636+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3735965973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad9ee0 msgr2=0x55960aae4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.636+0000 7fb89b400640 1 -- 172.21.15.102:0/3735965973 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8030d30 con 0x55960aad9ee0 2024-04-25T01:55:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.637+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad9ee0 0x55960aae4070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8000f80 tx=0x7fb8a80098b0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.637+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3735965973 shutdown_connections 2024-04-25T01:55:55.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.637+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960aae69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.637+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad9ee0 0x55960aae4070 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.637+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/3735965973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960aad99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.637+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3735965973 >> 172.21.15.102:0/3735965973 conn(0x55960aad58e0 msgr2=0x55960aad5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.637+0000 7fb8ad400640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb8940a3010 0x7fb8940a54d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55960aad6a70 tx=0x7fb89c00b2c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:55.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.638+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3735965973 shutdown_connections 2024-04-25T01:55:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.638+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3735965973 wait complete. 2024-04-25T01:55:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.639+0000 7fb8b7d69740 1 -- start start 2024-04-25T01:55:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.639+0000 7fb8b7d69740 1 Processor -- start 2024-04-25T01:55:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.640+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960aae1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.641+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aae1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.641+0000 7fb8ade00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960aae1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.641+0000 7fb8ad400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aae1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.641+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 0x55960aae1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.642+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55960aaefba0 con 0x55960aad9ee0 2024-04-25T01:55:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.642+0000 7fb8ad400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aae1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57210/0 (socket says 172.21.15.102:57210) 2024-04-25T01:55:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.642+0000 7fb8ad400640 1 -- 172.21.15.102:0/2538954006 learned_addr learned my addr 172.21.15.102:0/2538954006 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.642+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55960a9f9ec0 con 0x55960aad95c0 2024-04-25T01:55:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.642+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55960aaefd20 con 0x55960aae45b0 2024-04-25T01:55:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.642+0000 7fb8ae800640 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 0x55960aae1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.642+0000 7fb8b7d69740 10 client.0 _ll_register_callbacks cb 0x55960aa363f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.642+0000 7fb8ae800640 1 -- 172.21.15.102:0/2538954006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 msgr2=0x55960aae1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.643+0000 7fb8ae800640 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960aae1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.643+0000 7fb8ae800640 1 -- 172.21.15.102:0/2538954006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 msgr2=0x55960aae1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.643+0000 7fb8ae800640 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aae1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.643+0000 7fb8ae800640 1 -- 172.21.15.102:0/2538954006 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55960a81c7d0 con 0x55960aae45b0 2024-04-25T01:55:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.643+0000 7fb8ade00640 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960aae1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.643+0000 7fb8ae800640 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 0x55960aae1ec0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00cc100 tx=0x7fb8a0004800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d474b941248d3a34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.643+0000 7fb88fe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:55:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.644+0000 7fb88fe00640 1 -- 172.21.15.102:0/2538954006 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0020040 con 0x55960aae45b0 2024-04-25T01:55:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.644+0000 7fb88fe00640 1 -- 172.21.15.102:0/2538954006 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb8a00cc3b0 con 0x55960aae45b0 2024-04-25T01:55:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.644+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8a8036040 con 0x55960aae45b0 2024-04-25T01:55:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.645+0000 7fb88fe00640 1 -- 172.21.15.102:0/2538954006 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb8a009f1d0 con 0x55960aae45b0 2024-04-25T01:55:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.645+0000 7fb8b7d69740 10 client.25489 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.645+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55960aafbb80 con 0x55960aae45b0 2024-04-25T01:55:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.646+0000 7fb88fe00640 1 -- 172.21.15.102:0/2538954006 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8a8036040 con 0x55960aae45b0 2024-04-25T01:55:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.648+0000 7fb88fe00640 1 -- 172.21.15.102:0/2538954006 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fb8a009a270 con 0x55960aae45b0 2024-04-25T01:55:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.648+0000 7fb8b7d69740 10 client.25489 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.648+0000 7fb8b7d69740 10 client.25489 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:55:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.648+0000 7fb8b7d69740 20 client.25489 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.648+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55960aafcf90 0x55960aaff3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.649+0000 7fb8b7d69740 4 client.25489 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.649+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55960aa4ba20 con 0x55960aafcf90 2024-04-25T01:55:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.649+0000 7fb8ade00640 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55960aafcf90 0x55960aaff3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.650+0000 7fb8ade00640 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55960aafcf90 0x55960aaff3c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c059ca730c90b409 server_cookie=6be4b8afbaa0e825 in_seq=0 out_seq=0 2024-04-25T01:55:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.650+0000 7fb88fe00640 10 client.25489 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:55:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.651+0000 7fb88fe00640 1 -- 172.21.15.102:0/2538954006 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55960aa4ba20 con 0x55960aafcf90 2024-04-25T01:55:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.651+0000 7fb88fe00640 10 client.25489 handle_command_reply: tid=0 2024-04-25T01:55:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.652+0000 7fb8b7d69740 1 client.25489 shutdown 2024-04-25T01:55:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.652+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 msgr2=0x55960aae1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.652+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 0x55960aae1ec0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00cc100 tx=0x7fb8a0004800 comp rx=0 tx=0).stop 2024-04-25T01:55:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 shutdown_connections 2024-04-25T01:55:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 0x55960aae1ec0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aae1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55960aafcf90 0x55960aaff3c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/2538954006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960aae1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb88fe00640 0 client.25489 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:55:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 >> 172.21.15.102:0/2538954006 conn(0x55960aad58e0 msgr2=0x55960aadb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 shutdown_connections 2024-04-25T01:55:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb8b7d69740 1 -- 172.21.15.102:0/2538954006 wait complete. 2024-04-25T01:55:55.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.653+0000 7fb8b7d69740 20 client.25489 trim_cache size 0 max 16384 2024-04-25T01:55:55.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:55 smithi102 ceph-mon[27796]: pgmap v22: 97 pgs: 97 active+clean; 398 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 55 MiB/s wr, 227 op/s 2024-04-25T01:55:55.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.675+0000 7fb8b7d69740 1 Processor -- start 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.675+0000 7fb8b7d69740 1 -- start start 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.675+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad95c0 0x55960aaf8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.675+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad9ee0 0x55960aae1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.675+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960ab04c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.675+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b01adb00 con 0x55960aae45b0 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.675+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b01ab3a0 con 0x55960aad9ee0 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.675+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b01a9210 con 0x55960aad95c0 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.676+0000 7fb8ad400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad9ee0 0x55960aae1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.676+0000 7fb8ad400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad9ee0 0x55960aae1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45106/0 (socket says 172.21.15.102:45106) 2024-04-25T01:55:55.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.676+0000 7fb8ad400640 1 -- 172.21.15.102:0/3023486244 learned_addr learned my addr 172.21.15.102:0/3023486244 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:55.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.676+0000 7fb8ade00640 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad95c0 0x55960aaf8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.676+0000 7fb8ae800640 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960ab04c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.677+0000 7fb8ad400640 1 -- 172.21.15.102:0/3023486244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad95c0 msgr2=0x55960aaf8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:55.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.677+0000 7fb8ad400640 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad95c0 0x55960aaf8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.677+0000 7fb8ad400640 1 -- 172.21.15.102:0/3023486244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 msgr2=0x55960ab04c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:55.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.678+0000 7fb8ad400640 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960ab04c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.678+0000 7fb8ad400640 1 -- 172.21.15.102:0/3023486244 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55960aad9d10 con 0x55960aad9ee0 2024-04-25T01:55:55.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.678+0000 7fb8ae800640 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960ab04c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:55.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.679+0000 7fb8ad400640 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad9ee0 0x55960aae1ce0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb89c010620 tx=0x7fb89c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=794da8b8cdcb3526 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:55.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.680+0000 7fb89aa00640 1 -- 172.21.15.102:0/3023486244 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c02a070 con 0x55960aad9ee0 2024-04-25T01:55:55.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.680+0000 7fb89aa00640 1 -- 172.21.15.102:0/3023486244 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb89c01bba0 con 0x55960aad9ee0 2024-04-25T01:55:55.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.681+0000 7fb89aa00640 1 -- 172.21.15.102:0/3023486244 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c025d70 con 0x55960aad9ee0 2024-04-25T01:55:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.681+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3023486244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad9ee0 msgr2=0x55960aae1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.681+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad9ee0 0x55960aae1ce0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb89c010620 tx=0x7fb89c014000 comp rx=0 tx=0).stop 2024-04-25T01:55:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.682+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3023486244 shutdown_connections 2024-04-25T01:55:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.682+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aae45b0 0x55960ab04c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.682+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad9ee0 0x55960aae1ce0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.682+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/3023486244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aad95c0 0x55960aaf8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.682+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3023486244 >> 172.21.15.102:0/3023486244 conn(0x55960aad58e0 msgr2=0x55960aad4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.682+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3023486244 shutdown_connections 2024-04-25T01:55:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.682+0000 7fb8b7d69740 1 -- 172.21.15.102:0/3023486244 wait complete. 2024-04-25T01:55:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.683+0000 7fb8b7d69740 1 -- start start 2024-04-25T01:55:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.683+0000 7fb8b7d69740 1 Processor -- start 2024-04-25T01:55:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.684+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960ab5eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.684+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aaf5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.684+0000 7fb8b7d69740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 0x55960aaf5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.684+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b01a9860 con 0x55960aad9ee0 2024-04-25T01:55:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.684+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b01aacd0 con 0x55960aad95c0 2024-04-25T01:55:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.684+0000 7fb8b7d69740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b01a9f30 con 0x55960aae45b0 2024-04-25T01:55:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.684+0000 7fb8b7d69740 10 client.0 _ll_register_callbacks cb 0x7fb8a00cbf90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.684+0000 7fb8ae800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 0x55960aaf5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.684+0000 7fb8ade00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960ab5eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.685+0000 7fb8ade00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960ab5eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45118/0 (socket says 172.21.15.102:45118) 2024-04-25T01:55:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.685+0000 7fb8ad400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aaf5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.685+0000 7fb8ad400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aaf5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57232/0 (socket says 172.21.15.102:57232) 2024-04-25T01:55:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.685+0000 7fb8ad400640 1 -- 172.21.15.102:0/4281690758 learned_addr learned my addr 172.21.15.102:0/4281690758 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.685+0000 7fb8ad400640 1 -- 172.21.15.102:0/4281690758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 msgr2=0x55960aaf5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.685+0000 7fb8ad400640 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 0x55960aaf5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.685+0000 7fb8ad400640 1 -- 172.21.15.102:0/4281690758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 msgr2=0x55960ab5eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.685+0000 7fb8ad400640 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960ab5eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.685+0000 7fb8ad400640 1 -- 172.21.15.102:0/4281690758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55960aade150 con 0x55960aad9ee0 2024-04-25T01:55:55.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.686+0000 7fb8ad400640 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aaf5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00be60 tx=0x7fb89c025f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950ebd4b21227d1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:55.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.687+0000 7fb88fe00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:55.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.687+0000 7fb88fe00640 1 -- 172.21.15.102:0/4281690758 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89c02a040 con 0x55960aad9ee0 2024-04-25T01:55:55.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.688+0000 7fb88fe00640 1 -- 172.21.15.102:0/4281690758 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb89c01b4c0 con 0x55960aad9ee0 2024-04-25T01:55:55.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.688+0000 7fb8b7d69740 1 -- 172.21.15.102:0/4281690758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb89c01bba0 con 0x55960aad9ee0 2024-04-25T01:55:55.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.689+0000 7fb88fe00640 1 -- 172.21.15.102:0/4281690758 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb89c01b700 con 0x55960aad9ee0 2024-04-25T01:55:55.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.689+0000 7fb8b7d69740 10 client.15408 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:55.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.689+0000 7fb8b7d69740 1 -- 172.21.15.102:0/4281690758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55960ab74760 con 0x55960aad9ee0 2024-04-25T01:55:55.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.690+0000 7fb88fe00640 1 -- 172.21.15.102:0/4281690758 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb89c01bba0 con 0x55960aad9ee0 2024-04-25T01:55:55.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.691+0000 7fb88fe00640 1 -- 172.21.15.102:0/4281690758 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fb89c02d030 con 0x55960aad9ee0 2024-04-25T01:55:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.691+0000 7fb8b7d69740 10 client.15408 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.691+0000 7fb8b7d69740 10 client.15408 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:55:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.691+0000 7fb8b7d69740 20 client.15408 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.691+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55960ab74e80 0x55960ab772a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.691+0000 7fb8b7d69740 4 client.15408 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.691+0000 7fb8b7d69740 1 -- 172.21.15.102:0/4281690758 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55960aa45d70 con 0x55960ab74e80 2024-04-25T01:55:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.692+0000 7fb8ade00640 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55960ab74e80 0x55960ab772a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.693+0000 7fb8ade00640 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55960ab74e80 0x55960ab772a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=857e4f0138f2adc3 server_cookie=25ba72bf509d9235 in_seq=0 out_seq=0 2024-04-25T01:55:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.693+0000 7fb88fe00640 10 client.15408 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:55:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.694+0000 7fb88fe00640 1 -- 172.21.15.102:0/4281690758 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55960aa45d70 con 0x55960ab74e80 2024-04-25T01:55:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.694+0000 7fb88fe00640 10 client.15408 handle_command_reply: tid=0 2024-04-25T01:55:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.694+0000 7fb8b7d69740 1 client.15408 shutdown 2024-04-25T01:55:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.694+0000 7fb8b7d69740 1 -- 172.21.15.102:0/4281690758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 msgr2=0x55960aaf5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.694+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aaf5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb89c00be60 tx=0x7fb89c025f90 comp rx=0 tx=0).stop 2024-04-25T01:55:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.695+0000 7fb8b7d69740 1 -- 172.21.15.102:0/4281690758 shutdown_connections 2024-04-25T01:55:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.695+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55960aae45b0 0x55960aaf5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.695+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55960aad9ee0 0x55960aaf5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.695+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55960ab74e80 0x55960ab772a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.695+0000 7fb8b7d69740 1 --2- 172.21.15.102:0/4281690758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55960aad95c0 0x55960ab5eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.695+0000 7fb88fe00640 0 client.15408 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:55:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.695+0000 7fb8b7d69740 1 -- 172.21.15.102:0/4281690758 >> 172.21.15.102:0/4281690758 conn(0x55960aad58e0 msgr2=0x55960aae3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.695+0000 7fb8b7d69740 1 -- 172.21.15.102:0/4281690758 shutdown_connections 2024-04-25T01:55:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.695+0000 7fb8b7d69740 1 -- 172.21.15.102:0/4281690758 wait complete. 2024-04-25T01:55:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.696+0000 7fb8b7d69740 20 client.15408 trim_cache size 0 max 16384 2024-04-25T01:55:55.698 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:55:55.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.697+0000 7fb8b4e00640 1 -- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb8940a3010 msgr2=0x7fb8940a54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.697+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb8940a3010 0x7fb8940a54d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55960aad6a70 tx=0x7fb89c00b2c0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.697+0000 7fb8b4e00640 1 -- 172.21.15.102:0/3035063086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b0109990 msgr2=0x7fb8b019eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.697+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b0109990 0x7fb8b019eb80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000d680 tx=0x7fb8a000db40 comp rx=0 tx=0).stop 2024-04-25T01:55:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.698+0000 7fb8b4e00640 1 -- 172.21.15.102:0/3035063086 shutdown_connections 2024-04-25T01:55:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.698+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb8940a3010 0x7fb8940a54d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.698+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b0109990 0x7fb8b019eb80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.698+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8b0104340 0x7fb8b019a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.698+0000 7fb8b4e00640 1 --2- 172.21.15.102:0/3035063086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b0103980 0x7fb8b019a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.698+0000 7fb8b4e00640 1 -- 172.21.15.102:0/3035063086 >> 172.21.15.102:0/3035063086 conn(0x7fb8b00ff4d0 msgr2=0x7fb8b00ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:55.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.698+0000 7fb8b4e00640 1 -- 172.21.15.102:0/3035063086 shutdown_connections 2024-04-25T01:55:55.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:55.698+0000 7fb8b4e00640 1 -- 172.21.15.102:0/3035063086 wait complete. 2024-04-25T01:55:55.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:55:56.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.005+0000 7f0d62000640 1 -- 172.21.15.102:0/1273666212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c100d60 msgr2=0x7f0d5c10d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.005+0000 7f0d62000640 1 --2- 172.21.15.102:0/1273666212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c100d60 0x7f0d5c10d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0d48009900 tx=0x7f0d4802a420 comp rx=0 tx=0).stop 2024-04-25T01:55:56.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.006+0000 7f0d62000640 1 -- 172.21.15.102:0/1273666212 shutdown_connections 2024-04-25T01:55:56.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.006+0000 7f0d62000640 1 --2- 172.21.15.102:0/1273666212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d5c10df60 0x7f0d5c110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.006+0000 7f0d62000640 1 --2- 172.21.15.102:0/1273666212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c100d60 0x7f0d5c10d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.006+0000 7f0d62000640 1 --2- 172.21.15.102:0/1273666212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d5c1003b0 0x7f0d5c100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.006+0000 7f0d62000640 1 -- 172.21.15.102:0/1273666212 >> 172.21.15.102:0/1273666212 conn(0x7f0d5c0fbfc0 msgr2=0x7f0d5c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.006+0000 7f0d62000640 1 -- 172.21.15.102:0/1273666212 shutdown_connections 2024-04-25T01:55:56.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.006+0000 7f0d62000640 1 -- 172.21.15.102:0/1273666212 wait complete. 2024-04-25T01:55:56.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.008+0000 7f0d62000640 1 Processor -- start 2024-04-25T01:55:56.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.008+0000 7f0d62000640 1 -- start start 2024-04-25T01:55:56.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.008+0000 7f0d62000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c1003b0 0x7f0d5c108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.008+0000 7f0d62000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d5c100d60 0x7f0d5c1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.008+0000 7f0d62000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d5c10df60 0x7f0d5c103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d62000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c06bbb0 con 0x7f0d5c100d60 2024-04-25T01:55:56.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d62000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c06ba30 con 0x7f0d5c1003b0 2024-04-25T01:55:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d62000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c06bd30 con 0x7f0d5c10df60 2024-04-25T01:55:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d60c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c1003b0 0x7f0d5c108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d60c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c1003b0 0x7f0d5c108190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45134/0 (socket says 172.21.15.102:45134) 2024-04-25T01:55:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d60c00640 1 -- 172.21.15.102:0/4243377990 learned_addr learned my addr 172.21.15.102:0/4243377990 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d61600640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d5c10df60 0x7f0d5c103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d60c00640 1 -- 172.21.15.102:0/4243377990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d5c10df60 msgr2=0x7f0d5c103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d60c00640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d5c10df60 0x7f0d5c103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d60c00640 1 -- 172.21.15.102:0/4243377990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d5c100d60 msgr2=0x7f0d5c1086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d60c00640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d5c100d60 0x7f0d5c1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.009+0000 7f0d60c00640 1 -- 172.21.15.102:0/4243377990 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5c103bd0 con 0x7f0d5c1003b0 2024-04-25T01:55:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.010+0000 7f0d60c00640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c1003b0 0x7f0d5c108190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000c920 tx=0x7f0d5000cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.010+0000 7f0d61600640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d5c10df60 0x7f0d5c103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.010+0000 7f0d59600640 1 -- 172.21.15.102:0/4243377990 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d50007c70 con 0x7f0d5c1003b0 2024-04-25T01:55:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.010+0000 7f0d59600640 1 -- 172.21.15.102:0/4243377990 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d50007e10 con 0x7f0d5c1003b0 2024-04-25T01:55:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.010+0000 7f0d62000640 1 -- 172.21.15.102:0/4243377990 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d5c103ec0 con 0x7f0d5c1003b0 2024-04-25T01:55:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.011+0000 7f0d59600640 1 -- 172.21.15.102:0/4243377990 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d500188f0 con 0x7f0d5c1003b0 2024-04-25T01:55:56.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.011+0000 7f0d62000640 1 -- 172.21.15.102:0/4243377990 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d5c1b3950 con 0x7f0d5c1003b0 2024-04-25T01:55:56.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.015+0000 7f0d59600640 1 -- 172.21.15.102:0/4243377990 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f0d50018b30 con 0x7f0d5c1003b0 2024-04-25T01:55:56.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.015+0000 7f0d6641e740 1 Processor -- start 2024-04-25T01:55:56.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.015+0000 7f0d6641e740 1 -- start start 2024-04-25T01:55:56.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.016+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae4cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.016+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae57250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.016+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae59b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.016+0000 7f0d6641e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55afcadb14e0 con 0x55afcae4c7a0 2024-04-25T01:55:56.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.016+0000 7f0d6641e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55afcad59a70 con 0x55afcae57790 2024-04-25T01:55:56.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.016+0000 7f0d6641e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55afcadad4a0 con 0x55afcae4d0c0 2024-04-25T01:55:56.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.017+0000 7f0d59600640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d440a2ed0 0x7f0d440a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.017+0000 7f0d5be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae59b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.018+0000 7f0d59600640 1 -- 172.21.15.102:0/4243377990 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0d500cf750 con 0x7f0d5c1003b0 2024-04-25T01:55:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.018+0000 7f0d5be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae59b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45146/0 (socket says 172.21.15.102:45146) 2024-04-25T01:55:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.018+0000 7f0d5be00640 1 -- 172.21.15.102:0/2178075388 learned_addr learned my addr 172.21.15.102:0/2178075388 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.018+0000 7f0d61600640 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae4cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.018+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae57250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.019+0000 7f0d60c00640 1 -- 172.21.15.102:0/2178075388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 msgr2=0x55afcae59b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.019+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae59b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.019+0000 7f0d60c00640 1 -- 172.21.15.102:0/2178075388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 msgr2=0x55afcae4cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.019+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae4cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.019+0000 7f0d60c00640 1 -- 172.21.15.102:0/2178075388 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afcaac53f0 con 0x55afcae4d0c0 2024-04-25T01:55:56.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.019+0000 7f0d61600640 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae4cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:56.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.019+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae57250 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0d500ccdb0 tx=0x7f0d5009d360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8729dca57e3baf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.020+0000 7f0d4dc00640 1 -- 172.21.15.102:0/2178075388 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d500a5810 con 0x55afcae4d0c0 2024-04-25T01:55:56.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.020+0000 7f0d61600640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d440a2ed0 0x7f0d440a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.020+0000 7f0d4dc00640 1 -- 172.21.15.102:0/2178075388 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d5001fdb0 con 0x55afcae4d0c0 2024-04-25T01:55:56.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.020+0000 7f0d4dc00640 1 -- 172.21.15.102:0/2178075388 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d500cd2c0 con 0x55afcae4d0c0 2024-04-25T01:55:56.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.022+0000 7f0d6641e740 1 -- 172.21.15.102:0/2178075388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 msgr2=0x55afcae57250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.022+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae57250 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0d500ccdb0 tx=0x7f0d5009d360 comp rx=0 tx=0).stop 2024-04-25T01:55:56.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.022+0000 7f0d61600640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d440a2ed0 0x7f0d440a5390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c108db0 tx=0x7f0d54004a40 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.022+0000 7f0d6641e740 1 -- 172.21.15.102:0/2178075388 shutdown_connections 2024-04-25T01:55:56.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.023+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae59b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.023+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae57250 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.023+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2178075388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae4cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.023+0000 7f0d6641e740 1 -- 172.21.15.102:0/2178075388 >> 172.21.15.102:0/2178075388 conn(0x55afcae48ac0 msgr2=0x55afcae48ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.024+0000 7f0d6641e740 1 -- 172.21.15.102:0/2178075388 shutdown_connections 2024-04-25T01:55:56.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.024+0000 7f0d6641e740 1 -- 172.21.15.102:0/2178075388 wait complete. 2024-04-25T01:55:56.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.024+0000 7f0d6641e740 1 -- start start 2024-04-25T01:55:56.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.025+0000 7f0d6641e740 1 Processor -- start 2024-04-25T01:55:56.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.026+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae546e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.027+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae54c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.027+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae55160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.027+0000 7f0d6641e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55afcae5a580 con 0x55afcae4c7a0 2024-04-25T01:55:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.027+0000 7f0d61600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae55160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.027+0000 7f0d61600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae55160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45154/0 (socket says 172.21.15.102:45154) 2024-04-25T01:55:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.027+0000 7f0d61600640 1 -- 172.21.15.102:0/3873204445 learned_addr learned my addr 172.21.15.102:0/3873204445 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.027+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55afcad6cec0 con 0x55afcae57790 2024-04-25T01:55:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.027+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55afcae5a700 con 0x55afcae4d0c0 2024-04-25T01:55:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.028+0000 7f0d61600640 1 -- 172.21.15.102:0/3873204445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 msgr2=0x55afcae54c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:55:56.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.028+0000 7f0d6641e740 10 client.0 _ll_register_callbacks cb 0x55afcada93f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:56.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.028+0000 7f0d5be00640 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae54c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.029+0000 7f0d60c00640 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.029+0000 7f0d61600640 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae54c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.029+0000 7f0d61600640 1 -- 172.21.15.102:0/3873204445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 msgr2=0x55afcae546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.029+0000 7f0d61600640 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.029+0000 7f0d61600640 1 -- 172.21.15.102:0/3873204445 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afcab8f7d0 con 0x55afcae57790 2024-04-25T01:55:56.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.029+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d5001fdb0 con 0x55afcae57790 2024-04-25T01:55:56.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.029+0000 7f0d61600640 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae55160 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0d54012b00 tx=0x7f0d54004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=309a3ddffa4037e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.030+0000 7f0d42000640 10 client.25514 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:55:56.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.030+0000 7f0d60c00640 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:56.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.030+0000 7f0d5be00640 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae54c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:56.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.030+0000 7f0d42000640 1 -- 172.21.15.102:0/3873204445 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54023020 con 0x55afcae57790 2024-04-25T01:55:56.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.031+0000 7f0d42000640 1 -- 172.21.15.102:0/3873204445 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0d54013b80 con 0x55afcae57790 2024-04-25T01:55:56.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.031+0000 7f0d42000640 1 -- 172.21.15.102:0/3873204445 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0d54023020 con 0x55afcae57790 2024-04-25T01:55:56.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.031+0000 7f0d6641e740 10 client.25514 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:56.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.031+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55afcae66810 con 0x55afcae57790 2024-04-25T01:55:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.032+0000 7f0d42000640 1 -- 172.21.15.102:0/3873204445 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f0d5401f440 con 0x55afcae57790 2024-04-25T01:55:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.032+0000 7f0d6641e740 10 client.25514 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.032+0000 7f0d6641e740 10 client.25514 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:55:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.032+0000 7f0d6641e740 20 client.25514 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.032+0000 7f0d6641e740 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55afcae67c20 0x55afcae6a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.032+0000 7f0d6641e740 4 client.25514 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.032+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afcadbea20 con 0x55afcae67c20 2024-04-25T01:55:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.032+0000 7f0d60c00640 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55afcae67c20 0x55afcae6a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.033+0000 7f0d60c00640 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55afcae67c20 0x55afcae6a060 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5a2732ff1e6edf6e server_cookie=4c7cc229736cb18d in_seq=0 out_seq=0 2024-04-25T01:55:56.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.033+0000 7f0d42000640 10 client.25514 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:55:56.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.034+0000 7f0d42000640 1 -- 172.21.15.102:0/3873204445 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55afcadbea20 con 0x55afcae67c20 2024-04-25T01:55:56.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.034+0000 7f0d42000640 10 client.25514 handle_command_reply: tid=0 2024-04-25T01:55:56.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.034+0000 7f0d6641e740 1 client.25514 shutdown 2024-04-25T01:55:56.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.035+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 msgr2=0x55afcae55160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.035+0000 7f0d6641e740 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae55160 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0d54012b00 tx=0x7f0d54004b50 comp rx=0 tx=0).stop 2024-04-25T01:55:56.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.035+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 shutdown_connections 2024-04-25T01:55:56.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.035+0000 7f0d6641e740 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55afcae67c20 0x55afcae6a060 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.035+0000 7f0d42000640 0 client.25514 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:55:56.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.036+0000 7f0d6641e740 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae55160 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.036+0000 7f0d6641e740 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae54c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.036+0000 7f0d6641e740 1 --2- 172.21.15.102:0/3873204445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.036+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 >> 172.21.15.102:0/3873204445 conn(0x55afcae48ac0 msgr2=0x55afcae4e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.036+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 shutdown_connections 2024-04-25T01:55:56.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.036+0000 7f0d6641e740 1 -- 172.21.15.102:0/3873204445 wait complete. 2024-04-25T01:55:56.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.037+0000 7f0d6641e740 20 client.25514 trim_cache size 0 max 16384 2024-04-25T01:55:56.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.062+0000 7f0d6641e740 1 Processor -- start 2024-04-25T01:55:56.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.062+0000 7f0d6641e740 1 -- start start 2024-04-25T01:55:56.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.063+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae63470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.063+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae4ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.063+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.063+0000 7f0d6641e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c010ff0 con 0x55afcae4c7a0 2024-04-25T01:55:56.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.063+0000 7f0d6641e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c00e890 con 0x55afcae57790 2024-04-25T01:55:56.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.063+0000 7f0d6641e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c00c6e0 con 0x55afcae4d0c0 2024-04-25T01:55:56.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.064+0000 7f0d61600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.064+0000 7f0d61600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45164/0 (socket says 172.21.15.102:45164) 2024-04-25T01:55:56.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.065+0000 7f0d61600640 1 -- 172.21.15.102:0/2748352396 learned_addr learned my addr 172.21.15.102:0/2748352396 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.065+0000 7f0d5be00640 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae4ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.066+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae63470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.067+0000 7f0d61600640 1 -- 172.21.15.102:0/2748352396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 msgr2=0x55afcae4ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.067+0000 7f0d61600640 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae4ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.067+0000 7f0d61600640 1 -- 172.21.15.102:0/2748352396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 msgr2=0x55afcae63470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.067+0000 7f0d61600640 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae63470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.067+0000 7f0d61600640 1 -- 172.21.15.102:0/2748352396 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afcae42f90 con 0x55afcae57790 2024-04-25T01:55:56.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.067+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae63470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:56.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.068+0000 7f0d61600640 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae479c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0d540076d0 tx=0x7f0d5401e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb1c49976fd98f98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.068+0000 7f0d4d200640 1 -- 172.21.15.102:0/2748352396 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54012040 con 0x55afcae57790 2024-04-25T01:55:56.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.069+0000 7f0d4d200640 1 -- 172.21.15.102:0/2748352396 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d54037030 con 0x55afcae57790 2024-04-25T01:55:56.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.069+0000 7f0d4d200640 1 -- 172.21.15.102:0/2748352396 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54033b60 con 0x55afcae57790 2024-04-25T01:55:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.070+0000 7f0d6641e740 1 -- 172.21.15.102:0/2748352396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 msgr2=0x55afcae479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.070+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae479c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0d540076d0 tx=0x7f0d5401e000 comp rx=0 tx=0).stop 2024-04-25T01:55:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.070+0000 7f0d6641e740 1 -- 172.21.15.102:0/2748352396 shutdown_connections 2024-04-25T01:55:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.070+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae57790 0x55afcae479c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.070+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae4ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.070+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2748352396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae4c7a0 0x55afcae63470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.070+0000 7f0d6641e740 1 -- 172.21.15.102:0/2748352396 >> 172.21.15.102:0/2748352396 conn(0x55afcae48ac0 msgr2=0x55afcae472b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.070+0000 7f0d6641e740 1 -- 172.21.15.102:0/2748352396 shutdown_connections 2024-04-25T01:55:56.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.070+0000 7f0d6641e740 1 -- 172.21.15.102:0/2748352396 wait complete. 2024-04-25T01:55:56.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.071+0000 7f0d6641e740 1 -- start start 2024-04-25T01:55:56.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.071+0000 7f0d6641e740 1 Processor -- start 2024-04-25T01:55:56.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.072+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae4c7a0 0x55afcae5fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.073+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae5b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.073+0000 7f0d6641e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae57790 0x55afcae5b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.073+0000 7f0d6641e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c00cd50 con 0x55afcae57790 2024-04-25T01:55:56.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.073+0000 7f0d6641e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c00e1c0 con 0x55afcae4c7a0 2024-04-25T01:55:56.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.073+0000 7f0d6641e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c00d420 con 0x55afcae4d0c0 2024-04-25T01:55:56.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.073+0000 7f0d6641e740 10 client.0 _ll_register_callbacks cb 0x7f0d54011f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:56.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.073+0000 7f0d61600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae57790 0x55afcae5b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.074+0000 7f0d61600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae57790 0x55afcae5b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57290/0 (socket says 172.21.15.102:57290) 2024-04-25T01:55:56.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.074+0000 7f0d61600640 1 -- 172.21.15.102:0/2901232626 learned_addr learned my addr 172.21.15.102:0/2901232626 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.074+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae4c7a0 0x55afcae5fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.074+0000 7f0d5be00640 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae5b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.075+0000 7f0d61600640 1 -- 172.21.15.102:0/2901232626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 msgr2=0x55afcae5b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.075+0000 7f0d61600640 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae5b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.075+0000 7f0d61600640 1 -- 172.21.15.102:0/2901232626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae4c7a0 msgr2=0x55afcae5fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.075+0000 7f0d61600640 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae4c7a0 0x55afcae5fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.075+0000 7f0d61600640 1 -- 172.21.15.102:0/2901232626 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afcae5bd60 con 0x55afcae57790 2024-04-25T01:55:56.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.075+0000 7f0d5be00640 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae5b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:56.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.075+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae4c7a0 0x55afcae5fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:56.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.076+0000 7f0d61600640 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae57790 0x55afcae5b560 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0d5401f870 tx=0x7f0d540071d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd0054af0d945b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.076+0000 7f0d42000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:56.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.077+0000 7f0d42000640 1 -- 172.21.15.102:0/2901232626 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5403c420 con 0x55afcae57790 2024-04-25T01:55:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.077+0000 7f0d42000640 1 -- 172.21.15.102:0/2901232626 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0d5403c5c0 con 0x55afcae57790 2024-04-25T01:55:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.077+0000 7f0d6641e740 1 -- 172.21.15.102:0/2901232626 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d54037030 con 0x55afcae57790 2024-04-25T01:55:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.078+0000 7f0d42000640 1 -- 172.21.15.102:0/2901232626 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f0d5403c800 con 0x55afcae57790 2024-04-25T01:55:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.078+0000 7f0d6641e740 10 client.15417 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:56.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.078+0000 7f0d6641e740 1 -- 172.21.15.102:0/2901232626 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55afcaece110 con 0x55afcae57790 2024-04-25T01:55:56.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.079+0000 7f0d42000640 1 -- 172.21.15.102:0/2901232626 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d54037030 con 0x55afcae57790 2024-04-25T01:55:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.079+0000 7f0d42000640 1 -- 172.21.15.102:0/2901232626 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f0d5404e910 con 0x55afcae57790 2024-04-25T01:55:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.080+0000 7f0d6641e740 10 client.15417 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.080+0000 7f0d6641e740 10 client.15417 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:55:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.080+0000 7f0d6641e740 20 client.15417 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.080+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55afcae67c20 0x55afcaed0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.080+0000 7f0d6641e740 4 client.15417 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.080+0000 7f0d6641e740 1 -- 172.21.15.102:0/2901232626 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55afcae502e0 con 0x55afcae67c20 2024-04-25T01:55:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.080+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55afcae67c20 0x55afcaed0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.081+0000 7f0d42000640 10 client.15417 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:55:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.082+0000 7f0d60c00640 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55afcae67c20 0x55afcaed0780 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ffd5873ab7f0209d server_cookie=7f19a423dc69a906 in_seq=0 out_seq=0 2024-04-25T01:55:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.083+0000 7f0d42000640 1 -- 172.21.15.102:0/2901232626 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55afcae502e0 con 0x55afcae67c20 2024-04-25T01:55:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.083+0000 7f0d42000640 10 client.15417 handle_command_reply: tid=0 2024-04-25T01:55:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.083+0000 7f0d6641e740 1 client.15417 shutdown 2024-04-25T01:55:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.084+0000 7f0d6641e740 1 -- 172.21.15.102:0/2901232626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae57790 msgr2=0x55afcae5b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.084+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae57790 0x55afcae5b560 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0d5401f870 tx=0x7f0d540071d0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.084+0000 7f0d6641e740 1 -- 172.21.15.102:0/2901232626 shutdown_connections 2024-04-25T01:55:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.084+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55afcae67c20 0x55afcaed0780 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.084+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55afcae57790 0x55afcae5b560 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.084+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afcae4d0c0 0x55afcae5b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.084+0000 7f0d6641e740 1 --2- 172.21.15.102:0/2901232626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55afcae4c7a0 0x55afcae5fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.084+0000 7f0d6641e740 1 -- 172.21.15.102:0/2901232626 >> 172.21.15.102:0/2901232626 conn(0x55afcae48ac0 msgr2=0x55afcae54f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.084+0000 7f0d42000640 0 client.15417 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:55:56.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.085+0000 7f0d6641e740 1 -- 172.21.15.102:0/2901232626 shutdown_connections 2024-04-25T01:55:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.085+0000 7f0d6641e740 1 -- 172.21.15.102:0/2901232626 wait complete. 2024-04-25T01:55:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.085+0000 7f0d6641e740 20 client.15417 trim_cache size 0 max 16384 2024-04-25T01:55:56.089 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:55:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 -- 172.21.15.102:0/4243377990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d440a2ed0 msgr2=0x7f0d440a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d440a2ed0 0x7f0d440a5390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c108db0 tx=0x7f0d54004a40 comp rx=0 tx=0).stop 2024-04-25T01:55:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 -- 172.21.15.102:0/4243377990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c1003b0 msgr2=0x7f0d5c108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c1003b0 0x7f0d5c108190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000c920 tx=0x7f0d5000cde0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 -- 172.21.15.102:0/4243377990 shutdown_connections 2024-04-25T01:55:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d440a2ed0 0x7f0d440a5390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d5c10df60 0x7f0d5c103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d5c100d60 0x7f0d5c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 --2- 172.21.15.102:0/4243377990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c1003b0 0x7f0d5c108190 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000c920 tx=0x7f0d5000cde0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.087+0000 7f0d62000640 1 -- 172.21.15.102:0/4243377990 >> 172.21.15.102:0/4243377990 conn(0x7f0d5c0fbfc0 msgr2=0x7f0d5c0ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.088+0000 7f0d62000640 1 -- 172.21.15.102:0/4243377990 shutdown_connections 2024-04-25T01:55:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.088+0000 7f0d62000640 1 -- 172.21.15.102:0/4243377990 wait complete. 2024-04-25T01:55:56.122 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:55:56.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.399+0000 7f4071400640 1 -- 172.21.15.102:0/2341869518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f406c10dd00 msgr2=0x7f406c10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.399+0000 7f4071400640 1 --2- 172.21.15.102:0/2341869518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f406c10dd00 0x7f406c10e0e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f405c009900 tx=0x7f405c02a4b0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.399+0000 7f4071400640 1 -- 172.21.15.102:0/2341869518 shutdown_connections 2024-04-25T01:55:56.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.399+0000 7f4071400640 1 --2- 172.21.15.102:0/2341869518 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f406c108680 0x7f406c114aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.399+0000 7f4071400640 1 --2- 172.21.15.102:0/2341869518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f406c107ce0 0x7f406c108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.399+0000 7f4071400640 1 --2- 172.21.15.102:0/2341869518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f406c10dd00 0x7f406c10e0e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.399+0000 7f4071400640 1 -- 172.21.15.102:0/2341869518 >> 172.21.15.102:0/2341869518 conn(0x7f406c077560 msgr2=0x7f406c106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.400+0000 7f4071400640 1 -- 172.21.15.102:0/2341869518 shutdown_connections 2024-04-25T01:55:56.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.400+0000 7f4071400640 1 -- 172.21.15.102:0/2341869518 wait complete. 2024-04-25T01:55:56.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.402+0000 7f4071400640 1 Processor -- start 2024-04-25T01:55:56.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.402+0000 7f4071400640 1 -- start start 2024-04-25T01:55:56.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.403+0000 7f4071400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f406c107ce0 0x7f406c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.403+0000 7f4071400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f406c108680 0x7f406c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.403+0000 7f4071400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f406c10dd00 0x7f406c1a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.403+0000 7f4071400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f406c1180b0 con 0x7f406c10dd00 2024-04-25T01:55:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.403+0000 7f4071400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f406c117f30 con 0x7f406c108680 2024-04-25T01:55:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.403+0000 7f4071400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f406c118230 con 0x7f406c107ce0 2024-04-25T01:55:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.404+0000 7f406b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f406c108680 0x7f406c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.404+0000 7f406b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f406c108680 0x7f406c19ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45194/0 (socket says 172.21.15.102:45194) 2024-04-25T01:55:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.404+0000 7f406b400640 1 -- 172.21.15.102:0/3780955280 learned_addr learned my addr 172.21.15.102:0/3780955280 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.404+0000 7f4070a00640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f406c10dd00 0x7f406c1a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.404+0000 7f406be00640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f406c107ce0 0x7f406c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.405+0000 7f406b400640 1 -- 172.21.15.102:0/3780955280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f406c107ce0 msgr2=0x7f406c19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.405+0000 7f406b400640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f406c107ce0 0x7f406c19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.405+0000 7f406b400640 1 -- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f406c10dd00 msgr2=0x7f406c1a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.405+0000 7f406b400640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f406c10dd00 0x7f406c1a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.405+0000 7f406b400640 1 -- 172.21.15.102:0/3780955280 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c1a3500 con 0x7f406c108680 2024-04-25T01:55:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.405+0000 7f406be00640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f406c107ce0 0x7f406c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.405+0000 7f4070a00640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f406c10dd00 0x7f406c1a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:56.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.405+0000 7f406b400640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f406c108680 0x7f406c19ea50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f405800b450 tx=0x7f405800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.406+0000 7f4068c00640 1 -- 172.21.15.102:0/3780955280 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4058015020 con 0x7f406c108680 2024-04-25T01:55:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.406+0000 7f4068c00640 1 -- 172.21.15.102:0/3780955280 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4058011d70 con 0x7f406c108680 2024-04-25T01:55:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.406+0000 7f4068c00640 1 -- 172.21.15.102:0/3780955280 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405800f340 con 0x7f406c108680 2024-04-25T01:55:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.406+0000 7f4071400640 1 -- 172.21.15.102:0/3780955280 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406c1a37f0 con 0x7f406c108680 2024-04-25T01:55:56.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.406+0000 7f4071400640 1 -- 172.21.15.102:0/3780955280 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f406c1ab080 con 0x7f406c108680 2024-04-25T01:55:56.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.416+0000 7f4068c00640 1 -- 172.21.15.102:0/3780955280 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f4058018020 con 0x7f406c108680 2024-04-25T01:55:56.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.417+0000 7f407581d740 1 Processor -- start 2024-04-25T01:55:56.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.418+0000 7f407581d740 1 -- start start 2024-04-25T01:55:56.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.418+0000 7f407581d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 0x55fd2be4cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.418+0000 7f407581d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be57250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.418+0000 7f407581d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be57790 0x55fd2be59b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.418+0000 7f407581d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55fd2bdb14e0 con 0x55fd2be4c7a0 2024-04-25T01:55:56.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.418+0000 7f407581d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fd2bd59a70 con 0x55fd2be4d0c0 2024-04-25T01:55:56.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.418+0000 7f407581d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fd2bdad4a0 con 0x55fd2be57790 2024-04-25T01:55:56.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f406be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be57250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f406be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be57250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45208/0 (socket says 172.21.15.102:45208) 2024-04-25T01:55:56.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f406be00640 1 -- 172.21.15.102:0/2033179259 learned_addr learned my addr 172.21.15.102:0/2033179259 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f4070a00640 1 --2- 172.21.15.102:0/2033179259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 0x55fd2be4cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f406be00640 1 -- 172.21.15.102:0/2033179259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be57790 msgr2=0x55fd2be59b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:55:56.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f406be00640 1 --2- 172.21.15.102:0/2033179259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be57790 0x55fd2be59b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f406be00640 1 -- 172.21.15.102:0/2033179259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 msgr2=0x55fd2be4cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f406be00640 1 --2- 172.21.15.102:0/2033179259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 0x55fd2be4cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f406be00640 1 -- 172.21.15.102:0/2033179259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd2bac53f0 con 0x55fd2be4d0c0 2024-04-25T01:55:56.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.419+0000 7f4068c00640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40500a2e10 0x7f40500a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.420+0000 7f4068c00640 1 -- 172.21.15.102:0/3780955280 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f40580cef90 con 0x7f406c108680 2024-04-25T01:55:56.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.420+0000 7f406be00640 1 --2- 172.21.15.102:0/2033179259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be57250 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f405c004200 tx=0x7f405c009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=853966c6e2319d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.421+0000 7f4061200640 1 -- 172.21.15.102:0/2033179259 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c03f070 con 0x55fd2be4d0c0 2024-04-25T01:55:56.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.421+0000 7f4061200640 1 -- 172.21.15.102:0/2033179259 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f405c005520 con 0x55fd2be4d0c0 2024-04-25T01:55:56.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.421+0000 7f4061200640 1 -- 172.21.15.102:0/2033179259 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c03f070 con 0x55fd2be4d0c0 2024-04-25T01:55:56.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.423+0000 7f4070a00640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40500a2e10 0x7f40500a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:56 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2458593030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:56.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.425+0000 7f407581d740 1 -- 172.21.15.102:0/2033179259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 msgr2=0x55fd2be57250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.425+0000 7f407581d740 1 --2- 172.21.15.102:0/2033179259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be57250 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f405c004200 tx=0x7f405c009680 comp rx=0 tx=0).stop 2024-04-25T01:55:56.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.426+0000 7f407581d740 1 -- 172.21.15.102:0/2033179259 shutdown_connections 2024-04-25T01:55:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.426+0000 7f407581d740 1 --2- 172.21.15.102:0/2033179259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be57790 0x55fd2be59b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.426+0000 7f407581d740 1 --2- 172.21.15.102:0/2033179259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be57250 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.426+0000 7f407581d740 1 --2- 172.21.15.102:0/2033179259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 0x55fd2be4cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.426+0000 7f407581d740 1 -- 172.21.15.102:0/2033179259 >> 172.21.15.102:0/2033179259 conn(0x55fd2be48ac0 msgr2=0x55fd2be48ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.426+0000 7f4070a00640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40500a2e10 0x7f40500a52d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f406400af60 tx=0x7f406400a550 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.427+0000 7f407581d740 1 -- 172.21.15.102:0/2033179259 shutdown_connections 2024-04-25T01:55:56.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.427+0000 7f407581d740 1 -- 172.21.15.102:0/2033179259 wait complete. 2024-04-25T01:55:56.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.428+0000 7f407581d740 1 -- start start 2024-04-25T01:55:56.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.428+0000 7f407581d740 1 Processor -- start 2024-04-25T01:55:56.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f407581d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 0x55fd2be54770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f407581d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4d0c0 0x55fd2be54cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f407581d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be57790 0x55fd2be551f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f407581d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55fd2be5a520 con 0x55fd2be4c7a0 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f407581d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fd2bd6cec0 con 0x55fd2be57790 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f407581d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fd2be5a6a0 con 0x55fd2be4d0c0 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f4070a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be57790 0x55fd2be551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f4070a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be57790 0x55fd2be551f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45220/0 (socket says 172.21.15.102:45220) 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f4070a00640 1 -- 172.21.15.102:0/1166775072 learned_addr learned my addr 172.21.15.102:0/1166775072 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.430+0000 7f406b400640 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4d0c0 0x55fd2be54cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f407581d740 10 client.0 _ll_register_callbacks cb 0x55fd2bda93f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f4070a00640 1 -- 172.21.15.102:0/1166775072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4d0c0 msgr2=0x55fd2be54cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f4070a00640 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4d0c0 0x55fd2be54cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f4070a00640 1 -- 172.21.15.102:0/1166775072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 msgr2=0x55fd2be54770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f406be00640 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 0x55fd2be54770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f4070a00640 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 0x55fd2be54770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f4070a00640 1 -- 172.21.15.102:0/1166775072 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd2bb8f7d0 con 0x55fd2be57790 2024-04-25T01:55:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f407581d740 1 -- 172.21.15.102:0/1166775072 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f405c005520 con 0x55fd2be57790 2024-04-25T01:55:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f4070a00640 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be57790 0x55fd2be551f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4064011e40 tx=0x7f4064004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d33d4aeb6d2d042 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.431+0000 7f4055600640 10 client.25529 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.432+0000 7f4055600640 1 -- 172.21.15.102:0/1166775072 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406401e620 con 0x55fd2be57790 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.432+0000 7f4055600640 1 -- 172.21.15.102:0/1166775072 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f406401e7c0 con 0x55fd2be57790 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.432+0000 7f4055600640 1 -- 172.21.15.102:0/1166775072 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f406401e9e0 con 0x55fd2be57790 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.432+0000 7f407581d740 10 client.25529 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.432+0000 7f407581d740 1 -- 172.21.15.102:0/1166775072 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55fd2be667e0 con 0x55fd2be57790 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.433+0000 7f4055600640 1 -- 172.21.15.102:0/1166775072 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f406401eb80 con 0x55fd2be57790 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.433+0000 7f407581d740 10 client.25529 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.433+0000 7f407581d740 10 client.25529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.433+0000 7f407581d740 20 client.25529 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.433+0000 7f407581d740 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fd2be67be0 0x55fd2be6a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.433+0000 7f407581d740 4 client.25529 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.433+0000 7f407581d740 1 -- 172.21.15.102:0/1166775072 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd2bdbea20 con 0x55fd2be67be0 2024-04-25T01:55:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.434+0000 7f406be00640 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fd2be67be0 0x55fd2be6a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.436+0000 7f4055600640 10 client.25529 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.436+0000 7f406be00640 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fd2be67be0 0x55fd2be6a0a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dc812497d3145f2 server_cookie=e79e677602bf09d4 in_seq=0 out_seq=0 2024-04-25T01:55:56.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.437+0000 7f4055600640 1 -- 172.21.15.102:0/1166775072 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd2bdbea20 con 0x55fd2be67be0 2024-04-25T01:55:56.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.437+0000 7f4055600640 10 client.25529 handle_command_reply: tid=0 2024-04-25T01:55:56.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.437+0000 7f407581d740 1 client.25529 shutdown 2024-04-25T01:55:56.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.437+0000 7f407581d740 1 -- 172.21.15.102:0/1166775072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be57790 msgr2=0x55fd2be551f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.438+0000 7f407581d740 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be57790 0x55fd2be551f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4064011e40 tx=0x7f4064004a40 comp rx=0 tx=0).stop 2024-04-25T01:55:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.438+0000 7f407581d740 1 -- 172.21.15.102:0/1166775072 shutdown_connections 2024-04-25T01:55:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.438+0000 7f407581d740 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fd2be67be0 0x55fd2be6a0a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.438+0000 7f407581d740 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be57790 0x55fd2be551f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.438+0000 7f407581d740 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4d0c0 0x55fd2be54cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.438+0000 7f407581d740 1 --2- 172.21.15.102:0/1166775072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be4c7a0 0x55fd2be54770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.438+0000 7f407581d740 1 -- 172.21.15.102:0/1166775072 >> 172.21.15.102:0/1166775072 conn(0x55fd2be48ac0 msgr2=0x55fd2be4e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.438+0000 7f4055600640 0 client.25529 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:56.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.439+0000 7f407581d740 1 -- 172.21.15.102:0/1166775072 shutdown_connections 2024-04-25T01:55:56.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.439+0000 7f407581d740 1 -- 172.21.15.102:0/1166775072 wait complete. 2024-04-25T01:55:56.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.439+0000 7f407581d740 20 client.25529 trim_cache size 0 max 16384 2024-04-25T01:55:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.469+0000 7f407581d740 1 Processor -- start 2024-04-25T01:55:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.469+0000 7f407581d740 1 -- start start 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.469+0000 7f407581d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be63410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.469+0000 7f407581d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be6f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.469+0000 7f407581d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2be6feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.469+0000 7f407581d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f406c1b61c0 con 0x55fd2be57790 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.469+0000 7f407581d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f406c1b3a60 con 0x55fd2be4d0c0 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.469+0000 7f407581d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f406c1b18b0 con 0x55fd2be4c7a0 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.470+0000 7f406be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be63410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.470+0000 7f406be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be63410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55896/0 (socket says 172.21.15.102:55896) 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.470+0000 7f406be00640 1 -- 172.21.15.102:0/4071703197 learned_addr learned my addr 172.21.15.102:0/4071703197 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.470+0000 7f406b400640 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be6f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.470+0000 7f406be00640 1 -- 172.21.15.102:0/4071703197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 msgr2=0x55fd2be6f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.470+0000 7f4070a00640 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2be6feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.470+0000 7f406be00640 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be6f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.470+0000 7f406be00640 1 -- 172.21.15.102:0/4071703197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 msgr2=0x55fd2be6feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.471+0000 7f406be00640 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2be6feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.471+0000 7f406be00640 1 -- 172.21.15.102:0/4071703197 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd2be42f90 con 0x55fd2be4c7a0 2024-04-25T01:55:56.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.471+0000 7f406b400640 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be6f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.471+0000 7f4070a00640 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2be6feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:55:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.471+0000 7f406be00640 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be63410 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f405c004200 tx=0x7f405c004d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b022aa13270fa45a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.471+0000 7f4057e00640 1 -- 172.21.15.102:0/4071703197 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c03f040 con 0x55fd2be4c7a0 2024-04-25T01:55:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.472+0000 7f4057e00640 1 -- 172.21.15.102:0/4071703197 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f405c005350 con 0x55fd2be4c7a0 2024-04-25T01:55:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.472+0000 7f4057e00640 1 -- 172.21.15.102:0/4071703197 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c035030 con 0x55fd2be4c7a0 2024-04-25T01:55:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.473+0000 7f407581d740 1 -- 172.21.15.102:0/4071703197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 msgr2=0x55fd2be63410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.473+0000 7f407581d740 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be63410 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f405c004200 tx=0x7f405c004d40 comp rx=0 tx=0).stop 2024-04-25T01:55:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.473+0000 7f407581d740 1 -- 172.21.15.102:0/4071703197 shutdown_connections 2024-04-25T01:55:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.473+0000 7f407581d740 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2be6feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.473+0000 7f407581d740 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be6f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.473+0000 7f407581d740 1 --2- 172.21.15.102:0/4071703197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be63410 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.473+0000 7f407581d740 1 -- 172.21.15.102:0/4071703197 >> 172.21.15.102:0/4071703197 conn(0x55fd2be48ac0 msgr2=0x55fd2be4e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.474+0000 7f407581d740 1 -- 172.21.15.102:0/4071703197 shutdown_connections 2024-04-25T01:55:56.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.474+0000 7f407581d740 1 -- 172.21.15.102:0/4071703197 wait complete. 2024-04-25T01:55:56.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.475+0000 7f407581d740 1 -- start start 2024-04-25T01:55:56.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.475+0000 7f407581d740 1 Processor -- start 2024-04-25T01:55:56.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.476+0000 7f407581d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be51740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.476+0000 7f407581d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be51c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.476+0000 7f407581d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2becd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.476+0000 7f406be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be51740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.476+0000 7f406b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be51c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.476+0000 7f406be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be51740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55902/0 (socket says 172.21.15.102:55902) 2024-04-25T01:55:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.476+0000 7f406b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be51c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45238/0 (socket says 172.21.15.102:45238) 2024-04-25T01:55:56.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.477+0000 7f406be00640 1 -- 172.21.15.102:0/3081167832 learned_addr learned my addr 172.21.15.102:0/3081167832 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.477+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f406c1b1f20 con 0x55fd2be57790 2024-04-25T01:55:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.477+0000 7f4070a00640 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2becd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.477+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f406c1b3390 con 0x55fd2be4d0c0 2024-04-25T01:55:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.477+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f406c1b25f0 con 0x55fd2be4c7a0 2024-04-25T01:55:56.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.477+0000 7f407581d740 10 client.0 _ll_register_callbacks cb 0x7f406c06ae40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:56.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.478+0000 7f406be00640 1 -- 172.21.15.102:0/3081167832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 msgr2=0x55fd2be51c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.478+0000 7f406be00640 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be51c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.478+0000 7f406be00640 1 -- 172.21.15.102:0/3081167832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 msgr2=0x55fd2becd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.478+0000 7f406be00640 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2becd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.478+0000 7f406be00640 1 -- 172.21.15.102:0/3081167832 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd2be52510 con 0x55fd2be4c7a0 2024-04-25T01:55:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.479+0000 7f406b400640 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be51c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:56.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.479+0000 7f4070a00640 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2becd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:56.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.479+0000 7f406be00640 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be51740 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f405c046040 tx=0x7f405c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7077bdb5c8b272a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:56.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.479+0000 7f4055600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:55:56.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.480+0000 7f4055600640 1 -- 172.21.15.102:0/3081167832 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c05f060 con 0x55fd2be4c7a0 2024-04-25T01:55:56.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.480+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f405c005350 con 0x55fd2be4c7a0 2024-04-25T01:55:56.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.480+0000 7f4055600640 1 -- 172.21.15.102:0/3081167832 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f405c0503b0 con 0x55fd2be4c7a0 2024-04-25T01:55:56.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.480+0000 7f4055600640 1 -- 172.21.15.102:0/3081167832 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f405c03f050 con 0x55fd2be4c7a0 2024-04-25T01:55:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.481+0000 7f407581d740 10 client.25516 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.481+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fd2bece760 con 0x55fd2be4c7a0 2024-04-25T01:55:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.481+0000 7f4055600640 1 -- 172.21.15.102:0/3081167832 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f405c004db0 con 0x55fd2be4c7a0 2024-04-25T01:55:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.481+0000 7f4055600640 1 -- 172.21.15.102:0/3081167832 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f405c031960 con 0x55fd2be4c7a0 2024-04-25T01:55:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.481+0000 7f407581d740 10 client.25516 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:56.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.481+0000 7f407581d740 10 client.25516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:56.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.481+0000 7f407581d740 20 client.25516 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:56.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.482+0000 7f407581d740 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fd2becea40 0x55fd2bed0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:56.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.482+0000 7f407581d740 4 client.25516 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:55:56.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.482+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f405c005350 con 0x55fd2becea40 2024-04-25T01:55:56.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.482+0000 7f406b400640 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fd2becea40 0x55fd2bed0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:56.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.483+0000 7f4055600640 10 client.25516 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:56.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.483+0000 7f406b400640 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fd2becea40 0x55fd2bed0e30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d4e253d95a5ee0c server_cookie=3b185c898c65dc29 in_seq=0 out_seq=0 2024-04-25T01:55:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.484+0000 7f4055600640 1 -- 172.21.15.102:0/3081167832 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f405c005350 con 0x55fd2becea40 2024-04-25T01:55:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.484+0000 7f4055600640 10 client.25516 handle_command_reply: tid=0 2024-04-25T01:55:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.484+0000 7f407581d740 1 client.25516 shutdown 2024-04-25T01:55:56.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.486+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 msgr2=0x55fd2be51740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.486+0000 7f407581d740 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be51740 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f405c046040 tx=0x7f405c009260 comp rx=0 tx=0).stop 2024-04-25T01:55:56.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.486+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 shutdown_connections 2024-04-25T01:55:56.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:56 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2458593030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:56.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.487+0000 7f407581d740 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fd2becea40 0x55fd2bed0e30 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.487+0000 7f407581d740 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd2be57790 0x55fd2becd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.487+0000 7f407581d740 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fd2be4d0c0 0x55fd2be51c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.487+0000 7f407581d740 1 --2- 172.21.15.102:0/3081167832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd2be4c7a0 0x55fd2be51740 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.487+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 >> 172.21.15.102:0/3081167832 conn(0x55fd2be48ac0 msgr2=0x55fd2be54710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.487+0000 7f4055600640 0 client.25516 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:56.491 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:55:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.487+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 shutdown_connections 2024-04-25T01:55:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.488+0000 7f407581d740 1 -- 172.21.15.102:0/3081167832 wait complete. 2024-04-25T01:55:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.489+0000 7f407581d740 20 client.25516 trim_cache size 0 max 16384 2024-04-25T01:55:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 -- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40500a2e10 msgr2=0x7f40500a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40500a2e10 0x7f40500a52d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f406400af60 tx=0x7f406400a550 comp rx=0 tx=0).stop 2024-04-25T01:55:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 -- 172.21.15.102:0/3780955280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f406c108680 msgr2=0x7f406c19ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f406c108680 0x7f406c19ea50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f405800b450 tx=0x7f405800b910 comp rx=0 tx=0).stop 2024-04-25T01:55:56.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 -- 172.21.15.102:0/3780955280 shutdown_connections 2024-04-25T01:55:56.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40500a2e10 0x7f40500a52d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f406c10dd00 0x7f406c1a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f406c108680 0x7f406c19ea50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 --2- 172.21.15.102:0/3780955280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f406c107ce0 0x7f406c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:56.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.490+0000 7f4071400640 1 -- 172.21.15.102:0/3780955280 >> 172.21.15.102:0/3780955280 conn(0x7f406c077560 msgr2=0x7f406c0785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:56.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.491+0000 7f4071400640 1 -- 172.21.15.102:0/3780955280 shutdown_connections 2024-04-25T01:55:56.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:56.491+0000 7f4071400640 1 -- 172.21.15.102:0/3780955280 wait complete. 2024-04-25T01:55:56.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:56 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2458593030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:55:57.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:57 smithi110 ceph-mon[32848]: pgmap v23: 97 pgs: 97 active+clean; 398 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 46 MiB/s wr, 45 op/s 2024-04-25T01:55:57.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:57 smithi193 ceph-mon[32981]: pgmap v23: 97 pgs: 97 active+clean; 398 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 46 MiB/s wr, 45 op/s 2024-04-25T01:55:57.523 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:55:57.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:55:57.524 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:55:57.611 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:57 smithi102 ceph-mon[27796]: pgmap v23: 97 pgs: 97 active+clean; 398 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 46 MiB/s wr, 45 op/s 2024-04-25T01:55:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.787+0000 7fb406200640 1 -- 172.21.15.102:0/3957657544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb40010b160 msgr2=0x7fb40010b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.787+0000 7fb406200640 1 --2- 172.21.15.102:0/3957657544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb40010b160 0x7fb40010b540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009900 tx=0x7fb3f002a490 comp rx=0 tx=0).stop 2024-04-25T01:55:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.787+0000 7fb406200640 1 -- 172.21.15.102:0/3957657544 shutdown_connections 2024-04-25T01:55:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.787+0000 7fb406200640 1 --2- 172.21.15.102:0/3957657544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb400114530 0x7fb400116980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.787+0000 7fb406200640 1 --2- 172.21.15.102:0/3957657544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40010ba80 0x7fb400113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.787+0000 7fb406200640 1 --2- 172.21.15.102:0/3957657544 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb40010b160 0x7fb40010b540 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.787+0000 7fb406200640 1 -- 172.21.15.102:0/3957657544 >> 172.21.15.102:0/3957657544 conn(0x7fb400072bc0 msgr2=0x7fb400072fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:57.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.788+0000 7fb406200640 1 -- 172.21.15.102:0/3957657544 shutdown_connections 2024-04-25T01:55:57.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.788+0000 7fb406200640 1 -- 172.21.15.102:0/3957657544 wait complete. 2024-04-25T01:55:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.789+0000 7fb406200640 1 Processor -- start 2024-04-25T01:55:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.790+0000 7fb406200640 1 -- start start 2024-04-25T01:55:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.790+0000 7fb406200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40010b160 0x7fb400108db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.790+0000 7fb406200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb40010ba80 0x7fb4001092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.790+0000 7fb406200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb400114530 0x7fb400109830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.790+0000 7fb406200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb40011a2b0 con 0x7fb40010b160 2024-04-25T01:55:57.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.790+0000 7fb406200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb40011a130 con 0x7fb40010ba80 2024-04-25T01:55:57.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.790+0000 7fb406200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb40011a430 con 0x7fb400114530 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.791+0000 7fb3ff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40010b160 0x7fb400108db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.791+0000 7fb3ff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40010b160 0x7fb400108db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57346/0 (socket says 172.21.15.102:57346) 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.791+0000 7fb3ff400640 1 -- 172.21.15.102:0/3397402317 learned_addr learned my addr 172.21.15.102:0/3397402317 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.791+0000 7fb3fea00640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb40010ba80 0x7fb4001092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.791+0000 7fb3ffe00640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb400114530 0x7fb400109830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.791+0000 7fb3ff400640 1 -- 172.21.15.102:0/3397402317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb400114530 msgr2=0x7fb400109830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.791+0000 7fb3ff400640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb400114530 0x7fb400109830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.791+0000 7fb3ff400640 1 -- 172.21.15.102:0/3397402317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb40010ba80 msgr2=0x7fb4001092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.791+0000 7fb3ff400640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb40010ba80 0x7fb4001092f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.792+0000 7fb3ff400640 1 -- 172.21.15.102:0/3397402317 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4001aed70 con 0x7fb40010b160 2024-04-25T01:55:57.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.792+0000 7fb3fea00640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb40010ba80 0x7fb4001092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:57.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.792+0000 7fb3ff400640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40010b160 0x7fb400108db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0006010 tx=0x7fb3f00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.793+0000 7fb3f7e00640 1 -- 172.21.15.102:0/3397402317 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f003f030 con 0x7fb40010b160 2024-04-25T01:55:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.793+0000 7fb3f7e00640 1 -- 172.21.15.102:0/3397402317 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0003ce0 con 0x7fb40010b160 2024-04-25T01:55:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.793+0000 7fb3f7e00640 1 -- 172.21.15.102:0/3397402317 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0035070 con 0x7fb40010b160 2024-04-25T01:55:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.793+0000 7fb406200640 1 -- 172.21.15.102:0/3397402317 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4001af000 con 0x7fb40010b160 2024-04-25T01:55:57.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.793+0000 7fb406200640 1 -- 172.21.15.102:0/3397402317 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4001af4a0 con 0x7fb40010b160 2024-04-25T01:55:57.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.798+0000 7fb40902b740 1 Processor -- start 2024-04-25T01:55:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.798+0000 7fb40902b740 1 -- start start 2024-04-25T01:55:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.799+0000 7fb40902b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100daf6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.799+0000 7fb40902b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100db00810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.799+0000 7fb40902b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 0x56100db03140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.799+0000 7fb40902b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56100da54f40 con 0x56100daf5d60 2024-04-25T01:55:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.799+0000 7fb40902b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56100da5a190 con 0x56100daf6680 2024-04-25T01:55:57.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.799+0000 7fb40902b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56100da02a10 con 0x56100db00d50 2024-04-25T01:55:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.799+0000 7fb3fea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 0x56100db03140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.800+0000 7fb3f7e00640 1 -- 172.21.15.102:0/3397402317 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fb3f0042030 con 0x7fb40010b160 2024-04-25T01:55:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.800+0000 7fb3fea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 0x56100db03140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55948/0 (socket says 172.21.15.102:55948) 2024-04-25T01:55:57.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.800+0000 7fb3fea00640 1 -- 172.21.15.102:0/98204606 learned_addr learned my addr 172.21.15.102:0/98204606 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.800+0000 7fb3ff400640 1 --2- 172.21.15.102:0/98204606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100db00810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.801+0000 7fb3ff400640 1 -- 172.21.15.102:0/98204606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 msgr2=0x56100db03140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.801+0000 7fb3ff400640 1 --2- 172.21.15.102:0/98204606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 0x56100db03140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.801+0000 7fb3ff400640 1 -- 172.21.15.102:0/98204606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 msgr2=0x56100daf6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.801+0000 7fb3ff400640 1 --2- 172.21.15.102:0/98204606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100daf6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.801+0000 7fb3ff400640 1 -- 172.21.15.102:0/98204606 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56100d8367e0 con 0x56100daf6680 2024-04-25T01:55:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.802+0000 7fb3f7e00640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3e40a2ee0 0x7fb3e40a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.802+0000 7fb3ff400640 1 --2- 172.21.15.102:0/98204606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100db00810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00ece30 tx=0x7fb3f00f5c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab8cfec63d23b464 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.802+0000 7fb3ebe00640 1 -- 172.21.15.102:0/98204606 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00fe070 con 0x56100daf6680 2024-04-25T01:55:57.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.803+0000 7fb3ebe00640 1 -- 172.21.15.102:0/98204606 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f00f5e50 con 0x56100daf6680 2024-04-25T01:55:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.803+0000 7fb3f7e00640 1 -- 172.21.15.102:0/3397402317 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb3f00ef7d0 con 0x7fb40010b160 2024-04-25T01:55:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.803+0000 7fb3ffe00640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3e40a2ee0 0x7fb3e40a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.803+0000 7fb40902b740 1 -- 172.21.15.102:0/98204606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 msgr2=0x56100db00810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:57.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.803+0000 7fb40902b740 1 --2- 172.21.15.102:0/98204606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100db00810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00ece30 tx=0x7fb3f00f5c00 comp rx=0 tx=0).stop 2024-04-25T01:55:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.804+0000 7fb40902b740 1 -- 172.21.15.102:0/98204606 shutdown_connections 2024-04-25T01:55:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.804+0000 7fb40902b740 1 --2- 172.21.15.102:0/98204606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 0x56100db03140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.804+0000 7fb40902b740 1 --2- 172.21.15.102:0/98204606 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100db00810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.804+0000 7fb40902b740 1 --2- 172.21.15.102:0/98204606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100daf6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.805+0000 7fb40902b740 1 -- 172.21.15.102:0/98204606 >> 172.21.15.102:0/98204606 conn(0x56100daf2080 msgr2=0x56100daf2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:57.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.805+0000 7fb40902b740 1 -- 172.21.15.102:0/98204606 shutdown_connections 2024-04-25T01:55:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.805+0000 7fb40902b740 1 -- 172.21.15.102:0/98204606 wait complete. 2024-04-25T01:55:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.805+0000 7fb3ffe00640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3e40a2ee0 0x7fb3e40a53a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80092f0 tx=0x7fb3f8009280 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.805+0000 7fb40902b740 1 -- start start 2024-04-25T01:55:57.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.806+0000 7fb40902b740 1 Processor -- start 2024-04-25T01:55:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.806+0000 7fb40902b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100dafdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.806+0000 7fb40902b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100dafe100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.806+0000 7fb40902b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 0x56100dafe640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.806+0000 7fb40902b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56100da15e70 con 0x56100daf5d60 2024-04-25T01:55:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.806+0000 7fb40902b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56100da56150 con 0x56100daf6680 2024-04-25T01:55:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.806+0000 7fb40902b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56100db039a0 con 0x56100db00d50 2024-04-25T01:55:57.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.807+0000 7fb40902b740 10 client.0 _ll_register_callbacks cb 0x56100da511f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.807+0000 7fb3fea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100dafe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.807+0000 7fb3fea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100dafe100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45276/0 (socket says 172.21.15.102:45276) 2024-04-25T01:55:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.807+0000 7fb3fea00640 1 -- 172.21.15.102:0/337623938 learned_addr learned my addr 172.21.15.102:0/337623938 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.807+0000 7fb3ffe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 0x56100dafe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.807+0000 7fb3ff400640 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100dafdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.808+0000 7fb3fea00640 1 -- 172.21.15.102:0/337623938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 msgr2=0x56100dafe640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.808+0000 7fb3fea00640 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 0x56100dafe640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.808+0000 7fb3fea00640 1 -- 172.21.15.102:0/337623938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 msgr2=0x56100dafdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.808+0000 7fb3fea00640 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100dafdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.808+0000 7fb3fea00640 1 -- 172.21.15.102:0/337623938 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56100d896f70 con 0x56100daf6680 2024-04-25T01:55:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.808+0000 7fb3ff400640 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100dafdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:57.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.808+0000 7fb3fea00640 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100dafe100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec011a90 tx=0x7fb3ec011f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edba793623f17198 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.808+0000 7fb3e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:55:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.809+0000 7fb3e8c00640 1 -- 172.21.15.102:0/337623938 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec00fe70 con 0x56100daf6680 2024-04-25T01:55:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.809+0000 7fb3e8c00640 1 -- 172.21.15.102:0/337623938 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb3ec004020 con 0x56100daf6680 2024-04-25T01:55:57.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.809+0000 7fb40902b740 1 -- 172.21.15.102:0/337623938 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3f00f5e50 con 0x56100daf6680 2024-04-25T01:55:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.810+0000 7fb3e8c00640 1 -- 172.21.15.102:0/337623938 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb3ec004220 con 0x56100daf6680 2024-04-25T01:55:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.810+0000 7fb40902b740 10 client.25547 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.810+0000 7fb40902b740 1 -- 172.21.15.102:0/337623938 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56100d72e170 con 0x56100daf6680 2024-04-25T01:55:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.811+0000 7fb3e8c00640 1 -- 172.21.15.102:0/337623938 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3f00f5e50 con 0x56100daf6680 2024-04-25T01:55:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.811+0000 7fb3e8c00640 1 -- 172.21.15.102:0/337623938 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fb3ec018030 con 0x56100daf6680 2024-04-25T01:55:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.811+0000 7fb40902b740 10 client.25547 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:57.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.811+0000 7fb40902b740 10 client.25547 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.811+0000 7fb40902b740 20 client.25547 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.811+0000 7fb40902b740 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56100db10ea0 0x56100db13320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.811+0000 7fb40902b740 4 client.25547 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.811+0000 7fb40902b740 1 -- 172.21.15.102:0/337623938 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56100da676d0 con 0x56100db10ea0 2024-04-25T01:55:57.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.812+0000 7fb3ff400640 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56100db10ea0 0x56100db13320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.812+0000 7fb3ff400640 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56100db10ea0 0x56100db13320 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd6b195a2ef1c536 server_cookie=f97c11681ac953c1 in_seq=0 out_seq=0 2024-04-25T01:55:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.812+0000 7fb3e8c00640 10 client.25547 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.813+0000 7fb3e8c00640 1 -- 172.21.15.102:0/337623938 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56100da676d0 con 0x56100db10ea0 2024-04-25T01:55:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.813+0000 7fb3e8c00640 10 client.25547 handle_command_reply: tid=0 2024-04-25T01:55:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.813+0000 7fb40902b740 1 client.25547 shutdown 2024-04-25T01:55:57.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 -- 172.21.15.102:0/337623938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 msgr2=0x56100dafe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100dafe100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec011a90 tx=0x7fb3ec011f50 comp rx=0 tx=0).stop 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 -- 172.21.15.102:0/337623938 shutdown_connections 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56100db10ea0 0x56100db13320 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100db00d50 0x56100dafe640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf6680 0x56100dafe100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb3e8c00640 0 client.25547 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 --2- 172.21.15.102:0/337623938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100dafdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 -- 172.21.15.102:0/337623938 >> 172.21.15.102:0/337623938 conn(0x56100daf2080 msgr2=0x56100daf11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 -- 172.21.15.102:0/337623938 shutdown_connections 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 1 -- 172.21.15.102:0/337623938 wait complete. 2024-04-25T01:55:57.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.814+0000 7fb40902b740 20 client.25547 trim_cache size 0 max 16384 2024-04-25T01:55:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.853+0000 7fb40902b740 1 Processor -- start 2024-04-25T01:55:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.853+0000 7fb40902b740 1 -- start start 2024-04-25T01:55:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.853+0000 7fb40902b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100db0c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.853+0000 7fb40902b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100daf6680 0x56100daf1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.853+0000 7fb40902b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100db00d50 0x56100dafca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.853+0000 7fb40902b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001ba6c0 con 0x56100daf5d60 2024-04-25T01:55:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.853+0000 7fb40902b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001b7f60 con 0x56100db00d50 2024-04-25T01:55:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.853+0000 7fb40902b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001b5db0 con 0x56100daf6680 2024-04-25T01:55:57.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.854+0000 7fb3ffe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100db00d50 0x56100dafca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.854+0000 7fb3ffe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100db00d50 0x56100dafca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45290/0 (socket says 172.21.15.102:45290) 2024-04-25T01:55:57.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.855+0000 7fb3ffe00640 1 -- 172.21.15.102:0/3208631641 learned_addr learned my addr 172.21.15.102:0/3208631641 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:57.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.854+0000 7fb3fea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100daf6680 0x56100daf1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.855+0000 7fb3ff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100db0c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.855+0000 7fb3fea00640 1 -- 172.21.15.102:0/3208631641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100db00d50 msgr2=0x56100dafca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:57.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.855+0000 7fb3fea00640 1 --2- 172.21.15.102:0/3208631641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100db00d50 0x56100dafca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.855+0000 7fb3fea00640 1 -- 172.21.15.102:0/3208631641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 msgr2=0x56100db0c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.855+0000 7fb3fea00640 1 --2- 172.21.15.102:0/3208631641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100db0c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.855+0000 7fb3fea00640 1 -- 172.21.15.102:0/3208631641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56100daf6b80 con 0x56100daf6680 2024-04-25T01:55:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.856+0000 7fb3fea00640 1 --2- 172.21.15.102:0/3208631641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100daf6680 0x56100daf1960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0058c0 tx=0x7fb3ec00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7de38c2654800001 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.856+0000 7fb3eb400640 1 -- 172.21.15.102:0/3208631641 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec0043c0 con 0x56100daf6680 2024-04-25T01:55:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.857+0000 7fb3eb400640 1 -- 172.21.15.102:0/3208631641 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3ec005510 con 0x56100daf6680 2024-04-25T01:55:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.857+0000 7fb3eb400640 1 -- 172.21.15.102:0/3208631641 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec004660 con 0x56100daf6680 2024-04-25T01:55:57.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.858+0000 7fb40902b740 1 -- 172.21.15.102:0/3208631641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100daf6680 msgr2=0x56100daf1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:57.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.858+0000 7fb40902b740 1 --2- 172.21.15.102:0/3208631641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100daf6680 0x56100daf1960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0058c0 tx=0x7fb3ec00c330 comp rx=0 tx=0).stop 2024-04-25T01:55:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.858+0000 7fb40902b740 1 -- 172.21.15.102:0/3208631641 shutdown_connections 2024-04-25T01:55:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.858+0000 7fb40902b740 1 --2- 172.21.15.102:0/3208631641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100db00d50 0x56100dafca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.858+0000 7fb40902b740 1 --2- 172.21.15.102:0/3208631641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100daf6680 0x56100daf1960 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0058c0 tx=0x7fb3ec00c330 comp rx=0 tx=0).stop 2024-04-25T01:55:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.858+0000 7fb40902b740 1 --2- 172.21.15.102:0/3208631641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100daf5d60 0x56100db0c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.858+0000 7fb40902b740 1 -- 172.21.15.102:0/3208631641 >> 172.21.15.102:0/3208631641 conn(0x56100daf2080 msgr2=0x56100dafc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:57.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.859+0000 7fb40902b740 1 -- 172.21.15.102:0/3208631641 shutdown_connections 2024-04-25T01:55:57.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.859+0000 7fb40902b740 1 -- 172.21.15.102:0/3208631641 wait complete. 2024-04-25T01:55:57.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.859+0000 7fb40902b740 1 -- start start 2024-04-25T01:55:57.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.859+0000 7fb40902b740 1 Processor -- start 2024-04-25T01:55:57.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.861+0000 7fb40902b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf5d60 0x56100dafa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.861+0000 7fb40902b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100db00d50 0x56100dafaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.861+0000 7fb40902b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100dafb470 0x56100db07070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ff400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf5d60 0x56100dafa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ff400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf5d60 0x56100dafa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45302/0 (socket says 172.21.15.102:45302) 2024-04-25T01:55:57.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001b6420 con 0x56100db00d50 2024-04-25T01:55:57.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ff400640 1 -- 172.21.15.102:0/2844360735 learned_addr learned my addr 172.21.15.102:0/2844360735 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:57.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001b7890 con 0x56100daf5d60 2024-04-25T01:55:57.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb4001b6af0 con 0x56100dafb470 2024-04-25T01:55:57.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb40902b740 10 client.0 _ll_register_callbacks cb 0x7fb3ec00c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:57.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3fea00640 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100db00d50 0x56100dafaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ffe00640 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100dafb470 0x56100db07070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ff400640 1 -- 172.21.15.102:0/2844360735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100dafb470 msgr2=0x56100db07070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:57.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ff400640 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100dafb470 0x56100db07070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ff400640 1 -- 172.21.15.102:0/2844360735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100db00d50 msgr2=0x56100dafaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ff400640 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100db00d50 0x56100dafaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ff400640 1 -- 172.21.15.102:0/2844360735 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56100db076e0 con 0x56100daf5d60 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3ff400640 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf5d60 0x56100dafa9f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0004650 tx=0x7fb3f0009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=987b25e117fd7dfd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.862+0000 7fb3e8c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.863+0000 7fb3e8c00640 1 -- 172.21.15.102:0/2844360735 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0049040 con 0x56100daf5d60 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.863+0000 7fb3e8c00640 1 -- 172.21.15.102:0/2844360735 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb3f00b81c0 con 0x56100daf5d60 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.863+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3ec005510 con 0x56100daf5d60 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.863+0000 7fb3e8c00640 1 -- 172.21.15.102:0/2844360735 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb3f00fe040 con 0x56100daf5d60 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.863+0000 7fb40902b740 10 client.25553 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.863+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56100db80610 con 0x56100daf5d60 2024-04-25T01:55:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.864+0000 7fb3e8c00640 1 -- 172.21.15.102:0/2844360735 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3ec005510 con 0x56100daf5d60 2024-04-25T01:55:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.864+0000 7fb3e8c00640 1 -- 172.21.15.102:0/2844360735 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7fb3f004c060 con 0x56100daf5d60 2024-04-25T01:55:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.864+0000 7fb40902b740 10 client.25553 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.864+0000 7fb40902b740 10 client.25553 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.864+0000 7fb40902b740 20 client.25553 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.864+0000 7fb40902b740 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56100db809b0 0x56100db82da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.864+0000 7fb40902b740 4 client.25553 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:55:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.864+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56100da61a20 con 0x56100db809b0 2024-04-25T01:55:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.864+0000 7fb3fea00640 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56100db809b0 0x56100db82da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.865+0000 7fb3e8c00640 10 client.25553 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.867+0000 7fb3fea00640 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56100db809b0 0x56100db82da0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=887ba92dbb538731 server_cookie=805bb9840a74bd1 in_seq=0 out_seq=0 2024-04-25T01:55:57.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.868+0000 7fb3e8c00640 1 -- 172.21.15.102:0/2844360735 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56100da61a20 con 0x56100db809b0 2024-04-25T01:55:57.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.868+0000 7fb3e8c00640 10 client.25553 handle_command_reply: tid=0 2024-04-25T01:55:57.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.868+0000 7fb40902b740 1 client.25553 shutdown 2024-04-25T01:55:57.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf5d60 msgr2=0x56100dafa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb40902b740 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf5d60 0x56100dafa9f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0004650 tx=0x7fb3f0009880 comp rx=0 tx=0).stop 2024-04-25T01:55:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 shutdown_connections 2024-04-25T01:55:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb40902b740 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56100db809b0 0x56100db82da0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb40902b740 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56100dafb470 0x56100db07070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb40902b740 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56100db00d50 0x56100dafaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb40902b740 1 --2- 172.21.15.102:0/2844360735 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56100daf5d60 0x56100dafa9f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 >> 172.21.15.102:0/2844360735 conn(0x56100daf2080 msgr2=0x56100daffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:57.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb3e8c00640 0 client.25553 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.869+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 shutdown_connections 2024-04-25T01:55:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.870+0000 7fb40902b740 1 -- 172.21.15.102:0/2844360735 wait complete. 2024-04-25T01:55:57.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.870+0000 7fb40902b740 20 client.25553 trim_cache size 0 max 16384 2024-04-25T01:55:57.872 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:55:57.872 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T01:55:57.872 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "791ac905-bb40-4c51-8afc-61c6e71c3bd6", 2024-04-25T01:55:57.872 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T01:55:57.873 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:55:57.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 -- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3e40a2ee0 msgr2=0x7fb3e40a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:57.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3e40a2ee0 0x7fb3e40a53a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80092f0 tx=0x7fb3f8009280 comp rx=0 tx=0).stop 2024-04-25T01:55:57.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 -- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40010b160 msgr2=0x7fb400108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:57.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40010b160 0x7fb400108db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0006010 tx=0x7fb3f00042f0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 -- 172.21.15.102:0/3397402317 shutdown_connections 2024-04-25T01:55:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3e40a2ee0 0x7fb3e40a53a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb400114530 0x7fb400109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb40010ba80 0x7fb4001092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 --2- 172.21.15.102:0/3397402317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40010b160 0x7fb400108db0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 -- 172.21.15.102:0/3397402317 >> 172.21.15.102:0/3397402317 conn(0x7fb400072bc0 msgr2=0x7fb40006ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 -- 172.21.15.102:0/3397402317 shutdown_connections 2024-04-25T01:55:57.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:57.871+0000 7fb406200640 1 -- 172.21.15.102:0/3397402317 wait complete. 2024-04-25T01:55:57.898 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:55:57.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:55:58.070 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 5.0 s, 3329.6 tps, lat 28.644 ms stddev 29.257 2024-04-25T01:55:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.154+0000 7ffbf1a00640 1 -- 172.21.15.102:0/2844279413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 msgr2=0x7ffbec110850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.154+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/2844279413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 0x7ffbec110850 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4009cd0 tx=0x7ffbe402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.154+0000 7ffbf1a00640 1 -- 172.21.15.102:0/2844279413 shutdown_connections 2024-04-25T01:55:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.154+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/2844279413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 0x7ffbec110850 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.154+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/2844279413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec103980 0x7ffbec103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.154+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/2844279413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbec109990 0x7ffbec109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.154+0000 7ffbf1a00640 1 -- 172.21.15.102:0/2844279413 >> 172.21.15.102:0/2844279413 conn(0x7ffbec0ff4d0 msgr2=0x7ffbec1018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:58.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.154+0000 7ffbf1a00640 1 -- 172.21.15.102:0/2844279413 shutdown_connections 2024-04-25T01:55:58.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.155+0000 7ffbf1a00640 1 -- 172.21.15.102:0/2844279413 wait complete. 2024-04-25T01:55:58.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.156+0000 7ffbf1a00640 1 Processor -- start 2024-04-25T01:55:58.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.156+0000 7ffbf1a00640 1 -- start start 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.156+0000 7ffbf1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbec103980 0x7ffbec10f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.156+0000 7ffbf1a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 0x7ffbec10f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.156+0000 7ffbf1a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec109990 0x7ffbec1a3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.156+0000 7ffbf1a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec076ea0 con 0x7ffbec103980 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.156+0000 7ffbf1a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec076d20 con 0x7ffbec109990 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.156+0000 7ffbf1a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec077020 con 0x7ffbec104340 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 0x7ffbec10f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 0x7ffbec10f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56012/0 (socket says 172.21.15.102:56012) 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbe2000640 1 -- 172.21.15.102:0/3782479642 learned_addr learned my addr 172.21.15.102:0/3782479642 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbeb400640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec109990 0x7ffbec1a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbec103980 0x7ffbec10f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbe2000640 1 -- 172.21.15.102:0/3782479642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec109990 msgr2=0x7ffbec1a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:58.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbe2000640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec109990 0x7ffbec1a3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbe2000640 1 -- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbec103980 msgr2=0x7ffbec10f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:58.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbe2000640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbec103980 0x7ffbec10f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbe2000640 1 -- 172.21.15.102:0/3782479642 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbec1a3770 con 0x7ffbec104340 2024-04-25T01:55:58.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.157+0000 7ffbe2000640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 0x7ffbec10f9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9f0 tx=0x7ffbd400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:58.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.158+0000 7ffbe3e00640 1 -- 172.21.15.102:0/3782479642 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400cd80 con 0x7ffbec104340 2024-04-25T01:55:58.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.158+0000 7ffbe3e00640 1 -- 172.21.15.102:0/3782479642 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd4005500 con 0x7ffbec104340 2024-04-25T01:55:58.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.158+0000 7ffbf1a00640 1 -- 172.21.15.102:0/3782479642 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbec1a3a60 con 0x7ffbec104340 2024-04-25T01:55:58.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.158+0000 7ffbe3e00640 1 -- 172.21.15.102:0/3782479642 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4010690 con 0x7ffbec104340 2024-04-25T01:55:58.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.158+0000 7ffbf1a00640 1 -- 172.21.15.102:0/3782479642 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbec1ab330 con 0x7ffbec104340 2024-04-25T01:55:58.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.166+0000 7ffbe3e00640 1 -- 172.21.15.102:0/3782479642 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7ffbd40046c0 con 0x7ffbec104340 2024-04-25T01:55:58.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.166+0000 7ffbf47f9740 1 Processor -- start 2024-04-25T01:55:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.167+0000 7ffbf47f9740 1 -- start start 2024-04-25T01:55:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.167+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c19000 0x560921c193e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.167+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19920 0x560921c23ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.167+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c23ff0 0x560921c263e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.167+0000 7ffbf47f9740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560921b7d560 con 0x560921c19000 2024-04-25T01:55:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.167+0000 7ffbf47f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560921b25930 con 0x560921c19920 2024-04-25T01:55:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.167+0000 7ffbf47f9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560921b79520 con 0x560921c23ff0 2024-04-25T01:55:58.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.167+0000 7ffbeaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19920 0x560921c23ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.168+0000 7ffbeaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19920 0x560921c23ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45326/0 (socket says 172.21.15.102:45326) 2024-04-25T01:55:58.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.168+0000 7ffbeaa00640 1 -- 172.21.15.102:0/2321009955 learned_addr learned my addr 172.21.15.102:0/2321009955 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.168+0000 7ffbe2000640 1 --2- 172.21.15.102:0/2321009955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c23ff0 0x560921c263e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.168+0000 7ffbeaa00640 1 -- 172.21.15.102:0/2321009955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c23ff0 msgr2=0x560921c263e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.168+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/2321009955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c23ff0 0x560921c263e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.168+0000 7ffbeaa00640 1 -- 172.21.15.102:0/2321009955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c19000 msgr2=0x560921c193e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.168+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/2321009955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c19000 0x560921c193e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.168+0000 7ffbeaa00640 1 -- 172.21.15.102:0/2321009955 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56092195b7d0 con 0x560921c19920 2024-04-25T01:55:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.168+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/2321009955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19920 0x560921c23ab0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00b820 tx=0x7ffbdc00bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed0dbc6a8d66bba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.169+0000 7ffbdbe00640 1 -- 172.21.15.102:0/2321009955 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00d950 con 0x560921c19920 2024-04-25T01:55:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.169+0000 7ffbdbe00640 1 -- 172.21.15.102:0/2321009955 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc019070 con 0x560921c19920 2024-04-25T01:55:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.169+0000 7ffbdbe00640 1 -- 172.21.15.102:0/2321009955 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc0123e0 con 0x560921c19920 2024-04-25T01:55:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.169+0000 7ffbe3e00640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffbd00a2e10 0x7ffbd00a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.170+0000 7ffbe3e00640 1 -- 172.21.15.102:0/3782479642 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ffbd40cf9f0 con 0x7ffbec104340 2024-04-25T01:55:58.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.170+0000 7ffbeb400640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffbd00a2e10 0x7ffbd00a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.171+0000 7ffbf47f9740 1 -- 172.21.15.102:0/2321009955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19920 msgr2=0x560921c23ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:58.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.171+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/2321009955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19920 0x560921c23ab0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00b820 tx=0x7ffbdc00bce0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.172+0000 7ffbf47f9740 1 -- 172.21.15.102:0/2321009955 shutdown_connections 2024-04-25T01:55:58.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.172+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/2321009955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c23ff0 0x560921c263e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.172+0000 7ffbeb400640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffbd00a2e10 0x7ffbd00a52d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4005e30 tx=0x7ffbe4004a40 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:58.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.172+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/2321009955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19920 0x560921c23ab0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.172+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/2321009955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c19000 0x560921c193e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.172+0000 7ffbf47f9740 1 -- 172.21.15.102:0/2321009955 >> 172.21.15.102:0/2321009955 conn(0x560921c15320 msgr2=0x560921c15700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:58.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.175+0000 7ffbf47f9740 1 -- 172.21.15.102:0/2321009955 shutdown_connections 2024-04-25T01:55:58.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.175+0000 7ffbf47f9740 1 -- 172.21.15.102:0/2321009955 wait complete. 2024-04-25T01:55:58.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.175+0000 7ffbf47f9740 1 -- start start 2024-04-25T01:55:58.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.176+0000 7ffbf47f9740 1 Processor -- start 2024-04-25T01:55:58.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.176+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c20f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.176+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c214d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.176+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 0x560921c21a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c214d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c214d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56040/0 (socket says 172.21.15.102:56040) 2024-04-25T01:55:58.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbeaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c20f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbeaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c20f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45338/0 (socket says 172.21.15.102:45338) 2024-04-25T01:55:58.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbeaa00640 1 -- 172.21.15.102:0/3385330412 learned_addr learned my addr 172.21.15.102:0/3385330412 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:58.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560921c37df0 con 0x560921c23ff0 2024-04-25T01:55:58.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560921b38d90 con 0x560921c19000 2024-04-25T01:55:58.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560921c37f70 con 0x560921c19920 2024-04-25T01:55:58.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbf47f9740 10 client.0 _ll_register_callbacks cb 0x560921b38780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:58.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.177+0000 7ffbeb400640 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 0x560921c21a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.178+0000 7ffbe2000640 1 -- 172.21.15.102:0/3385330412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 msgr2=0x560921c20f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:58.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.178+0000 7ffbe2000640 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c20f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.178+0000 7ffbe2000640 1 -- 172.21.15.102:0/3385330412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 msgr2=0x560921c21a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:58.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.178+0000 7ffbe2000640 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 0x560921c21a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.178+0000 7ffbe2000640 1 -- 172.21.15.102:0/3385330412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609218913f0 con 0x560921c19920 2024-04-25T01:55:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.178+0000 7ffbe2000640 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c214d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40cd0b0 tx=0x7ffbd409bba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43c0bd5b369d9966 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.178+0000 7ffbd8c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:55:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.178+0000 7ffbd8c00640 1 -- 172.21.15.102:0/3385330412 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd40a47b0 con 0x560921c19920 2024-04-25T01:55:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbdc019070 con 0x560921c19920 2024-04-25T01:55:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbd8c00640 1 -- 172.21.15.102:0/3385330412 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ffbd409b220 con 0x560921c19920 2024-04-25T01:55:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbd8c00640 1 -- 172.21.15.102:0/3385330412 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ffbd40cd570 con 0x560921c19920 2024-04-25T01:55:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbf47f9740 10 client.25543 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5609218510f0 con 0x560921c19920 2024-04-25T01:55:58.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbd8c00640 1 -- 172.21.15.102:0/3385330412 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc019070 con 0x560921c19920 2024-04-25T01:55:58.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbd8c00640 1 -- 172.21.15.102:0/3385330412 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7ffbd40b1030 con 0x560921c19920 2024-04-25T01:55:58.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbf47f9740 10 client.25543 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:58.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbf47f9740 10 client.25543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:58.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbf47f9740 20 client.25543 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:58.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.179+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560921c452e0 0x560921c476d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.180+0000 7ffbf47f9740 4 client.25543 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:55:58.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.180+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560921b90d90 con 0x560921c452e0 2024-04-25T01:55:58.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.180+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560921c452e0 0x560921c476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.182+0000 7ffbd8c00640 10 client.25543 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:58.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.182+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560921c452e0 0x560921c476d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c3ba4a7ae7fef2d server_cookie=1a79f04b8d367522 in_seq=0 out_seq=0 2024-04-25T01:55:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.183+0000 7ffbd8c00640 1 -- 172.21.15.102:0/3385330412 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560921b90d90 con 0x560921c452e0 2024-04-25T01:55:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.183+0000 7ffbd8c00640 10 client.25543 handle_command_reply: tid=0 2024-04-25T01:55:58.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.184+0000 7ffbf47f9740 1 client.25543 shutdown 2024-04-25T01:55:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.185+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 msgr2=0x560921c214d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.185+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c214d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40cd0b0 tx=0x7ffbd409bba0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.185+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 shutdown_connections 2024-04-25T01:55:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.185+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560921c452e0 0x560921c476d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.185+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 0x560921c21a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.185+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c214d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.185+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/3385330412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c20f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.185+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 >> 172.21.15.102:0/3385330412 conn(0x560921c15320 msgr2=0x560921c1b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:58.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.185+0000 7ffbd8c00640 0 client.25543 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:58.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.186+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 shutdown_connections 2024-04-25T01:55:58.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.186+0000 7ffbf47f9740 1 -- 172.21.15.102:0/3385330412 wait complete. 2024-04-25T01:55:58.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.186+0000 7ffbf47f9740 20 client.25543 trim_cache size 0 max 16384 2024-04-25T01:55:58.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.217+0000 7ffbf47f9740 1 Processor -- start 2024-04-25T01:55:58.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.217+0000 7ffbf47f9740 1 -- start start 2024-04-25T01:55:58.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.218+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c19000 0x560921c40cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.218+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.218+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c23ff0 0x560921c14300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.218+0000 7ffbf47f9740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec1adac0 con 0x560921c19000 2024-04-25T01:55:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.218+0000 7ffbf47f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec11b0f0 con 0x560921c23ff0 2024-04-25T01:55:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.218+0000 7ffbf47f9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec118f60 con 0x560921c19920 2024-04-25T01:55:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.218+0000 7ffbeb400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c23ff0 0x560921c14300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.218+0000 7ffbeb400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c23ff0 0x560921c14300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45342/0 (socket says 172.21.15.102:45342) 2024-04-25T01:55:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.218+0000 7ffbeb400640 1 -- 172.21.15.102:0/424281979 learned_addr learned my addr 172.21.15.102:0/424281979 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:58.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.219+0000 7ffbe2000640 1 --2- 172.21.15.102:0/424281979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.219+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/424281979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c19000 0x560921c40cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.219+0000 7ffbeb400640 1 -- 172.21.15.102:0/424281979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 msgr2=0x560921c1b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:58.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.219+0000 7ffbeb400640 1 --2- 172.21.15.102:0/424281979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.219+0000 7ffbeb400640 1 -- 172.21.15.102:0/424281979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c19000 msgr2=0x560921c40cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:58.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.219+0000 7ffbeb400640 1 --2- 172.21.15.102:0/424281979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c19000 0x560921c40cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.219+0000 7ffbeb400640 1 -- 172.21.15.102:0/424281979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560921c1f4e0 con 0x560921c23ff0 2024-04-25T01:55:58.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.220+0000 7ffbeb400640 1 --2- 172.21.15.102:0/424281979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c23ff0 0x560921c14300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4035000 tx=0x7ffbe403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52265ad7df94254d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:58.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.222+0000 7ffbdb400640 1 -- 172.21.15.102:0/424281979 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe4003b70 con 0x560921c23ff0 2024-04-25T01:55:58.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.222+0000 7ffbdb400640 1 -- 172.21.15.102:0/424281979 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbe4036040 con 0x560921c23ff0 2024-04-25T01:55:58.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.222+0000 7ffbdb400640 1 -- 172.21.15.102:0/424281979 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe40309e0 con 0x560921c23ff0 2024-04-25T01:55:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.223+0000 7ffbf47f9740 1 -- 172.21.15.102:0/424281979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c23ff0 msgr2=0x560921c14300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.223+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/424281979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c23ff0 0x560921c14300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4035000 tx=0x7ffbe403f000 comp rx=0 tx=0).stop 2024-04-25T01:55:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.223+0000 7ffbf47f9740 1 -- 172.21.15.102:0/424281979 shutdown_connections 2024-04-25T01:55:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.223+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/424281979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c23ff0 0x560921c14300 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.223+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/424281979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.223+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/424281979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c19000 0x560921c40cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.223+0000 7ffbf47f9740 1 -- 172.21.15.102:0/424281979 >> 172.21.15.102:0/424281979 conn(0x560921c15320 msgr2=0x560921c13bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.224+0000 7ffbf47f9740 1 -- 172.21.15.102:0/424281979 shutdown_connections 2024-04-25T01:55:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.224+0000 7ffbf47f9740 1 -- 172.21.15.102:0/424281979 wait complete. 2024-04-25T01:55:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.224+0000 7ffbf47f9740 1 -- start start 2024-04-25T01:55:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.224+0000 7ffbf47f9740 1 Processor -- start 2024-04-25T01:55:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c1dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbf47f9740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 0x560921c1e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbf47f9740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec1195b0 con 0x560921c23ff0 2024-04-25T01:55:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbf47f9740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec11aa20 con 0x560921c19000 2024-04-25T01:55:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbf47f9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec119c80 con 0x560921c19920 2024-04-25T01:55:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbf47f9740 10 client.0 _ll_register_callbacks cb 0x7ffbd4004ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:55:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbeb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 0x560921c1e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56050/0 (socket says 172.21.15.102:56050) 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.225+0000 7ffbe2000640 1 -- 172.21.15.102:0/4150292790 learned_addr learned my addr 172.21.15.102:0/4150292790 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.226+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c1dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.226+0000 7ffbeb400640 1 -- 172.21.15.102:0/4150292790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 msgr2=0x560921c1e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.226+0000 7ffbeb400640 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.226+0000 7ffbeb400640 1 -- 172.21.15.102:0/4150292790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 msgr2=0x560921c1dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.226+0000 7ffbeb400640 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c1dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.226+0000 7ffbeb400640 1 -- 172.21.15.102:0/4150292790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560921caa980 con 0x560921c23ff0 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.226+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c1dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.226+0000 7ffbe2000640 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.227+0000 7ffbeb400640 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 0x560921c1e7f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffbe402ae00 tx=0x7ffbe4004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26d6860f00c4131 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:55:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.227+0000 7ffbd8c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:55:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.230+0000 7ffbd8c00640 1 -- 172.21.15.102:0/4150292790 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe4004450 con 0x560921c23ff0 2024-04-25T01:55:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.231+0000 7ffbd8c00640 1 -- 172.21.15.102:0/4150292790 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ffbe4030720 con 0x560921c23ff0 2024-04-25T01:55:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.231+0000 7ffbf47f9740 1 -- 172.21.15.102:0/4150292790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbe4036040 con 0x560921c23ff0 2024-04-25T01:55:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.231+0000 7ffbd8c00640 1 -- 172.21.15.102:0/4150292790 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ffbe40045f0 con 0x560921c23ff0 2024-04-25T01:55:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.231+0000 7ffbf47f9740 10 client.15450 fetch_fsmap learned FSMap version 26 2024-04-25T01:55:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.231+0000 7ffbf47f9740 1 -- 172.21.15.102:0/4150292790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560921cabc70 con 0x560921c23ff0 2024-04-25T01:55:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.232+0000 7ffbd8c00640 1 -- 172.21.15.102:0/4150292790 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbe4036040 con 0x560921c23ff0 2024-04-25T01:55:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.232+0000 7ffbd8c00640 1 -- 172.21.15.102:0/4150292790 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7ffbe4030070 con 0x560921c23ff0 2024-04-25T01:55:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.232+0000 7ffbf47f9740 10 client.15450 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T01:55:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.232+0000 7ffbf47f9740 10 client.15450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:55:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.232+0000 7ffbf47f9740 20 client.15450 populate_metadata read hostname 'smithi102' 2024-04-25T01:55:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.232+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560921cac320 0x560921cae740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:55:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.232+0000 7ffbf47f9740 4 client.15450 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:55:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.232+0000 7ffbf47f9740 1 -- 172.21.15.102:0/4150292790 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560921b8aaa0 con 0x560921cac320 2024-04-25T01:55:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.233+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560921cac320 0x560921cae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:55:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.234+0000 7ffbd8c00640 10 client.15450 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.235+0000 7ffbeaa00640 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560921cac320 0x560921cae740 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68f1afb302c716a2 server_cookie=f92532b40db695ba in_seq=0 out_seq=0 2024-04-25T01:55:58.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbd8c00640 1 -- 172.21.15.102:0/4150292790 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560921b8aaa0 con 0x560921cac320 2024-04-25T01:55:58.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbd8c00640 10 client.15450 handle_command_reply: tid=0 2024-04-25T01:55:58.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbf47f9740 1 client.15450 shutdown 2024-04-25T01:55:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbf47f9740 1 -- 172.21.15.102:0/4150292790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 msgr2=0x560921c1e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 0x560921c1e7f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffbe402ae00 tx=0x7ffbe4004990 comp rx=0 tx=0).stop 2024-04-25T01:55:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbf47f9740 1 -- 172.21.15.102:0/4150292790 shutdown_connections 2024-04-25T01:55:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560921cac320 0x560921cae740 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560921c23ff0 0x560921c1e7f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560921c19920 0x560921c1e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbf47f9740 1 --2- 172.21.15.102:0/4150292790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560921c19000 0x560921c1dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.236+0000 7ffbf47f9740 1 -- 172.21.15.102:0/4150292790 >> 172.21.15.102:0/4150292790 conn(0x560921c15320 msgr2=0x560921c14480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.237+0000 7ffbd8c00640 0 client.15450 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:55:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.237+0000 7ffbf47f9740 1 -- 172.21.15.102:0/4150292790 shutdown_connections 2024-04-25T01:55:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.237+0000 7ffbf47f9740 1 -- 172.21.15.102:0/4150292790 wait complete. 2024-04-25T01:55:58.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.238+0000 7ffbf47f9740 20 client.15450 trim_cache size 0 max 16384 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout: "791ac905-bb40-4c51-8afc-61c6e71c3bd6": { 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "791ac905-bb40-4c51-8afc-61c6e71c3bd6", 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:55:58.240 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:55:58.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 -- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffbd00a2e10 msgr2=0x7ffbd00a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:58.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffbd00a2e10 0x7ffbd00a52d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4005e30 tx=0x7ffbe4004a40 comp rx=0 tx=0).stop 2024-04-25T01:55:58.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 -- 172.21.15.102:0/3782479642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 msgr2=0x7ffbec10f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:55:58.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 0x7ffbec10f9f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9f0 tx=0x7ffbd400edf0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 -- 172.21.15.102:0/3782479642 shutdown_connections 2024-04-25T01:55:58.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffbd00a2e10 0x7ffbd00a52d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec109990 0x7ffbec1a3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbec104340 0x7ffbec10f9f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 --2- 172.21.15.102:0/3782479642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbec103980 0x7ffbec10f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:55:58.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 -- 172.21.15.102:0/3782479642 >> 172.21.15.102:0/3782479642 conn(0x7ffbec0ff4d0 msgr2=0x7ffbec108ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:55:58.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 -- 172.21.15.102:0/3782479642 shutdown_connections 2024-04-25T01:55:58.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:55:58.239+0000 7ffbf1a00640 1 -- 172.21.15.102:0/3782479642 wait complete. 2024-04-25T01:55:58.271 INFO:tasks.cephfs.filesystem:scrub status for tag:791ac905-bb40-4c51-8afc-61c6e71c3bd6 - {'path': '/', 'tag': '791ac905-bb40-4c51-8afc-61c6e71c3bd6', 'options': 'recursive,force'} 2024-04-25T01:55:58.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:58 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:55:58.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:58 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:58.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:58 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:55:58.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:58 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:58.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:58 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:55:58.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:58 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:58.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:58 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:55:58.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:58 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:58.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:58 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:55:58.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:58 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:58.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:58 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:55:58.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:58 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:59.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:59 smithi110 ceph-mon[32848]: pgmap v24: 97 pgs: 97 active+clean; 753 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 61 MiB/s wr, 187 op/s 2024-04-25T01:55:59.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:59 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T01:55:59.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:59 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:59.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:59 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:55:59.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:59 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:55:59.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:59 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:59.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:55:59 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:59.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:59 smithi193 ceph-mon[32981]: pgmap v24: 97 pgs: 97 active+clean; 753 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 61 MiB/s wr, 187 op/s 2024-04-25T01:55:59.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:59 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T01:55:59.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:59 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:59.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:59 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:55:59.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:59 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:55:59.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:59 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:59.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:55:59 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:59.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:59 smithi102 ceph-mon[27796]: pgmap v24: 97 pgs: 97 active+clean; 753 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 61 MiB/s wr, 187 op/s 2024-04-25T01:55:59.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:59 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T01:55:59.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:59 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:59.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:59 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:55:59.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:59 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:55:59.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:59 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:55:59.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:55:59 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:01.175 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:56:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:56:00.726+0000 7f5011200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:56:01.175 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:56:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:56:00.726+0000 7f5011200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:56:01.175 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:56:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:56:00.726+0000 7f5011200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:56:01.175 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:56:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:56:00.726+0000 7f5011200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:56:01.175 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 01:56:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T01:56:00.726+0000 7f5011200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T01:56:01.488 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:01 smithi110 ceph-mon[32848]: pgmap v25: 97 pgs: 97 active+clean; 753 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 34 MiB/s wr, 167 op/s 2024-04-25T01:56:01.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:01 smithi193 ceph-mon[32981]: pgmap v25: 97 pgs: 97 active+clean; 753 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 34 MiB/s wr, 167 op/s 2024-04-25T01:56:01.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:01 smithi102 ceph-mon[27796]: pgmap v25: 97 pgs: 97 active+clean; 753 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 34 MiB/s wr, 167 op/s 2024-04-25T01:56:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 10.0 s, 2054.0 tps, lat 47.671 ms stddev 79.047 2024-04-25T01:56:03.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:03 smithi193 ceph-mon[32981]: pgmap v26: 97 pgs: 97 active+clean; 849 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 35 MiB/s wr, 395 op/s 2024-04-25T01:56:03.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:03 smithi193 ceph-mon[32981]: mgrmap e25: x(active, since 46s), standbys: z, y 2024-04-25T01:56:03.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:03 smithi102 ceph-mon[27796]: pgmap v26: 97 pgs: 97 active+clean; 849 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 35 MiB/s wr, 395 op/s 2024-04-25T01:56:03.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:03 smithi102 ceph-mon[27796]: mgrmap e25: x(active, since 46s), standbys: z, y 2024-04-25T01:56:03.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:03 smithi110 ceph-mon[32848]: pgmap v26: 97 pgs: 97 active+clean; 849 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 35 MiB/s wr, 395 op/s 2024-04-25T01:56:03.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:03 smithi110 ceph-mon[32848]: mgrmap e25: x(active, since 46s), standbys: z, y 2024-04-25T01:56:04.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:04 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:56:04.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:04 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:56:04.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:04 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:56:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.554+0000 7f78a3e00640 1 -- 172.21.15.102:0/3555125805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a41058f0 msgr2=0x7f78a4105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.554+0000 7f78a3e00640 1 --2- 172.21.15.102:0/3555125805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a41058f0 0x7f78a4105d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f789c009be0 tx=0x7f789c02a750 comp rx=0 tx=0).stop 2024-04-25T01:56:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.555+0000 7f78a3e00640 1 -- 172.21.15.102:0/3555125805 shutdown_connections 2024-04-25T01:56:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.555+0000 7f78a3e00640 1 --2- 172.21.15.102:0/3555125805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a41058f0 0x7f78a4105d80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.555+0000 7f78a3e00640 1 --2- 172.21.15.102:0/3555125805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78a4100e10 0x7f78a41052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.555+0000 7f78a3e00640 1 --2- 172.21.15.102:0/3555125805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78a41004f0 0x7f78a41008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.555+0000 7f78a3e00640 1 -- 172.21.15.102:0/3555125805 >> 172.21.15.102:0/3555125805 conn(0x7f78a40fbf60 msgr2=0x7f78a40fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.555+0000 7f78a3e00640 1 -- 172.21.15.102:0/3555125805 shutdown_connections 2024-04-25T01:56:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.555+0000 7f78a3e00640 1 -- 172.21.15.102:0/3555125805 wait complete. 2024-04-25T01:56:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.556+0000 7f78a3e00640 1 Processor -- start 2024-04-25T01:56:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.556+0000 7f78a3e00640 1 -- start start 2024-04-25T01:56:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78a41004f0 0x7f78a419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a4100e10 0x7f78a419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78a41058f0 0x7f78a41a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4117ed0 con 0x7f78a41058f0 2024-04-25T01:56:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a3e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4117d50 con 0x7f78a4100e10 2024-04-25T01:56:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a3e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4118050 con 0x7f78a41004f0 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a4100e10 0x7f78a419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a4100e10 0x7f78a419e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48458/0 (socket says 172.21.15.102:48458) 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a2000640 1 -- 172.21.15.102:0/4288142684 learned_addr learned my addr 172.21.15.102:0/4288142684 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a2a00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78a41004f0 0x7f78a419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a3400640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78a41058f0 0x7f78a41a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a2000640 1 -- 172.21.15.102:0/4288142684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78a41004f0 msgr2=0x7f78a419e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.557+0000 7f78a2000640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78a41004f0 0x7f78a419e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.558+0000 7f78a2000640 1 -- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78a41058f0 msgr2=0x7f78a41a2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.558+0000 7f78a2000640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78a41058f0 0x7f78a41a2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.558+0000 7f78a2000640 1 -- 172.21.15.102:0/4288142684 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a41a33e0 con 0x7f78a4100e10 2024-04-25T01:56:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.558+0000 7f78a2000640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a4100e10 0x7f78a419e930 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f789000d740 tx=0x7f789000db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.558+0000 7f78a8a00640 1 -- 172.21.15.102:0/4288142684 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7890017070 con 0x7f78a4100e10 2024-04-25T01:56:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.558+0000 7f78a8a00640 1 -- 172.21.15.102:0/4288142684 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7890010460 con 0x7f78a4100e10 2024-04-25T01:56:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.558+0000 7f78a8a00640 1 -- 172.21.15.102:0/4288142684 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789000f620 con 0x7f78a4100e10 2024-04-25T01:56:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.558+0000 7f78a3e00640 1 -- 172.21.15.102:0/4288142684 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a41a36d0 con 0x7f78a4100e10 2024-04-25T01:56:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.558+0000 7f78a3e00640 1 -- 172.21.15.102:0/4288142684 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a41aaf60 con 0x7f78a4100e10 2024-04-25T01:56:05.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.563+0000 7f78a8a00640 1 -- 172.21.15.102:0/4288142684 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f7890009a20 con 0x7f78a4100e10 2024-04-25T01:56:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.563+0000 7f78acc50740 1 Processor -- start 2024-04-25T01:56:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.563+0000 7f78acc50740 1 -- start start 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.563+0000 7f78acc50740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78acc50740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78acc50740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917fe580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78acc50740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ee91754ab0 con 0x55ee917fc190 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78acc50740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ee916fd640 con 0x55ee917f0fb0 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78acc50740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ee91750a70 con 0x55ee917f18d0 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48476/0 (socket says 172.21.15.102:48476) 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a3400640 1 -- 172.21.15.102:0/256510134 learned_addr learned my addr 172.21.15.102:0/256510134 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a2000640 1 --2- 172.21.15.102:0/256510134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917fe580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a2a00640 1 --2- 172.21.15.102:0/256510134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a3400640 1 -- 172.21.15.102:0/256510134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 msgr2=0x55ee917fbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a3400640 1 --2- 172.21.15.102:0/256510134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917fbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a3400640 1 -- 172.21.15.102:0/256510134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 msgr2=0x55ee917fe580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a3400640 1 --2- 172.21.15.102:0/256510134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917fe580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.564+0000 7f78a3400640 1 -- 172.21.15.102:0/256510134 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee91469210 con 0x55ee917f0fb0 2024-04-25T01:56:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.565+0000 7f78a3400640 1 --2- 172.21.15.102:0/256510134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f1390 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f789c034040 tx=0x7f789c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ce0b16753bdaae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.565+0000 7f788fe00640 1 -- 172.21.15.102:0/256510134 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c040030 con 0x55ee917f0fb0 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.565+0000 7f788fe00640 1 -- 172.21.15.102:0/256510134 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c030ca0 con 0x55ee917f0fb0 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.565+0000 7f788fe00640 1 -- 172.21.15.102:0/256510134 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c036040 con 0x55ee917f0fb0 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.565+0000 7f78a8a00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78880a2e90 0x7f78880a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.565+0000 7f78a8a00640 1 -- 172.21.15.102:0/4288142684 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f78900cf210 con 0x7f78a4100e10 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78a2a00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78880a2e90 0x7f78880a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78acc50740 1 -- 172.21.15.102:0/256510134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 msgr2=0x55ee917f1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78acc50740 1 --2- 172.21.15.102:0/256510134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f1390 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f789c034040 tx=0x7f789c009310 comp rx=0 tx=0).stop 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78a2a00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78880a2e90 0x7f78880a5350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x55ee917f2550 tx=0x7f7894009260 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78acc50740 1 -- 172.21.15.102:0/256510134 shutdown_connections 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78acc50740 1 --2- 172.21.15.102:0/256510134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917fe580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78acc50740 1 --2- 172.21.15.102:0/256510134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78acc50740 1 --2- 172.21.15.102:0/256510134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f1390 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78acc50740 1 -- 172.21.15.102:0/256510134 >> 172.21.15.102:0/256510134 conn(0x55ee917ed4c0 msgr2=0x55ee917ed8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78acc50740 1 -- 172.21.15.102:0/256510134 shutdown_connections 2024-04-25T01:56:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.566+0000 7f78acc50740 1 -- 172.21.15.102:0/256510134 wait complete. 2024-04-25T01:56:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.567+0000 7f78acc50740 1 -- start start 2024-04-25T01:56:05.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.567+0000 7f78acc50740 1 Processor -- start 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78acc50740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78acc50740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917f9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78acc50740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917f9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78acc50740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ee917fed60 con 0x55ee917fc190 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78acc50740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ee91710990 con 0x55ee917f0fb0 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78acc50740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ee917feee0 con 0x55ee917f18d0 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78acc50740 10 client.0 _ll_register_callbacks cb 0x55ee91712950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78a2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78a2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48494/0 (socket says 172.21.15.102:48494) 2024-04-25T01:56:05.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78a2a00640 1 -- 172.21.15.102:0/4061552848 learned_addr learned my addr 172.21.15.102:0/4061552848 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78a3400640 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917f9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.568+0000 7f78a2000640 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917f9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f78a2a00640 1 -- 172.21.15.102:0/4061552848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 msgr2=0x55ee917f9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f78a2a00640 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917f9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f78a2a00640 1 -- 172.21.15.102:0/4061552848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 msgr2=0x55ee917f9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f78a2a00640 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917f9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f78a2a00640 1 -- 172.21.15.102:0/4061552848 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee91535620 con 0x55ee917f0fb0 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f78a3400640 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917f9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f78a2a00640 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f90d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f789401e9d0 tx=0x7f7894004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a35669026ec5647 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f788cc00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f788cc00640 1 -- 172.21.15.102:0/4061552848 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789402e060 con 0x55ee917f0fb0 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f788cc00640 1 -- 172.21.15.102:0/4061552848 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7894018030 con 0x55ee917f0fb0 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.569+0000 7f78acc50740 1 -- 172.21.15.102:0/4061552848 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f789c030ca0 con 0x55ee917f0fb0 2024-04-25T01:56:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.570+0000 7f788cc00640 1 -- 172.21.15.102:0/4061552848 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7894005020 con 0x55ee917f0fb0 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.570+0000 7f78acc50740 10 client.25589 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.570+0000 7f78acc50740 1 -- 172.21.15.102:0/4061552848 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee9180ac40 con 0x55ee917f0fb0 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.570+0000 7f788cc00640 1 -- 172.21.15.102:0/4061552848 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c030ca0 con 0x55ee917f0fb0 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.570+0000 7f788cc00640 1 -- 172.21.15.102:0/4061552848 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f789400f3c0 con 0x55ee917f0fb0 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.571+0000 7f78acc50740 10 client.25589 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.571+0000 7f78acc50740 10 client.25589 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.571+0000 7f78acc50740 20 client.25589 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.571+0000 7f78acc50740 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ee9180c150 0x55ee9180e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.571+0000 7f78acc50740 4 client.25589 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.571+0000 7f78acc50740 1 -- 172.21.15.102:0/4061552848 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee91761ff0 con 0x55ee9180c150 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.571+0000 7f78a2000640 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ee9180c150 0x55ee9180e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.571+0000 7f788cc00640 10 client.25589 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.571+0000 7f78a2000640 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ee9180c150 0x55ee9180e560 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4767597486e5130 server_cookie=f166ec4af2ae786f in_seq=0 out_seq=0 2024-04-25T01:56:05.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.572+0000 7f788cc00640 1 -- 172.21.15.102:0/4061552848 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee91761ff0 con 0x55ee9180c150 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.572+0000 7f788cc00640 10 client.25589 handle_command_reply: tid=0 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.572+0000 7f78acc50740 1 client.25589 shutdown 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.572+0000 7f78acc50740 1 -- 172.21.15.102:0/4061552848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 msgr2=0x55ee917f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.572+0000 7f78acc50740 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f90d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f789401e9d0 tx=0x7f7894004730 comp rx=0 tx=0).stop 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f78acc50740 1 -- 172.21.15.102:0/4061552848 shutdown_connections 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f78acc50740 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ee9180c150 0x55ee9180e560 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f78acc50740 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917f9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f78acc50740 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917f9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f78acc50740 1 --2- 172.21.15.102:0/4061552848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917f90d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f78acc50740 1 -- 172.21.15.102:0/4061552848 >> 172.21.15.102:0/4061552848 conn(0x55ee917ed4c0 msgr2=0x55ee917f31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f788cc00640 0 client.25589 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f78acc50740 1 -- 172.21.15.102:0/4061552848 shutdown_connections 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f78acc50740 1 -- 172.21.15.102:0/4061552848 wait complete. 2024-04-25T01:56:05.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.573+0000 7f78acc50740 20 client.25589 trim_cache size 0 max 16384 2024-04-25T01:56:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.592+0000 7f78acc50740 1 Processor -- start 2024-04-25T01:56:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.592+0000 7f78acc50740 1 -- start start 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78acc50740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee91807c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78acc50740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917ed8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78acc50740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917f9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78acc50740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41b6100 con 0x55ee917fc190 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78acc50740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41b3970 con 0x55ee917f0fb0 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78acc50740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41b17c0 con 0x55ee917f18d0 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917ed8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee91807c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917f9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917ed8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:48318/0 (socket says 172.21.15.102:48318) 2024-04-25T01:56:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a2000640 1 -- 172.21.15.102:0/77218278 learned_addr learned my addr 172.21.15.102:0/77218278 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a2000640 1 -- 172.21.15.102:0/77218278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 msgr2=0x55ee91807c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a2000640 1 --2- 172.21.15.102:0/77218278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee91807c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a2000640 1 -- 172.21.15.102:0/77218278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 msgr2=0x55ee917f9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a2000640 1 --2- 172.21.15.102:0/77218278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917f9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.593+0000 7f78a2000640 1 -- 172.21.15.102:0/77218278 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee917f8080 con 0x55ee917f18d0 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.594+0000 7f78a2000640 1 --2- 172.21.15.102:0/77218278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917ed8a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7890097910 tx=0x7f78900d2ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=198b358f1ea645d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.594+0000 7f788f400640 1 -- 172.21.15.102:0/77218278 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7890020040 con 0x55ee917f18d0 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.594+0000 7f788f400640 1 -- 172.21.15.102:0/77218278 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789009bbb0 con 0x55ee917f18d0 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.594+0000 7f788f400640 1 -- 172.21.15.102:0/77218278 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78900109a0 con 0x55ee917f18d0 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.595+0000 7f78acc50740 1 -- 172.21.15.102:0/77218278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 msgr2=0x55ee917ed8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.595+0000 7f78acc50740 1 --2- 172.21.15.102:0/77218278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917ed8a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7890097910 tx=0x7f78900d2ca0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.595+0000 7f78acc50740 1 -- 172.21.15.102:0/77218278 shutdown_connections 2024-04-25T01:56:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.595+0000 7f78acc50740 1 --2- 172.21.15.102:0/77218278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917fc190 0x55ee917f9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.595+0000 7f78acc50740 1 --2- 172.21.15.102:0/77218278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917f18d0 0x55ee917ed8a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.595+0000 7f78acc50740 1 --2- 172.21.15.102:0/77218278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee91807c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.595+0000 7f78acc50740 1 -- 172.21.15.102:0/77218278 >> 172.21.15.102:0/77218278 conn(0x55ee917ed4c0 msgr2=0x55ee917ec620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.595+0000 7f78acc50740 1 -- 172.21.15.102:0/77218278 shutdown_connections 2024-04-25T01:56:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.595+0000 7f78acc50740 1 -- 172.21.15.102:0/77218278 wait complete. 2024-04-25T01:56:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.596+0000 7f78acc50740 1 -- start start 2024-04-25T01:56:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.596+0000 7f78acc50740 1 Processor -- start 2024-04-25T01:56:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.596+0000 7f78acc50740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917ffc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.596+0000 7f78acc50740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917f18d0 0x55ee918001d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.596+0000 7f78acc50740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917fc190 0x55ee91800710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.596+0000 7f78acc50740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41b1e30 con 0x55ee917f18d0 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.596+0000 7f78acc50740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41b32a0 con 0x55ee917f0fb0 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.596+0000 7f78acc50740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41b2500 con 0x55ee917fc190 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.596+0000 7f78acc50740 10 client.0 _ll_register_callbacks cb 0x7f7894009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917fc190 0x55ee91800710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917f18d0 0x55ee918001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917f18d0 0x55ee918001d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50928/0 (socket says 172.21.15.102:50928) 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2000640 1 -- 172.21.15.102:0/968904820 learned_addr learned my addr 172.21.15.102:0/968904820 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2a00640 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917ffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2000640 1 -- 172.21.15.102:0/968904820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917fc190 msgr2=0x55ee91800710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2000640 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917fc190 0x55ee91800710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2000640 1 -- 172.21.15.102:0/968904820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 msgr2=0x55ee917ffc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2000640 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917ffc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2000640 1 -- 172.21.15.102:0/968904820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee918717e0 con 0x55ee917f18d0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2a00640 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a3400640 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917fc190 0x55ee91800710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f78a2000640 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917f18d0 0x55ee918001d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7890009610 tx=0x7f789009b0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7564a66c7043e4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.597+0000 7f788cc00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.598+0000 7f788cc00640 1 -- 172.21.15.102:0/968904820 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7890020070 con 0x55ee917f18d0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.598+0000 7f788cc00640 1 -- 172.21.15.102:0/968904820 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7890009d10 con 0x55ee917f18d0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.598+0000 7f78acc50740 1 -- 172.21.15.102:0/968904820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f789009bbb0 con 0x55ee917f18d0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.598+0000 7f788cc00640 1 -- 172.21.15.102:0/968904820 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f789009ae10 con 0x55ee917f18d0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.598+0000 7f78acc50740 10 client.15471 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.598+0000 7f78acc50740 1 -- 172.21.15.102:0/968904820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee91872ad0 con 0x55ee917f18d0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.598+0000 7f788cc00640 1 -- 172.21.15.102:0/968904820 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789009bbb0 con 0x55ee917f18d0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.598+0000 7f788cc00640 1 -- 172.21.15.102:0/968904820 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7890010610 con 0x55ee917f18d0 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.599+0000 7f78acc50740 10 client.15471 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.599+0000 7f78acc50740 10 client.15471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.599+0000 7f78acc50740 20 client.15471 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.599+0000 7f78acc50740 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ee91873280 0x55ee918756a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.599+0000 7f78acc50740 4 client.15471 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:56:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.599+0000 7f78acc50740 1 -- 172.21.15.102:0/968904820 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee9175c340 con 0x55ee91873280 2024-04-25T01:56:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.599+0000 7f78a2a00640 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ee91873280 0x55ee918756a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.600+0000 7f78a2a00640 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ee91873280 0x55ee918756a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee75865611ba9d98 server_cookie=1af77e9af154471e in_seq=0 out_seq=0 2024-04-25T01:56:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.600+0000 7f788cc00640 10 client.15471 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f788cc00640 1 -- 172.21.15.102:0/968904820 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee9175c340 con 0x55ee91873280 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f788cc00640 10 client.15471 handle_command_reply: tid=0 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f78acc50740 1 client.15471 shutdown 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f78acc50740 1 -- 172.21.15.102:0/968904820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917f18d0 msgr2=0x55ee918001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f78acc50740 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917f18d0 0x55ee918001d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7890009610 tx=0x7f789009b0b0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f78acc50740 1 -- 172.21.15.102:0/968904820 shutdown_connections 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f78acc50740 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ee91873280 0x55ee918756a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f78acc50740 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ee917fc190 0x55ee91800710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f78acc50740 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee917f18d0 0x55ee918001d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f788cc00640 0 client.15471 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f78acc50740 1 --2- 172.21.15.102:0/968904820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ee917f0fb0 0x55ee917ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.617+0000 7f78acc50740 1 -- 172.21.15.102:0/968904820 >> 172.21.15.102:0/968904820 conn(0x55ee917ed4c0 msgr2=0x55ee917ec620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.618+0000 7f78acc50740 1 -- 172.21.15.102:0/968904820 shutdown_connections 2024-04-25T01:56:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.618+0000 7f78acc50740 1 -- 172.21.15.102:0/968904820 wait complete. 2024-04-25T01:56:05.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.618+0000 7f78acc50740 20 client.15471 trim_cache size 0 max 16384 2024-04-25T01:56:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:56:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:56:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:56:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:56:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 -- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78880a2e90 msgr2=0x7f78880a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78880a2e90 0x7f78880a5350 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x55ee917f2550 tx=0x7f7894009260 comp rx=0 tx=0).stop 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 -- 172.21.15.102:0/4288142684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a4100e10 msgr2=0x7f78a419e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a4100e10 0x7f78a419e930 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f789000d740 tx=0x7f789000db40 comp rx=0 tx=0).stop 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 -- 172.21.15.102:0/4288142684 shutdown_connections 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78880a2e90 0x7f78880a5350 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78a41058f0 0x7f78a41a2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a4100e10 0x7f78a419e930 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 --2- 172.21.15.102:0/4288142684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78a41004f0 0x7f78a419e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 -- 172.21.15.102:0/4288142684 >> 172.21.15.102:0/4288142684 conn(0x7f78a40fbf60 msgr2=0x7f78a40fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 -- 172.21.15.102:0/4288142684 shutdown_connections 2024-04-25T01:56:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:05.619+0000 7f78a3e00640 1 -- 172.21.15.102:0/4288142684 wait complete. 2024-04-25T01:56:05.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:05 smithi102 ceph-mon[27796]: pgmap v27: 97 pgs: 97 active+clean; 903 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 31 MiB/s wr, 548 op/s 2024-04-25T01:56:05.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:05 smithi110 ceph-mon[32848]: pgmap v27: 97 pgs: 97 active+clean; 903 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 31 MiB/s wr, 548 op/s 2024-04-25T01:56:05.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:05 smithi193 ceph-mon[32981]: pgmap v27: 97 pgs: 97 active+clean; 903 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 31 MiB/s wr, 548 op/s 2024-04-25T01:56:07.540 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:07 smithi110 ceph-mon[32848]: pgmap v28: 97 pgs: 97 active+clean; 903 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 18 MiB/s wr, 537 op/s 2024-04-25T01:56:07.540 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:07 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:56:07.540 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:07 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:56:07.540 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:07 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:56:07.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:07 smithi102 ceph-mon[27796]: pgmap v28: 97 pgs: 97 active+clean; 903 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 18 MiB/s wr, 537 op/s 2024-04-25T01:56:07.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:07 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:56:07.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:07 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:56:07.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:07 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:56:07.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:07 smithi193 ceph-mon[32981]: pgmap v28: 97 pgs: 97 active+clean; 903 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 18 MiB/s wr, 537 op/s 2024-04-25T01:56:07.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:07 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:56:07.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:07 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:56:07.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:07 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:56:08.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 15.0 s, 199.0 tps, lat 476.191 ms stddev 290.443 2024-04-25T01:56:09.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:09 smithi102 ceph-mon[27796]: pgmap v29: 97 pgs: 97 active+clean; 910 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 19 MiB/s wr, 753 op/s 2024-04-25T01:56:09.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:09 smithi110 ceph-mon[32848]: pgmap v29: 97 pgs: 97 active+clean; 910 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 19 MiB/s wr, 753 op/s 2024-04-25T01:56:09.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:09 smithi193 ceph-mon[32981]: pgmap v29: 97 pgs: 97 active+clean; 910 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 19 MiB/s wr, 753 op/s 2024-04-25T01:56:11.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:11 smithi102 ceph-mon[27796]: pgmap v30: 97 pgs: 97 active+clean; 910 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.9 MiB/s wr, 612 op/s 2024-04-25T01:56:11.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:11 smithi110 ceph-mon[32848]: pgmap v30: 97 pgs: 97 active+clean; 910 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.9 MiB/s wr, 612 op/s 2024-04-25T01:56:11.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:11 smithi193 ceph-mon[32981]: pgmap v30: 97 pgs: 97 active+clean; 910 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.9 MiB/s wr, 612 op/s 2024-04-25T01:56:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 20.0 s, 2608.1 tps, lat 41.419 ms stddev 122.347 2024-04-25T01:56:13.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:13 smithi102 ceph-mon[27796]: pgmap v31: 97 pgs: 97 active+clean; 935 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 30 MiB/s wr, 1.04k op/s 2024-04-25T01:56:13.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:13 smithi110 ceph-mon[32848]: pgmap v31: 97 pgs: 97 active+clean; 935 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 30 MiB/s wr, 1.04k op/s 2024-04-25T01:56:13.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:13 smithi193 ceph-mon[32981]: pgmap v31: 97 pgs: 97 active+clean; 935 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 30 MiB/s wr, 1.04k op/s 2024-04-25T01:56:15.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:15 smithi102 ceph-mon[27796]: pgmap v32: 97 pgs: 97 active+clean; 954 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 39 MiB/s wr, 976 op/s 2024-04-25T01:56:15.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:15 smithi110 ceph-mon[32848]: pgmap v32: 97 pgs: 97 active+clean; 954 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 39 MiB/s wr, 976 op/s 2024-04-25T01:56:15.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:15 smithi193 ceph-mon[32981]: pgmap v32: 97 pgs: 97 active+clean; 954 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 39 MiB/s wr, 976 op/s 2024-04-25T01:56:17.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:17 smithi102 ceph-mon[27796]: pgmap v33: 97 pgs: 97 active+clean; 954 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 MiB/s wr, 809 op/s 2024-04-25T01:56:17.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:17 smithi110 ceph-mon[32848]: pgmap v33: 97 pgs: 97 active+clean; 954 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 MiB/s wr, 809 op/s 2024-04-25T01:56:17.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:17 smithi193 ceph-mon[32981]: pgmap v33: 97 pgs: 97 active+clean; 954 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 MiB/s wr, 809 op/s 2024-04-25T01:56:18.070 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 25.0 s, 3944.8 tps, lat 25.289 ms stddev 25.547 2024-04-25T01:56:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.812+0000 7fed4a400640 1 -- 172.21.15.102:0/1348874498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed4410e100 msgr2=0x7fed441104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.812+0000 7fed4a400640 1 --2- 172.21.15.102:0/1348874498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed4410e100 0x7fed441104f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009900 tx=0x7fed3c02a420 comp rx=0 tx=0).stop 2024-04-25T01:56:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.814+0000 7fed4a400640 1 -- 172.21.15.102:0/1348874498 shutdown_connections 2024-04-25T01:56:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.814+0000 7fed4a400640 1 --2- 172.21.15.102:0/1348874498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed4410e100 0x7fed441104f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.814+0000 7fed4a400640 1 --2- 172.21.15.102:0/1348874498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed44100e70 0x7fed4410dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.814+0000 7fed4a400640 1 --2- 172.21.15.102:0/1348874498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed44100550 0x7fed44100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.817 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.814+0000 7fed4a400640 1 -- 172.21.15.102:0/1348874498 >> 172.21.15.102:0/1348874498 conn(0x7fed440777c0 msgr2=0x7fed44077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.815+0000 7fed4a400640 1 -- 172.21.15.102:0/1348874498 shutdown_connections 2024-04-25T01:56:18.818 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.815+0000 7fed4a400640 1 -- 172.21.15.102:0/1348874498 wait complete. 2024-04-25T01:56:18.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.816+0000 7fed4a400640 1 Processor -- start 2024-04-25T01:56:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.818+0000 7fed4a400640 1 -- start start 2024-04-25T01:56:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.818+0000 7fed4a400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed44100550 0x7fed44102830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.818+0000 7fed4a400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed44100e70 0x7fed44102d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.818+0000 7fed4a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed4410e100 0x7fed441032b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.818+0000 7fed4a400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fed44113d80 con 0x7fed44100550 2024-04-25T01:56:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.818+0000 7fed4a400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fed44113c00 con 0x7fed44100e70 2024-04-25T01:56:18.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.818+0000 7fed4a400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fed44113f00 con 0x7fed4410e100 2024-04-25T01:56:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.818+0000 7fed42a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed44100e70 0x7fed44102d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.819+0000 7fed43e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed4410e100 0x7fed441032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.819+0000 7fed43e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed4410e100 0x7fed441032b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34972/0 (socket says 172.21.15.102:34972) 2024-04-25T01:56:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.819+0000 7fed43e00640 1 -- 172.21.15.102:0/2851587430 learned_addr learned my addr 172.21.15.102:0/2851587430 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.819+0000 7fed43400640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed44100550 0x7fed44102830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.819+0000 7fed42a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed44100e70 0x7fed44102d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48552/0 (socket says 172.21.15.102:48552) 2024-04-25T01:56:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.820+0000 7fed43e00640 1 -- 172.21.15.102:0/2851587430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed44100e70 msgr2=0x7fed44102d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.820+0000 7fed43e00640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed44100e70 0x7fed44102d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.823 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.820+0000 7fed43e00640 1 -- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed44100550 msgr2=0x7fed44102830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.820+0000 7fed43e00640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed44100550 0x7fed44102830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.820+0000 7fed43e00640 1 -- 172.21.15.102:0/2851587430 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed441aa650 con 0x7fed4410e100 2024-04-25T01:56:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.821+0000 7fed43e00640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed4410e100 0x7fed441032b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fed3c02a940 tx=0x7fed3c003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:18.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.821+0000 7fed3be00640 1 -- 172.21.15.102:0/2851587430 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c03f070 con 0x7fed4410e100 2024-04-25T01:56:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.821+0000 7fed4a400640 1 -- 172.21.15.102:0/2851587430 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed441aa8e0 con 0x7fed4410e100 2024-04-25T01:56:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.821+0000 7fed3be00640 1 -- 172.21.15.102:0/2851587430 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed3c031c20 con 0x7fed4410e100 2024-04-25T01:56:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.821+0000 7fed3be00640 1 -- 172.21.15.102:0/2851587430 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c035030 con 0x7fed4410e100 2024-04-25T01:56:18.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.821+0000 7fed4a400640 1 -- 172.21.15.102:0/2851587430 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed441aae20 con 0x7fed4410e100 2024-04-25T01:56:18.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.825+0000 7fed4d1ba740 1 Processor -- start 2024-04-25T01:56:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.827+0000 7fed3be00640 1 -- 172.21.15.102:0/2851587430 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7fed3c042070 con 0x7fed4410e100 2024-04-25T01:56:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.830+0000 7fed4d1ba740 1 -- start start 2024-04-25T01:56:18.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.830+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e083de0 0x55a16e0841c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.830+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e084700 0x55a16e08ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.830+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e08efc0 0x55a16e0913b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.830+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a16dfe7ab0 con 0x55a16e084700 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.830+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a16df90640 con 0x55a16e083de0 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.830+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a16dfe3a70 con 0x55a16e08efc0 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.830+0000 7fed3be00640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fed280a2f40 0x7fed280a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.831+0000 7fed3be00640 1 -- 172.21.15.102:0/2851587430 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fed3c0efdf0 con 0x7fed4410e100 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.831+0000 7fed43400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e083de0 0x55a16e0841c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.831+0000 7fed43400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e083de0 0x55a16e0841c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48568/0 (socket says 172.21.15.102:48568) 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.831+0000 7fed43400640 1 -- 172.21.15.102:0/4179510590 learned_addr learned my addr 172.21.15.102:0/4179510590 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:18.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.831+0000 7fed43e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e08efc0 0x55a16e0913b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.832+0000 7fed42a00640 1 --2- 172.21.15.102:0/4179510590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e084700 0x55a16e08ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.832+0000 7fed43e00640 1 -- 172.21.15.102:0/4179510590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e083de0 msgr2=0x55a16e0841c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.832+0000 7fed43e00640 1 --2- 172.21.15.102:0/4179510590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e083de0 0x55a16e0841c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.832+0000 7fed43e00640 1 -- 172.21.15.102:0/4179510590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e084700 msgr2=0x55a16e08ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.832+0000 7fed43e00640 1 --2- 172.21.15.102:0/4179510590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e084700 0x55a16e08ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.832+0000 7fed43e00640 1 -- 172.21.15.102:0/4179510590 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a16dcfc210 con 0x55a16e08efc0 2024-04-25T01:56:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.832+0000 7fed42a00640 1 --2- 172.21.15.102:0/4179510590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e084700 0x55a16e08ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.833+0000 7fed43e00640 1 --2- 172.21.15.102:0/4179510590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e08efc0 0x55a16e0913b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fed3c0ed390 tx=0x7fed3c0bbd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e349be046f90b46e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.833+0000 7fed38c00640 1 -- 172.21.15.102:0/4179510590 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c049070 con 0x55a16e08efc0 2024-04-25T01:56:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.833+0000 7fed38c00640 1 -- 172.21.15.102:0/4179510590 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed3c0c0510 con 0x55a16e08efc0 2024-04-25T01:56:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.833+0000 7fed38c00640 1 -- 172.21.15.102:0/4179510590 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0c4940 con 0x55a16e08efc0 2024-04-25T01:56:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.833+0000 7fed42a00640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fed280a2f40 0x7fed280a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.834+0000 7fed4d1ba740 1 -- 172.21.15.102:0/4179510590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e08efc0 msgr2=0x55a16e0913b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.834+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/4179510590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e08efc0 0x55a16e0913b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fed3c0ed390 tx=0x7fed3c0bbd50 comp rx=0 tx=0).stop 2024-04-25T01:56:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.834+0000 7fed4d1ba740 1 -- 172.21.15.102:0/4179510590 shutdown_connections 2024-04-25T01:56:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.834+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/4179510590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e08efc0 0x55a16e0913b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.834+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/4179510590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e084700 0x55a16e08ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.834+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/4179510590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e083de0 0x55a16e0841c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.834+0000 7fed4d1ba740 1 -- 172.21.15.102:0/4179510590 >> 172.21.15.102:0/4179510590 conn(0x55a16e0802f0 msgr2=0x55a16e0806d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.834+0000 7fed4d1ba740 1 -- 172.21.15.102:0/4179510590 shutdown_connections 2024-04-25T01:56:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.835+0000 7fed4d1ba740 1 -- 172.21.15.102:0/4179510590 wait complete. 2024-04-25T01:56:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.835+0000 7fed4d1ba740 1 -- start start 2024-04-25T01:56:18.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.835+0000 7fed4d1ba740 1 Processor -- start 2024-04-25T01:56:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.836+0000 7fed42a00640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fed280a2f40 0x7fed280a5400 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55a16e085380 tx=0x7fed2c004730 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.836+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e08bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.836+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.836+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 0x55a16e08c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.836+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a16e0919b0 con 0x55a16e08efc0 2024-04-25T01:56:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.836+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a16dfa3990 con 0x55a16e084700 2024-04-25T01:56:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.836+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a16e091b30 con 0x55a16e083de0 2024-04-25T01:56:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.836+0000 7fed4d1ba740 10 client.0 _ll_register_callbacks cb 0x55a16dfa5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.837+0000 7fed43400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e08bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.837+0000 7fed43400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e08bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34996/0 (socket says 172.21.15.102:34996) 2024-04-25T01:56:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.837+0000 7fed43400640 1 -- 172.21.15.102:0/1375299180 learned_addr learned my addr 172.21.15.102:0/1375299180 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.837+0000 7fed42a00640 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.837+0000 7fed43400640 1 -- 172.21.15.102:0/1375299180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 msgr2=0x55a16e08c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.837+0000 7fed43400640 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.837+0000 7fed43400640 1 -- 172.21.15.102:0/1375299180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 msgr2=0x55a16e08c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.837+0000 7fed43400640 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 0x55a16e08c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.837+0000 7fed43400640 1 -- 172.21.15.102:0/1375299180 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a16ddc8620 con 0x55a16e083de0 2024-04-25T01:56:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.838+0000 7fed42a00640 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.838+0000 7fed43400640 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e08bf20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fed34010c00 tx=0x7fed3400f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6aaffa7ae445ba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.838+0000 7fed31600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:56:18.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.838+0000 7fed31600640 1 -- 172.21.15.102:0/1375299180 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3400e5c0 con 0x55a16e083de0 2024-04-25T01:56:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.838+0000 7fed31600640 1 -- 172.21.15.102:0/1375299180 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fed3400e760 con 0x55a16e083de0 2024-04-25T01:56:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.838+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1375299180 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed3c0c0510 con 0x55a16e083de0 2024-04-25T01:56:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.839+0000 7fed31600640 1 -- 172.21.15.102:0/1375299180 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed3400e960 con 0x55a16e083de0 2024-04-25T01:56:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.839+0000 7fed4d1ba740 10 client.25582 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:18.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.839+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1375299180 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55a16e0ae6d0 con 0x55a16e083de0 2024-04-25T01:56:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.839+0000 7fed31600640 1 -- 172.21.15.102:0/1375299180 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fed3400eb00 con 0x55a16e083de0 2024-04-25T01:56:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.839+0000 7fed4d1ba740 10 client.25582 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.840+0000 7fed4d1ba740 10 client.25582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.840+0000 7fed4d1ba740 20 client.25582 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.840+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a16e0afad0 0x55a16e0b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.840+0000 7fed4d1ba740 4 client.25582 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:18.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.840+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1375299180 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a16dff4ff0 con 0x55a16e0afad0 2024-04-25T01:56:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.840+0000 7fed42a00640 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a16e0afad0 0x55a16e0b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.840+0000 7fed42a00640 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a16e0afad0 0x55a16e0b1f90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14eec78f3d24751b server_cookie=a639d04fa253ec1 in_seq=0 out_seq=0 2024-04-25T01:56:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.841+0000 7fed31600640 10 client.25582 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.841+0000 7fed31600640 1 -- 172.21.15.102:0/1375299180 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a16dff4ff0 con 0x55a16e0afad0 2024-04-25T01:56:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.841+0000 7fed31600640 10 client.25582 handle_command_reply: tid=0 2024-04-25T01:56:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.842+0000 7fed4d1ba740 1 client.25582 shutdown 2024-04-25T01:56:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.843+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1375299180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 msgr2=0x55a16e08bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:18.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.843+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e08bf20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fed34010c00 tx=0x7fed3400f7f0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.844+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1375299180 shutdown_connections 2024-04-25T01:56:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.844+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a16e0afad0 0x55a16e0b1f90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.844+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 0x55a16e08c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.844+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.844+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1375299180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e08bf20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.844+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1375299180 >> 172.21.15.102:0/1375299180 conn(0x55a16e0802f0 msgr2=0x55a16e086070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.844+0000 7fed31600640 0 client.25582 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.844+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1375299180 shutdown_connections 2024-04-25T01:56:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.845+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1375299180 wait complete. 2024-04-25T01:56:18.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.845+0000 7fed4d1ba740 20 client.25582 trim_cache size 0 max 16384 2024-04-25T01:56:18.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.868+0000 7fed4d1ba740 1 Processor -- start 2024-04-25T01:56:18.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.868+0000 7fed4d1ba740 1 -- start start 2024-04-25T01:56:18.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.868+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e09a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.868+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.868+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 0x55a16e0b78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.868+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fed441ad9c0 con 0x55a16e08efc0 2024-04-25T01:56:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.868+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fed441ab230 con 0x55a16e084700 2024-04-25T01:56:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.868+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fed4404f500 con 0x55a16e083de0 2024-04-25T01:56:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.869+0000 7fed42a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.869+0000 7fed42a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48578/0 (socket says 172.21.15.102:48578) 2024-04-25T01:56:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.869+0000 7fed42a00640 1 -- 172.21.15.102:0/1614154013 learned_addr learned my addr 172.21.15.102:0/1614154013 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:18.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.869+0000 7fed43400640 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e09a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.869+0000 7fed42a00640 1 -- 172.21.15.102:0/1614154013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 msgr2=0x55a16e09a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.869+0000 7fed42a00640 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e09a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.869+0000 7fed42a00640 1 -- 172.21.15.102:0/1614154013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 msgr2=0x55a16e0b78a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.870+0000 7fed43e00640 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 0x55a16e0b78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.870+0000 7fed42a00640 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 0x55a16e0b78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.870+0000 7fed42a00640 1 -- 172.21.15.102:0/1614154013 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a16e091720 con 0x55a16e084700 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.870+0000 7fed43e00640 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 0x55a16e0b78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.870+0000 7fed42a00640 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c7c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fed44103ff0 tx=0x7fed2c014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6271e31d3c252a9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.871+0000 7fed33e00640 1 -- 172.21.15.102:0/1614154013 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2c026d50 con 0x55a16e084700 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.871+0000 7fed33e00640 1 -- 172.21.15.102:0/1614154013 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed2c026310 con 0x55a16e084700 2024-04-25T01:56:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.871+0000 7fed33e00640 1 -- 172.21.15.102:0/1614154013 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2c01e880 con 0x55a16e084700 2024-04-25T01:56:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.872+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1614154013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 msgr2=0x55a16e08c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.872+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c7c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fed44103ff0 tx=0x7fed2c014040 comp rx=0 tx=0).stop 2024-04-25T01:56:18.875 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.872+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1614154013 shutdown_connections 2024-04-25T01:56:18.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.872+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e08efc0 0x55a16e0b78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.872+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e084700 0x55a16e08c7c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.872+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/1614154013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e083de0 0x55a16e09a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.872+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1614154013 >> 172.21.15.102:0/1614154013 conn(0x55a16e0802f0 msgr2=0x55a16e07f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:18.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.872+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1614154013 shutdown_connections 2024-04-25T01:56:18.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.873+0000 7fed4d1ba740 1 -- 172.21.15.102:0/1614154013 wait complete. 2024-04-25T01:56:18.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.873+0000 7fed4d1ba740 1 -- start start 2024-04-25T01:56:18.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.873+0000 7fed4d1ba740 1 Processor -- start 2024-04-25T01:56:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e083de0 0x55a16e096e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e084700 0x55a16e0887d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed4d1ba740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e08efc0 0x55a16e088d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fed44071f40 con 0x55a16e083de0 2024-04-25T01:56:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fed440733b0 con 0x55a16e08efc0 2024-04-25T01:56:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed4d1ba740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fed44072610 con 0x55a16e084700 2024-04-25T01:56:18.877 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed4d1ba740 10 client.0 _ll_register_callbacks cb 0x7fed34010810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed42a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e084700 0x55a16e0887d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed42a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e084700 0x55a16e0887d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35014/0 (socket says 172.21.15.102:35014) 2024-04-25T01:56:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed42a00640 1 -- 172.21.15.102:0/3765853758 learned_addr learned my addr 172.21.15.102:0/3765853758 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed43400640 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e083de0 0x55a16e096e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.874+0000 7fed43e00640 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e08efc0 0x55a16e088d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.875+0000 7fed43400640 1 -- 172.21.15.102:0/3765853758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e084700 msgr2=0x55a16e0887d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.875+0000 7fed43400640 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e084700 0x55a16e0887d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.875+0000 7fed43400640 1 -- 172.21.15.102:0/3765853758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e08efc0 msgr2=0x55a16e088d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:18.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.875+0000 7fed43400640 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e08efc0 0x55a16e088d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.875+0000 7fed43400640 1 -- 172.21.15.102:0/3765853758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a16e0892d0 con 0x55a16e083de0 2024-04-25T01:56:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.876+0000 7fed43400640 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e083de0 0x55a16e096e80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fed34010bd0 tx=0x7fed34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abaab4f479f42974 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.876+0000 7fed31600640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:56:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.876+0000 7fed31600640 1 -- 172.21.15.102:0/3765853758 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed34007300 con 0x55a16e083de0 2024-04-25T01:56:18.880 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.876+0000 7fed4d1ba740 1 -- 172.21.15.102:0/3765853758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed2c026310 con 0x55a16e083de0 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.876+0000 7fed31600640 1 -- 172.21.15.102:0/3765853758 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fed340247d0 con 0x55a16e083de0 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.876+0000 7fed31600640 1 -- 172.21.15.102:0/3765853758 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fed340249d0 con 0x55a16e083de0 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.876+0000 7fed4d1ba740 10 client.15480 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.876+0000 7fed4d1ba740 1 -- 172.21.15.102:0/3765853758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a16e115e40 con 0x55a16e083de0 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.877+0000 7fed31600640 1 -- 172.21.15.102:0/3765853758 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed2c026310 con 0x55a16e083de0 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.877+0000 7fed31600640 1 -- 172.21.15.102:0/3765853758 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fed34004130 con 0x55a16e083de0 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.877+0000 7fed4d1ba740 10 client.15480 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.877+0000 7fed4d1ba740 10 client.15480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.877+0000 7fed4d1ba740 20 client.15480 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.877+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a16e116550 0x55a16e118940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.877+0000 7fed4d1ba740 4 client.15480 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:56:18.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.877+0000 7fed4d1ba740 1 -- 172.21.15.102:0/3765853758 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a16dfef340 con 0x55a16e116550 2024-04-25T01:56:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.878+0000 7fed42a00640 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a16e116550 0x55a16e118940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.878+0000 7fed31600640 10 client.15480 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:18.882 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.879+0000 7fed42a00640 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a16e116550 0x55a16e118940 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fead72dddfeea58b server_cookie=c56b1fb6e234e93e in_seq=0 out_seq=0 2024-04-25T01:56:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.895+0000 7fed31600640 1 -- 172.21.15.102:0/3765853758 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a16dfef340 con 0x55a16e116550 2024-04-25T01:56:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.895+0000 7fed31600640 10 client.15480 handle_command_reply: tid=0 2024-04-25T01:56:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.895+0000 7fed4d1ba740 1 client.15480 shutdown 2024-04-25T01:56:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.896+0000 7fed4d1ba740 1 -- 172.21.15.102:0/3765853758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e083de0 msgr2=0x55a16e096e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.896+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e083de0 0x55a16e096e80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fed34010bd0 tx=0x7fed34004a40 comp rx=0 tx=0).stop 2024-04-25T01:56:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.896+0000 7fed4d1ba740 1 -- 172.21.15.102:0/3765853758 shutdown_connections 2024-04-25T01:56:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.896+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a16e116550 0x55a16e118940 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.896+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a16e08efc0 0x55a16e088d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.896+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a16e084700 0x55a16e0887d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.896+0000 7fed4d1ba740 1 --2- 172.21.15.102:0/3765853758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a16e083de0 0x55a16e096e80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.896+0000 7fed4d1ba740 1 -- 172.21.15.102:0/3765853758 >> 172.21.15.102:0/3765853758 conn(0x55a16e0802f0 msgr2=0x55a16e07f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.897+0000 7fed31600640 0 client.15480 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.897+0000 7fed4d1ba740 1 -- 172.21.15.102:0/3765853758 shutdown_connections 2024-04-25T01:56:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.897+0000 7fed4d1ba740 1 -- 172.21.15.102:0/3765853758 wait complete. 2024-04-25T01:56:18.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.897+0000 7fed4d1ba740 20 client.15480 trim_cache size 0 max 16384 2024-04-25T01:56:18.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:56:18.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:56:18.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:56:18.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:56:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.899+0000 7fed4a400640 1 -- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fed280a2f40 msgr2=0x7fed280a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.899+0000 7fed4a400640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fed280a2f40 0x7fed280a5400 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55a16e085380 tx=0x7fed2c004730 comp rx=0 tx=0).stop 2024-04-25T01:56:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.900+0000 7fed4a400640 1 -- 172.21.15.102:0/2851587430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed4410e100 msgr2=0x7fed441032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.900+0000 7fed4a400640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed4410e100 0x7fed441032b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fed3c02a940 tx=0x7fed3c003b90 comp rx=0 tx=0).stop 2024-04-25T01:56:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.900+0000 7fed4a400640 1 -- 172.21.15.102:0/2851587430 shutdown_connections 2024-04-25T01:56:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.900+0000 7fed4a400640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fed280a2f40 0x7fed280a5400 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.900+0000 7fed4a400640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed4410e100 0x7fed441032b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.900+0000 7fed4a400640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed44100e70 0x7fed44102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.900+0000 7fed4a400640 1 --2- 172.21.15.102:0/2851587430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed44100550 0x7fed44102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.900+0000 7fed4a400640 1 -- 172.21.15.102:0/2851587430 >> 172.21.15.102:0/2851587430 conn(0x7fed440777c0 msgr2=0x7fed4410f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.901+0000 7fed4a400640 1 -- 172.21.15.102:0/2851587430 shutdown_connections 2024-04-25T01:56:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:18.901+0000 7fed4a400640 1 -- 172.21.15.102:0/2851587430 wait complete. 2024-04-25T01:56:19.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:19 smithi102 ceph-mon[27796]: pgmap v34: 97 pgs: 97 active+clean; 996 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 43 MiB/s wr, 1.36k op/s 2024-04-25T01:56:19.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:19 smithi110 ceph-mon[32848]: pgmap v34: 97 pgs: 97 active+clean; 996 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 43 MiB/s wr, 1.36k op/s 2024-04-25T01:56:19.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:19 smithi193 ceph-mon[32981]: pgmap v34: 97 pgs: 97 active+clean; 996 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 43 MiB/s wr, 1.36k op/s 2024-04-25T01:56:21.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:21 smithi102 ceph-mon[27796]: pgmap v35: 97 pgs: 97 active+clean; 996 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 42 MiB/s wr, 1.14k op/s 2024-04-25T01:56:21.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:21 smithi110 ceph-mon[32848]: pgmap v35: 97 pgs: 97 active+clean; 996 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 42 MiB/s wr, 1.14k op/s 2024-04-25T01:56:21.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:21 smithi193 ceph-mon[32981]: pgmap v35: 97 pgs: 97 active+clean; 996 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 42 MiB/s wr, 1.14k op/s 2024-04-25T01:56:23.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 30.0 s, 3885.1 tps, lat 25.670 ms stddev 29.349 2024-04-25T01:56:23.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:23 smithi102 ceph-mon[27796]: pgmap v36: 97 pgs: 97 active+clean; 1017 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 45 MiB/s wr, 1.39k op/s 2024-04-25T01:56:23.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:23 smithi110 ceph-mon[32848]: pgmap v36: 97 pgs: 97 active+clean; 1017 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 45 MiB/s wr, 1.39k op/s 2024-04-25T01:56:23.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:23 smithi193 ceph-mon[32981]: pgmap v36: 97 pgs: 97 active+clean; 1017 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 45 MiB/s wr, 1.39k op/s 2024-04-25T01:56:25.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:25 smithi102 ceph-mon[27796]: pgmap v37: 97 pgs: 97 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 21 MiB/s wr, 1.06k op/s 2024-04-25T01:56:25.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:25 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:56:25.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:25 smithi110 ceph-mon[32848]: pgmap v37: 97 pgs: 97 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 21 MiB/s wr, 1.06k op/s 2024-04-25T01:56:25.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:25 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:56:25.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:25 smithi193 ceph-mon[32981]: pgmap v37: 97 pgs: 97 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 21 MiB/s wr, 1.06k op/s 2024-04-25T01:56:25.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:25 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:56:27.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:27 smithi102 ceph-mon[27796]: pgmap v38: 97 pgs: 97 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 891 op/s 2024-04-25T01:56:27.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:27 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:56:27.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:27 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:56:27.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:27 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:56:27.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:27 smithi110 ceph-mon[32848]: pgmap v38: 97 pgs: 97 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 891 op/s 2024-04-25T01:56:27.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:27 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:56:27.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:27 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:56:27.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:27 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:56:27.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:27 smithi193 ceph-mon[32981]: pgmap v38: 97 pgs: 97 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 10 MiB/s wr, 891 op/s 2024-04-25T01:56:27.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:27 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:56:27.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:27 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:56:27.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:27 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:56:28.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 35.0 s, 2955.7 tps, lat 33.775 ms stddev 34.678 2024-04-25T01:56:28.271 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:56:28.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:56:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.525+0000 7f074d000640 1 -- 172.21.15.102:0/4121906646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 msgr2=0x7f074810dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.525+0000 7f074d000640 1 --2- 172.21.15.102:0/4121906646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 0x7f074810dae0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0738009900 tx=0x7f073802a4a0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.525+0000 7f074d000640 1 -- 172.21.15.102:0/4121906646 shutdown_connections 2024-04-25T01:56:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.525+0000 7f074d000640 1 --2- 172.21.15.102:0/4121906646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f074810e020 0x7f0748110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.525+0000 7f074d000640 1 --2- 172.21.15.102:0/4121906646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 0x7f074810dae0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.525+0000 7f074d000640 1 --2- 172.21.15.102:0/4121906646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0748073a10 0x7f0748073df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.525+0000 7f074d000640 1 -- 172.21.15.102:0/4121906646 >> 172.21.15.102:0/4121906646 conn(0x7f0748078b90 msgr2=0x7f0748078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:28.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.525+0000 7f074d000640 1 -- 172.21.15.102:0/4121906646 shutdown_connections 2024-04-25T01:56:28.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.526+0000 7f074d000640 1 -- 172.21.15.102:0/4121906646 wait complete. 2024-04-25T01:56:28.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.527+0000 7f074d000640 1 Processor -- start 2024-04-25T01:56:28.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.527+0000 7f074d000640 1 -- start start 2024-04-25T01:56:28.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.527+0000 7f074d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748073a10 0x7f0748101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.527+0000 7f074d000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 0x7f0748101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.527+0000 7f074d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f074810e020 0x7f07481021c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f0745e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748073a10 0x7f0748101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f0745e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748073a10 0x7f0748101740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50272/0 (socket says 172.21.15.102:50272) 2024-04-25T01:56:28.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f0746800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f074810e020 0x7f07481021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f0746800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f074810e020 0x7f07481021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49480/0 (socket says 172.21.15.102:49480) 2024-04-25T01:56:28.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f0745400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 0x7f0748101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f0745400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 0x7f0748101c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59198/0 (socket says 172.21.15.102:59198) 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0748113d70 con 0x7f0748073a10 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0748113bf0 con 0x7f0748074330 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0748113ef0 con 0x7f074810e020 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.528+0000 7f0745e00640 1 -- 172.21.15.102:0/89899683 learned_addr learned my addr 172.21.15.102:0/89899683 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.529+0000 7f0746800640 1 -- 172.21.15.102:0/89899683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 msgr2=0x7f0748101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.529+0000 7f0746800640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 0x7f0748101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.529+0000 7f0746800640 1 -- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748073a10 msgr2=0x7f0748101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.529+0000 7f0746800640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748073a10 0x7f0748101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.529+0000 7f0746800640 1 -- 172.21.15.102:0/89899683 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07481aa920 con 0x7f074810e020 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.529+0000 7f0745e00640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748073a10 0x7f0748101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.529+0000 7f0745400640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 0x7f0748101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.530+0000 7f0746800640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f074810e020 0x7f07481021c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0740005fd0 tx=0x7f074000dea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.530+0000 7f073ea00640 1 -- 172.21.15.102:0/89899683 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074000bd40 con 0x7f074810e020 2024-04-25T01:56:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.530+0000 7f073ea00640 1 -- 172.21.15.102:0/89899683 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0740003e50 con 0x7f074810e020 2024-04-25T01:56:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.530+0000 7f073ea00640 1 -- 172.21.15.102:0/89899683 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0740010750 con 0x7f074810e020 2024-04-25T01:56:28.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.531+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07481aac10 con 0x7f074810e020 2024-04-25T01:56:28.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.531+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07481ab0e0 con 0x7f074810e020 2024-04-25T01:56:28.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.535+0000 7f073ea00640 1 -- 172.21.15.102:0/89899683 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f07400042e0 con 0x7f074810e020 2024-04-25T01:56:28.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.543+0000 7f073ea00640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f072c0a3030 0x7f072c0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.544+0000 7f0745e00640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f072c0a3030 0x7f072c0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.544+0000 7f073ea00640 1 -- 172.21.15.102:0/89899683 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f07400cf1a0 con 0x7f074810e020 2024-04-25T01:56:28.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.545+0000 7f074fe54740 1 Processor -- start 2024-04-25T01:56:28.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.545+0000 7f0745e00640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f072c0a3030 0x7f072c0a54f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0730009ce0 tx=0x7f0730005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.546+0000 7f074fe54740 1 -- start start 2024-04-25T01:56:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.547+0000 7f074fe54740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe2463e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.548+0000 7f074fe54740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe250ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.548+0000 7f074fe54740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 0x5588fe2533e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.548+0000 7f074fe54740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5588fe1aa560 con 0x5588fe246000 2024-04-25T01:56:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.548+0000 7f074fe54740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5588fe152930 con 0x5588fe246920 2024-04-25T01:56:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.548+0000 7f074fe54740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5588fe1a6520 con 0x5588fe250ff0 2024-04-25T01:56:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.548+0000 7f0745400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe250ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.548+0000 7f0745400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe250ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59206/0 (socket says 172.21.15.102:59206) 2024-04-25T01:56:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.548+0000 7f0745400640 1 -- 172.21.15.102:0/1601026643 learned_addr learned my addr 172.21.15.102:0/1601026643 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.549+0000 7f0746800640 1 --2- 172.21.15.102:0/1601026643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 0x5588fe2533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.549+0000 7f0745400640 1 -- 172.21.15.102:0/1601026643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 msgr2=0x5588fe2533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.549+0000 7f0745400640 1 --2- 172.21.15.102:0/1601026643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 0x5588fe2533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.549+0000 7f0745400640 1 -- 172.21.15.102:0/1601026643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 msgr2=0x5588fe2463e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:56:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.550+0000 7f0745e00640 1 --2- 172.21.15.102:0/1601026643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe2463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.550+0000 7f0745400640 1 --2- 172.21.15.102:0/1601026643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe2463e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.550+0000 7f0745400640 1 -- 172.21.15.102:0/1601026643 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588fdf887d0 con 0x5588fe246920 2024-04-25T01:56:28.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.550+0000 7f0745400640 1 --2- 172.21.15.102:0/1601026643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe250ab0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0738009230 tx=0x7f07380043f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34842192cb60a6a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:28.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.551+0000 7f0737400640 1 -- 172.21.15.102:0/1601026643 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738035040 con 0x5588fe246920 2024-04-25T01:56:28.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.551+0000 7f0737400640 1 -- 172.21.15.102:0/1601026643 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0738004700 con 0x5588fe246920 2024-04-25T01:56:28.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.551+0000 7f0737400640 1 -- 172.21.15.102:0/1601026643 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738035040 con 0x5588fe246920 2024-04-25T01:56:28.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.554+0000 7f074fe54740 1 -- 172.21.15.102:0/1601026643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 msgr2=0x5588fe250ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.554+0000 7f074fe54740 1 --2- 172.21.15.102:0/1601026643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe250ab0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0738009230 tx=0x7f07380043f0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.554+0000 7f074fe54740 1 -- 172.21.15.102:0/1601026643 shutdown_connections 2024-04-25T01:56:28.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.554+0000 7f074fe54740 1 --2- 172.21.15.102:0/1601026643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 0x5588fe2533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.554+0000 7f074fe54740 1 --2- 172.21.15.102:0/1601026643 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe250ab0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.554+0000 7f074fe54740 1 --2- 172.21.15.102:0/1601026643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe2463e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.554+0000 7f074fe54740 1 -- 172.21.15.102:0/1601026643 >> 172.21.15.102:0/1601026643 conn(0x5588fe242320 msgr2=0x5588fe242700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:28.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.554+0000 7f074fe54740 1 -- 172.21.15.102:0/1601026643 shutdown_connections 2024-04-25T01:56:28.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.554+0000 7f074fe54740 1 -- 172.21.15.102:0/1601026643 wait complete. 2024-04-25T01:56:28.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.555+0000 7f074fe54740 1 -- start start 2024-04-25T01:56:28.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.555+0000 7f074fe54740 1 Processor -- start 2024-04-25T01:56:28.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.556+0000 7f074fe54740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe24dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.556+0000 7f074fe54740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe24e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.556+0000 7f074fe54740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 0x5588fe24ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.556+0000 7f074fe54740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5588fe253d80 con 0x5588fe250ff0 2024-04-25T01:56:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.556+0000 7f074fe54740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5588fe165d90 con 0x5588fe246920 2024-04-25T01:56:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.556+0000 7f074fe54740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5588fe253f00 con 0x5588fe246000 2024-04-25T01:56:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.556+0000 7f074fe54740 10 client.0 _ll_register_callbacks cb 0x5588fe165780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe24e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe24e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59220/0 (socket says 172.21.15.102:59220) 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745400640 1 -- 172.21.15.102:0/1870016355 learned_addr learned my addr 172.21.15.102:0/1870016355 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745e00640 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe24dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0746800640 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 0x5588fe24ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745400640 1 -- 172.21.15.102:0/1870016355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 msgr2=0x5588fe24dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745400640 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe24dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745400640 1 -- 172.21.15.102:0/1870016355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 msgr2=0x5588fe24ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745400640 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 0x5588fe24ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745400640 1 -- 172.21.15.102:0/1870016355 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588fdebe3f0 con 0x5588fe246920 2024-04-25T01:56:28.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.557+0000 7f0745e00640 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe24dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:28.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.558+0000 7f0745400640 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe24e4f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0738009850 tx=0x7f0738003900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeb4f02e9738ffee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.558+0000 7f072be00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.558+0000 7f072be00640 1 -- 172.21.15.102:0/1870016355 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738035040 con 0x5588fe246920 2024-04-25T01:56:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.558+0000 7f074fe54740 1 -- 172.21.15.102:0/1870016355 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0738004700 con 0x5588fe246920 2024-04-25T01:56:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.559+0000 7f072be00640 1 -- 172.21.15.102:0/1870016355 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0738003dc0 con 0x5588fe246920 2024-04-25T01:56:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.559+0000 7f072be00640 1 -- 172.21.15.102:0/1870016355 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0738035040 con 0x5588fe246920 2024-04-25T01:56:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.559+0000 7f074fe54740 10 client.25622 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:28.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.559+0000 7f074fe54740 1 -- 172.21.15.102:0/1870016355 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5588fde7e0f0 con 0x5588fe246920 2024-04-25T01:56:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.559+0000 7f072be00640 1 -- 172.21.15.102:0/1870016355 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0738004700 con 0x5588fe246920 2024-04-25T01:56:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.559+0000 7f072be00640 1 -- 172.21.15.102:0/1870016355 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f073804c070 con 0x5588fe246920 2024-04-25T01:56:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.560+0000 7f074fe54740 10 client.25622 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.560+0000 7f074fe54740 10 client.25622 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.560+0000 7f074fe54740 20 client.25622 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.560+0000 7f074fe54740 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5588fe2611d0 0x5588fe2635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.560+0000 7f074fe54740 4 client.25622 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.560+0000 7f074fe54740 1 -- 172.21.15.102:0/1870016355 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588fe1bdd90 con 0x5588fe2611d0 2024-04-25T01:56:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.560+0000 7f0745e00640 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5588fe2611d0 0x5588fe2635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.561+0000 7f0745e00640 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5588fe2611d0 0x5588fe2635c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=232adcac19507517 server_cookie=de263c964811fb5d in_seq=0 out_seq=0 2024-04-25T01:56:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.561+0000 7f072be00640 10 client.25622 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:28.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.562+0000 7f072be00640 1 -- 172.21.15.102:0/1870016355 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588fe1bdd90 con 0x5588fe2611d0 2024-04-25T01:56:28.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.562+0000 7f072be00640 10 client.25622 handle_command_reply: tid=0 2024-04-25T01:56:28.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.562+0000 7f074fe54740 1 client.25622 shutdown 2024-04-25T01:56:28.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.562+0000 7f074fe54740 1 -- 172.21.15.102:0/1870016355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 msgr2=0x5588fe24e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.562+0000 7f074fe54740 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe24e4f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0738009850 tx=0x7f0738003900 comp rx=0 tx=0).stop 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.562+0000 7f074fe54740 1 -- 172.21.15.102:0/1870016355 shutdown_connections 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.562+0000 7f074fe54740 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5588fe2611d0 0x5588fe2635c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.562+0000 7f074fe54740 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 0x5588fe24ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.562+0000 7f074fe54740 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe24e4f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.563+0000 7f074fe54740 1 --2- 172.21.15.102:0/1870016355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe24dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.563+0000 7f074fe54740 1 -- 172.21.15.102:0/1870016355 >> 172.21.15.102:0/1870016355 conn(0x5588fe242320 msgr2=0x5588fe248120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.563+0000 7f072be00640 0 client.25622 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.563+0000 7f074fe54740 1 -- 172.21.15.102:0/1870016355 shutdown_connections 2024-04-25T01:56:28.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.563+0000 7f074fe54740 1 -- 172.21.15.102:0/1870016355 wait complete. 2024-04-25T01:56:28.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.563+0000 7f074fe54740 20 client.25622 trim_cache size 0 max 16384 2024-04-25T01:56:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.593+0000 7f074fe54740 1 Processor -- start 2024-04-25T01:56:28.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.593+0000 7f074fe54740 1 -- start start 2024-04-25T01:56:28.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.593+0000 7f074fe54740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe25cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.593+0000 7f074fe54740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe268bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.594+0000 7f0745400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe268bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.594+0000 7f0745e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe25cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.594+0000 7f074fe54740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 0x5588fe269100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.594+0000 7f0745e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe25cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49504/0 (socket says 172.21.15.102:49504) 2024-04-25T01:56:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.594+0000 7f0745e00640 1 -- 172.21.15.102:0/1828114442 learned_addr learned my addr 172.21.15.102:0/1828114442 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.594+0000 7f074fe54740 1 -- 172.21.15.102:0/1828114442 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f07481b6340 con 0x5588fe250ff0 2024-04-25T01:56:28.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.594+0000 7f074fe54740 1 -- 172.21.15.102:0/1828114442 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f07481b3bb0 con 0x5588fe246920 2024-04-25T01:56:28.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.594+0000 7f074fe54740 1 -- 172.21.15.102:0/1828114442 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f07481b1a00 con 0x5588fe246000 2024-04-25T01:56:28.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.594+0000 7f0746800640 1 --2- 172.21.15.102:0/1828114442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 0x5588fe269100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.595+0000 7f0745400640 1 -- 172.21.15.102:0/1828114442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 msgr2=0x5588fe25cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.595+0000 7f0745400640 1 --2- 172.21.15.102:0/1828114442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe25cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.595+0000 7f0745400640 1 -- 172.21.15.102:0/1828114442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 msgr2=0x5588fe269100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.595+0000 7f0745400640 1 --2- 172.21.15.102:0/1828114442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 0x5588fe269100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.595+0000 7f0745400640 1 -- 172.21.15.102:0/1828114442 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588fe246750 con 0x5588fe246920 2024-04-25T01:56:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.595+0000 7f0745400640 1 --2- 172.21.15.102:0/1828114442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe268bc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f073802aa90 tx=0x7f07380043f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57644da596bb8294 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.596+0000 7f0736a00640 1 -- 172.21.15.102:0/1828114442 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073803cce0 con 0x5588fe246920 2024-04-25T01:56:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.596+0000 7f0736a00640 1 -- 172.21.15.102:0/1828114442 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0738005520 con 0x5588fe246920 2024-04-25T01:56:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.596+0000 7f0736a00640 1 -- 172.21.15.102:0/1828114442 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738004c60 con 0x5588fe246920 2024-04-25T01:56:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.597+0000 7f074fe54740 1 -- 172.21.15.102:0/1828114442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 msgr2=0x5588fe268bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.597+0000 7f074fe54740 1 --2- 172.21.15.102:0/1828114442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe268bc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f073802aa90 tx=0x7f07380043f0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.597+0000 7f074fe54740 1 -- 172.21.15.102:0/1828114442 shutdown_connections 2024-04-25T01:56:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.597+0000 7f074fe54740 1 --2- 172.21.15.102:0/1828114442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe250ff0 0x5588fe269100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.597+0000 7f074fe54740 1 --2- 172.21.15.102:0/1828114442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe268bc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.597+0000 7f074fe54740 1 --2- 172.21.15.102:0/1828114442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe246000 0x5588fe25cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.597+0000 7f074fe54740 1 -- 172.21.15.102:0/1828114442 >> 172.21.15.102:0/1828114442 conn(0x5588fe242320 msgr2=0x5588fe249270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.597+0000 7f074fe54740 1 -- 172.21.15.102:0/1828114442 shutdown_connections 2024-04-25T01:56:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.597+0000 7f074fe54740 1 -- 172.21.15.102:0/1828114442 wait complete. 2024-04-25T01:56:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.598+0000 7f074fe54740 1 -- start start 2024-04-25T01:56:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.598+0000 7f074fe54740 1 Processor -- start 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.599+0000 7f074fe54740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe259390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f074fe54740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe2540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f074fe54740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 0x5588fe254620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f0745e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe259390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f0745e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe259390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50310/0 (socket says 172.21.15.102:50310) 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f0745e00640 1 -- 172.21.15.102:0/956903126 learned_addr learned my addr 172.21.15.102:0/956903126 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f07481b2070 con 0x5588fe246000 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f07481b34e0 con 0x5588fe246920 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f07481b2740 con 0x5588fe250ff0 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f074fe54740 10 client.0 _ll_register_callbacks cb 0x7f0738009480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.600+0000 7f0745400640 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe2540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.601+0000 7f0746800640 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 0x5588fe254620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.601+0000 7f0745e00640 1 -- 172.21.15.102:0/956903126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 msgr2=0x5588fe254620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.601+0000 7f0745e00640 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 0x5588fe254620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.602+0000 7f0745e00640 1 -- 172.21.15.102:0/956903126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 msgr2=0x5588fe2540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.602+0000 7f0745e00640 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe2540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.602+0000 7f0745e00640 1 -- 172.21.15.102:0/956903126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588fe254c60 con 0x5588fe246000 2024-04-25T01:56:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.602+0000 7f072be00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:56:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.602+0000 7f0745e00640 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe259390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f07300128a0 tx=0x7f0730004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d620e650bb76fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.603+0000 7f072be00640 1 -- 172.21.15.102:0/956903126 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073002b800 con 0x5588fe246000 2024-04-25T01:56:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.603+0000 7f072be00640 1 -- 172.21.15.102:0/956903126 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f073001f420 con 0x5588fe246000 2024-04-25T01:56:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.603+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0738005520 con 0x5588fe246000 2024-04-25T01:56:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.604+0000 7f072be00640 1 -- 172.21.15.102:0/956903126 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f073001f640 con 0x5588fe246000 2024-04-25T01:56:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.604+0000 7f074fe54740 10 client.15495 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.604+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5588fe2c6300 con 0x5588fe246000 2024-04-25T01:56:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.605+0000 7f072be00640 1 -- 172.21.15.102:0/956903126 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0738005520 con 0x5588fe246000 2024-04-25T01:56:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.605+0000 7f072be00640 1 -- 172.21.15.102:0/956903126 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f073003d980 con 0x5588fe246000 2024-04-25T01:56:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.606+0000 7f074fe54740 10 client.15495 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.606+0000 7f074fe54740 10 client.15495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.606+0000 7f074fe54740 20 client.15495 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.607+0000 7f074fe54740 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5588fe2c69b0 0x5588fe2c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.607+0000 7f074fe54740 4 client.15495 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:56:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.607+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5588fe1b7aa0 con 0x5588fe2c69b0 2024-04-25T01:56:28.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.607+0000 7f0745400640 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5588fe2c69b0 0x5588fe2c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.608+0000 7f0745400640 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5588fe2c69b0 0x5588fe2c8dd0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a78cece68bf9272f server_cookie=a25cdebc74d54a2 in_seq=0 out_seq=0 2024-04-25T01:56:28.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.608+0000 7f072be00640 10 client.15495 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:28.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.608+0000 7f072be00640 1 -- 172.21.15.102:0/956903126 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5588fe1b7aa0 con 0x5588fe2c69b0 2024-04-25T01:56:28.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.608+0000 7f072be00640 10 client.15495 handle_command_reply: tid=0 2024-04-25T01:56:28.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.608+0000 7f074fe54740 1 client.15495 shutdown 2024-04-25T01:56:28.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.608+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 msgr2=0x5588fe259390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.608+0000 7f074fe54740 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe259390 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f07300128a0 tx=0x7f0730004730 comp rx=0 tx=0).stop 2024-04-25T01:56:28.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.609+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 shutdown_connections 2024-04-25T01:56:28.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.609+0000 7f074fe54740 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5588fe2c69b0 0x5588fe2c8dd0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.609+0000 7f074fe54740 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588fe250ff0 0x5588fe254620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.609+0000 7f074fe54740 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5588fe246920 0x5588fe2540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.609+0000 7f074fe54740 1 --2- 172.21.15.102:0/956903126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5588fe246000 0x5588fe259390 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.609+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 >> 172.21.15.102:0/956903126 conn(0x5588fe242320 msgr2=0x5588fe24c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.609+0000 7f072be00640 0 client.15495 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.609+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 shutdown_connections 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.609+0000 7f074fe54740 1 -- 172.21.15.102:0/956903126 wait complete. 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.610+0000 7f074fe54740 20 client.15495 trim_cache size 0 max 16384 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T01:56:28.612 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T01:56:28.613 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:56:28.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f072c0a3030 msgr2=0x7f072c0a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f072c0a3030 0x7f072c0a54f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0730009ce0 tx=0x7f0730005f90 comp rx=0 tx=0).stop 2024-04-25T01:56:28.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f074810e020 msgr2=0x7f07481021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f074810e020 0x7f07481021c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0740005fd0 tx=0x7f074000dea0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 shutdown_connections 2024-04-25T01:56:28.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f072c0a3030 0x7f072c0a54f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f074810e020 0x7f07481021c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0748074330 0x7f0748101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 --2- 172.21.15.102:0/89899683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748073a10 0x7f0748101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 >> 172.21.15.102:0/89899683 conn(0x7f0748078b90 msgr2=0x7f07480fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:28.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 shutdown_connections 2024-04-25T01:56:28.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.611+0000 7f074d000640 1 -- 172.21.15.102:0/89899683 wait complete. 2024-04-25T01:56:28.648 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:56:28.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:56:28.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.894+0000 7f3858400640 1 -- 172.21.15.102:0/1853292797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850114890 msgr2=0x7f3850116c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.894+0000 7f3858400640 1 --2- 172.21.15.102:0/1853292797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850114890 0x7f3850116c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3848009cd0 tx=0x7f384802a910 comp rx=0 tx=0).stop 2024-04-25T01:56:28.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.894+0000 7f3858400640 1 -- 172.21.15.102:0/1853292797 shutdown_connections 2024-04-25T01:56:28.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.894+0000 7f3858400640 1 --2- 172.21.15.102:0/1853292797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850114890 0x7f3850116c80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.894+0000 7f3858400640 1 --2- 172.21.15.102:0/1853292797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850109290 0x7f3850114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.894+0000 7f3858400640 1 --2- 172.21.15.102:0/1853292797 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3850108970 0x7f3850108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.894+0000 7f3858400640 1 -- 172.21.15.102:0/1853292797 >> 172.21.15.102:0/1853292797 conn(0x7f385006c860 msgr2=0x7f385006cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:28.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.894+0000 7f3858400640 1 -- 172.21.15.102:0/1853292797 shutdown_connections 2024-04-25T01:56:28.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.894+0000 7f3858400640 1 -- 172.21.15.102:0/1853292797 wait complete. 2024-04-25T01:56:28.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.895+0000 7f3858400640 1 Processor -- start 2024-04-25T01:56:28.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.896+0000 7f3858400640 1 -- start start 2024-04-25T01:56:28.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.896+0000 7f3858400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3850108970 0x7f38501a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.896+0000 7f3858400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850109290 0x7f38501a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.896+0000 7f3858400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850114890 0x7f38501a2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.896+0000 7f3858400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f385011a390 con 0x7f3850114890 2024-04-25T01:56:28.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.896+0000 7f3858400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f385011a210 con 0x7f3850108970 2024-04-25T01:56:28.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.896+0000 7f3858400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f385011a510 con 0x7f3850109290 2024-04-25T01:56:28.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.897+0000 7f3855a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3850108970 0x7f38501a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.897+0000 7f3855000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850109290 0x7f38501a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.897+0000 7f3855000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850109290 0x7f38501a8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49544/0 (socket says 172.21.15.102:49544) 2024-04-25T01:56:28.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.897+0000 7f3855000640 1 -- 172.21.15.102:0/3575215553 learned_addr learned my addr 172.21.15.102:0/3575215553 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:28.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.897+0000 7f3856400640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850114890 0x7f38501a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.898+0000 7f3855a00640 1 -- 172.21.15.102:0/3575215553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850109290 msgr2=0x7f38501a8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.898+0000 7f3855a00640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850109290 0x7f38501a8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.898+0000 7f3855a00640 1 -- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850114890 msgr2=0x7f38501a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:28.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.898+0000 7f3855a00640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850114890 0x7f38501a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:28.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.898+0000 7f3855a00640 1 -- 172.21.15.102:0/3575215553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38501a3000 con 0x7f3850108970 2024-04-25T01:56:28.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.898+0000 7f3855000640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850109290 0x7f38501a8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:28.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.898+0000 7f3856400640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850114890 0x7f38501a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:28.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.899+0000 7f3855a00640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3850108970 0x7f38501a86c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3844009fd0 tx=0x7f384400b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.900+0000 7f384e000640 1 -- 172.21.15.102:0/3575215553 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3844015020 con 0x7f3850108970 2024-04-25T01:56:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.900+0000 7f384e000640 1 -- 172.21.15.102:0/3575215553 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3844011d10 con 0x7f3850108970 2024-04-25T01:56:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.900+0000 7f384e000640 1 -- 172.21.15.102:0/3575215553 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384400f2e0 con 0x7f3850108970 2024-04-25T01:56:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.900+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38501a32f0 con 0x7f3850108970 2024-04-25T01:56:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.900+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38501af510 con 0x7f3850108970 2024-04-25T01:56:28.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.902+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f385006d460 con 0x7f3850108970 2024-04-25T01:56:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.909+0000 7f384e000640 1 -- 172.21.15.102:0/3575215553 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f3844018020 con 0x7f3850108970 2024-04-25T01:56:28.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.913+0000 7f384e000640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38380a2f70 0x7f38380a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:28.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.913+0000 7f384e000640 1 -- 172.21.15.102:0/3575215553 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f38440cef30 con 0x7f3850108970 2024-04-25T01:56:28.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.913+0000 7f3855000640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38380a2f70 0x7f38380a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:28.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.914+0000 7f3855000640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38380a2f70 0x7f38380a5430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f385010c1d0 tx=0x7f383c00a680 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:28.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:28.916+0000 7f384e000640 1 -- 172.21.15.102:0/3575215553 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f384401f020 con 0x7f3850108970 2024-04-25T01:56:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.340+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3850068560 con 0x7f3850108970 2024-04-25T01:56:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.341+0000 7f384e000640 1 -- 172.21.15.102:0/3575215553 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11901 (secure 0 0 0) 0x7f384409bac0 con 0x7f3850108970 2024-04-25T01:56:29.346 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:56:29.346 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:56:02.805319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24541,24587,24533]},"id":1}]} 2024-04-25T01:56:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-25T01:56:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38380a2f70 msgr2=0x7f38380a5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38380a2f70 0x7f38380a5430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f385010c1d0 tx=0x7f383c00a680 comp rx=0 tx=0).stop 2024-04-25T01:56:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3850108970 msgr2=0x7f38501a86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3850108970 0x7f38501a86c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3844009fd0 tx=0x7f384400b870 comp rx=0 tx=0).stop 2024-04-25T01:56:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 shutdown_connections 2024-04-25T01:56:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38380a2f70 0x7f38380a5430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3850114890 0x7f38501a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850109290 0x7f38501a8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 --2- 172.21.15.102:0/3575215553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3850108970 0x7f38501a86c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 >> 172.21.15.102:0/3575215553 conn(0x7f385006c860 msgr2=0x7f3850070240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 shutdown_connections 2024-04-25T01:56:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.345+0000 7f3858400640 1 -- 172.21.15.102:0/3575215553 wait complete. 2024-04-25T01:56:29.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:56:29.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.639+0000 7f99c8c00640 1 -- 172.21.15.102:0/1654122197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c4105f00 msgr2=0x7f99c41062e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.639+0000 7f99c8c00640 1 --2- 172.21.15.102:0/1654122197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c4105f00 0x7f99c41062e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009900 tx=0x7f99b402a400 comp rx=0 tx=0).stop 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.639+0000 7f99c8c00640 1 -- 172.21.15.102:0/1654122197 shutdown_connections 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.639+0000 7f99c8c00640 1 --2- 172.21.15.102:0/1654122197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c4100ab0 0x7f99c410cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.639+0000 7f99c8c00640 1 --2- 172.21.15.102:0/1654122197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c4100110 0x7f99c4100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.639+0000 7f99c8c00640 1 --2- 172.21.15.102:0/1654122197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c4105f00 0x7f99c41062e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.639+0000 7f99c8c00640 1 -- 172.21.15.102:0/1654122197 >> 172.21.15.102:0/1654122197 conn(0x7f99c40767a0 msgr2=0x7f99c40fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.639+0000 7f99c8c00640 1 -- 172.21.15.102:0/1654122197 shutdown_connections 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.639+0000 7f99c8c00640 1 -- 172.21.15.102:0/1654122197 wait complete. 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.640+0000 7f99c8c00640 1 Processor -- start 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.641+0000 7f99c8c00640 1 -- start start 2024-04-25T01:56:29.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.641+0000 7f99c8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c4100110 0x7f99c4106cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.641+0000 7f99c8c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c4100ab0 0x7f99c41071f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.641+0000 7f99c8c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c4105f00 0x7f99c4107730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.641+0000 7f99c8c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99c4111c00 con 0x7f99c4100110 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.641+0000 7f99c8c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99c4111a80 con 0x7f99c4100ab0 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.641+0000 7f99c8c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99c4111d80 con 0x7f99c4105f00 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.641+0000 7f99c2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c4105f00 0x7f99c4107730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.642+0000 7f99c2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c4105f00 0x7f99c4107730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49572/0 (socket says 172.21.15.102:49572) 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.642+0000 7f99c2800640 1 -- 172.21.15.102:0/2272226693 learned_addr learned my addr 172.21.15.102:0/2272226693 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.642+0000 7f99c1400640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c4100ab0 0x7f99c41071f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.642+0000 7f99c1e00640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c4100110 0x7f99c4106cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.642+0000 7f99c1400640 1 -- 172.21.15.102:0/2272226693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c4105f00 msgr2=0x7f99c4107730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.642+0000 7f99c1400640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c4105f00 0x7f99c4107730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.642+0000 7f99c1400640 1 -- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c4100110 msgr2=0x7f99c4106cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.643+0000 7f99c1400640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c4100110 0x7f99c4106cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.643+0000 7f99c1400640 1 -- 172.21.15.102:0/2272226693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c41aa920 con 0x7f99c4100ab0 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.643+0000 7f99c1e00640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c4100110 0x7f99c4106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.643+0000 7f99c1400640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c4100ab0 0x7f99c41071f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f99b000ea20 tx=0x7f99b000eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.643+0000 7f99baa00640 1 -- 172.21.15.102:0/2272226693 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b000ccc0 con 0x7f99c4100ab0 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.643+0000 7f99baa00640 1 -- 172.21.15.102:0/2272226693 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b000ce60 con 0x7f99c4100ab0 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.643+0000 7f99baa00640 1 -- 172.21.15.102:0/2272226693 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b0004a00 con 0x7f99c4100ab0 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.643+0000 7f99c8c00640 1 -- 172.21.15.102:0/2272226693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c41aac10 con 0x7f99c4100ab0 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.644+0000 7f99c8c00640 1 -- 172.21.15.102:0/2272226693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c41ab110 con 0x7f99c4100ab0 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.649+0000 7f99cb97d740 1 Processor -- start 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.650+0000 7f99baa00640 1 -- 172.21.15.102:0/2272226693 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f99b0005140 con 0x7f99c4100ab0 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.650+0000 7f99cb97d740 1 -- start start 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd15c0 0x556554bd19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554bdc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bdc5b0 0x556554bde9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99cb97d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x556554b364e0 con 0x556554bd15c0 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99cb97d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556554adea70 con 0x556554bd1ee0 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99cb97d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556554b324a0 con 0x556554bdc5b0 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99c2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd15c0 0x556554bd19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99c1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bdc5b0 0x556554bde9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99c1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bdc5b0 0x556554bde9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49576/0 (socket says 172.21.15.102:49576) 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99c1400640 1 -- 172.21.15.102:0/3852730931 learned_addr learned my addr 172.21.15.102:0/3852730931 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.651+0000 7f99c1e00640 1 --2- 172.21.15.102:0/3852730931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554bdc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.652+0000 7f99c1400640 1 -- 172.21.15.102:0/3852730931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 msgr2=0x556554bdc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.652+0000 7f99c1400640 1 --2- 172.21.15.102:0/3852730931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554bdc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.652+0000 7f99c1400640 1 -- 172.21.15.102:0/3852730931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd15c0 msgr2=0x556554bd19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.652+0000 7f99c1400640 1 --2- 172.21.15.102:0/3852730931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd15c0 0x556554bd19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.652+0000 7f99c1400640 1 -- 172.21.15.102:0/3852730931 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55655484a3f0 con 0x556554bdc5b0 2024-04-25T01:56:29.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.652+0000 7f99c2800640 1 --2- 172.21.15.102:0/3852730931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd15c0 0x556554bd19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.652+0000 7f99c1400640 1 --2- 172.21.15.102:0/3852730931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bdc5b0 0x556554bde9a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f99b00ccbd0 tx=0x7f99b00d5730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0f27a8d5ad0ea29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.654+0000 7f99baa00640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a80a2ff0 0x7f99a80a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.656+0000 7f99af000640 1 -- 172.21.15.102:0/3852730931 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b00e3070 con 0x556554bdc5b0 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.656+0000 7f99af000640 1 -- 172.21.15.102:0/3852730931 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b00cfd60 con 0x556554bdc5b0 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.656+0000 7f99af000640 1 -- 172.21.15.102:0/3852730931 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b00db9d0 con 0x556554bdc5b0 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.657+0000 7f99c2800640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a80a2ff0 0x7f99a80a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.657+0000 7f99baa00640 1 -- 172.21.15.102:0/2272226693 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f99b00cf5a0 con 0x7f99c4100ab0 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.657+0000 7f99cb97d740 1 -- 172.21.15.102:0/3852730931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bdc5b0 msgr2=0x556554bde9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.657+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3852730931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bdc5b0 0x556554bde9a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f99b00ccbd0 tx=0x7f99b00d5730 comp rx=0 tx=0).stop 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.658+0000 7f99cb97d740 1 -- 172.21.15.102:0/3852730931 shutdown_connections 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.658+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3852730931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bdc5b0 0x556554bde9a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.658+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3852730931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554bdc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.658+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3852730931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd15c0 0x556554bd19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.658+0000 7f99cb97d740 1 -- 172.21.15.102:0/3852730931 >> 172.21.15.102:0/3852730931 conn(0x556554bcd8e0 msgr2=0x556554bcdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.658+0000 7f99c2800640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a80a2ff0 0x7f99a80a54b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f99bc011580 tx=0x7f99bc00b040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.658+0000 7f99cb97d740 1 -- 172.21.15.102:0/3852730931 shutdown_connections 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.658+0000 7f99cb97d740 1 -- 172.21.15.102:0/3852730931 wait complete. 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.659+0000 7f99cb97d740 1 -- start start 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.659+0000 7f99cb97d740 1 Processor -- start 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99cb97d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x556554bf0400 con 0x556554bd1ee0 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99cb97d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556554af1ec0 con 0x556554bdc5b0 2024-04-25T01:56:29.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99cb97d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556554bf0580 con 0x556554bd15c0 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99c2800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99c1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99c1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49592/0 (socket says 172.21.15.102:49592) 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99c1e00640 1 -- 172.21.15.102:0/2442476577 learned_addr learned my addr 172.21.15.102:0/2442476577 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.660+0000 7f99cb97d740 10 client.0 _ll_register_callbacks cb 0x556554b2e3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.661+0000 7f99c1400640 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.661+0000 7f99c2800640 1 -- 172.21.15.102:0/2442476577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 msgr2=0x556554bd9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.661+0000 7f99c2800640 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.661+0000 7f99c2800640 1 -- 172.21.15.102:0/2442476577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 msgr2=0x556554bd9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.661+0000 7f99c2800640 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.662+0000 7f99c2800640 1 -- 172.21.15.102:0/2442476577 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565549147d0 con 0x556554bdc5b0 2024-04-25T01:56:29.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.662+0000 7f99c2800640 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd9e80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f99bc017e10 tx=0x7f99bc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7f3d5db32ee91f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.662+0000 7f99a3400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.663+0000 7f99a3400640 1 -- 172.21.15.102:0/2442476577 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc01b070 con 0x556554bdc5b0 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.663+0000 7f99a3400640 1 -- 172.21.15.102:0/2442476577 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f99bc022670 con 0x556554bdc5b0 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.663+0000 7f99cb97d740 1 -- 172.21.15.102:0/2442476577 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b00cfd60 con 0x556554bdc5b0 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.663+0000 7f99a3400640 1 -- 172.21.15.102:0/2442476577 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f99bc017e40 con 0x556554bdc5b0 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.663+0000 7f99cb97d740 10 client.25640 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.663+0000 7f99cb97d740 1 -- 172.21.15.102:0/2442476577 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556554bfc680 con 0x556554bdc5b0 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.664+0000 7f99a3400640 1 -- 172.21.15.102:0/2442476577 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b00cfd60 con 0x556554bdc5b0 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.666+0000 7f99a3400640 1 -- 172.21.15.102:0/2442476577 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f99bc02bab0 con 0x556554bdc5b0 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.666+0000 7f99cb97d740 10 client.25640 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.666+0000 7f99cb97d740 10 client.25640 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.666+0000 7f99cb97d740 20 client.25640 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.666+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x556554bfdbe0 0x556554c00000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.666+0000 7f99cb97d740 4 client.25640 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:29.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.666+0000 7f99cb97d740 1 -- 172.21.15.102:0/2442476577 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556554b43a20 con 0x556554bfdbe0 2024-04-25T01:56:29.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.666+0000 7f99c1e00640 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x556554bfdbe0 0x556554c00000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.667+0000 7f99c1e00640 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x556554bfdbe0 0x556554c00000 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6f26f3af0ff6b7d8 server_cookie=b370f9d7ae84440b in_seq=0 out_seq=0 2024-04-25T01:56:29.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.667+0000 7f99a3400640 10 client.25640 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:56:29.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.668+0000 7f99a3400640 1 -- 172.21.15.102:0/2442476577 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556554b43a20 con 0x556554bfdbe0 2024-04-25T01:56:29.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.668+0000 7f99a3400640 10 client.25640 handle_command_reply: tid=0 2024-04-25T01:56:29.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.668+0000 7f99cb97d740 1 client.25640 shutdown 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.668+0000 7f99cb97d740 1 -- 172.21.15.102:0/2442476577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 msgr2=0x556554bd9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.668+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd9e80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f99bc017e10 tx=0x7f99bc004a40 comp rx=0 tx=0).stop 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.669+0000 7f99cb97d740 1 -- 172.21.15.102:0/2442476577 shutdown_connections 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.669+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x556554bfdbe0 0x556554c00000 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.669+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd9e80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.669+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.669+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2442476577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.669+0000 7f99cb97d740 1 -- 172.21.15.102:0/2442476577 >> 172.21.15.102:0/2442476577 conn(0x556554bcd8e0 msgr2=0x556554bccb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.669+0000 7f99a3400640 0 client.25640 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.669+0000 7f99cb97d740 1 -- 172.21.15.102:0/2442476577 shutdown_connections 2024-04-25T01:56:29.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.669+0000 7f99cb97d740 1 -- 172.21.15.102:0/2442476577 wait complete. 2024-04-25T01:56:29.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.670+0000 7f99cb97d740 20 client.25640 trim_cache size 0 max 16384 2024-04-25T01:56:29.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:29 smithi102 ceph-mon[27796]: pgmap v39: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 19 MiB/s wr, 1.23k op/s 2024-04-25T01:56:29.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.693+0000 7f99cb97d740 1 Processor -- start 2024-04-25T01:56:29.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.693+0000 7f99cb97d740 1 -- start start 2024-04-25T01:56:29.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.693+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.693+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554c058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.693+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bdc5b0 0x556554c05e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.693+0000 7f99cb97d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99c41adb10 con 0x556554bdc5b0 2024-04-25T01:56:29.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.693+0000 7f99cb97d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99c41ab3a0 con 0x556554bd1ee0 2024-04-25T01:56:29.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.693+0000 7f99cb97d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99c40689f0 con 0x556554bd15c0 2024-04-25T01:56:29.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.694+0000 7f99c1400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554c058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.694+0000 7f99c1400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554c058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59330/0 (socket says 172.21.15.102:59330) 2024-04-25T01:56:29.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.694+0000 7f99c1400640 1 -- 172.21.15.102:0/2972707822 learned_addr learned my addr 172.21.15.102:0/2972707822 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:29.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.694+0000 7f99c1e00640 1 --2- 172.21.15.102:0/2972707822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.694+0000 7f99c1400640 1 -- 172.21.15.102:0/2972707822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 msgr2=0x556554bd9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:29.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.694+0000 7f99c1400640 1 --2- 172.21.15.102:0/2972707822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.694+0000 7f99c1400640 1 -- 172.21.15.102:0/2972707822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bdc5b0 msgr2=0x556554c05e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:29.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.694+0000 7f99c1400640 1 --2- 172.21.15.102:0/2972707822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bdc5b0 0x556554c05e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.694+0000 7f99c1400640 1 -- 172.21.15.102:0/2972707822 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556554bd4240 con 0x556554bd1ee0 2024-04-25T01:56:29.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.695+0000 7f99c1400640 1 --2- 172.21.15.102:0/2972707822 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554c058d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f99b0098210 tx=0x7f99b00d5f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd4a5947e33109e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:29.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.695+0000 7f99ae600640 1 -- 172.21.15.102:0/2972707822 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b009b610 con 0x556554bd1ee0 2024-04-25T01:56:29.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.695+0000 7f99ae600640 1 -- 172.21.15.102:0/2972707822 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b009bf40 con 0x556554bd1ee0 2024-04-25T01:56:29.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.695+0000 7f99ae600640 1 -- 172.21.15.102:0/2972707822 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b009b1a0 con 0x556554bd1ee0 2024-04-25T01:56:29.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.696+0000 7f99cb97d740 1 -- 172.21.15.102:0/2972707822 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 msgr2=0x556554c058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.696+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2972707822 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554c058d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f99b0098210 tx=0x7f99b00d5f30 comp rx=0 tx=0).stop 2024-04-25T01:56:29.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.696+0000 7f99cb97d740 1 -- 172.21.15.102:0/2972707822 shutdown_connections 2024-04-25T01:56:29.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.696+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2972707822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bdc5b0 0x556554c05e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.696+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2972707822 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bd1ee0 0x556554c058d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.696+0000 7f99cb97d740 1 --2- 172.21.15.102:0/2972707822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.696+0000 7f99cb97d740 1 -- 172.21.15.102:0/2972707822 >> 172.21.15.102:0/2972707822 conn(0x556554bcd8e0 msgr2=0x556554bcc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:29.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.697+0000 7f99cb97d740 1 -- 172.21.15.102:0/2972707822 shutdown_connections 2024-04-25T01:56:29.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.697+0000 7f99cb97d740 1 -- 172.21.15.102:0/2972707822 wait complete. 2024-04-25T01:56:29.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.697+0000 7f99cb97d740 1 -- start start 2024-04-25T01:56:29.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.698+0000 7f99cb97d740 1 Processor -- start 2024-04-25T01:56:29.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.698+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.698+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99cb97d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99cb97d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99c41a0c90 con 0x556554bd1ee0 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99cb97d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99c41a2100 con 0x556554bdc5b0 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99cb97d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99c41a1360 con 0x556554bd15c0 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99cb97d740 10 client.0 _ll_register_callbacks cb 0x7f99bc009c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99c1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99c1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49620/0 (socket says 172.21.15.102:49620) 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99c1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99c1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50386/0 (socket says 172.21.15.102:50386) 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99c1400640 1 -- 172.21.15.102:0/3460850721 learned_addr learned my addr 172.21.15.102:0/3460850721 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:29.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.699+0000 7f99c2800640 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.700+0000 7f99c1400640 1 -- 172.21.15.102:0/3460850721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 msgr2=0x556554bd6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.700+0000 7f99c1400640 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.700+0000 7f99c1400640 1 -- 172.21.15.102:0/3460850721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 msgr2=0x556554bd6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.700+0000 7f99c1400640 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.700+0000 7f99c1400640 1 -- 172.21.15.102:0/3460850721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556554c63310 con 0x556554bd1ee0 2024-04-25T01:56:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.700+0000 7f99c2800640 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.700+0000 7f99c1400640 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd6850 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f99b00a3f80 tx=0x7f99b009b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e2980b89508a18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:29.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.700+0000 7f99a3400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.701+0000 7f99a3400640 1 -- 172.21.15.102:0/3460850721 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b009b380 con 0x556554bd1ee0 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.701+0000 7f99a3400640 1 -- 172.21.15.102:0/3460850721 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f99b00cfb60 con 0x556554bd1ee0 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.701+0000 7f99cb97d740 1 -- 172.21.15.102:0/3460850721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b009bf40 con 0x556554bd1ee0 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.701+0000 7f99a3400640 1 -- 172.21.15.102:0/3460850721 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f99b009b1a0 con 0x556554bd1ee0 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.701+0000 7f99cb97d740 10 client.15510 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.701+0000 7f99cb97d740 1 -- 172.21.15.102:0/3460850721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556554c645d0 con 0x556554bd1ee0 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.702+0000 7f99a3400640 1 -- 172.21.15.102:0/3460850721 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b009bf40 con 0x556554bd1ee0 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.702+0000 7f99a3400640 1 -- 172.21.15.102:0/3460850721 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f99b009baa0 con 0x556554bd1ee0 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.702+0000 7f99cb97d740 10 client.15510 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.702+0000 7f99cb97d740 10 client.15510 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.702+0000 7f99cb97d740 20 client.15510 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:29.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.702+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x556554c64800 0x556554c66bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.702+0000 7f99cb97d740 4 client.15510 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.702+0000 7f99c1e00640 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x556554c64800 0x556554c66bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.702+0000 7f99cb97d740 1 -- 172.21.15.102:0/3460850721 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556554b3dd70 con 0x556554c64800 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99c1e00640 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x556554c64800 0x556554c66bf0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a7302c6bfac685f server_cookie=f621fe430af396f9 in_seq=0 out_seq=0 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99a3400640 10 client.15510 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99a3400640 1 -- 172.21.15.102:0/3460850721 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556554b3dd70 con 0x556554c64800 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99a3400640 10 client.15510 handle_command_reply: tid=0 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99cb97d740 1 client.15510 shutdown 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99cb97d740 1 -- 172.21.15.102:0/3460850721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 msgr2=0x556554bd6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd6850 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f99b00a3f80 tx=0x7f99b009b840 comp rx=0 tx=0).stop 2024-04-25T01:56:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99cb97d740 1 -- 172.21.15.102:0/3460850721 shutdown_connections 2024-04-25T01:56:29.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x556554c64800 0x556554c66bf0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556554bdc5b0 0x556554bd6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556554bd1ee0 0x556554bd6850 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99cb97d740 1 --2- 172.21.15.102:0/3460850721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556554bd15c0 0x556554bd6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.703+0000 7f99cb97d740 1 -- 172.21.15.102:0/3460850721 >> 172.21.15.102:0/3460850721 conn(0x556554bcd8e0 msgr2=0x556554bdb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:29.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.704+0000 7f99a3400640 0 client.15510 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:56:29.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.704+0000 7f99cb97d740 1 -- 172.21.15.102:0/3460850721 shutdown_connections 2024-04-25T01:56:29.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.704+0000 7f99cb97d740 1 -- 172.21.15.102:0/3460850721 wait complete. 2024-04-25T01:56:29.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.704+0000 7f99cb97d740 20 client.15510 trim_cache size 0 max 16384 2024-04-25T01:56:29.708 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 -- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a80a2ff0 msgr2=0x7f99a80a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a80a2ff0 0x7f99a80a54b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f99bc011580 tx=0x7f99bc00b040 comp rx=0 tx=0).stop 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 -- 172.21.15.102:0/2272226693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c4100ab0 msgr2=0x7f99c41071f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c4100ab0 0x7f99c41071f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f99b000ea20 tx=0x7f99b000eee0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 -- 172.21.15.102:0/2272226693 shutdown_connections 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a80a2ff0 0x7f99a80a54b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c4105f00 0x7f99c4107730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c4100ab0 0x7f99c41071f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 --2- 172.21.15.102:0/2272226693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c4100110 0x7f99c4106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:29.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.705+0000 7f99c8c00640 1 -- 172.21.15.102:0/2272226693 >> 172.21.15.102:0/2272226693 conn(0x7f99c40767a0 msgr2=0x7f99c40fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:29.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.706+0000 7f99c8c00640 1 -- 172.21.15.102:0/2272226693 shutdown_connections 2024-04-25T01:56:29.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:29.706+0000 7f99c8c00640 1 -- 172.21.15.102:0/2272226693 wait complete. 2024-04-25T01:56:29.738 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:29 smithi110 ceph-mon[32848]: pgmap v39: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 19 MiB/s wr, 1.23k op/s 2024-04-25T01:56:29.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:56:29.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:29 smithi193 ceph-mon[32981]: pgmap v39: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 19 MiB/s wr, 1.23k op/s 2024-04-25T01:56:30.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.034+0000 7f7f19a00640 1 -- 172.21.15.102:0/1463327949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f141032e0 msgr2=0x7f7f1410b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.034+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1463327949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f141032e0 0x7f7f1410b7c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7f00009900 tx=0x7f7f0002a510 comp rx=0 tx=0).stop 2024-04-25T01:56:30.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.034+0000 7f7f19a00640 1 -- 172.21.15.102:0/1463327949 shutdown_connections 2024-04-25T01:56:30.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.034+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1463327949 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1410bd00 0x7f7f1410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.034+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1463327949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f141032e0 0x7f7f1410b7c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.034+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1463327949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f141029c0 0x7f7f14102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.034+0000 7f7f19a00640 1 -- 172.21.15.102:0/1463327949 >> 172.21.15.102:0/1463327949 conn(0x7f7f14077740 msgr2=0x7f7f14077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.034+0000 7f7f19a00640 1 -- 172.21.15.102:0/1463327949 shutdown_connections 2024-04-25T01:56:30.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.036+0000 7f7f19a00640 1 -- 172.21.15.102:0/1463327949 wait complete. 2024-04-25T01:56:30.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.037+0000 7f7f19a00640 1 Processor -- start 2024-04-25T01:56:30.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.037+0000 7f7f19a00640 1 -- start start 2024-04-25T01:56:30.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f19a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f141029c0 0x7f7f1419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f19a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f141032e0 0x7f7f1419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f19a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f1410bd00 0x7f7f1419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f19a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14111b50 con 0x7f7f141032e0 2024-04-25T01:56:30.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f19a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141119d0 con 0x7f7f141029c0 2024-04-25T01:56:30.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f19a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14111cd0 con 0x7f7f1410bd00 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f13400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f1410bd00 0x7f7f1419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f12000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f141032e0 0x7f7f1419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f13400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f1410bd00 0x7f7f1419ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49642/0 (socket says 172.21.15.102:49642) 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.038+0000 7f7f13400640 1 -- 172.21.15.102:0/1295180759 learned_addr learned my addr 172.21.15.102:0/1295180759 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.039+0000 7f7f12a00640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f141029c0 0x7f7f1419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.039+0000 7f7f13400640 1 -- 172.21.15.102:0/1295180759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f141029c0 msgr2=0x7f7f1419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.039+0000 7f7f13400640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f141029c0 0x7f7f1419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.039+0000 7f7f13400640 1 -- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f141032e0 msgr2=0x7f7f1419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.039+0000 7f7f13400640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f141032e0 0x7f7f1419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.039+0000 7f7f13400640 1 -- 172.21.15.102:0/1295180759 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f1419f1a0 con 0x7f7f1410bd00 2024-04-25T01:56:30.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.040+0000 7f7f12000640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f141032e0 0x7f7f1419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:30.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.040+0000 7f7f12a00640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f141029c0 0x7f7f1419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:30.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.040+0000 7f7f13400640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f1410bd00 0x7f7f1419ea70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00cce0 tx=0x7f7f0c00a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.040+0000 7f7f0b400640 1 -- 172.21.15.102:0/1295180759 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c022070 con 0x7f7f1410bd00 2024-04-25T01:56:30.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.040+0000 7f7f19a00640 1 -- 172.21.15.102:0/1295180759 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f1419f490 con 0x7f7f1410bd00 2024-04-25T01:56:30.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.041+0000 7f7f0b400640 1 -- 172.21.15.102:0/1295180759 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f0c015d50 con 0x7f7f1410bd00 2024-04-25T01:56:30.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.041+0000 7f7f0b400640 1 -- 172.21.15.102:0/1295180759 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c01b5a0 con 0x7f7f1410bd00 2024-04-25T01:56:30.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.041+0000 7f7f19a00640 1 -- 172.21.15.102:0/1295180759 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f14103760 con 0x7f7f1410bd00 2024-04-25T01:56:30.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.050+0000 7f7f1c7f3740 1 Processor -- start 2024-04-25T01:56:30.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.052+0000 7f7f0b400640 1 -- 172.21.15.102:0/1295180759 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f7f0c025070 con 0x7f7f1410bd00 2024-04-25T01:56:30.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.053+0000 7f7f1c7f3740 1 -- start start 2024-04-25T01:56:30.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f7a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f85070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f855b0 0x5564c9f879a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5564c9edf4e0 con 0x5564c9f7a5c0 2024-04-25T01:56:30.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5564c9e87a70 con 0x5564c9f855b0 2024-04-25T01:56:30.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5564c9edb4a0 con 0x5564c9f7aee0 2024-04-25T01:56:30.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f12a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f85070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f12a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f85070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49658/0 (socket says 172.21.15.102:49658) 2024-04-25T01:56:30.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f12a00640 1 -- 172.21.15.102:0/3093317549 learned_addr learned my addr 172.21.15.102:0/3093317549 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.054+0000 7f7f12000640 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f7a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.055+0000 7f7f12000640 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f855b0 0x5564c9f879a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.055+0000 7f7f12a00640 1 -- 172.21.15.102:0/3093317549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f855b0 msgr2=0x5564c9f879a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.055+0000 7f7f12a00640 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f855b0 0x5564c9f879a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.055+0000 7f7f12a00640 1 -- 172.21.15.102:0/3093317549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 msgr2=0x5564c9f7a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.056+0000 7f7f12a00640 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f7a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.056+0000 7f7f12a00640 1 -- 172.21.15.102:0/3093317549 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564c9bf33f0 con 0x5564c9f7aee0 2024-04-25T01:56:30.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.056+0000 7f7f12000640 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f7a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:30.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.057+0000 7f7f12a00640 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f85070 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009d50 tx=0x7f7f040117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83b290a471a44faf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.058+0000 7f7effe00640 1 -- 172.21.15.102:0/3093317549 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04012650 con 0x5564c9f7aee0 2024-04-25T01:56:30.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.058+0000 7f7effe00640 1 -- 172.21.15.102:0/3093317549 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f04009e40 con 0x5564c9f7aee0 2024-04-25T01:56:30.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.058+0000 7f7effe00640 1 -- 172.21.15.102:0/3093317549 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0400fc60 con 0x5564c9f7aee0 2024-04-25T01:56:30.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.059+0000 7f7f0b400640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7ef80a2ef0 0x7f7ef80a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.059+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3093317549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 msgr2=0x5564c9f85070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.059+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f85070 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009d50 tx=0x7f7f040117b0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.059+0000 7f7f0b400640 1 -- 172.21.15.102:0/1295180759 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7f0c0d79c0 con 0x7f7f1410bd00 2024-04-25T01:56:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.060+0000 7f7f12000640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7ef80a2ef0 0x7f7ef80a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.060+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3093317549 shutdown_connections 2024-04-25T01:56:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.060+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f855b0 0x5564c9f879a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.060+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f85070 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.060+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3093317549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f7a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.060+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3093317549 >> 172.21.15.102:0/3093317549 conn(0x5564c9f768e0 msgr2=0x5564c9f76cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.061+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3093317549 shutdown_connections 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.061+0000 7f7f12000640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7ef80a2ef0 0x7f7ef80a53b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5564c9f77a70 tx=0x7f7f00000bc0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.061+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3093317549 wait complete. 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.061+0000 7f7f1c7f3740 1 -- start start 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.061+0000 7f7f1c7f3740 1 Processor -- start 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f82520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7aee0 0x5564c9f82a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 0x5564c9f82fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5564c9f883a0 con 0x5564c9f7a5c0 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5564c9e9aec0 con 0x5564c9f7aee0 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5564c9f88520 con 0x5564c9f855b0 2024-04-25T01:56:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f1c7f3740 10 client.0 _ll_register_callbacks cb 0x5564c9ed73f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f12000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7aee0 0x5564c9f82a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f12000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7aee0 0x5564c9f82a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59394/0 (socket says 172.21.15.102:59394) 2024-04-25T01:56:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.062+0000 7f7f12000640 1 -- 172.21.15.102:0/3667133837 learned_addr learned my addr 172.21.15.102:0/3667133837 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.063+0000 7f7f13400640 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 0x5564c9f82fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.063+0000 7f7f12a00640 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f82520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.064+0000 7f7f13400640 1 -- 172.21.15.102:0/3667133837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7aee0 msgr2=0x5564c9f82a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.064+0000 7f7f13400640 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7aee0 0x5564c9f82a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.064+0000 7f7f13400640 1 -- 172.21.15.102:0/3667133837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 msgr2=0x5564c9f82520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.064+0000 7f7f13400640 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f82520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.064+0000 7f7f13400640 1 -- 172.21.15.102:0/3667133837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564c9cbd7d0 con 0x5564c9f855b0 2024-04-25T01:56:30.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.064+0000 7f7f12000640 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7aee0 0x5564c9f82a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:30.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.064+0000 7f7f12a00640 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f82520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:30.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.064+0000 7f7f13400640 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 0x5564c9f82fa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0d50a0 tx=0x7f7f0c0a37f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe8d71fcf63b4160 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.064+0000 7f7efcc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:56:30.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.065+0000 7f7efcc00640 1 -- 172.21.15.102:0/3667133837 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c0ac7b0 con 0x5564c9f855b0 2024-04-25T01:56:30.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.065+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3667133837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5564c9eeca20 con 0x5564c9f855b0 2024-04-25T01:56:30.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.065+0000 7f7efcc00640 1 -- 172.21.15.102:0/3667133837 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7f0c0d52a0 con 0x5564c9f855b0 2024-04-25T01:56:30.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.065+0000 7f7efcc00640 1 -- 172.21.15.102:0/3667133837 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7f0c0aca40 con 0x5564c9f855b0 2024-04-25T01:56:30.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.065+0000 7f7f1c7f3740 10 client.25642 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:30.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.065+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3667133837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5564c9f94700 con 0x5564c9f855b0 2024-04-25T01:56:30.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.065+0000 7f7efcc00640 1 -- 172.21.15.102:0/3667133837 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5564c9eeca20 con 0x5564c9f855b0 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.066+0000 7f7efcc00640 1 -- 172.21.15.102:0/3667133837 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7f0c0b9030 con 0x5564c9f855b0 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.066+0000 7f7f1c7f3740 10 client.25642 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.066+0000 7f7f1c7f3740 10 client.25642 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.066+0000 7f7f1c7f3740 20 client.25642 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.066+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5564c9f95b10 0x5564c9f97f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.066+0000 7f7f1c7f3740 4 client.25642 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.066+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3667133837 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564c9ee6d70 con 0x5564c9f95b10 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.066+0000 7f7f12a00640 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5564c9f95b10 0x5564c9f97f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.067+0000 7f7f12a00640 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5564c9f95b10 0x5564c9f97f00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=937630591ae9e787 server_cookie=7fe6ecd423f2fe5e in_seq=0 out_seq=0 2024-04-25T01:56:30.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.067+0000 7f7efcc00640 10 client.25642 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:56:30.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7efcc00640 1 -- 172.21.15.102:0/3667133837 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5564c9ee6d70 con 0x5564c9f95b10 2024-04-25T01:56:30.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7efcc00640 10 client.25642 handle_command_reply: tid=0 2024-04-25T01:56:30.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 client.25642 shutdown 2024-04-25T01:56:30.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3667133837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 msgr2=0x5564c9f82fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 0x5564c9f82fa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0d50a0 tx=0x7f7f0c0a37f0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3667133837 shutdown_connections 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5564c9f95b10 0x5564c9f97f00 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 0x5564c9f82fa0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7aee0 0x5564c9f82a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/3667133837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7a5c0 0x5564c9f82520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7efcc00640 0 client.25642 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3667133837 >> 172.21.15.102:0/3667133837 conn(0x5564c9f768e0 msgr2=0x5564c9f75b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.068+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3667133837 shutdown_connections 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.069+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/3667133837 wait complete. 2024-04-25T01:56:30.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.069+0000 7f7f1c7f3740 20 client.25642 trim_cache size 0 max 16384 2024-04-25T01:56:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.095+0000 7f7f1c7f3740 1 Processor -- start 2024-04-25T01:56:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.095+0000 7f7f1c7f3740 1 -- start start 2024-04-25T01:56:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.096+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f762c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.096+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7aee0 0x5564c9f82d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.096+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 0x5564c9f9d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.096+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141ad8f0 con 0x5564c9f7aee0 2024-04-25T01:56:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.096+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141ab160 con 0x5564c9f7a5c0 2024-04-25T01:56:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.096+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141a9080 con 0x5564c9f855b0 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.096+0000 7f7f12a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.096+0000 7f7f13400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 0x5564c9f9d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.097+0000 7f7f12a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59400/0 (socket says 172.21.15.102:59400) 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.097+0000 7f7f12a00640 1 -- 172.21.15.102:0/845788337 learned_addr learned my addr 172.21.15.102:0/845788337 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.097+0000 7f7f12a00640 1 -- 172.21.15.102:0/845788337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 msgr2=0x5564c9f9d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.097+0000 7f7f12a00640 1 --2- 172.21.15.102:0/845788337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 0x5564c9f9d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.097+0000 7f7f12a00640 1 -- 172.21.15.102:0/845788337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7aee0 msgr2=0x5564c9f82d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.097+0000 7f7f12a00640 1 --2- 172.21.15.102:0/845788337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7aee0 0x5564c9f82d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.097+0000 7f7f12a00640 1 -- 172.21.15.102:0/845788337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564c9f87bd0 con 0x5564c9f7a5c0 2024-04-25T01:56:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.097+0000 7f7f12a00640 1 --2- 172.21.15.102:0/845788337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f762c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009c70 tx=0x7f7f0400f8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39f94d6cd4afe746 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.098+0000 7f7eff400640 1 -- 172.21.15.102:0/845788337 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04009960 con 0x5564c9f7a5c0 2024-04-25T01:56:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.098+0000 7f7eff400640 1 -- 172.21.15.102:0/845788337 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f0400fe30 con 0x5564c9f7a5c0 2024-04-25T01:56:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.098+0000 7f7eff400640 1 -- 172.21.15.102:0/845788337 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f040293f0 con 0x5564c9f7a5c0 2024-04-25T01:56:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.099+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/845788337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 msgr2=0x5564c9f762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.099+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/845788337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f762c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009c70 tx=0x7f7f0400f8e0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.099+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/845788337 shutdown_connections 2024-04-25T01:56:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.099+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/845788337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f855b0 0x5564c9f9d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.099+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/845788337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f7aee0 0x5564c9f82d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.099+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/845788337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f762c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.099+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/845788337 >> 172.21.15.102:0/845788337 conn(0x5564c9f768e0 msgr2=0x5564c9f7bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.099+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/845788337 shutdown_connections 2024-04-25T01:56:30.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.100+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/845788337 wait complete. 2024-04-25T01:56:30.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.100+0000 7f7f1c7f3740 1 -- start start 2024-04-25T01:56:30.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.100+0000 7f7f1c7f3740 1 Processor -- start 2024-04-25T01:56:30.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.101+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f895b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.102+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f89af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.102+0000 7f7f1c7f3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f855b0 0x5564c9ffb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.102+0000 7f7f13400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f855b0 0x5564c9ffb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.102+0000 7f7f1c7f3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141a9640 con 0x5564c9f855b0 2024-04-25T01:56:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.103+0000 7f7f12a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f895b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.103+0000 7f7f12a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f895b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59406/0 (socket says 172.21.15.102:59406) 2024-04-25T01:56:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.103+0000 7f7f12a00640 1 -- 172.21.15.102:0/2907809712 learned_addr learned my addr 172.21.15.102:0/2907809712 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.103+0000 7f7f12000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f89af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.103+0000 7f7f13400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f855b0 0x5564c9ffb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50444/0 (socket says 172.21.15.102:50444) 2024-04-25T01:56:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.103+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141aaa90 con 0x5564c9f7a5c0 2024-04-25T01:56:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.103+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141a9cf0 con 0x5564c9f7aee0 2024-04-25T01:56:30.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.103+0000 7f7f1c7f3740 10 client.0 _ll_register_callbacks cb 0x7f7f0c007800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:30.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.104+0000 7f7f12a00640 1 -- 172.21.15.102:0/2907809712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 msgr2=0x5564c9f89af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.104+0000 7f7f12a00640 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f89af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.104+0000 7f7f12a00640 1 -- 172.21.15.102:0/2907809712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f855b0 msgr2=0x5564c9ffb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.104+0000 7f7f12a00640 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f855b0 0x5564c9ffb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.104+0000 7f7f12a00640 1 -- 172.21.15.102:0/2907809712 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564c9ffb580 con 0x5564c9f7a5c0 2024-04-25T01:56:30.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.104+0000 7f7f13400640 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f855b0 0x5564c9ffb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:30.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.104+0000 7f7f12a00640 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f895b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7f04005b80 tx=0x7f7f040117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bf9cda32a71e32d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.104+0000 7f7efcc00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:30.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.105+0000 7f7efcc00640 1 -- 172.21.15.102:0/2907809712 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04037060 con 0x5564c9f7a5c0 2024-04-25T01:56:30.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.105+0000 7f7efcc00640 1 -- 172.21.15.102:0/2907809712 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7f0400f950 con 0x5564c9f7a5c0 2024-04-25T01:56:30.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.105+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f0400fe30 con 0x5564c9f7a5c0 2024-04-25T01:56:30.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.106+0000 7f7efcc00640 1 -- 172.21.15.102:0/2907809712 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7f0400fb90 con 0x5564c9f7a5c0 2024-04-25T01:56:30.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.106+0000 7f7f1c7f3740 10 client.25664 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:30.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.106+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5564c9ffc700 con 0x5564c9f7a5c0 2024-04-25T01:56:30.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.107+0000 7f7efcc00640 1 -- 172.21.15.102:0/2907809712 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f0400fe30 con 0x5564c9f7a5c0 2024-04-25T01:56:30.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.107+0000 7f7efcc00640 1 -- 172.21.15.102:0/2907809712 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7f04003850 con 0x5564c9f7a5c0 2024-04-25T01:56:30.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.107+0000 7f7f1c7f3740 10 client.25664 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:30.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.107+0000 7f7f1c7f3740 10 client.25664 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:56:30.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.107+0000 7f7f1c7f3740 20 client.25664 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:30.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.107+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5564c9ffcc50 0x5564c9fff070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.107+0000 7f7f1c7f3740 4 client.25664 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:30.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.107+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564c9e646e0 con 0x5564c9ffcc50 2024-04-25T01:56:30.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.108+0000 7f7f12000640 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5564c9ffcc50 0x5564c9fff070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.109+0000 7f7f12000640 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5564c9ffcc50 0x5564c9fff070 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=42f4605ee7f58217 server_cookie=ffe29bfded57ea08 in_seq=0 out_seq=0 2024-04-25T01:56:30.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.109+0000 7f7efcc00640 10 client.25664 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:56:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.110+0000 7f7efcc00640 1 -- 172.21.15.102:0/2907809712 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564c9e646e0 con 0x5564c9ffcc50 2024-04-25T01:56:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.110+0000 7f7efcc00640 10 client.25664 handle_command_reply: tid=0 2024-04-25T01:56:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.110+0000 7f7f1c7f3740 1 client.25664 shutdown 2024-04-25T01:56:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.110+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 msgr2=0x5564c9f895b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f895b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7f04005b80 tx=0x7f7f040117b0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 shutdown_connections 2024-04-25T01:56:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5564c9ffcc50 0x5564c9fff070 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5564c9f855b0 0x5564c9ffb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5564c9f7aee0 0x5564c9f89af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 1 --2- 172.21.15.102:0/2907809712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5564c9f7a5c0 0x5564c9f895b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 >> 172.21.15.102:0/2907809712 conn(0x5564c9f768e0 msgr2=0x5564c9f7bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7efcc00640 0 client.25664 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:56:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 shutdown_connections 2024-04-25T01:56:30.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 1 -- 172.21.15.102:0/2907809712 wait complete. 2024-04-25T01:56:30.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.111+0000 7f7f1c7f3740 20 client.25664 trim_cache size 0 max 16384 2024-04-25T01:56:30.114 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:56:30.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.113+0000 7f7f19a00640 1 -- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7ef80a2ef0 msgr2=0x7f7ef80a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.113+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7ef80a2ef0 0x7f7ef80a53b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5564c9f77a70 tx=0x7f7f00000bc0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.113+0000 7f7f19a00640 1 -- 172.21.15.102:0/1295180759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f1410bd00 msgr2=0x7f7f1419ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.113+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f1410bd00 0x7f7f1419ea70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c00cce0 tx=0x7f7f0c00a620 comp rx=0 tx=0).stop 2024-04-25T01:56:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.114+0000 7f7f19a00640 1 -- 172.21.15.102:0/1295180759 shutdown_connections 2024-04-25T01:56:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.114+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7ef80a2ef0 0x7f7ef80a53b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.114+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f1410bd00 0x7f7f1419ea70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.114+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f141032e0 0x7f7f1419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.114+0000 7f7f19a00640 1 --2- 172.21.15.102:0/1295180759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f141029c0 0x7f7f1419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.114+0000 7f7f19a00640 1 -- 172.21.15.102:0/1295180759 >> 172.21.15.102:0/1295180759 conn(0x7f7f14077740 msgr2=0x7f7f14101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.114+0000 7f7f19a00640 1 -- 172.21.15.102:0/1295180759 shutdown_connections 2024-04-25T01:56:30.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.114+0000 7f7f19a00640 1 -- 172.21.15.102:0/1295180759 wait complete. 2024-04-25T01:56:30.147 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:56:30.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:30 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3575215553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:56:30.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.426+0000 7f0e33200640 1 -- 172.21.15.102:0/3263884349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 msgr2=0x7f0e2c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.426+0000 7f0e33200640 1 --2- 172.21.15.102:0/3263884349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 0x7f0e2c10b7c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009860 tx=0x7f0e1802a460 comp rx=0 tx=0).stop 2024-04-25T01:56:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.427+0000 7f0e33200640 1 -- 172.21.15.102:0/3263884349 shutdown_connections 2024-04-25T01:56:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.427+0000 7f0e33200640 1 --2- 172.21.15.102:0/3263884349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e2c10bd00 0x7f0e2c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.427+0000 7f0e33200640 1 --2- 172.21.15.102:0/3263884349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 0x7f0e2c10b7c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.427+0000 7f0e33200640 1 --2- 172.21.15.102:0/3263884349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c1029c0 0x7f0e2c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.427+0000 7f0e33200640 1 -- 172.21.15.102:0/3263884349 >> 172.21.15.102:0/3263884349 conn(0x7f0e2c077740 msgr2=0x7f0e2c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.427+0000 7f0e33200640 1 -- 172.21.15.102:0/3263884349 shutdown_connections 2024-04-25T01:56:30.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.427+0000 7f0e33200640 1 -- 172.21.15.102:0/3263884349 wait complete. 2024-04-25T01:56:30.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.429+0000 7f0e33200640 1 Processor -- start 2024-04-25T01:56:30.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.429+0000 7f0e33200640 1 -- start start 2024-04-25T01:56:30.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e33200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c1029c0 0x7f0e2c195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e33200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 0x7f0e2c196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e33200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e2c10bd00 0x7f0e2c19a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 0x7f0e2c196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e33200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10f930 con 0x7f0e2c1029c0 2024-04-25T01:56:30.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e33200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10f7b0 con 0x7f0e2c10bd00 2024-04-25T01:56:30.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e33200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c10fab0 con 0x7f0e2c1032e0 2024-04-25T01:56:30.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 0x7f0e2c196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49698/0 (socket says 172.21.15.102:49698) 2024-04-25T01:56:30.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e2b400640 1 -- 172.21.15.102:0/3339330131 learned_addr learned my addr 172.21.15.102:0/3339330131 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.430+0000 7f0e2be00640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c1029c0 0x7f0e2c195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.431+0000 7f0e31200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e2c10bd00 0x7f0e2c19a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.431+0000 7f0e2b400640 1 -- 172.21.15.102:0/3339330131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e2c10bd00 msgr2=0x7f0e2c19a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.431+0000 7f0e2b400640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e2c10bd00 0x7f0e2c19a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.431+0000 7f0e2b400640 1 -- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c1029c0 msgr2=0x7f0e2c195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.431+0000 7f0e2b400640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c1029c0 0x7f0e2c195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.431+0000 7f0e2b400640 1 -- 172.21.15.102:0/3339330131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c19af00 con 0x7f0e2c1032e0 2024-04-25T01:56:30.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.431+0000 7f0e31200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e2c10bd00 0x7f0e2c19a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:30.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.431+0000 7f0e2b400640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 0x7f0e2c196450 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0e1802a980 tx=0x7f0e180307b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.431+0000 7f0e28c00640 1 -- 172.21.15.102:0/3339330131 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e18040020 con 0x7f0e2c1032e0 2024-04-25T01:56:30.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.432+0000 7f0e33200640 1 -- 172.21.15.102:0/3339330131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2c19b190 con 0x7f0e2c1032e0 2024-04-25T01:56:30.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.432+0000 7f0e33200640 1 -- 172.21.15.102:0/3339330131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e2c1a2a80 con 0x7f0e2c1032e0 2024-04-25T01:56:30.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.432+0000 7f0e28c00640 1 -- 172.21.15.102:0/3339330131 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e18004460 con 0x7f0e2c1032e0 2024-04-25T01:56:30.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.432+0000 7f0e28c00640 1 -- 172.21.15.102:0/3339330131 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e180046e0 con 0x7f0e2c1032e0 2024-04-25T01:56:30.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.436+0000 7f0e28c00640 1 -- 172.21.15.102:0/3339330131 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f0e18043020 con 0x7f0e2c1032e0 2024-04-25T01:56:30.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.436+0000 7f0e3602a740 1 Processor -- start 2024-04-25T01:56:30.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.436+0000 7f0e3602a740 1 -- start start 2024-04-25T01:56:30.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.437+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55912857e5c0 0x55912857e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.437+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x559128589070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.437+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5591285895b0 0x55912858b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.437+0000 7f0e3602a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5591284e34e0 con 0x5591285895b0 2024-04-25T01:56:30.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.437+0000 7f0e3602a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55912848ba70 con 0x55912857eee0 2024-04-25T01:56:30.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.437+0000 7f0e3602a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5591284df4a0 con 0x55912857e5c0 2024-04-25T01:56:30.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.438+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x559128589070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.438+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x559128589070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59426/0 (socket says 172.21.15.102:59426) 2024-04-25T01:56:30.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.438+0000 7f0e2be00640 1 -- 172.21.15.102:0/3489835008 learned_addr learned my addr 172.21.15.102:0/3489835008 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.438+0000 7f0e31200640 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55912857e5c0 0x55912857e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.438+0000 7f0e2b400640 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5591285895b0 0x55912858b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.439+0000 7f0e28c00640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0e100a2f20 0x7f0e100a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.439+0000 7f0e28c00640 1 -- 172.21.15.102:0/3339330131 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0e180f0630 con 0x7f0e2c1032e0 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.440+0000 7f0e2be00640 1 -- 172.21.15.102:0/3489835008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55912857e5c0 msgr2=0x55912857e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.440+0000 7f0e2be00640 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55912857e5c0 0x55912857e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.440+0000 7f0e2be00640 1 -- 172.21.15.102:0/3489835008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5591285895b0 msgr2=0x55912858b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.440+0000 7f0e2be00640 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5591285895b0 0x55912858b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.440+0000 7f0e2be00640 1 -- 172.21.15.102:0/3489835008 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591281f73f0 con 0x55912857eee0 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.440+0000 7f0e2b400640 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5591285895b0 0x55912858b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:30.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.440+0000 7f0e31200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0e100a2f20 0x7f0e100a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.440+0000 7f0e2be00640 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x559128589070 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00b820 tx=0x7f0e1c00bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=124cc612938d0d45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.440+0000 7f0e21600640 1 -- 172.21.15.102:0/3489835008 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c00d950 con 0x55912857eee0 2024-04-25T01:56:30.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.441+0000 7f0e31200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0e100a2f20 0x7f0e100a53e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005fd0 tx=0x7f0e2400f040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.441+0000 7f0e21600640 1 -- 172.21.15.102:0/3489835008 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1c019070 con 0x55912857eee0 2024-04-25T01:56:30.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.441+0000 7f0e21600640 1 -- 172.21.15.102:0/3489835008 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c0123e0 con 0x55912857eee0 2024-04-25T01:56:30.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.443+0000 7f0e3602a740 1 -- 172.21.15.102:0/3489835008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 msgr2=0x559128589070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.443+0000 7f0e3602a740 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x559128589070 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00b820 tx=0x7f0e1c00bce0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.444+0000 7f0e3602a740 1 -- 172.21.15.102:0/3489835008 shutdown_connections 2024-04-25T01:56:30.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.444+0000 7f0e3602a740 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5591285895b0 0x55912858b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.444+0000 7f0e3602a740 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x559128589070 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.444+0000 7f0e3602a740 1 --2- 172.21.15.102:0/3489835008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55912857e5c0 0x55912857e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.444+0000 7f0e3602a740 1 -- 172.21.15.102:0/3489835008 >> 172.21.15.102:0/3489835008 conn(0x55912857a8e0 msgr2=0x55912857acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.444+0000 7f0e3602a740 1 -- 172.21.15.102:0/3489835008 shutdown_connections 2024-04-25T01:56:30.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.445+0000 7f0e3602a740 1 -- 172.21.15.102:0/3489835008 wait complete. 2024-04-25T01:56:30.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.446+0000 7f0e3602a740 1 -- start start 2024-04-25T01:56:30.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.448+0000 7f0e3602a740 1 Processor -- start 2024-04-25T01:56:30.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.450+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857e5c0 0x5591285864a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.450+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 0x5591285869e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.450+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128586f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.451+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857e5c0 0x5591285864a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.451+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857e5c0 0x5591285864a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59428/0 (socket says 172.21.15.102:59428) 2024-04-25T01:56:30.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.451+0000 7f0e31200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128586f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.451+0000 7f0e31200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128586f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49722/0 (socket says 172.21.15.102:49722) 2024-04-25T01:56:30.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.451+0000 7f0e2b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 0x5591285869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.452+0000 7f0e3602a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559128594bb0 con 0x55912857eee0 2024-04-25T01:56:30.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.452+0000 7f0e3602a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55912849eec0 con 0x55912857e5c0 2024-04-25T01:56:30.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.452+0000 7f0e3602a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559128594d30 con 0x5591285895b0 2024-04-25T01:56:30.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.452+0000 7f0e3602a740 10 client.0 _ll_register_callbacks cb 0x5591284db3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:30.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.452+0000 7f0e2be00640 1 -- 172.21.15.102:0/1250917348 learned_addr learned my addr 172.21.15.102:0/1250917348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.452+0000 7f0e2be00640 1 -- 172.21.15.102:0/1250917348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 msgr2=0x559128586f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.452+0000 7f0e2be00640 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128586f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.452+0000 7f0e2be00640 1 -- 172.21.15.102:0/1250917348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 msgr2=0x5591285869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.452+0000 7f0e2be00640 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 0x5591285869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.453+0000 7f0e2be00640 1 -- 172.21.15.102:0/1250917348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591282c17d0 con 0x55912857e5c0 2024-04-25T01:56:30.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.453+0000 7f0e31200640 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128586f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:30.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.453+0000 7f0e2be00640 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857e5c0 0x5591285864a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c004810 tx=0x7f0e1c014660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e91ab3058b575155 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.453+0000 7f0e16000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:30.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.454+0000 7f0e16000640 1 -- 172.21.15.102:0/1250917348 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c023070 con 0x55912857e5c0 2024-04-25T01:56:30.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.454+0000 7f0e16000640 1 -- 172.21.15.102:0/1250917348 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0e1c00dd60 con 0x55912857e5c0 2024-04-25T01:56:30.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.454+0000 7f0e3602a740 1 -- 172.21.15.102:0/1250917348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e1c019070 con 0x55912857e5c0 2024-04-25T01:56:30.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.454+0000 7f0e16000640 1 -- 172.21.15.102:0/1250917348 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e1c02d050 con 0x55912857e5c0 2024-04-25T01:56:30.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e3602a740 10 client.25676 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:30.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e3602a740 1 -- 172.21.15.102:0/1250917348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5591285a0e10 con 0x55912857e5c0 2024-04-25T01:56:30.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e16000640 1 -- 172.21.15.102:0/1250917348 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e1c019070 con 0x55912857e5c0 2024-04-25T01:56:30.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e16000640 1 -- 172.21.15.102:0/1250917348 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0e1c013070 con 0x55912857e5c0 2024-04-25T01:56:30.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e3602a740 10 client.25676 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:30.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e3602a740 10 client.25676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:30.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e3602a740 20 client.25676 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:30.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5591285a22d0 0x5591285a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e3602a740 4 client.25676 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:30.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.455+0000 7f0e3602a740 1 -- 172.21.15.102:0/1250917348 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591284f0a20 con 0x5591285a22d0 2024-04-25T01:56:30.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.456+0000 7f0e2b400640 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5591285a22d0 0x5591285a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.456+0000 7f0e2b400640 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5591285a22d0 0x5591285a46c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a5244e6750ba3d8 server_cookie=3bc5073b99a88bf4 in_seq=0 out_seq=0 2024-04-25T01:56:30.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.456+0000 7f0e16000640 10 client.25676 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:30.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.457+0000 7f0e16000640 1 -- 172.21.15.102:0/1250917348 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5591284f0a20 con 0x5591285a22d0 2024-04-25T01:56:30.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.457+0000 7f0e16000640 10 client.25676 handle_command_reply: tid=0 2024-04-25T01:56:30.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.457+0000 7f0e3602a740 1 client.25676 shutdown 2024-04-25T01:56:30.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 -- 172.21.15.102:0/1250917348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857e5c0 msgr2=0x5591285864a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857e5c0 0x5591285864a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c004810 tx=0x7f0e1c014660 comp rx=0 tx=0).stop 2024-04-25T01:56:30.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 -- 172.21.15.102:0/1250917348 shutdown_connections 2024-04-25T01:56:30.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5591285a22d0 0x5591285a46c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128586f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 0x5591285869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e16000640 0 client.25676 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:30.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1250917348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857e5c0 0x5591285864a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 -- 172.21.15.102:0/1250917348 >> 172.21.15.102:0/1250917348 conn(0x55912857a8e0 msgr2=0x559128579b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 -- 172.21.15.102:0/1250917348 shutdown_connections 2024-04-25T01:56:30.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.458+0000 7f0e3602a740 1 -- 172.21.15.102:0/1250917348 wait complete. 2024-04-25T01:56:30.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.459+0000 7f0e3602a740 20 client.25676 trim_cache size 0 max 16384 2024-04-25T01:56:30.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.491+0000 7f0e3602a740 1 Processor -- start 2024-04-25T01:56:30.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.492+0000 7f0e3602a740 1 -- start start 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.492+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x55912859dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.492+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128585600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.492+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559128585f90 0x5591285799c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.492+0000 7f0e3602a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c010ff0 con 0x559128585f90 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.492+0000 7f0e3602a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c00e860 con 0x55912857eee0 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.492+0000 7f0e3602a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c00c6b0 con 0x5591285895b0 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.493+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x55912859dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.493+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x55912859dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59434/0 (socket says 172.21.15.102:59434) 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.493+0000 7f0e2be00640 1 -- 172.21.15.102:0/2560122359 learned_addr learned my addr 172.21.15.102:0/2560122359 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.493+0000 7f0e2b400640 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128585600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.493+0000 7f0e31200640 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559128585f90 0x5591285799c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.494+0000 7f0e2b400640 1 -- 172.21.15.102:0/2560122359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 msgr2=0x55912859dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.494+0000 7f0e2b400640 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x55912859dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.494+0000 7f0e2b400640 1 -- 172.21.15.102:0/2560122359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559128585f90 msgr2=0x5591285799c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.494+0000 7f0e2b400640 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559128585f90 0x5591285799c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.494+0000 7f0e2b400640 1 -- 172.21.15.102:0/2560122359 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591285a4c50 con 0x5591285895b0 2024-04-25T01:56:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.494+0000 7f0e2be00640 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x55912859dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.495+0000 7f0e31200640 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559128585f90 0x5591285799c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.495+0000 7f0e2b400640 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128585600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e180edbd0 tx=0x7f0e180f2a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1332cc090b17eaf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.495+0000 7f0e20c00640 1 -- 172.21.15.102:0/2560122359 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e180bc300 con 0x5591285895b0 2024-04-25T01:56:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.495+0000 7f0e20c00640 1 -- 172.21.15.102:0/2560122359 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e180b8ce0 con 0x5591285895b0 2024-04-25T01:56:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.495+0000 7f0e20c00640 1 -- 172.21.15.102:0/2560122359 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e180c5180 con 0x5591285895b0 2024-04-25T01:56:30.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.497+0000 7f0e3602a740 1 -- 172.21.15.102:0/2560122359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 msgr2=0x559128585600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.497+0000 7f0e3602a740 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128585600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e180edbd0 tx=0x7f0e180f2a00 comp rx=0 tx=0).stop 2024-04-25T01:56:30.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.497+0000 7f0e3602a740 1 -- 172.21.15.102:0/2560122359 shutdown_connections 2024-04-25T01:56:30.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.497+0000 7f0e3602a740 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559128585f90 0x5591285799c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.497+0000 7f0e3602a740 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5591285895b0 0x559128585600 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.497+0000 7f0e3602a740 1 --2- 172.21.15.102:0/2560122359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55912857eee0 0x55912859dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.498+0000 7f0e3602a740 1 -- 172.21.15.102:0/2560122359 >> 172.21.15.102:0/2560122359 conn(0x55912857a8e0 msgr2=0x5591285792b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.498+0000 7f0e3602a740 1 -- 172.21.15.102:0/2560122359 shutdown_connections 2024-04-25T01:56:30.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.498+0000 7f0e3602a740 1 -- 172.21.15.102:0/2560122359 wait complete. 2024-04-25T01:56:30.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.498+0000 7f0e3602a740 1 -- start start 2024-04-25T01:56:30.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.499+0000 7f0e3602a740 1 Processor -- start 2024-04-25T01:56:30.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.500+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 0x559128603db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.501+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559128585f90 0x559128599fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.501+0000 7f0e3602a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5591285895b0 0x55912859a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.501+0000 7f0e3602a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c00cd20 con 0x55912857eee0 2024-04-25T01:56:30.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.501+0000 7f0e3602a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c00e190 con 0x5591285895b0 2024-04-25T01:56:30.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.501+0000 7f0e3602a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c00d3f0 con 0x559128585f90 2024-04-25T01:56:30.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.501+0000 7f0e3602a740 10 client.0 _ll_register_callbacks cb 0x7f0e180f3f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:30.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e31200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5591285895b0 0x55912859a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e31200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5591285895b0 0x55912859a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59438/0 (socket says 172.21.15.102:59438) 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e31200640 1 -- 172.21.15.102:0/1395965655 learned_addr learned my addr 172.21.15.102:0/1395965655 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e2b400640 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559128585f90 0x559128599fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e2be00640 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 0x559128603db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e2b400640 1 -- 172.21.15.102:0/1395965655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5591285895b0 msgr2=0x55912859a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e2b400640 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5591285895b0 0x55912859a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e2b400640 1 -- 172.21.15.102:0/1395965655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 msgr2=0x559128603db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e2b400640 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 0x559128603db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.502+0000 7f0e2b400640 1 -- 172.21.15.102:0/1395965655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591286011c0 con 0x559128585f90 2024-04-25T01:56:30.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.503+0000 7f0e2b400640 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559128585f90 0x559128599fe0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55912857fc10 tx=0x7f0e180b92c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ff5846d1fbdda1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:30.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.503+0000 7f0e16000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:56:30.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.503+0000 7f0e16000640 1 -- 172.21.15.102:0/1395965655 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e180bc300 con 0x559128585f90 2024-04-25T01:56:30.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.503+0000 7f0e16000640 1 -- 172.21.15.102:0/1395965655 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0e180c5180 con 0x559128585f90 2024-04-25T01:56:30.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.503+0000 7f0e3602a740 1 -- 172.21.15.102:0/1395965655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e180b8ce0 con 0x559128585f90 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.504+0000 7f0e16000640 1 -- 172.21.15.102:0/1395965655 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e180bc300 con 0x559128585f90 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.504+0000 7f0e3602a740 10 client.25666 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.504+0000 7f0e3602a740 1 -- 172.21.15.102:0/1395965655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559128619980 con 0x559128585f90 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.504+0000 7f0e16000640 1 -- 172.21.15.102:0/1395965655 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e180bc9a0 con 0x559128585f90 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.504+0000 7f0e16000640 1 -- 172.21.15.102:0/1395965655 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0e180bbdf0 con 0x559128585f90 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.505+0000 7f0e3602a740 10 client.25666 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.505+0000 7f0e3602a740 10 client.25666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.505+0000 7f0e3602a740 20 client.25666 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.505+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55912861a060 0x55912861c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.505+0000 7f0e3602a740 4 client.25666 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:56:30.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.505+0000 7f0e3602a740 1 -- 172.21.15.102:0/1395965655 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0e180b8ce0 con 0x55912861a060 2024-04-25T01:56:30.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.505+0000 7f0e2be00640 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55912861a060 0x55912861c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:30.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.506+0000 7f0e16000640 10 client.25666 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:30.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e2be00640 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55912861a060 0x55912861c480 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca058ad43ba7fcfe server_cookie=44a4a88719d71908 in_seq=0 out_seq=0 2024-04-25T01:56:30.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e16000640 1 -- 172.21.15.102:0/1395965655 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0e180b8ce0 con 0x55912861a060 2024-04-25T01:56:30.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e16000640 10 client.25666 handle_command_reply: tid=0 2024-04-25T01:56:30.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e3602a740 1 client.25666 shutdown 2024-04-25T01:56:30.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e3602a740 1 -- 172.21.15.102:0/1395965655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559128585f90 msgr2=0x559128599fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559128585f90 0x559128599fe0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x55912857fc10 tx=0x7f0e180b92c0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e3602a740 1 -- 172.21.15.102:0/1395965655 shutdown_connections 2024-04-25T01:56:30.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55912861a060 0x55912861c480 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e16000640 0 client.25666 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:30.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5591285895b0 0x55912859a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559128585f90 0x559128599fe0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e3602a740 1 --2- 172.21.15.102:0/1395965655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55912857eee0 0x559128603db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.507+0000 7f0e3602a740 1 -- 172.21.15.102:0/1395965655 >> 172.21.15.102:0/1395965655 conn(0x55912857a8e0 msgr2=0x5591285863f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.508+0000 7f0e3602a740 1 -- 172.21.15.102:0/1395965655 shutdown_connections 2024-04-25T01:56:30.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.508+0000 7f0e3602a740 1 -- 172.21.15.102:0/1395965655 wait complete. 2024-04-25T01:56:30.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.508+0000 7f0e3602a740 20 client.25666 trim_cache size 0 max 16384 2024-04-25T01:56:30.511 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:56:30.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 -- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0e100a2f20 msgr2=0x7f0e100a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0e100a2f20 0x7f0e100a53e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005fd0 tx=0x7f0e2400f040 comp rx=0 tx=0).stop 2024-04-25T01:56:30.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 -- 172.21.15.102:0/3339330131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 msgr2=0x7f0e2c196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:30.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 0x7f0e2c196450 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0e1802a980 tx=0x7f0e180307b0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 -- 172.21.15.102:0/3339330131 shutdown_connections 2024-04-25T01:56:30.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0e100a2f20 0x7f0e100a53e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e2c10bd00 0x7f0e2c19a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e2c1032e0 0x7f0e2c196450 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 --2- 172.21.15.102:0/3339330131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c1029c0 0x7f0e2c195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:30.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.510+0000 7f0e33200640 1 -- 172.21.15.102:0/3339330131 >> 172.21.15.102:0/3339330131 conn(0x7f0e2c077740 msgr2=0x7f0e2c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:30.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.511+0000 7f0e33200640 1 -- 172.21.15.102:0/3339330131 shutdown_connections 2024-04-25T01:56:30.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:30.511+0000 7f0e33200640 1 -- 172.21.15.102:0/3339330131 wait complete. 2024-04-25T01:56:30.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:30 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3575215553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:56:30.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:30 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3575215553' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:56:31.539 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:56:31.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:56:31.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:56:31.623 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:31 smithi102 ceph-mon[27796]: pgmap v40: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 14 MiB/s wr, 676 op/s 2024-04-25T01:56:31.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:31 smithi110 ceph-mon[32848]: pgmap v40: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 14 MiB/s wr, 676 op/s 2024-04-25T01:56:31.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:31 smithi193 ceph-mon[32981]: pgmap v40: 97 pgs: 97 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 14 MiB/s wr, 676 op/s 2024-04-25T01:56:31.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.840+0000 7f289ba00640 1 -- 172.21.15.102:0/1713289812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 msgr2=0x7f2894115040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.840+0000 7f289ba00640 1 --2- 172.21.15.102:0/1713289812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 0x7f2894115040 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f288c005fd0 tx=0x7f288c02b3f0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.841+0000 7f289ba00640 1 -- 172.21.15.102:0/1713289812 shutdown_connections 2024-04-25T01:56:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.841+0000 7f289ba00640 1 --2- 172.21.15.102:0/1713289812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 0x7f2894115040 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.841+0000 7f289ba00640 1 --2- 172.21.15.102:0/1713289812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2894109360 0x7f289410cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.841+0000 7f289ba00640 1 --2- 172.21.15.102:0/1713289812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28941089b0 0x7f2894108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.841+0000 7f289ba00640 1 -- 172.21.15.102:0/1713289812 >> 172.21.15.102:0/1713289812 conn(0x7f289406c850 msgr2=0x7f289406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.841+0000 7f289ba00640 1 -- 172.21.15.102:0/1713289812 shutdown_connections 2024-04-25T01:56:31.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.841+0000 7f289ba00640 1 -- 172.21.15.102:0/1713289812 wait complete. 2024-04-25T01:56:31.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.842+0000 7f289ba00640 1 Processor -- start 2024-04-25T01:56:31.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.843+0000 7f289ba00640 1 -- start start 2024-04-25T01:56:31.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.843+0000 7f289ba00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28941089b0 0x7f289419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.843+0000 7f289ba00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 0x7f289419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.843+0000 7f289ba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28941a2e70 0x7f28941a3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.844+0000 7f2899000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28941089b0 0x7f289419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.844+0000 7f289ba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f28941180b0 con 0x7f28941a2e70 2024-04-25T01:56:31.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.844+0000 7f289ba00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2894117f30 con 0x7f289410d470 2024-04-25T01:56:31.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.844+0000 7f289ba00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2894118230 con 0x7f28941089b0 2024-04-25T01:56:31.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.844+0000 7f2893e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 0x7f289419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.844+0000 7f2899a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28941a2e70 0x7f28941a3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.844+0000 7f2893e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 0x7f289419eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59470/0 (socket says 172.21.15.102:59470) 2024-04-25T01:56:31.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.844+0000 7f2893e00640 1 -- 172.21.15.102:0/3932586726 learned_addr learned my addr 172.21.15.102:0/3932586726 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:31.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.845+0000 7f2899000640 1 -- 172.21.15.102:0/3932586726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 msgr2=0x7f289419eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:31.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.845+0000 7f2899000640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 0x7f289419eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.845+0000 7f2899000640 1 -- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28941a2e70 msgr2=0x7f28941a3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:31.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.845+0000 7f2899000640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28941a2e70 0x7f28941a3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.845+0000 7f2899000640 1 -- 172.21.15.102:0/3932586726 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28941a3960 con 0x7f28941089b0 2024-04-25T01:56:31.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.845+0000 7f2893e00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 0x7f289419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:31.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.845+0000 7f2899a00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28941a2e70 0x7f28941a3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:31.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.846+0000 7f2899000640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28941089b0 0x7f289419e5b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f288800c970 tx=0x7f288800ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:31.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.846+0000 7f2891600640 1 -- 172.21.15.102:0/3932586726 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2888007bb0 con 0x7f28941089b0 2024-04-25T01:56:31.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.846+0000 7f2891600640 1 -- 172.21.15.102:0/3932586726 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2888007d50 con 0x7f28941089b0 2024-04-25T01:56:31.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.846+0000 7f2891600640 1 -- 172.21.15.102:0/3932586726 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28880187b0 con 0x7f28941089b0 2024-04-25T01:56:31.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.848+0000 7f289ba00640 1 -- 172.21.15.102:0/3932586726 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28941a3c50 con 0x7f28941089b0 2024-04-25T01:56:31.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.848+0000 7f289ba00640 1 -- 172.21.15.102:0/3932586726 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28941af7d0 con 0x7f28941089b0 2024-04-25T01:56:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.855+0000 7f2891600640 1 -- 172.21.15.102:0/3932586726 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f288800f450 con 0x7f28941089b0 2024-04-25T01:56:31.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.859+0000 7f2891600640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f287c0a2fa0 0x7f287c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.860+0000 7f2893e00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f287c0a2fa0 0x7f287c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.860+0000 7f2891600640 1 -- 172.21.15.102:0/3932586726 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f28880cf5c0 con 0x7f28941089b0 2024-04-25T01:56:31.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.861+0000 7f2893e00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f287c0a2fa0 0x7f287c0a5460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2894072e80 tx=0x7f2880005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:31.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.863+0000 7f289e80b740 1 Processor -- start 2024-04-25T01:56:31.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.864+0000 7f289e80b740 1 -- start start 2024-04-25T01:56:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.864+0000 7f289e80b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb735950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.864+0000 7f289e80b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb740020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.864+0000 7f289e80b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb742950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.864+0000 7f289e80b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d8cb68bf40 con 0x55d8cb735570 2024-04-25T01:56:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.864+0000 7f289e80b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d8cb691190 con 0x55d8cb740560 2024-04-25T01:56:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.864+0000 7f289e80b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d8cb639a10 con 0x55d8cb735e90 2024-04-25T01:56:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.865+0000 7f2893e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb740020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.865+0000 7f2893e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb740020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49760/0 (socket says 172.21.15.102:49760) 2024-04-25T01:56:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.865+0000 7f2893e00640 1 -- 172.21.15.102:0/259446689 learned_addr learned my addr 172.21.15.102:0/259446689 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.865+0000 7f2899000640 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb735950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.866+0000 7f2899a00640 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb742950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.867+0000 7f2899a00640 1 -- 172.21.15.102:0/259446689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 msgr2=0x55d8cb740020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.867+0000 7f2899a00640 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb740020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.867+0000 7f2899a00640 1 -- 172.21.15.102:0/259446689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 msgr2=0x55d8cb735950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.867+0000 7f2899a00640 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb735950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.867+0000 7f2899a00640 1 -- 172.21.15.102:0/259446689 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8cb46d7e0 con 0x55d8cb740560 2024-04-25T01:56:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.868+0000 7f2893e00640 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb740020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.868+0000 7f2899a00640 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb742950 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f288c007780 tx=0x7f288c007b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fcf0e1a46135e18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.869+0000 7f2886000640 1 -- 172.21.15.102:0/259446689 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c037040 con 0x55d8cb740560 2024-04-25T01:56:31.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.869+0000 7f2886000640 1 -- 172.21.15.102:0/259446689 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f288c007dc0 con 0x55d8cb740560 2024-04-25T01:56:31.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.869+0000 7f2886000640 1 -- 172.21.15.102:0/259446689 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c008040 con 0x55d8cb740560 2024-04-25T01:56:31.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.872+0000 7f289e80b740 1 -- 172.21.15.102:0/259446689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 msgr2=0x55d8cb742950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:31.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.873+0000 7f289e80b740 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb742950 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f288c007780 tx=0x7f288c007b50 comp rx=0 tx=0).stop 2024-04-25T01:56:31.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.873+0000 7f289e80b740 1 -- 172.21.15.102:0/259446689 shutdown_connections 2024-04-25T01:56:31.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.873+0000 7f289e80b740 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb742950 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.873+0000 7f289e80b740 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb740020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.873+0000 7f289e80b740 1 --2- 172.21.15.102:0/259446689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb735950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.873+0000 7f289e80b740 1 -- 172.21.15.102:0/259446689 >> 172.21.15.102:0/259446689 conn(0x55d8cb731890 msgr2=0x55d8cb731c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:31.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.874+0000 7f289e80b740 1 -- 172.21.15.102:0/259446689 shutdown_connections 2024-04-25T01:56:31.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.874+0000 7f289e80b740 1 -- 172.21.15.102:0/259446689 wait complete. 2024-04-25T01:56:31.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.875+0000 7f289e80b740 1 -- start start 2024-04-25T01:56:31.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.878+0000 7f289e80b740 1 Processor -- start 2024-04-25T01:56:31.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.878+0000 7f289e80b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb73d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.878+0000 7f289e80b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb73daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.879+0000 7f289e80b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb73dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.879+0000 7f289e80b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d8cb64ce70 con 0x55d8cb735570 2024-04-25T01:56:31.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.880+0000 7f289e80b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d8cb68d150 con 0x55d8cb740560 2024-04-25T01:56:31.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.880+0000 7f289e80b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d8cb743390 con 0x55d8cb735e90 2024-04-25T01:56:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.880+0000 7f289e80b740 10 client.0 _ll_register_callbacks cb 0x55d8cb6881f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.880+0000 7f2899a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb73dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.880+0000 7f2899a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb73dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59482/0 (socket says 172.21.15.102:59482) 2024-04-25T01:56:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.880+0000 7f2899a00640 1 -- 172.21.15.102:0/550299068 learned_addr learned my addr 172.21.15.102:0/550299068 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.880+0000 7f2893e00640 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb73daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.880+0000 7f2899000640 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb73d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.881+0000 7f2899a00640 1 -- 172.21.15.102:0/550299068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 msgr2=0x55d8cb73daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.881+0000 7f2899a00640 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb73daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.881+0000 7f2899a00640 1 -- 172.21.15.102:0/550299068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 msgr2=0x55d8cb73d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.881+0000 7f2899a00640 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb73d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.882+0000 7f2899a00640 1 -- 172.21.15.102:0/550299068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8cb4cdf70 con 0x55d8cb740560 2024-04-25T01:56:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.882+0000 7f2899000640 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb73d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.882+0000 7f287aa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.882+0000 7f2899a00640 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb73dfe0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f288c035040 tx=0x7f288c007500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4f1008840f319d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.883+0000 7f287aa00640 1 -- 172.21.15.102:0/550299068 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c037040 con 0x55d8cb740560 2024-04-25T01:56:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.883+0000 7f287aa00640 1 -- 172.21.15.102:0/550299068 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f288c00a2e0 con 0x55d8cb740560 2024-04-25T01:56:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.883+0000 7f289e80b740 1 -- 172.21.15.102:0/550299068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f288c007dc0 con 0x55d8cb740560 2024-04-25T01:56:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.884+0000 7f287aa00640 1 -- 172.21.15.102:0/550299068 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f288c037040 con 0x55d8cb740560 2024-04-25T01:56:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.884+0000 7f289e80b740 10 client.25694 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.884+0000 7f289e80b740 1 -- 172.21.15.102:0/550299068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d8cb365170 con 0x55d8cb740560 2024-04-25T01:56:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.884+0000 7f287aa00640 1 -- 172.21.15.102:0/550299068 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f288c007dc0 con 0x55d8cb740560 2024-04-25T01:56:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.884+0000 7f287aa00640 1 -- 172.21.15.102:0/550299068 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f288c03cad0 con 0x55d8cb740560 2024-04-25T01:56:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.885+0000 7f289e80b740 10 client.25694 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.885+0000 7f289e80b740 10 client.25694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.885+0000 7f289e80b740 20 client.25694 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.885+0000 7f289e80b740 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d8cb750690 0x55d8cb752a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.885+0000 7f289e80b740 4 client.25694 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.885+0000 7f289e80b740 1 -- 172.21.15.102:0/550299068 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8cb69e6d0 con 0x55d8cb750690 2024-04-25T01:56:31.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.885+0000 7f2899000640 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d8cb750690 0x55d8cb752a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.886+0000 7f287aa00640 10 client.25694 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.886+0000 7f2899000640 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d8cb750690 0x55d8cb752a80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2353039430d46c50 server_cookie=de9be69ba015f1ed in_seq=0 out_seq=0 2024-04-25T01:56:31.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.938+0000 7f287aa00640 1 -- 172.21.15.102:0/550299068 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8cb69e6d0 con 0x55d8cb750690 2024-04-25T01:56:31.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.938+0000 7f287aa00640 10 client.25694 handle_command_reply: tid=0 2024-04-25T01:56:31.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.938+0000 7f289e80b740 1 client.25694 shutdown 2024-04-25T01:56:31.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.939+0000 7f289e80b740 1 -- 172.21.15.102:0/550299068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 msgr2=0x55d8cb73dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:31.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.939+0000 7f289e80b740 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb73dfe0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f288c035040 tx=0x7f288c007500 comp rx=0 tx=0).stop 2024-04-25T01:56:31.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f289e80b740 1 -- 172.21.15.102:0/550299068 shutdown_connections 2024-04-25T01:56:31.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f289e80b740 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d8cb750690 0x55d8cb752a80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f289e80b740 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb73dfe0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f289e80b740 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb73daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f289e80b740 1 --2- 172.21.15.102:0/550299068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb73d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f289e80b740 1 -- 172.21.15.102:0/550299068 >> 172.21.15.102:0/550299068 conn(0x55d8cb731890 msgr2=0x55d8cb737690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:31.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f287aa00640 0 client.25694 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:31.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f289e80b740 1 -- 172.21.15.102:0/550299068 shutdown_connections 2024-04-25T01:56:31.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f289e80b740 1 -- 172.21.15.102:0/550299068 wait complete. 2024-04-25T01:56:31.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.943+0000 7f289e80b740 20 client.25694 trim_cache size 0 max 16384 2024-04-25T01:56:31.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.985+0000 7f289e80b740 1 Processor -- start 2024-04-25T01:56:31.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.985+0000 7f289e80b740 1 -- start start 2024-04-25T01:56:31.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.986+0000 7f289e80b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb735570 0x55d8cb74c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.986+0000 7f289e80b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb728060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.986+0000 7f289e80b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb740560 0x55d8cb758110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.987+0000 7f289e80b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f28941b6230 con 0x55d8cb740560 2024-04-25T01:56:31.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.987+0000 7f289e80b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f28941b3aa0 con 0x55d8cb735570 2024-04-25T01:56:31.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.987+0000 7f289e80b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28941b1910 con 0x55d8cb735e90 2024-04-25T01:56:31.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.987+0000 7f2899000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb735570 0x55d8cb74c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.987+0000 7f2899000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb735570 0x55d8cb74c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59496/0 (socket says 172.21.15.102:59496) 2024-04-25T01:56:31.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.987+0000 7f2899000640 1 -- 172.21.15.102:0/3917356300 learned_addr learned my addr 172.21.15.102:0/3917356300 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:31.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.987+0000 7f2893e00640 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb728060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.988+0000 7f2899a00640 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb740560 0x55d8cb758110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.988+0000 7f2899000640 1 -- 172.21.15.102:0/3917356300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 msgr2=0x55d8cb728060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:31.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.988+0000 7f2899000640 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb728060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.988+0000 7f2899000640 1 -- 172.21.15.102:0/3917356300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb740560 msgr2=0x55d8cb758110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:31.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.989+0000 7f2899000640 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb740560 0x55d8cb758110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.989+0000 7f2899000640 1 -- 172.21.15.102:0/3917356300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8cb736390 con 0x55d8cb735570 2024-04-25T01:56:31.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.989+0000 7f2893e00640 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb728060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:31.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.989+0000 7f2899a00640 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb740560 0x55d8cb758110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:31.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.989+0000 7f2899000640 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb735570 0x55d8cb74c0d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f288809b5e0 tx=0x7f288809e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b321bb700aad434f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:31.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.989+0000 7f2885600640 1 -- 172.21.15.102:0/3917356300 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28880a9020 con 0x55d8cb735570 2024-04-25T01:56:31.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.989+0000 7f2885600640 1 -- 172.21.15.102:0/3917356300 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2888099980 con 0x55d8cb735570 2024-04-25T01:56:31.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.989+0000 7f2885600640 1 -- 172.21.15.102:0/3917356300 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28880cd0a0 con 0x55d8cb735570 2024-04-25T01:56:31.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.991+0000 7f289e80b740 1 -- 172.21.15.102:0/3917356300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb735570 msgr2=0x55d8cb74c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:31.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.991+0000 7f289e80b740 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb735570 0x55d8cb74c0d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f288809b5e0 tx=0x7f288809e040 comp rx=0 tx=0).stop 2024-04-25T01:56:31.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.991+0000 7f289e80b740 1 -- 172.21.15.102:0/3917356300 shutdown_connections 2024-04-25T01:56:31.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.991+0000 7f289e80b740 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb740560 0x55d8cb758110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.991+0000 7f289e80b740 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb728060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.992+0000 7f289e80b740 1 --2- 172.21.15.102:0/3917356300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb735570 0x55d8cb74c0d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:31.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.992+0000 7f289e80b740 1 -- 172.21.15.102:0/3917356300 >> 172.21.15.102:0/3917356300 conn(0x55d8cb731890 msgr2=0x55d8cb727950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:31.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.992+0000 7f289e80b740 1 -- 172.21.15.102:0/3917356300 shutdown_connections 2024-04-25T01:56:31.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.992+0000 7f289e80b740 1 -- 172.21.15.102:0/3917356300 wait complete. 2024-04-25T01:56:31.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.992+0000 7f289e80b740 1 -- start start 2024-04-25T01:56:31.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.993+0000 7f289e80b740 1 Processor -- start 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.995+0000 7f289e80b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb7440c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.995+0000 7f289e80b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb744600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.995+0000 7f289e80b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb744b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.995+0000 7f2893e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb744600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.995+0000 7f2893e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb744600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49786/0 (socket says 172.21.15.102:49786) 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.995+0000 7f2899a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb744b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.995+0000 7f2899a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb744b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59510/0 (socket says 172.21.15.102:59510) 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.995+0000 7f2893e00640 1 -- 172.21.15.102:0/2817187348 learned_addr learned my addr 172.21.15.102:0/2817187348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.995+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f28941b1f60 con 0x55d8cb735570 2024-04-25T01:56:31.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f28941b33d0 con 0x55d8cb740560 2024-04-25T01:56:31.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f28941b2630 con 0x55d8cb735e90 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f2899a00640 1 -- 172.21.15.102:0/2817187348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 msgr2=0x55d8cb744600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f2899a00640 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb744600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f2899a00640 1 -- 172.21.15.102:0/2817187348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 msgr2=0x55d8cb7440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f2899a00640 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb7440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f2899a00640 1 -- 172.21.15.102:0/2817187348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8cb7b59b0 con 0x55d8cb740560 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f289e80b740 10 client.0 _ll_register_callbacks cb 0x7f28800098a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2888099980 con 0x55d8cb740560 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.996+0000 7f2893e00640 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb744600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.997+0000 7f287aa00640 10 client.25700 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.997+0000 7f2899a00640 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb744b40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f288c054040 tx=0x7f288c007df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=312dc430229ffb9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.997+0000 7f287aa00640 1 -- 172.21.15.102:0/2817187348 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c00a4f0 con 0x55d8cb740560 2024-04-25T01:56:32.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.998+0000 7f287aa00640 1 -- 172.21.15.102:0/2817187348 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f288c060020 con 0x55d8cb740560 2024-04-25T01:56:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.998+0000 7f287aa00640 1 -- 172.21.15.102:0/2817187348 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f288c00a4f0 con 0x55d8cb740560 2024-04-25T01:56:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.998+0000 7f289e80b740 10 client.25700 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:32.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.998+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55d8cb7b6bf0 con 0x55d8cb740560 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.999+0000 7f287aa00640 1 -- 172.21.15.102:0/2817187348 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f288c032850 con 0x55d8cb740560 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.999+0000 7f289e80b740 10 client.25700 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.999+0000 7f289e80b740 10 client.25700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.999+0000 7f289e80b740 20 client.25700 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.999+0000 7f289e80b740 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d8cb7b70c0 0x55d8cb7b9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.999+0000 7f289e80b740 4 client.25700 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.999+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d8cb698a20 con 0x55d8cb7b70c0 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:31.999+0000 7f2899000640 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d8cb7b70c0 0x55d8cb7b9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.000+0000 7f2899000640 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d8cb7b70c0 0x55d8cb7b9500 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48d02e0893bcf93e server_cookie=966606ecaf0bad53 in_seq=0 out_seq=0 2024-04-25T01:56:32.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.000+0000 7f287aa00640 10 client.25700 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.001+0000 7f287aa00640 1 -- 172.21.15.102:0/2817187348 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d8cb698a20 con 0x55d8cb7b70c0 2024-04-25T01:56:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.001+0000 7f287aa00640 10 client.25700 handle_command_reply: tid=0 2024-04-25T01:56:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.001+0000 7f289e80b740 1 client.25700 shutdown 2024-04-25T01:56:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.002+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 msgr2=0x55d8cb744b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.002+0000 7f289e80b740 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb744b40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f288c054040 tx=0x7f288c007df0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.003+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 shutdown_connections 2024-04-25T01:56:32.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.003+0000 7f289e80b740 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d8cb7b70c0 0x55d8cb7b9500 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.003+0000 7f289e80b740 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d8cb740560 0x55d8cb744b40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.003+0000 7f289e80b740 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8cb735e90 0x55d8cb744600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.003+0000 7f289e80b740 1 --2- 172.21.15.102:0/2817187348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d8cb735570 0x55d8cb7440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.003+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 >> 172.21.15.102:0/2817187348 conn(0x55d8cb731890 msgr2=0x55d8cb7281e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.003+0000 7f287aa00640 0 client.25700 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.003+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 shutdown_connections 2024-04-25T01:56:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.003+0000 7f289e80b740 1 -- 172.21.15.102:0/2817187348 wait complete. 2024-04-25T01:56:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.004+0000 7f289e80b740 20 client.25700 trim_cache size 0 max 16384 2024-04-25T01:56:32.008 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:56:32.008 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T01:56:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "3dfcb22e-7cb2-499e-9ba9-7e213f121726", 2024-04-25T01:56:32.009 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T01:56:32.009 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:56:32.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.008+0000 7f289ba00640 1 -- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f287c0a2fa0 msgr2=0x7f287c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.008+0000 7f289ba00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f287c0a2fa0 0x7f287c0a5460 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2894072e80 tx=0x7f2880005f90 comp rx=0 tx=0).stop 2024-04-25T01:56:32.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.008+0000 7f289ba00640 1 -- 172.21.15.102:0/3932586726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28941089b0 msgr2=0x7f289419e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.008+0000 7f289ba00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28941089b0 0x7f289419e5b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f288800c970 tx=0x7f288800ce30 comp rx=0 tx=0).stop 2024-04-25T01:56:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.009+0000 7f289ba00640 1 -- 172.21.15.102:0/3932586726 shutdown_connections 2024-04-25T01:56:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.009+0000 7f289ba00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f287c0a2fa0 0x7f287c0a5460 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.009+0000 7f289ba00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28941a2e70 0x7f28941a3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.009+0000 7f289ba00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f289410d470 0x7f289419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.009+0000 7f289ba00640 1 --2- 172.21.15.102:0/3932586726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f28941089b0 0x7f289419e5b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.009+0000 7f289ba00640 1 -- 172.21.15.102:0/3932586726 >> 172.21.15.102:0/3932586726 conn(0x7f289406c850 msgr2=0x7f28940714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.009+0000 7f289ba00640 1 -- 172.21.15.102:0/3932586726 shutdown_connections 2024-04-25T01:56:32.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.009+0000 7f289ba00640 1 -- 172.21.15.102:0/3932586726 wait complete. 2024-04-25T01:56:32.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:56:32.037 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:56:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.122+0000 7f963ce00640 1 -- 172.21.15.102:0/1046893376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9638112590 msgr2=0x7f9638114980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.124 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.122+0000 7f963ce00640 1 --2- 172.21.15.102:0/1046893376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9638112590 0x7f9638114980 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9630009ca0 tx=0x7f963002a980 comp rx=0 tx=0).stop 2024-04-25T01:56:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.123+0000 7f963ce00640 1 -- 172.21.15.102:0/1046893376 shutdown_connections 2024-04-25T01:56:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.123+0000 7f963ce00640 1 --2- 172.21.15.102:0/1046893376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9638112590 0x7f9638114980 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.123+0000 7f963ce00640 1 --2- 172.21.15.102:0/1046893376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638074400 0x7f9638112050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.123+0000 7f963ce00640 1 --2- 172.21.15.102:0/1046893376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9638073ae0 0x7f9638073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.123+0000 7f963ce00640 1 -- 172.21.15.102:0/1046893376 >> 172.21.15.102:0/1046893376 conn(0x7f96380fd090 msgr2=0x7f96380ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.123+0000 7f963ce00640 1 -- 172.21.15.102:0/1046893376 shutdown_connections 2024-04-25T01:56:32.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.123+0000 7f963ce00640 1 -- 172.21.15.102:0/1046893376 wait complete. 2024-04-25T01:56:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.125+0000 7f963ce00640 1 Processor -- start 2024-04-25T01:56:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.126+0000 7f963ce00640 1 -- start start 2024-04-25T01:56:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.126+0000 7f963ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9638073ae0 0x7f963819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.126+0000 7f963ce00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9638074400 0x7f963819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.126+0000 7f963ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638112590 0x7f96381a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.126+0000 7f963ce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9638118070 con 0x7f9638073ae0 2024-04-25T01:56:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f963ce00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9638117ef0 con 0x7f9638074400 2024-04-25T01:56:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f963ce00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f96381181f0 con 0x7f9638112590 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9637e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638112590 0x7f96381a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9637e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638112590 0x7f96381a2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49794/0 (socket says 172.21.15.102:49794) 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9637e00640 1 -- 172.21.15.102:0/3980268342 learned_addr learned my addr 172.21.15.102:0/3980268342 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9636a00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9638074400 0x7f963819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9637400640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9638073ae0 0x7f963819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9637e00640 1 -- 172.21.15.102:0/3980268342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9638074400 msgr2=0x7f963819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9637e00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9638074400 0x7f963819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9637e00640 1 -- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9638073ae0 msgr2=0x7f963819e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9637e00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9638073ae0 0x7f963819e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.127+0000 7f9637e00640 1 -- 172.21.15.102:0/3980268342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96381a3520 con 0x7f9638112590 2024-04-25T01:56:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.128+0000 7f9637e00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638112590 0x7f96381a2e20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9630006010 tx=0x7f9630030930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.128+0000 7f962fe00640 1 -- 172.21.15.102:0/3980268342 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630047070 con 0x7f9638112590 2024-04-25T01:56:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.128+0000 7f962fe00640 1 -- 172.21.15.102:0/3980268342 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9630004040 con 0x7f9638112590 2024-04-25T01:56:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.128+0000 7f963ce00640 1 -- 172.21.15.102:0/3980268342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96381a37b0 con 0x7f9638112590 2024-04-25T01:56:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.129+0000 7f962fe00640 1 -- 172.21.15.102:0/3980268342 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630034070 con 0x7f9638112590 2024-04-25T01:56:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.129+0000 7f963ce00640 1 -- 172.21.15.102:0/3980268342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f963806b160 con 0x7f9638112590 2024-04-25T01:56:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.136+0000 7f96413dd740 1 Processor -- start 2024-04-25T01:56:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.137+0000 7f962fe00640 1 -- 172.21.15.102:0/3980268342 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f963004a070 con 0x7f9638112590 2024-04-25T01:56:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.139+0000 7f96413dd740 1 -- start start 2024-04-25T01:56:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.140+0000 7f962fe00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f961c0a2f10 0x7f961c0a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.140+0000 7f96413dd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.140+0000 7f962fe00640 1 -- 172.21.15.102:0/3980268342 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f96300f05e0 con 0x7f9638112590 2024-04-25T01:56:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.141+0000 7f96413dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.141+0000 7f96413dd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 0x55ce9e8e1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.142+0000 7f96413dd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e837ab0 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.142+0000 7f96413dd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e7e0640 con 0x55ce9e8df190 2024-04-25T01:56:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.142+0000 7f96413dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e833a70 con 0x55ce9e8d48d0 2024-04-25T01:56:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.142+0000 7f9636a00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f961c0a2f10 0x7f961c0a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.142+0000 7f9637400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.143+0000 7f9637400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50584/0 (socket says 172.21.15.102:50584) 2024-04-25T01:56:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.143+0000 7f9637400640 1 -- 172.21.15.102:0/1526551300 learned_addr learned my addr 172.21.15.102:0/1526551300 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.143+0000 7f9637e00640 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 0x55ce9e8e1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.144+0000 7f9637e00640 1 -- 172.21.15.102:0/1526551300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 msgr2=0x55ce9e8dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.144+0000 7f9636a00640 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.144+0000 7f9637e00640 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.144+0000 7f9637e00640 1 -- 172.21.15.102:0/1526551300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 msgr2=0x55ce9e8d4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.145+0000 7f9637e00640 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.145+0000 7f9637e00640 1 -- 172.21.15.102:0/1526551300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce9e54c210 con 0x55ce9e8df190 2024-04-25T01:56:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.145+0000 7f9636a00640 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.145+0000 7f9637400640 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:32.148 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.146+0000 7f9637e00640 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 0x55ce9e8e1580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f96300edb80 tx=0x7f96300bcf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b0d05f2fbcf6ea4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.149 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.147+0000 7f9636a00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f961c0a2f10 0x7f961c0a53d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55ce9e8d5550 tx=0x7f96240092b0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.150+0000 7f962cc00640 1 -- 172.21.15.102:0/1526551300 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96300d5070 con 0x55ce9e8df190 2024-04-25T01:56:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.150+0000 7f962cc00640 1 -- 172.21.15.102:0/1526551300 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96300c54f0 con 0x55ce9e8df190 2024-04-25T01:56:32.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.150+0000 7f962cc00640 1 -- 172.21.15.102:0/1526551300 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96300c52c0 con 0x55ce9e8df190 2024-04-25T01:56:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.152+0000 7f96413dd740 1 -- 172.21.15.102:0/1526551300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 msgr2=0x55ce9e8e1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.153+0000 7f96413dd740 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 0x55ce9e8e1580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f96300edb80 tx=0x7f96300bcf80 comp rx=0 tx=0).stop 2024-04-25T01:56:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.153+0000 7f96413dd740 1 -- 172.21.15.102:0/1526551300 shutdown_connections 2024-04-25T01:56:32.155 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.154+0000 7f96413dd740 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 0x55ce9e8e1580 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.154+0000 7f96413dd740 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.154+0000 7f96413dd740 1 --2- 172.21.15.102:0/1526551300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.156 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.154+0000 7f96413dd740 1 -- 172.21.15.102:0/1526551300 >> 172.21.15.102:0/1526551300 conn(0x55ce9e8d04c0 msgr2=0x55ce9e8d08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.155+0000 7f96413dd740 1 -- 172.21.15.102:0/1526551300 shutdown_connections 2024-04-25T01:56:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.155+0000 7f96413dd740 1 -- 172.21.15.102:0/1526551300 wait complete. 2024-04-25T01:56:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.157+0000 7f96413dd740 1 -- start start 2024-04-25T01:56:32.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.157+0000 7f96413dd740 1 Processor -- start 2024-04-25T01:56:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.158+0000 7f96413dd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8dc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.160 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.158+0000 7f96413dd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.158+0000 7f96413dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 0x55ce9e8dccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.158+0000 7f96413dd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e7e0640 con 0x55ce9e8d48d0 2024-04-25T01:56:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.158+0000 7f96413dd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e7f3990 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.158+0000 7f96413dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ce9e8e1b80 con 0x55ce9e8df190 2024-04-25T01:56:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.158+0000 7f96413dd740 10 client.0 _ll_register_callbacks cb 0x55ce9e7f5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.160+0000 7f9636a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.160+0000 7f9636a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50600/0 (socket says 172.21.15.102:50600) 2024-04-25T01:56:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.161+0000 7f9637400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8dc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.161+0000 7f9636a00640 1 -- 172.21.15.102:0/3472332591 learned_addr learned my addr 172.21.15.102:0/3472332591 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.161+0000 7f9637e00640 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 0x55ce9e8dccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.162+0000 7f9637e00640 1 -- 172.21.15.102:0/3472332591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 msgr2=0x55ce9e8dc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.162+0000 7f9637e00640 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8dc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.162+0000 7f9637e00640 1 -- 172.21.15.102:0/3472332591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 msgr2=0x55ce9e8dc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.162+0000 7f9637e00640 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.162+0000 7f9637e00640 1 -- 172.21.15.102:0/3472332591 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce9e618620 con 0x55ce9e8df190 2024-04-25T01:56:32.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.162+0000 7f9636a00640 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-25T01:56:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.165+0000 7f9637e00640 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 0x55ce9e8dccc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f96300bc580 tx=0x7f9630005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4df4d5bad258961 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.165+0000 7f9621600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:56:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.169+0000 7f9621600640 1 -- 172.21.15.102:0/3472332591 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96300d5050 con 0x55ce9e8df190 2024-04-25T01:56:32.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.169+0000 7f9621600640 1 -- 172.21.15.102:0/3472332591 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f96300c5270 con 0x55ce9e8df190 2024-04-25T01:56:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.171+0000 7f96413dd740 1 -- 172.21.15.102:0/3472332591 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96300c54f0 con 0x55ce9e8df190 2024-04-25T01:56:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.171+0000 7f9621600640 1 -- 172.21.15.102:0/3472332591 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f96300a0050 con 0x55ce9e8df190 2024-04-25T01:56:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.172+0000 7f96413dd740 10 client.25690 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.172+0000 7f96413dd740 1 -- 172.21.15.102:0/3472332591 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce9e8ed6d0 con 0x55ce9e8df190 2024-04-25T01:56:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.173+0000 7f9621600640 1 -- 172.21.15.102:0/3472332591 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96300c54f0 con 0x55ce9e8df190 2024-04-25T01:56:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.173+0000 7f9621600640 1 -- 172.21.15.102:0/3472332591 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9630041030 con 0x55ce9e8df190 2024-04-25T01:56:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.173+0000 7f96413dd740 10 client.25690 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.173+0000 7f96413dd740 10 client.25690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.174+0000 7f96413dd740 20 client.25690 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.174+0000 7f96413dd740 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ce9e8ee850 0x55ce9e8f0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.174+0000 7f96413dd740 4 client.25690 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.174+0000 7f96413dd740 1 -- 172.21.15.102:0/3472332591 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce9e844ff0 con 0x55ce9e8ee850 2024-04-25T01:56:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.174+0000 7f9637400640 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ce9e8ee850 0x55ce9e8f0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.176+0000 7f9621600640 10 client.25690 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.176+0000 7f9637400640 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ce9e8ee850 0x55ce9e8f0c40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68b663c599c89b3e server_cookie=a0b54e02bb568642 in_seq=0 out_seq=0 2024-04-25T01:56:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.177+0000 7f9621600640 1 -- 172.21.15.102:0/3472332591 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce9e844ff0 con 0x55ce9e8ee850 2024-04-25T01:56:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.177+0000 7f9621600640 10 client.25690 handle_command_reply: tid=0 2024-04-25T01:56:32.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.177+0000 7f96413dd740 1 client.25690 shutdown 2024-04-25T01:56:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.178+0000 7f96413dd740 1 -- 172.21.15.102:0/3472332591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 msgr2=0x55ce9e8dccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.179+0000 7f96413dd740 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 0x55ce9e8dccc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f96300bc580 tx=0x7f9630005ef0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.179+0000 7f96413dd740 1 -- 172.21.15.102:0/3472332591 shutdown_connections 2024-04-25T01:56:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.179+0000 7f96413dd740 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ce9e8ee850 0x55ce9e8f0c40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.179+0000 7f96413dd740 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 0x55ce9e8dccc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.179+0000 7f96413dd740 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8dc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.179+0000 7f9621600640 0 client.25690 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.179+0000 7f96413dd740 1 --2- 172.21.15.102:0/3472332591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8dc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.179+0000 7f96413dd740 1 -- 172.21.15.102:0/3472332591 >> 172.21.15.102:0/3472332591 conn(0x55ce9e8d04c0 msgr2=0x55ce9e8d6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.180+0000 7f96413dd740 1 -- 172.21.15.102:0/3472332591 shutdown_connections 2024-04-25T01:56:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.180+0000 7f96413dd740 1 -- 172.21.15.102:0/3472332591 wait complete. 2024-04-25T01:56:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.180+0000 7f96413dd740 20 client.25690 trim_cache size 0 max 16384 2024-04-25T01:56:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.214+0000 7f96413dd740 1 Processor -- start 2024-04-25T01:56:32.216 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.214+0000 7f96413dd740 1 -- start start 2024-04-25T01:56:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.215+0000 7f96413dd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8ea810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.215+0000 7f96413dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.215+0000 7f96413dd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 0x55ce9e8f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.215+0000 7f96413dd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f96381beaf0 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.215+0000 7f96413dd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f96381bc360 con 0x55ce9e8df190 2024-04-25T01:56:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.215+0000 7f96413dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9638071a40 con 0x55ce9e8d48d0 2024-04-25T01:56:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.215+0000 7f9636a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.215+0000 7f9636a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49824/0 (socket says 172.21.15.102:49824) 2024-04-25T01:56:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.215+0000 7f9636a00640 1 -- 172.21.15.102:0/4035574871 learned_addr learned my addr 172.21.15.102:0/4035574871 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.218 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.216+0000 7f9637e00640 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 0x55ce9e8f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.216+0000 7f9637400640 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.217+0000 7f9636a00640 1 -- 172.21.15.102:0/4035574871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 msgr2=0x55ce9e8f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.217+0000 7f9636a00640 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 0x55ce9e8f64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.217+0000 7f9636a00640 1 -- 172.21.15.102:0/4035574871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 msgr2=0x55ce9e8ea810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.217+0000 7f9636a00640 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8ea810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.217+0000 7f9636a00640 1 -- 172.21.15.102:0/4035574871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce9e8e1580 con 0x55ce9e8d48d0 2024-04-25T01:56:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.218+0000 7f9637400640 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.219+0000 7f9636a00640 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d60c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9624011c80 tx=0x7f962400e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c006b8cef28ee31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.220+0000 7f9623e00640 1 -- 172.21.15.102:0/4035574871 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9624020610 con 0x55ce9e8d48d0 2024-04-25T01:56:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.220+0000 7f9623e00640 1 -- 172.21.15.102:0/4035574871 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9624025070 con 0x55ce9e8d48d0 2024-04-25T01:56:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.220+0000 7f9623e00640 1 -- 172.21.15.102:0/4035574871 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962401f9d0 con 0x55ce9e8d48d0 2024-04-25T01:56:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.221+0000 7f96413dd740 1 -- 172.21.15.102:0/4035574871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 msgr2=0x55ce9e8d60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.221+0000 7f96413dd740 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d60c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9624011c80 tx=0x7f962400e000 comp rx=0 tx=0).stop 2024-04-25T01:56:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.222+0000 7f96413dd740 1 -- 172.21.15.102:0/4035574871 shutdown_connections 2024-04-25T01:56:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.222+0000 7f96413dd740 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8df190 0x55ce9e8f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.222+0000 7f96413dd740 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d60c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.222+0000 7f96413dd740 1 --2- 172.21.15.102:0/4035574871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8ea810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.222+0000 7f96413dd740 1 -- 172.21.15.102:0/4035574871 >> 172.21.15.102:0/4035574871 conn(0x55ce9e8d04c0 msgr2=0x55ce9e8d59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.222+0000 7f96413dd740 1 -- 172.21.15.102:0/4035574871 shutdown_connections 2024-04-25T01:56:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.223+0000 7f96413dd740 1 -- 172.21.15.102:0/4035574871 wait complete. 2024-04-25T01:56:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.223+0000 7f96413dd740 1 -- start start 2024-04-25T01:56:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.224+0000 7f96413dd740 1 Processor -- start 2024-04-25T01:56:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.225+0000 7f96413dd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.225+0000 7f96413dd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.225+0000 7f96413dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 0x55ce9e8d99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.225+0000 7f9637400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.225+0000 7f9637400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59542/0 (socket says 172.21.15.102:59542) 2024-04-25T01:56:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.225+0000 7f96413dd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f96380720b0 con 0x55ce9e8d48d0 2024-04-25T01:56:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.225+0000 7f96413dd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9638073520 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.225+0000 7f96413dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9638072780 con 0x55ce9e8df190 2024-04-25T01:56:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.225+0000 7f9637400640 1 -- 172.21.15.102:0/2957769367 learned_addr learned my addr 172.21.15.102:0/2957769367 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f9637e00640 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 0x55ce9e8d99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f9636a00640 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f96413dd740 10 client.0 _ll_register_callbacks cb 0x7f96300d3070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f9637400640 1 -- 172.21.15.102:0/2957769367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 msgr2=0x55ce9e8d99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f9637400640 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 0x55ce9e8d99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f9637400640 1 -- 172.21.15.102:0/2957769367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 msgr2=0x55ce9e8d9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f9637400640 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f9637400640 1 -- 172.21.15.102:0/2957769367 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce9e953d10 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f96413dd740 1 -- 172.21.15.102:0/2957769367 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9624025070 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f9637400640 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d8f30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f962800da10 tx=0x7f962800d530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c776f1ef8632dad2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.226+0000 7f9621600640 10 client.25715 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.227+0000 7f9621600640 1 -- 172.21.15.102:0/2957769367 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962802a050 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.227+0000 7f9621600640 1 -- 172.21.15.102:0/2957769367 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f962801c1d0 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.227+0000 7f9621600640 1 -- 172.21.15.102:0/2957769367 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f962801c3d0 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.227+0000 7f96413dd740 10 client.25715 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.227+0000 7f96413dd740 1 -- 172.21.15.102:0/2957769367 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55ce9e954ef0 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.228+0000 7f9621600640 1 -- 172.21.15.102:0/2957769367 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f962801c570 con 0x55ce9e8d3fb0 2024-04-25T01:56:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.228+0000 7f96413dd740 10 client.25715 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.228+0000 7f96413dd740 10 client.25715 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.228+0000 7f96413dd740 20 client.25715 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.228+0000 7f96413dd740 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ce9e9555e0 0x55ce9e957a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.228+0000 7f96413dd740 4 client.25715 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:56:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.228+0000 7f96413dd740 1 -- 172.21.15.102:0/2957769367 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ce9e83f340 con 0x55ce9e9555e0 2024-04-25T01:56:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.228+0000 7f9636a00640 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ce9e9555e0 0x55ce9e957a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.229+0000 7f9621600640 10 client.25715 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.229+0000 7f9636a00640 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ce9e9555e0 0x55ce9e957a00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5860080aafe1556e server_cookie=7a2f983e2cabc325 in_seq=0 out_seq=0 2024-04-25T01:56:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.271+0000 7f9621600640 1 -- 172.21.15.102:0/2957769367 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ce9e83f340 con 0x55ce9e9555e0 2024-04-25T01:56:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.271+0000 7f9621600640 10 client.25715 handle_command_reply: tid=0 2024-04-25T01:56:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.271+0000 7f96413dd740 1 client.25715 shutdown 2024-04-25T01:56:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.272+0000 7f96413dd740 1 -- 172.21.15.102:0/2957769367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 msgr2=0x55ce9e8d8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.272+0000 7f96413dd740 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d8f30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f962800da10 tx=0x7f962800d530 comp rx=0 tx=0).stop 2024-04-25T01:56:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.272+0000 7f96413dd740 1 -- 172.21.15.102:0/2957769367 shutdown_connections 2024-04-25T01:56:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.272+0000 7f96413dd740 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ce9e9555e0 0x55ce9e957a00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.272+0000 7f96413dd740 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce9e8df190 0x55ce9e8d99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.272+0000 7f96413dd740 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce9e8d48d0 0x55ce9e8d9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.272+0000 7f96413dd740 1 --2- 172.21.15.102:0/2957769367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ce9e8d3fb0 0x55ce9e8d8f30 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.272+0000 7f96413dd740 1 -- 172.21.15.102:0/2957769367 >> 172.21.15.102:0/2957769367 conn(0x55ce9e8d04c0 msgr2=0x55ce9e8dca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.272+0000 7f9621600640 0 client.25715 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.273+0000 7f96413dd740 1 -- 172.21.15.102:0/2957769367 shutdown_connections 2024-04-25T01:56:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.273+0000 7f96413dd740 1 -- 172.21.15.102:0/2957769367 wait complete. 2024-04-25T01:56:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.273+0000 7f96413dd740 20 client.25715 trim_cache size 0 max 16384 2024-04-25T01:56:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:56:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:56:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:56:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:56:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.275+0000 7f963ce00640 1 -- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f961c0a2f10 msgr2=0x7f961c0a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.275+0000 7f963ce00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f961c0a2f10 0x7f961c0a53d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x55ce9e8d5550 tx=0x7f96240092b0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.275+0000 7f963ce00640 1 -- 172.21.15.102:0/3980268342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638112590 msgr2=0x7f96381a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.275+0000 7f963ce00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638112590 0x7f96381a2e20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9630006010 tx=0x7f9630030930 comp rx=0 tx=0).stop 2024-04-25T01:56:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.276+0000 7f963ce00640 1 -- 172.21.15.102:0/3980268342 shutdown_connections 2024-04-25T01:56:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.276+0000 7f963ce00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f961c0a2f10 0x7f961c0a53d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.276+0000 7f963ce00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638112590 0x7f96381a2e20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.276+0000 7f963ce00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9638074400 0x7f963819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.276+0000 7f963ce00640 1 --2- 172.21.15.102:0/3980268342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9638073ae0 0x7f963819e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.276+0000 7f963ce00640 1 -- 172.21.15.102:0/3980268342 >> 172.21.15.102:0/3980268342 conn(0x7f96380fd090 msgr2=0x7f96380fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.280+0000 7f963ce00640 1 -- 172.21.15.102:0/3980268342 shutdown_connections 2024-04-25T01:56:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:32.280+0000 7f963ce00640 1 -- 172.21.15.102:0/3980268342 wait complete. 2024-04-25T01:56:32.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.372+0000 7fdce4a00640 1 -- 172.21.15.102:0/2832624634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce0100d30 msgr2=0x7fdce010d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.372+0000 7fdce4a00640 1 --2- 172.21.15.102:0/2832624634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce0100d30 0x7fdce010d9f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009860 tx=0x7fdccc02a340 comp rx=0 tx=0).stop 2024-04-25T01:56:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.373+0000 7fdce4a00640 1 -- 172.21.15.102:0/2832624634 shutdown_connections 2024-04-25T01:56:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.373+0000 7fdce4a00640 1 --2- 172.21.15.102:0/2832624634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce010df30 0x7fdce0110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.373+0000 7fdce4a00640 1 --2- 172.21.15.102:0/2832624634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce0100d30 0x7fdce010d9f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.373+0000 7fdce4a00640 1 --2- 172.21.15.102:0/2832624634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdce0100380 0x7fdce0100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.373+0000 7fdce4a00640 1 -- 172.21.15.102:0/2832624634 >> 172.21.15.102:0/2832624634 conn(0x7fdce00fbfc0 msgr2=0x7fdce00fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.373+0000 7fdce4a00640 1 -- 172.21.15.102:0/2832624634 shutdown_connections 2024-04-25T01:56:32.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.373+0000 7fdce4a00640 1 -- 172.21.15.102:0/2832624634 wait complete. 2024-04-25T01:56:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.374+0000 7fdce4a00640 1 Processor -- start 2024-04-25T01:56:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.375+0000 7fdce4a00640 1 -- start start 2024-04-25T01:56:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.375+0000 7fdce4a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce0100380 0x7fdce010d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.375+0000 7fdce4a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0100d30 0x7fdce010a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.375+0000 7fdce4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdce010df30 0x7fdce010afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.375+0000 7fdce4a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdce0113c30 con 0x7fdce010df30 2024-04-25T01:56:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.375+0000 7fdce4a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdce0113ab0 con 0x7fdce0100d30 2024-04-25T01:56:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.375+0000 7fdce4a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdce0113db0 con 0x7fdce0100380 2024-04-25T01:56:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0100d30 0x7fdce010a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0100d30 0x7fdce010a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59564/0 (socket says 172.21.15.102:59564) 2024-04-25T01:56:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdde00640 1 -- 172.21.15.102:0/303810776 learned_addr learned my addr 172.21.15.102:0/303810776 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcde800640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce0100380 0x7fdce010d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdf200640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdce010df30 0x7fdce010afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdde00640 1 -- 172.21.15.102:0/303810776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce0100380 msgr2=0x7fdce010d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdde00640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce0100380 0x7fdce010d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdde00640 1 -- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdce010df30 msgr2=0x7fdce010afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdde00640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdce010df30 0x7fdce010afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdde00640 1 -- 172.21.15.102:0/303810776 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce01aa950 con 0x7fdce0100d30 2024-04-25T01:56:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.376+0000 7fdcdde00640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0100d30 0x7fdce010a9e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdccc02a8b0 tx=0x7fdccc033d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.377+0000 7fdcd7400640 1 -- 172.21.15.102:0/303810776 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc037040 con 0x7fdce0100d30 2024-04-25T01:56:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.377+0000 7fdce4a00640 1 -- 172.21.15.102:0/303810776 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce01aab20 con 0x7fdce0100d30 2024-04-25T01:56:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.377+0000 7fdce4a00640 1 -- 172.21.15.102:0/303810776 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce01aafe0 con 0x7fdce0100d30 2024-04-25T01:56:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.378+0000 7fdcd7400640 1 -- 172.21.15.102:0/303810776 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdccc030550 con 0x7fdce0100d30 2024-04-25T01:56:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.378+0000 7fdcd7400640 1 -- 172.21.15.102:0/303810776 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc003850 con 0x7fdce0100d30 2024-04-25T01:56:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.382+0000 7fdcd7400640 1 -- 172.21.15.102:0/303810776 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7fdccc042030 con 0x7fdce0100d30 2024-04-25T01:56:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.383+0000 7fdce816c740 1 Processor -- start 2024-04-25T01:56:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.383+0000 7fdce816c740 1 -- start start 2024-04-25T01:56:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.383+0000 7fdce816c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b1f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.383+0000 7fdce816c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b201c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.383+0000 7fdce816c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 0x55ba8b2045c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.383+0000 7fdce816c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ba8b15b560 con 0x55ba8b2021d0 2024-04-25T01:56:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.383+0000 7fdce816c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ba8b103930 con 0x55ba8b1f7b00 2024-04-25T01:56:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.383+0000 7fdce816c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ba8b157520 con 0x55ba8b1f71e0 2024-04-25T01:56:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.384+0000 7fdcde800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b201c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.384+0000 7fdcde800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b201c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59568/0 (socket says 172.21.15.102:59568) 2024-04-25T01:56:32.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.384+0000 7fdcde800640 1 -- 172.21.15.102:0/873452558 learned_addr learned my addr 172.21.15.102:0/873452558 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.384+0000 7fdcd7400640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdcc40a2ef0 0x7fdcc40a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.384+0000 7fdcde800640 1 -- 172.21.15.102:0/873452558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 msgr2=0x55ba8b1f75c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:56:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.384+0000 7fdcde800640 1 --2- 172.21.15.102:0/873452558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b1f75c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.384+0000 7fdcde800640 1 -- 172.21.15.102:0/873452558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 msgr2=0x55ba8b2045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.384+0000 7fdcde800640 1 --2- 172.21.15.102:0/873452558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 0x55ba8b2045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.384+0000 7fdcde800640 1 -- 172.21.15.102:0/873452558 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba8af397d0 con 0x55ba8b1f7b00 2024-04-25T01:56:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.385+0000 7fdcd7400640 1 -- 172.21.15.102:0/303810776 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fdccc0ee7b0 con 0x7fdce0100d30 2024-04-25T01:56:32.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.385+0000 7fdcde800640 1 --2- 172.21.15.102:0/873452558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b201c90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00103f0 tx=0x7fdcd00108b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9532a95b0d50e9e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.385+0000 7fdccb400640 1 -- 172.21.15.102:0/873452558 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0016070 con 0x55ba8b1f7b00 2024-04-25T01:56:32.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.385+0000 7fdccb400640 1 -- 172.21.15.102:0/873452558 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcd0004d20 con 0x55ba8b1f7b00 2024-04-25T01:56:32.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.385+0000 7fdccb400640 1 -- 172.21.15.102:0/873452558 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0016070 con 0x55ba8b1f7b00 2024-04-25T01:56:32.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.386+0000 7fdce816c740 1 -- 172.21.15.102:0/873452558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 msgr2=0x55ba8b201c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.386+0000 7fdce816c740 1 --2- 172.21.15.102:0/873452558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b201c90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00103f0 tx=0x7fdcd00108b0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.386+0000 7fdcdf200640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdcc40a2ef0 0x7fdcc40a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.389+0000 7fdcdf200640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdcc40a2ef0 0x7fdcc40a53b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdce0102560 tx=0x7fdcd8009260 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.389+0000 7fdce816c740 1 -- 172.21.15.102:0/873452558 shutdown_connections 2024-04-25T01:56:32.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.389+0000 7fdce816c740 1 --2- 172.21.15.102:0/873452558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 0x55ba8b2045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.389+0000 7fdce816c740 1 --2- 172.21.15.102:0/873452558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b201c90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.389+0000 7fdce816c740 1 --2- 172.21.15.102:0/873452558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b1f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.389+0000 7fdce816c740 1 -- 172.21.15.102:0/873452558 >> 172.21.15.102:0/873452558 conn(0x55ba8b1f3500 msgr2=0x55ba8b1f38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.390+0000 7fdce816c740 1 -- 172.21.15.102:0/873452558 shutdown_connections 2024-04-25T01:56:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.390+0000 7fdce816c740 1 -- 172.21.15.102:0/873452558 wait complete. 2024-04-25T01:56:32.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.391+0000 7fdce816c740 1 -- start start 2024-04-25T01:56:32.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.391+0000 7fdce816c740 1 Processor -- start 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.392+0000 7fdce816c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f71e0 0x55ba8b1ff080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.392+0000 7fdce816c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1ff5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.392+0000 7fdce816c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b2021d0 0x55ba8b1ffb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.392+0000 7fdce816c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ba8b204fc0 con 0x55ba8b1f7b00 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.392+0000 7fdce816c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ba8b116d90 con 0x55ba8b1f71e0 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.392+0000 7fdce816c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ba8b205140 con 0x55ba8b2021d0 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.392+0000 7fdce816c740 10 client.0 _ll_register_callbacks cb 0x55ba8b116780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcde800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f71e0 0x55ba8b1ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcde800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f71e0 0x55ba8b1ff080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59570/0 (socket says 172.21.15.102:59570) 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcde800640 1 -- 172.21.15.102:0/1619332866 learned_addr learned my addr 172.21.15.102:0/1619332866 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcdf200640 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b2021d0 0x55ba8b1ffb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcdde00640 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1ff5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcde800640 1 -- 172.21.15.102:0/1619332866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b2021d0 msgr2=0x55ba8b1ffb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcde800640 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b2021d0 0x55ba8b1ffb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcde800640 1 -- 172.21.15.102:0/1619332866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 msgr2=0x55ba8b1ff5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcde800640 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1ff5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.393+0000 7fdcde800640 1 -- 172.21.15.102:0/1619332866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba8ae6f3f0 con 0x55ba8b1f71e0 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.394+0000 7fdcde800640 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f71e0 0x55ba8b1ff080 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0003bd0 tx=0x7fdcd0003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3609e0562ca7d73c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.394+0000 7fdcbfe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.394+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1619332866 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0016070 con 0x55ba8b1f71e0 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.394+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1619332866 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fdcd0012720 con 0x55ba8b1f71e0 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.394+0000 7fdce816c740 1 -- 172.21.15.102:0/1619332866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcd0004d20 con 0x55ba8b1f71e0 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.395+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1619332866 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdcd0009c90 con 0x55ba8b1f71e0 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.395+0000 7fdce816c740 10 client.25727 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:32.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.395+0000 7fdce816c740 1 -- 172.21.15.102:0/1619332866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ba8ae2f0f0 con 0x55ba8b1f71e0 2024-04-25T01:56:32.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.395+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1619332866 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcd0004d20 con 0x55ba8b1f71e0 2024-04-25T01:56:32.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.395+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1619332866 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdcd000f040 con 0x55ba8b1f71e0 2024-04-25T01:56:32.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.396+0000 7fdce816c740 10 client.25727 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:32.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.396+0000 7fdce816c740 10 client.25727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:32.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.396+0000 7fdce816c740 20 client.25727 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:32.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.396+0000 7fdce816c740 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ba8b2124b0 0x55ba8b2148a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.396+0000 7fdce816c740 4 client.25727 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:32.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.396+0000 7fdce816c740 1 -- 172.21.15.102:0/1619332866 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba8b16ed90 con 0x55ba8b2124b0 2024-04-25T01:56:32.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.397+0000 7fdcdde00640 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ba8b2124b0 0x55ba8b2148a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.397+0000 7fdcdde00640 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ba8b2124b0 0x55ba8b2148a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=687affcfa476d3e5 server_cookie=bb8b86439c0c6939 in_seq=0 out_seq=0 2024-04-25T01:56:32.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.397+0000 7fdcbfe00640 10 client.25727 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.398+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1619332866 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ba8b16ed90 con 0x55ba8b2124b0 2024-04-25T01:56:32.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.398+0000 7fdcbfe00640 10 client.25727 handle_command_reply: tid=0 2024-04-25T01:56:32.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.398+0000 7fdce816c740 1 client.25727 shutdown 2024-04-25T01:56:32.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.399+0000 7fdce816c740 1 -- 172.21.15.102:0/1619332866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f71e0 msgr2=0x55ba8b1ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.399+0000 7fdce816c740 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f71e0 0x55ba8b1ff080 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0003bd0 tx=0x7fdcd0003c00 comp rx=0 tx=0).stop 2024-04-25T01:56:32.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.399+0000 7fdce816c740 1 -- 172.21.15.102:0/1619332866 shutdown_connections 2024-04-25T01:56:32.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.399+0000 7fdce816c740 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ba8b2124b0 0x55ba8b2148a0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.399+0000 7fdce816c740 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b2021d0 0x55ba8b1ffb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.399+0000 7fdce816c740 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1ff5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.399+0000 7fdce816c740 1 --2- 172.21.15.102:0/1619332866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f71e0 0x55ba8b1ff080 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.399+0000 7fdce816c740 1 -- 172.21.15.102:0/1619332866 >> 172.21.15.102:0/1619332866 conn(0x55ba8b1f3500 msgr2=0x55ba8b1f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.399+0000 7fdcbfe00640 0 client.25727 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.400+0000 7fdce816c740 1 -- 172.21.15.102:0/1619332866 shutdown_connections 2024-04-25T01:56:32.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.400+0000 7fdce816c740 1 -- 172.21.15.102:0/1619332866 wait complete. 2024-04-25T01:56:32.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.400+0000 7fdce816c740 20 client.25727 trim_cache size 0 max 16384 2024-04-25T01:56:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.430+0000 7fdce816c740 1 Processor -- start 2024-04-25T01:56:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.430+0000 7fdce816c740 1 -- start start 2024-04-25T01:56:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.431+0000 7fdce816c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.431+0000 7fdce816c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1f9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.431+0000 7fdce816c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b2021d0 0x55ba8b1f2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.431+0000 7fdce816c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc00110a0 con 0x55ba8b1f7b00 2024-04-25T01:56:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.431+0000 7fdce816c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc000e910 con 0x55ba8b2021d0 2024-04-25T01:56:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.431+0000 7fdce816c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc000c720 con 0x55ba8b1f71e0 2024-04-25T01:56:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.431+0000 7fdcde800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.431+0000 7fdcde800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49882/0 (socket says 172.21.15.102:49882) 2024-04-25T01:56:32.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.431+0000 7fdcde800640 1 -- 172.21.15.102:0/379911247 learned_addr learned my addr 172.21.15.102:0/379911247 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.432+0000 7fdcde800640 1 -- 172.21.15.102:0/379911247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b2021d0 msgr2=0x55ba8b1f2680 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:56:32.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.432+0000 7fdcde800640 1 --2- 172.21.15.102:0/379911247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b2021d0 0x55ba8b1f2680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.432+0000 7fdcde800640 1 -- 172.21.15.102:0/379911247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 msgr2=0x55ba8b1f9960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:56:32.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.432+0000 7fdcdde00640 1 --2- 172.21.15.102:0/379911247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1f9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.432+0000 7fdcde800640 1 --2- 172.21.15.102:0/379911247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1f9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.432+0000 7fdcde800640 1 -- 172.21.15.102:0/379911247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba8b1fd5d0 con 0x55ba8b1f71e0 2024-04-25T01:56:32.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.432+0000 7fdcdde00640 1 --2- 172.21.15.102:0/379911247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1f9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:32.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.433+0000 7fdcde800640 1 --2- 172.21.15.102:0/379911247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20df20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0003ba0 tx=0x7fdcd0005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3d542857cef08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.433+0000 7fdccaa00640 1 -- 172.21.15.102:0/379911247 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0009c90 con 0x55ba8b1f71e0 2024-04-25T01:56:32.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.433+0000 7fdccaa00640 1 -- 172.21.15.102:0/379911247 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcd0029070 con 0x55ba8b1f71e0 2024-04-25T01:56:32.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.433+0000 7fdccaa00640 1 -- 172.21.15.102:0/379911247 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0004b10 con 0x55ba8b1f71e0 2024-04-25T01:56:32.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.434+0000 7fdce816c740 1 -- 172.21.15.102:0/379911247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 msgr2=0x55ba8b20df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.434+0000 7fdce816c740 1 --2- 172.21.15.102:0/379911247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20df20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0003ba0 tx=0x7fdcd0005f10 comp rx=0 tx=0).stop 2024-04-25T01:56:32.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.435+0000 7fdce816c740 1 -- 172.21.15.102:0/379911247 shutdown_connections 2024-04-25T01:56:32.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.435+0000 7fdce816c740 1 --2- 172.21.15.102:0/379911247 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b2021d0 0x55ba8b1f2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.435+0000 7fdce816c740 1 --2- 172.21.15.102:0/379911247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1f9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.435+0000 7fdce816c740 1 --2- 172.21.15.102:0/379911247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20df20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.435+0000 7fdce816c740 1 -- 172.21.15.102:0/379911247 >> 172.21.15.102:0/379911247 conn(0x55ba8b1f3500 msgr2=0x55ba8b1f1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.435+0000 7fdce816c740 1 -- 172.21.15.102:0/379911247 shutdown_connections 2024-04-25T01:56:32.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.435+0000 7fdce816c740 1 -- 172.21.15.102:0/379911247 wait complete. 2024-04-25T01:56:32.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.436+0000 7fdce816c740 1 -- start start 2024-04-25T01:56:32.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.437+0000 7fdce816c740 1 Processor -- start 2024-04-25T01:56:32.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.439+0000 7fdce816c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.439+0000 7fdce816c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1fb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.439+0000 7fdce816c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 0x55ba8b1fbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.439+0000 7fdce816c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc000cdd0 con 0x55ba8b2021d0 2024-04-25T01:56:32.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.439+0000 7fdce816c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc000e240 con 0x55ba8b1f7b00 2024-04-25T01:56:32.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.439+0000 7fdce816c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc000d4a0 con 0x55ba8b1f71e0 2024-04-25T01:56:32.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.439+0000 7fdce816c740 10 client.0 _ll_register_callbacks cb 0x7fdcd0004ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:32.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.441+0000 7fdcde800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.441+0000 7fdcdde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1fb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.441+0000 7fdcdde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1fb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59580/0 (socket says 172.21.15.102:59580) 2024-04-25T01:56:32.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.441+0000 7fdcdde00640 1 -- 172.21.15.102:0/1372024368 learned_addr learned my addr 172.21.15.102:0/1372024368 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:32.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.441+0000 7fdcdf200640 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 0x55ba8b1fbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.442+0000 7fdcde800640 1 -- 172.21.15.102:0/1372024368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 msgr2=0x55ba8b1fb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.442+0000 7fdcde800640 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1fb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.442+0000 7fdcde800640 1 -- 172.21.15.102:0/1372024368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 msgr2=0x55ba8b1fbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:32.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.442+0000 7fdcde800640 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 0x55ba8b1fbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.442+0000 7fdcde800640 1 -- 172.21.15.102:0/1372024368 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba8b1fc310 con 0x55ba8b1f71e0 2024-04-25T01:56:32.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.442+0000 7fdcdf200640 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 0x55ba8b1fbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:56:32.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.443+0000 7fdcdde00640 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1fb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:32.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.443+0000 7fdcde800640 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20a620 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00097f0 tx=0x7fdcd00129c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=574f89c1f07d783c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:32.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.443+0000 7fdcbfe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:56:32.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.444+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1372024368 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0004380 con 0x55ba8b1f71e0 2024-04-25T01:56:32.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.444+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1372024368 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fdcd0004520 con 0x55ba8b1f71e0 2024-04-25T01:56:32.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.445+0000 7fdce816c740 1 -- 172.21.15.102:0/1372024368 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcd0029070 con 0x55ba8b1f71e0 2024-04-25T01:56:32.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.445+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1372024368 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdcd0004720 con 0x55ba8b1f71e0 2024-04-25T01:56:32.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.445+0000 7fdce816c740 10 client.25711 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:32.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.445+0000 7fdce816c740 1 -- 172.21.15.102:0/1372024368 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ba8b278590 con 0x55ba8b1f71e0 2024-04-25T01:56:32.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.446+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1372024368 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcd0004b50 con 0x55ba8b1f71e0 2024-04-25T01:56:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.446+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1372024368 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdcd0036030 con 0x55ba8b1f71e0 2024-04-25T01:56:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.447+0000 7fdce816c740 10 client.25711 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.447+0000 7fdce816c740 10 client.25711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.447+0000 7fdce816c740 20 client.25711 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.447+0000 7fdce816c740 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ba8b278c20 0x55ba8b27b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.447+0000 7fdce816c740 4 client.25711 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:56:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.447+0000 7fdce816c740 1 -- 172.21.15.102:0/1372024368 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fdcd0029070 con 0x55ba8b278c20 2024-04-25T01:56:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.447+0000 7fdcdde00640 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ba8b278c20 0x55ba8b27b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.448+0000 7fdcdde00640 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ba8b278c20 0x55ba8b27b040 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22341dcf0e523138 server_cookie=2aaf67a418bf1a7 in_seq=0 out_seq=0 2024-04-25T01:56:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.448+0000 7fdcbfe00640 10 client.25711 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.448+0000 7fdcbfe00640 1 -- 172.21.15.102:0/1372024368 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fdcd0029070 con 0x55ba8b278c20 2024-04-25T01:56:32.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.448+0000 7fdcbfe00640 10 client.25711 handle_command_reply: tid=0 2024-04-25T01:56:32.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.448+0000 7fdce816c740 1 client.25711 shutdown 2024-04-25T01:56:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.448+0000 7fdce816c740 1 -- 172.21.15.102:0/1372024368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 msgr2=0x55ba8b20a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.448+0000 7fdce816c740 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20a620 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdcd00097f0 tx=0x7fdcd00129c0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.449+0000 7fdce816c740 1 -- 172.21.15.102:0/1372024368 shutdown_connections 2024-04-25T01:56:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.449+0000 7fdce816c740 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ba8b278c20 0x55ba8b27b040 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.449+0000 7fdce816c740 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ba8b2021d0 0x55ba8b1fbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.449+0000 7fdce816c740 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ba8b1f7b00 0x55ba8b1fb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.449+0000 7fdcbfe00640 0 client.25711 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.449+0000 7fdce816c740 1 --2- 172.21.15.102:0/1372024368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba8b1f71e0 0x55ba8b20a620 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.449+0000 7fdce816c740 1 -- 172.21.15.102:0/1372024368 >> 172.21.15.102:0/1372024368 conn(0x55ba8b1f3500 msgr2=0x55ba8b1ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.449+0000 7fdce816c740 1 -- 172.21.15.102:0/1372024368 shutdown_connections 2024-04-25T01:56:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.450+0000 7fdce816c740 1 -- 172.21.15.102:0/1372024368 wait complete. 2024-04-25T01:56:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.450+0000 7fdce816c740 20 client.25711 trim_cache size 0 max 16384 2024-04-25T01:56:32.452 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:56:32.452 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:56:32.452 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T01:56:32.453 INFO:teuthology.orchestra.run.smithi102.stdout: "3dfcb22e-7cb2-499e-9ba9-7e213f121726": { 2024-04-25T01:56:32.453 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T01:56:32.453 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "3dfcb22e-7cb2-499e-9ba9-7e213f121726", 2024-04-25T01:56:32.453 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T01:56:32.453 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:56:32.453 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:56:32.453 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 -- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdcc40a2ef0 msgr2=0x7fdcc40a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdcc40a2ef0 0x7fdcc40a53b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdce0102560 tx=0x7fdcd8009260 comp rx=0 tx=0).stop 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 -- 172.21.15.102:0/303810776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0100d30 msgr2=0x7fdce010a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0100d30 0x7fdce010a9e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdccc02a8b0 tx=0x7fdccc033d10 comp rx=0 tx=0).stop 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 -- 172.21.15.102:0/303810776 shutdown_connections 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdcc40a2ef0 0x7fdcc40a53b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdce010df30 0x7fdce010afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdce0100d30 0x7fdce010a9e0 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdccc02a8b0 tx=0x7fdccc033d10 comp rx=0 tx=0).stop 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 --2- 172.21.15.102:0/303810776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce0100380 0x7fdce010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 -- 172.21.15.102:0/303810776 >> 172.21.15.102:0/303810776 conn(0x7fdce00fbfc0 msgr2=0x7fdce00fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.451+0000 7fdce4a00640 1 -- 172.21.15.102:0/303810776 shutdown_connections 2024-04-25T01:56:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:56:32.452+0000 7fdce4a00640 1 -- 172.21.15.102:0/303810776 wait complete. 2024-04-25T01:56:32.487 INFO:tasks.cephfs.filesystem:scrub status for tag:3dfcb22e-7cb2-499e-9ba9-7e213f121726 - {'path': '/', 'tag': '3dfcb22e-7cb2-499e-9ba9-7e213f121726', 'options': 'recursive,force'} 2024-04-25T01:56:33.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 40.0 s, 3664.4 tps, lat 27.327 ms stddev 29.736 2024-04-25T01:56:33.674 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: pgmap v41: 97 pgs: 97 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 18 MiB/s wr, 971 op/s 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:56:33.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:56:33.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:33 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: pgmap v41: 97 pgs: 97 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 18 MiB/s wr, 971 op/s 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:56:33.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:56:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:56:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:56:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:33 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: pgmap v41: 97 pgs: 97 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 18 MiB/s wr, 971 op/s 2024-04-25T01:56:33.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:56:33.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:33.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:33 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:56:34.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:34 smithi102 ceph-mon[27796]: pgmap v42: 97 pgs: 97 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 MiB/s wr, 753 op/s 2024-04-25T01:56:34.739 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:34 smithi110 ceph-mon[32848]: pgmap v42: 97 pgs: 97 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 MiB/s wr, 753 op/s 2024-04-25T01:56:34.756 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:34 smithi193 ceph-mon[32981]: pgmap v42: 97 pgs: 97 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 MiB/s wr, 753 op/s 2024-04-25T01:56:36.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:36 smithi110 ceph-mon[32848]: pgmap v43: 97 pgs: 97 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 660 op/s 2024-04-25T01:56:37.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:36 smithi193 ceph-mon[32981]: pgmap v43: 97 pgs: 97 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 660 op/s 2024-04-25T01:56:37.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:36 smithi102 ceph-mon[27796]: pgmap v43: 97 pgs: 97 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 660 op/s 2024-04-25T01:56:38.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 45.0 s, 3986.0 tps, lat 25.105 ms stddev 26.077 2024-04-25T01:56:38.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:38 smithi110 ceph-mon[32848]: pgmap v44: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 17 MiB/s wr, 1.10k op/s 2024-04-25T01:56:39.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:38 smithi193 ceph-mon[32981]: pgmap v44: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 17 MiB/s wr, 1.10k op/s 2024-04-25T01:56:39.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:38 smithi102 ceph-mon[27796]: pgmap v44: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 17 MiB/s wr, 1.10k op/s 2024-04-25T01:56:40.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:40 smithi110 ceph-mon[32848]: pgmap v45: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 8.5 MiB/s wr, 761 op/s 2024-04-25T01:56:41.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:40 smithi193 ceph-mon[32981]: pgmap v45: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 8.5 MiB/s wr, 761 op/s 2024-04-25T01:56:41.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:40 smithi102 ceph-mon[27796]: pgmap v45: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 8.5 MiB/s wr, 761 op/s 2024-04-25T01:56:42.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:42 smithi110 ceph-mon[32848]: pgmap v46: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 12 MiB/s wr, 1.01k op/s 2024-04-25T01:56:42.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:42 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:56:42.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:42 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:56:42.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:42 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:56:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:42 smithi193 ceph-mon[32981]: pgmap v46: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 12 MiB/s wr, 1.01k op/s 2024-04-25T01:56:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:42 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:56:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:42 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:56:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:42 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:56:43.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 50.0 s, 3511.6 tps, lat 28.461 ms stddev 29.561 2024-04-25T01:56:43.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:42 smithi102 ceph-mon[27796]: pgmap v46: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 12 MiB/s wr, 1.01k op/s 2024-04-25T01:56:43.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:42 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:56:43.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:42 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:56:43.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:42 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:56:44.990 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:44 smithi110 ceph-mon[32848]: pgmap v47: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 8.5 MiB/s wr, 793 op/s 2024-04-25T01:56:45.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:44 smithi193 ceph-mon[32981]: pgmap v47: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 8.5 MiB/s wr, 793 op/s 2024-04-25T01:56:45.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:44 smithi102 ceph-mon[27796]: pgmap v47: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 8.5 MiB/s wr, 793 op/s 2024-04-25T01:56:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.466+0000 7f4dd0400640 1 -- 172.21.15.102:0/2508970744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 msgr2=0x7f4dc810e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.466+0000 7f4dd0400640 1 --2- 172.21.15.102:0/2508970744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 0x7f4dc810e620 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009930 tx=0x7f4dc002a420 comp rx=0 tx=0).stop 2024-04-25T01:56:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.466+0000 7f4dd0400640 1 -- 172.21.15.102:0/2508970744 shutdown_connections 2024-04-25T01:56:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.466+0000 7f4dd0400640 1 --2- 172.21.15.102:0/2508970744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 0x7f4dc810e620 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.466+0000 7f4dd0400640 1 --2- 172.21.15.102:0/2508970744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dc8101750 0x7f4dc8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.466+0000 7f4dd0400640 1 --2- 172.21.15.102:0/2508970744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4dc8107760 0x7f4dc8107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.466+0000 7f4dd0400640 1 -- 172.21.15.102:0/2508970744 >> 172.21.15.102:0/2508970744 conn(0x7f4dc80fd240 msgr2=0x7f4dc80ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.467+0000 7f4dd0400640 1 -- 172.21.15.102:0/2508970744 shutdown_connections 2024-04-25T01:56:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.467+0000 7f4dd0400640 1 -- 172.21.15.102:0/2508970744 wait complete. 2024-04-25T01:56:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.468+0000 7f4dd0400640 1 Processor -- start 2024-04-25T01:56:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dd0400640 1 -- start start 2024-04-25T01:56:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dd0400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dc8101750 0x7f4dc8199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dd0400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 0x7f4dc819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dd0400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4dc8107760 0x7f4dc819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dd0400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc8111b30 con 0x7f4dc8107760 2024-04-25T01:56:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dd0400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81119b0 con 0x7f4dc8101750 2024-04-25T01:56:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dd0400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc8111cb0 con 0x7f4dc8102110 2024-04-25T01:56:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dcd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 0x7f4dc819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dce400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4dc8107760 0x7f4dc819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dcd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 0x7f4dc819a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:57416/0 (socket says 172.21.15.102:57416) 2024-04-25T01:56:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.469+0000 7f4dcd000640 1 -- 172.21.15.102:0/3140879570 learned_addr learned my addr 172.21.15.102:0/3140879570 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.470+0000 7f4dcda00640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dc8101750 0x7f4dc8199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.470+0000 7f4dcd000640 1 -- 172.21.15.102:0/3140879570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dc8101750 msgr2=0x7f4dc8199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.470+0000 7f4dcd000640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dc8101750 0x7f4dc8199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.470+0000 7f4dcd000640 1 -- 172.21.15.102:0/3140879570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4dc8107760 msgr2=0x7f4dc819e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.470+0000 7f4dcd000640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4dc8107760 0x7f4dc819e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.470+0000 7f4dcd000640 1 -- 172.21.15.102:0/3140879570 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc819ef20 con 0x7f4dc8102110 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.470+0000 7f4dcd000640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 0x7f4dc819a470 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009f50 tx=0x7f4db400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.471+0000 7f4dc6000640 1 -- 172.21.15.102:0/3140879570 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db4024070 con 0x7f4dc8102110 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.471+0000 7f4dc6000640 1 -- 172.21.15.102:0/3140879570 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4db401f440 con 0x7f4dc8102110 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.471+0000 7f4dc6000640 1 -- 172.21.15.102:0/3140879570 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db401d5a0 con 0x7f4dc8102110 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.471+0000 7f4dd0400640 1 -- 172.21.15.102:0/3140879570 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc819f210 con 0x7f4dc8102110 2024-04-25T01:56:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.471+0000 7f4dd0400640 1 -- 172.21.15.102:0/3140879570 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc81085d0 con 0x7f4dc8102110 2024-04-25T01:56:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.475+0000 7f4dd31b2740 1 Processor -- start 2024-04-25T01:56:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.475+0000 7f4dd31b2740 1 -- start start 2024-04-25T01:56:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151d61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 0x55cf151e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151e0fe0 0x55cf151e33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dd31b2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cf15139ab0 con 0x55cf151d6720 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dd31b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cf150e2640 con 0x55cf151d5e00 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dd31b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cf15135a70 con 0x55cf151e0fe0 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dce400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151d61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dce400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151d61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48034/0 (socket says 172.21.15.102:48034) 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dce400640 1 -- 172.21.15.102:0/2079625653 learned_addr learned my addr 172.21.15.102:0/2079625653 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dce400640 1 -- 172.21.15.102:0/2079625653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151e0fe0 msgr2=0x55cf151e33d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dce400640 1 --2- 172.21.15.102:0/2079625653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151e0fe0 0x55cf151e33d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dce400640 1 -- 172.21.15.102:0/2079625653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 msgr2=0x55cf151e0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.476+0000 7f4dce400640 1 --2- 172.21.15.102:0/2079625653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 0x55cf151e0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.477+0000 7f4dce400640 1 -- 172.21.15.102:0/2079625653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf14e4e210 con 0x55cf151d5e00 2024-04-25T01:56:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.478+0000 7f4dce400640 1 --2- 172.21.15.102:0/2079625653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151d61e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0033000 tx=0x7f4dc0005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36422ef38d28e7d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.478+0000 7f4dbaa00640 1 -- 172.21.15.102:0/2079625653 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0040070 con 0x55cf151d5e00 2024-04-25T01:56:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.478+0000 7f4dbaa00640 1 -- 172.21.15.102:0/2079625653 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc0005520 con 0x55cf151d5e00 2024-04-25T01:56:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.478+0000 7f4dbaa00640 1 -- 172.21.15.102:0/2079625653 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0035040 con 0x55cf151d5e00 2024-04-25T01:56:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.479+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2079625653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 msgr2=0x55cf151d61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.479+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2079625653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151d61e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0033000 tx=0x7f4dc0005e00 comp rx=0 tx=0).stop 2024-04-25T01:56:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.480+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2079625653 shutdown_connections 2024-04-25T01:56:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.480+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2079625653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151e0fe0 0x55cf151e33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.480+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2079625653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 0x55cf151e0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.480+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2079625653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151d61e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.480+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2079625653 >> 172.21.15.102:0/2079625653 conn(0x55cf151d2310 msgr2=0x55cf151d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.480+0000 7f4dc6000640 1 -- 172.21.15.102:0/3140879570 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f4db4027070 con 0x7f4dc8102110 2024-04-25T01:56:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.482+0000 7f4dc6000640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4db00a2f40 0x7f4db00a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.482+0000 7f4dc6000640 1 -- 172.21.15.102:0/3140879570 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f4db40d7a60 con 0x7f4dc8102110 2024-04-25T01:56:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.482+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2079625653 shutdown_connections 2024-04-25T01:56:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.483+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2079625653 wait complete. 2024-04-25T01:56:45.485 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.483+0000 7f4dcda00640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4db00a2f40 0x7f4db00a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.484+0000 7f4dd31b2740 1 -- start start 2024-04-25T01:56:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.484+0000 7f4dcda00640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4db00a2f40 0x7f4db00a5400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc005c40 tx=0x7f4dbc005bd0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.485+0000 7f4dd31b2740 1 Processor -- start 2024-04-25T01:56:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.485+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151de0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.485+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 0x55cf151de610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.485+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 0x55cf151ed020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.485+0000 7f4dd31b2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cf151e3bb0 con 0x55cf151d6720 2024-04-25T01:56:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.485+0000 7f4dd31b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cf150f5990 con 0x55cf151d5e00 2024-04-25T01:56:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.485+0000 7f4dd31b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cf151e3d30 con 0x55cf151deb50 2024-04-25T01:56:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.486+0000 7f4dd31b2740 10 client.0 _ll_register_callbacks cb 0x55cf150f7950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.486+0000 7f4dcd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 0x55cf151de610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.486+0000 7f4dcd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 0x55cf151de610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34868/0 (socket says 172.21.15.102:34868) 2024-04-25T01:56:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.486+0000 7f4dcd000640 1 -- 172.21.15.102:0/2675853832 learned_addr learned my addr 172.21.15.102:0/2675853832 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.486+0000 7f4dce400640 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 0x55cf151ed020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.486+0000 7f4dcda00640 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151de0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.487+0000 7f4dcd000640 1 -- 172.21.15.102:0/2675853832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 msgr2=0x55cf151ed020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.487+0000 7f4dcd000640 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 0x55cf151ed020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.487+0000 7f4dcd000640 1 -- 172.21.15.102:0/2675853832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 msgr2=0x55cf151de0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.487+0000 7f4dcd000640 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151de0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.487+0000 7f4dcd000640 1 -- 172.21.15.102:0/2675853832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf14f1a620 con 0x55cf151d6720 2024-04-25T01:56:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.488+0000 7f4dcd000640 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 0x55cf151de610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4db40d5280 tx=0x7f4db401f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da15d8883821e126 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.488+0000 7f4daf400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:56:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.488+0000 7f4daf400640 1 -- 172.21.15.102:0/2675853832 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db40ac850 con 0x55cf151d6720 2024-04-25T01:56:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.488+0000 7f4daf400640 1 -- 172.21.15.102:0/2675853832 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f4db40d5340 con 0x55cf151d6720 2024-04-25T01:56:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.488+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2675853832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dc0005520 con 0x55cf151d6720 2024-04-25T01:56:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.489+0000 7f4daf400640 1 -- 172.21.15.102:0/2675853832 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4db40acae0 con 0x55cf151d6720 2024-04-25T01:56:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.489+0000 7f4dd31b2740 10 client.15570 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.489+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2675853832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cf151f0080 con 0x55cf151d6720 2024-04-25T01:56:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.491+0000 7f4daf400640 1 -- 172.21.15.102:0/2675853832 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc0005520 con 0x55cf151d6720 2024-04-25T01:56:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.491+0000 7f4daf400640 1 -- 172.21.15.102:0/2675853832 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4db40c2b40 con 0x55cf151d6720 2024-04-25T01:56:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.492+0000 7f4dd31b2740 10 client.15570 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.492+0000 7f4dd31b2740 10 client.15570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.492+0000 7f4dd31b2740 20 client.15570 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.493+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cf151f1540 0x55cf151f3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.493+0000 7f4dd31b2740 4 client.15570 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.493+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2675853832 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf15146ff0 con 0x55cf151f1540 2024-04-25T01:56:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.494+0000 7f4dcda00640 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cf151f1540 0x55cf151f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.495+0000 7f4dcda00640 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cf151f1540 0x55cf151f3930 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2289750252c60a6 server_cookie=c41ac8b2e474b4bd in_seq=0 out_seq=0 2024-04-25T01:56:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.495+0000 7f4daf400640 10 client.15570 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.496+0000 7f4daf400640 1 -- 172.21.15.102:0/2675853832 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf15146ff0 con 0x55cf151f1540 2024-04-25T01:56:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.496+0000 7f4daf400640 10 client.15570 handle_command_reply: tid=0 2024-04-25T01:56:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.496+0000 7f4dd31b2740 1 client.15570 shutdown 2024-04-25T01:56:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.497+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2675853832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 msgr2=0x55cf151de610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.497+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 0x55cf151de610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4db40d5280 tx=0x7f4db401f5f0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.497+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2675853832 shutdown_connections 2024-04-25T01:56:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.497+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cf151f1540 0x55cf151f3930 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.497+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 0x55cf151ed020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.497+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d6720 0x55cf151de610 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.497+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/2675853832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151d5e00 0x55cf151de0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.497+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2675853832 >> 172.21.15.102:0/2675853832 conn(0x55cf151d2310 msgr2=0x55cf151d8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.497+0000 7f4daf400640 0 client.15570 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.498+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2675853832 shutdown_connections 2024-04-25T01:56:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.498+0000 7f4dd31b2740 1 -- 172.21.15.102:0/2675853832 wait complete. 2024-04-25T01:56:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.498+0000 7f4dd31b2740 20 client.15570 trim_cache size 0 max 16384 2024-04-25T01:56:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.536+0000 7f4dd31b2740 1 Processor -- start 2024-04-25T01:56:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.536+0000 7f4dd31b2740 1 -- start start 2024-04-25T01:56:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d5e00 0x55cf151d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 0x55cf151f91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 0x55cf151f96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dd31b2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81ad980 con 0x55cf151d5e00 2024-04-25T01:56:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dd31b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81ab1f0 con 0x55cf151e0910 2024-04-25T01:56:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dd31b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81a9060 con 0x55cf151deb50 2024-04-25T01:56:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dcda00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d5e00 0x55cf151d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dce400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 0x55cf151f96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dce400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 0x55cf151f96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48044/0 (socket says 172.21.15.102:48044) 2024-04-25T01:56:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dce400640 1 -- 172.21.15.102:0/1210140903 learned_addr learned my addr 172.21.15.102:0/1210140903 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.537+0000 7f4dcd000640 1 --2- 172.21.15.102:0/1210140903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 0x55cf151f91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.538+0000 7f4dcda00640 1 -- 172.21.15.102:0/1210140903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 msgr2=0x55cf151f91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.538+0000 7f4dcda00640 1 --2- 172.21.15.102:0/1210140903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 0x55cf151f91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.538+0000 7f4dcda00640 1 -- 172.21.15.102:0/1210140903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 msgr2=0x55cf151f96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.538+0000 7f4dcda00640 1 --2- 172.21.15.102:0/1210140903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 0x55cf151f96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.538+0000 7f4dcda00640 1 -- 172.21.15.102:0/1210140903 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf151d1aa0 con 0x55cf151d5e00 2024-04-25T01:56:45.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.539+0000 7f4dcda00640 1 --2- 172.21.15.102:0/1210140903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d5e00 0x55cf151d0bf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55cf151f4120 tx=0x7f4dbc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d1884c0750a200 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.540+0000 7f4dba000640 1 -- 172.21.15.102:0/1210140903 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc02c070 con 0x55cf151d5e00 2024-04-25T01:56:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.540+0000 7f4dba000640 1 -- 172.21.15.102:0/1210140903 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dbc026730 con 0x55cf151d5e00 2024-04-25T01:56:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.540+0000 7f4dba000640 1 -- 172.21.15.102:0/1210140903 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc01f980 con 0x55cf151d5e00 2024-04-25T01:56:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.541+0000 7f4dd31b2740 1 -- 172.21.15.102:0/1210140903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d5e00 msgr2=0x55cf151d0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.541+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/1210140903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d5e00 0x55cf151d0bf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55cf151f4120 tx=0x7f4dbc016000 comp rx=0 tx=0).stop 2024-04-25T01:56:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.541+0000 7f4dd31b2740 1 -- 172.21.15.102:0/1210140903 shutdown_connections 2024-04-25T01:56:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.541+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/1210140903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 0x55cf151f96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.542+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/1210140903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151deb50 0x55cf151f91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.542+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/1210140903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151d5e00 0x55cf151d0bf0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.542+0000 7f4dd31b2740 1 -- 172.21.15.102:0/1210140903 >> 172.21.15.102:0/1210140903 conn(0x55cf151d2310 msgr2=0x55cf151de1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.542+0000 7f4dd31b2740 1 -- 172.21.15.102:0/1210140903 shutdown_connections 2024-04-25T01:56:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.542+0000 7f4dd31b2740 1 -- 172.21.15.102:0/1210140903 wait complete. 2024-04-25T01:56:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.543+0000 7f4dd31b2740 1 -- start start 2024-04-25T01:56:45.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.543+0000 7f4dd31b2740 1 Processor -- start 2024-04-25T01:56:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.544+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151d5e00 0x55cf151e8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.546 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.545+0000 7f4dd31b2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151deb50 0x55cf151e94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.545+0000 7f4dcda00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151d5e00 0x55cf151e8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.545+0000 7f4dcda00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151d5e00 0x55cf151e8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:57450/0 (socket says 172.21.15.102:57450) 2024-04-25T01:56:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.545+0000 7f4dcda00640 1 -- 172.21.15.102:0/3380513825 learned_addr learned my addr 172.21.15.102:0/3380513825 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.545+0000 7f4dcd000640 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151deb50 0x55cf151e94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.545+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 0x55cf151e5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.545+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81a96b0 con 0x55cf151deb50 2024-04-25T01:56:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.545+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81aab20 con 0x55cf151e0910 2024-04-25T01:56:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.545+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81a9d80 con 0x55cf151d5e00 2024-04-25T01:56:45.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.546+0000 7f4dd31b2740 10 client.0 _ll_register_callbacks cb 0x7f4db401fc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.546+0000 7f4dce400640 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 0x55cf151e5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.546+0000 7f4dcda00640 1 -- 172.21.15.102:0/3380513825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 msgr2=0x55cf151e5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.546+0000 7f4dcda00640 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 0x55cf151e5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.546+0000 7f4dcda00640 1 -- 172.21.15.102:0/3380513825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151deb50 msgr2=0x55cf151e94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.547+0000 7f4dcda00640 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151deb50 0x55cf151e94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.547+0000 7f4dcda00640 1 -- 172.21.15.102:0/3380513825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf151e6510 con 0x55cf151d5e00 2024-04-25T01:56:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.547+0000 7f4dcd000640 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151deb50 0x55cf151e94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.547+0000 7f4dcda00640 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151d5e00 0x55cf151e8f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc026b20 tx=0x7f4dbc013b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71ee3e88d8dfebb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.547+0000 7f4daf400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:56:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.548+0000 7f4daf400640 1 -- 172.21.15.102:0/3380513825 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc035060 con 0x55cf151d5e00 2024-04-25T01:56:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.548+0000 7f4daf400640 1 -- 172.21.15.102:0/3380513825 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f4dbc004c40 con 0x55cf151d5e00 2024-04-25T01:56:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.548+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dbc026730 con 0x55cf151d5e00 2024-04-25T01:56:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.548+0000 7f4daf400640 1 -- 172.21.15.102:0/3380513825 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4dbc02c070 con 0x55cf151d5e00 2024-04-25T01:56:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.548+0000 7f4dd31b2740 10 client.25726 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.548+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cf15258300 con 0x55cf151d5e00 2024-04-25T01:56:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.549+0000 7f4daf400640 1 -- 172.21.15.102:0/3380513825 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dbc026730 con 0x55cf151d5e00 2024-04-25T01:56:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.549+0000 7f4daf400640 1 -- 172.21.15.102:0/3380513825 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4dbc00f040 con 0x55cf151d5e00 2024-04-25T01:56:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.549+0000 7f4dd31b2740 10 client.25726 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.549+0000 7f4dd31b2740 10 client.25726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.549+0000 7f4dd31b2740 20 client.25726 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.549+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cf152589a0 0x55cf1525adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.549+0000 7f4dd31b2740 4 client.25726 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:56:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.549+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf15141340 con 0x55cf152589a0 2024-04-25T01:56:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.549+0000 7f4dcd000640 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cf152589a0 0x55cf1525adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.550+0000 7f4dcd000640 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cf152589a0 0x55cf1525adc0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cf21b3481e87c7d server_cookie=18e96b30a5536a5f in_seq=0 out_seq=0 2024-04-25T01:56:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.550+0000 7f4daf400640 10 client.25726 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.567+0000 7f4daf400640 1 -- 172.21.15.102:0/3380513825 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cf15141340 con 0x55cf152589a0 2024-04-25T01:56:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.567+0000 7f4daf400640 10 client.25726 handle_command_reply: tid=0 2024-04-25T01:56:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.567+0000 7f4dd31b2740 1 client.25726 shutdown 2024-04-25T01:56:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.567+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151d5e00 msgr2=0x55cf151e8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.567+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151d5e00 0x55cf151e8f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc026b20 tx=0x7f4dbc013b90 comp rx=0 tx=0).stop 2024-04-25T01:56:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.568+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 shutdown_connections 2024-04-25T01:56:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.568+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cf152589a0 0x55cf1525adc0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.568+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cf151e0910 0x55cf151e5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.568+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cf151deb50 0x55cf151e94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.568+0000 7f4dd31b2740 1 --2- 172.21.15.102:0/3380513825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cf151d5e00 0x55cf151e8f80 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc026b20 tx=0x7f4dbc013b90 comp rx=0 tx=0).stop 2024-04-25T01:56:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.568+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 >> 172.21.15.102:0/3380513825 conn(0x55cf151d2310 msgr2=0x55cf151dd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.568+0000 7f4daf400640 0 client.25726 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.568+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 shutdown_connections 2024-04-25T01:56:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.568+0000 7f4dd31b2740 1 -- 172.21.15.102:0/3380513825 wait complete. 2024-04-25T01:56:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.569+0000 7f4dd31b2740 20 client.25726 trim_cache size 0 max 16384 2024-04-25T01:56:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:56:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:56:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:56:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:56:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.569+0000 7f4dd0400640 1 -- 172.21.15.102:0/3140879570 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4db00a2f40 msgr2=0x7f4db00a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.569+0000 7f4dd0400640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4db00a2f40 0x7f4db00a5400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc005c40 tx=0x7f4dbc005bd0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.569+0000 7f4dd0400640 1 -- 172.21.15.102:0/3140879570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 msgr2=0x7f4dc819a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.570+0000 7f4dd0400640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 0x7f4dc819a470 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009f50 tx=0x7f4db400ee50 comp rx=0 tx=0).stop 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.570+0000 7f4dd0400640 1 -- 172.21.15.102:0/3140879570 shutdown_connections 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.570+0000 7f4dd0400640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4db00a2f40 0x7f4db00a5400 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.570+0000 7f4dd0400640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4dc8107760 0x7f4dc819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.570+0000 7f4dd0400640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4dc8102110 0x7f4dc819a470 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.570+0000 7f4dd0400640 1 --2- 172.21.15.102:0/3140879570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dc8101750 0x7f4dc8199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.570+0000 7f4dd0400640 1 -- 172.21.15.102:0/3140879570 >> 172.21.15.102:0/3140879570 conn(0x7f4dc80fd240 msgr2=0x7f4dc810cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.570+0000 7f4dd0400640 1 -- 172.21.15.102:0/3140879570 shutdown_connections 2024-04-25T01:56:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:45.570+0000 7f4dd0400640 1 -- 172.21.15.102:0/3140879570 wait complete. 2024-04-25T01:56:46.977 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:46 smithi102 ceph-mon[27796]: pgmap v48: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 8.2 MiB/s wr, 763 op/s 2024-04-25T01:56:46.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:46 smithi110 ceph-mon[32848]: pgmap v48: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 8.2 MiB/s wr, 763 op/s 2024-04-25T01:56:47.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:46 smithi193 ceph-mon[32981]: pgmap v48: 97 pgs: 97 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 8.2 MiB/s wr, 763 op/s 2024-04-25T01:56:48.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 55.0 s, 3240.4 tps, lat 30.884 ms stddev 33.472 2024-04-25T01:56:48.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:48 smithi110 ceph-mon[32848]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-25T01:56:48.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:48 smithi110 ceph-mon[32848]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T01:56:49.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:48 smithi193 ceph-mon[32981]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-25T01:56:49.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:48 smithi193 ceph-mon[32981]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T01:56:49.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:48 smithi102 ceph-mon[27796]: pgmap v49: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-25T01:56:49.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:48 smithi102 ceph-mon[27796]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-25T01:56:50.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:50 smithi110 ceph-mon[32848]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 8.4 MiB/s wr, 663 op/s 2024-04-25T01:56:51.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:50 smithi193 ceph-mon[32981]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 8.4 MiB/s wr, 663 op/s 2024-04-25T01:56:51.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:50 smithi102 ceph-mon[27796]: pgmap v50: 97 pgs: 97 active+clean; 1.1 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 8.4 MiB/s wr, 663 op/s 2024-04-25T01:56:52.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:52 smithi110 ceph-mon[32848]: pgmap v51: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 13 MiB/s wr, 983 op/s 2024-04-25T01:56:53.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:52 smithi193 ceph-mon[32981]: pgmap v51: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 13 MiB/s wr, 983 op/s 2024-04-25T01:56:53.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 60.0 s, 3638.8 tps, lat 27.469 ms stddev 29.968 2024-04-25T01:56:53.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:52 smithi102 ceph-mon[27796]: pgmap v51: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 13 MiB/s wr, 983 op/s 2024-04-25T01:56:54.989 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:54 smithi110 ceph-mon[32848]: pgmap v52: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 9.8 MiB/s wr, 733 op/s 2024-04-25T01:56:55.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:54 smithi193 ceph-mon[32981]: pgmap v52: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 9.8 MiB/s wr, 733 op/s 2024-04-25T01:56:55.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:54 smithi102 ceph-mon[27796]: pgmap v52: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 9.8 MiB/s wr, 733 op/s 2024-04-25T01:56:57.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:56 smithi193 ceph-mon[32981]: pgmap v53: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 8.3 MiB/s wr, 657 op/s 2024-04-25T01:56:57.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:56 smithi102 ceph-mon[27796]: pgmap v53: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 8.3 MiB/s wr, 657 op/s 2024-04-25T01:56:57.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:56 smithi110 ceph-mon[32848]: pgmap v53: 97 pgs: 97 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 8.3 MiB/s wr, 657 op/s 2024-04-25T01:56:58.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 65.0 s, 3151.2 tps, lat 31.553 ms stddev 35.832 2024-04-25T01:56:58.753 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.750+0000 7f4436c00640 1 -- 172.21.15.102:0/1315182389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 msgr2=0x7f4430108380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:58.753 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.750+0000 7f4436c00640 1 --2- 172.21.15.102:0/1315182389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 0x7f4430108380 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4420009900 tx=0x7f442002a3f0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.753+0000 7f4436c00640 1 -- 172.21.15.102:0/1315182389 shutdown_connections 2024-04-25T01:56:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.753+0000 7f4436c00640 1 --2- 172.21.15.102:0/1315182389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f443010d070 0x7f4430114c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.753+0000 7f4436c00640 1 --2- 172.21.15.102:0/1315182389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4430108950 0x7f443010c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.753+0000 7f4436c00640 1 --2- 172.21.15.102:0/1315182389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 0x7f4430108380 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.753+0000 7f4436c00640 1 -- 172.21.15.102:0/1315182389 >> 172.21.15.102:0/1315182389 conn(0x7f4430075cd0 msgr2=0x7f44300780f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.753+0000 7f4436c00640 1 -- 172.21.15.102:0/1315182389 shutdown_connections 2024-04-25T01:56:58.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.753+0000 7f4436c00640 1 -- 172.21.15.102:0/1315182389 wait complete. 2024-04-25T01:56:58.758 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.755+0000 7f4436c00640 1 Processor -- start 2024-04-25T01:56:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.756+0000 7f4436c00640 1 -- start start 2024-04-25T01:56:58.759 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.756+0000 7f4436c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 0x7f443019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.756+0000 7f4436c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f443010d070 0x7f443019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.756+0000 7f4436c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44301a2e30 0x7f44301a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.756+0000 7f4436c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4430118150 con 0x7f44301a2e30 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.756+0000 7f4436c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4430117fd0 con 0x7f443010d070 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.756+0000 7f4436c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f44301182d0 con 0x7f4430107fa0 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f442fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 0x7f443019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f442f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f443010d070 0x7f443019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f442fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 0x7f443019e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58934/0 (socket says 172.21.15.102:58934) 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f442fe00640 1 -- 172.21.15.102:0/492901838 learned_addr learned my addr 172.21.15.102:0/492901838 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:58.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f4434c00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44301a2e30 0x7f44301a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f442fe00640 1 -- 172.21.15.102:0/492901838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f443010d070 msgr2=0x7f443019eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f442fe00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f443010d070 0x7f443019eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f442fe00640 1 -- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44301a2e30 msgr2=0x7f44301a32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f442fe00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44301a2e30 0x7f44301a32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f442fe00640 1 -- 172.21.15.102:0/492901838 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44301a3920 con 0x7f4430107fa0 2024-04-25T01:56:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.757+0000 7f4434c00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44301a2e30 0x7f44301a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.758+0000 7f442fe00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 0x7f443019e570 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f442002a960 tx=0x7f4420003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.759+0000 7f442cc00640 1 -- 172.21.15.102:0/492901838 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f442003f070 con 0x7f4430107fa0 2024-04-25T01:56:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.759+0000 7f442cc00640 1 -- 172.21.15.102:0/492901838 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4420031c90 con 0x7f4430107fa0 2024-04-25T01:56:58.762 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.759+0000 7f442cc00640 1 -- 172.21.15.102:0/492901838 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420035030 con 0x7f4430107fa0 2024-04-25T01:56:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.759+0000 7f4436c00640 1 -- 172.21.15.102:0/492901838 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44301a3bb0 con 0x7f4430107fa0 2024-04-25T01:56:58.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.759+0000 7f4436c00640 1 -- 172.21.15.102:0/492901838 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44301ab320 con 0x7f4430107fa0 2024-04-25T01:56:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.763+0000 7f442cc00640 1 -- 172.21.15.102:0/492901838 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4420042070 con 0x7f4430107fa0 2024-04-25T01:56:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.765+0000 7f442cc00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f44140a2fd0 0x7f44140a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.765+0000 7f442cc00640 1 -- 172.21.15.102:0/492901838 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f44200efb20 con 0x7f4430107fa0 2024-04-25T01:56:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.765+0000 7f442f400640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f44140a2fd0 0x7f44140a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.765+0000 7f4439a1b740 1 Processor -- start 2024-04-25T01:56:58.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.767+0000 7f442f400640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f44140a2fd0 0x7f44140a5490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4430107040 tx=0x7f441c00b410 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:58.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.767+0000 7f4439a1b740 1 -- start start 2024-04-25T01:56:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.768+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321be00 0x5649c321c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.768+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c321c720 0x5649c3226aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.768+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3226fe0 0x5649c32293d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.768+0000 7f4439a1b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5649c317fab0 con 0x5649c3226fe0 2024-04-25T01:56:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.768+0000 7f4439a1b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5649c3128640 con 0x5649c321c720 2024-04-25T01:56:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.768+0000 7f4439a1b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649c317ba70 con 0x5649c321be00 2024-04-25T01:56:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.768+0000 7f4434c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3226fe0 0x5649c32293d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.768+0000 7f442fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321be00 0x5649c321c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.768+0000 7f442fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321be00 0x5649c321c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58944/0 (socket says 172.21.15.102:58944) 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.769+0000 7f442fe00640 1 -- 172.21.15.102:0/840914824 learned_addr learned my addr 172.21.15.102:0/840914824 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.769+0000 7f442fe00640 1 -- 172.21.15.102:0/840914824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c321c720 msgr2=0x5649c3226aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.769+0000 7f442fe00640 1 --2- 172.21.15.102:0/840914824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c321c720 0x5649c3226aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.769+0000 7f442fe00640 1 -- 172.21.15.102:0/840914824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3226fe0 msgr2=0x5649c32293d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.769+0000 7f442fe00640 1 --2- 172.21.15.102:0/840914824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3226fe0 0x5649c32293d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.769+0000 7f442fe00640 1 -- 172.21.15.102:0/840914824 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c2e94210 con 0x5649c321be00 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.769+0000 7f4434c00640 1 --2- 172.21.15.102:0/840914824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3226fe0 0x5649c32293d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.770+0000 7f442fe00640 1 --2- 172.21.15.102:0/840914824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321be00 0x5649c321c1e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f44200b9970 tx=0x7f442003b720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b7b678017a6a1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.770+0000 7f4425200640 1 -- 172.21.15.102:0/840914824 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4420049070 con 0x5649c321be00 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.770+0000 7f4425200640 1 -- 172.21.15.102:0/840914824 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44200c5c10 con 0x5649c321be00 2024-04-25T01:56:58.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.770+0000 7f4425200640 1 -- 172.21.15.102:0/840914824 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44200c5780 con 0x5649c321be00 2024-04-25T01:56:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.772+0000 7f4439a1b740 1 -- 172.21.15.102:0/840914824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321be00 msgr2=0x5649c321c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.772+0000 7f4439a1b740 1 --2- 172.21.15.102:0/840914824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321be00 0x5649c321c1e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f44200b9970 tx=0x7f442003b720 comp rx=0 tx=0).stop 2024-04-25T01:56:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.772+0000 7f4439a1b740 1 -- 172.21.15.102:0/840914824 shutdown_connections 2024-04-25T01:56:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.772+0000 7f4439a1b740 1 --2- 172.21.15.102:0/840914824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3226fe0 0x5649c32293d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.772+0000 7f4439a1b740 1 --2- 172.21.15.102:0/840914824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c321c720 0x5649c3226aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.772+0000 7f4439a1b740 1 --2- 172.21.15.102:0/840914824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321be00 0x5649c321c1e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.772+0000 7f4439a1b740 1 -- 172.21.15.102:0/840914824 >> 172.21.15.102:0/840914824 conn(0x5649c3218310 msgr2=0x5649c32186f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.772+0000 7f4439a1b740 1 -- 172.21.15.102:0/840914824 shutdown_connections 2024-04-25T01:56:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.772+0000 7f4439a1b740 1 -- 172.21.15.102:0/840914824 wait complete. 2024-04-25T01:56:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.773+0000 7f4439a1b740 1 -- start start 2024-04-25T01:56:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.773+0000 7f4439a1b740 1 Processor -- start 2024-04-25T01:56:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.774+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c32244d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.774+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3224a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.774+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 0x5649c3233150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.774+0000 7f4439a1b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5649c32299d0 con 0x5649c321c720 2024-04-25T01:56:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.774+0000 7f4439a1b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5649c313b990 con 0x5649c3226fe0 2024-04-25T01:56:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.774+0000 7f4439a1b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649c3229b50 con 0x5649c3230d60 2024-04-25T01:56:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.774+0000 7f4439a1b740 10 client.0 _ll_register_callbacks cb 0x5649c313d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f442f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3224a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f442f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3224a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48028/0 (socket says 172.21.15.102:48028) 2024-04-25T01:56:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f442f400640 1 -- 172.21.15.102:0/414506936 learned_addr learned my addr 172.21.15.102:0/414506936 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f4434c00640 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 0x5649c3233150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f442fe00640 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c32244d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f4434c00640 1 -- 172.21.15.102:0/414506936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 msgr2=0x5649c3224a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f4434c00640 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3224a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f4434c00640 1 -- 172.21.15.102:0/414506936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 msgr2=0x5649c32244d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f4434c00640 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c32244d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.775+0000 7f4434c00640 1 -- 172.21.15.102:0/414506936 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c2f60620 con 0x5649c3230d60 2024-04-25T01:56:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.776+0000 7f4434c00640 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 0x5649c3233150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x5649c32194a0 tx=0x7f44280107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8a19311acf9d5d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.776+0000 7f4419600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:56:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.776+0000 7f4419600640 1 -- 172.21.15.102:0/414506936 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428009e10 con 0x5649c3230d60 2024-04-25T01:56:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.777+0000 7f4419600640 1 -- 172.21.15.102:0/414506936 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f4428004b20 con 0x5649c3230d60 2024-04-25T01:56:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.777+0000 7f4439a1b740 1 -- 172.21.15.102:0/414506936 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44200c5c10 con 0x5649c3230d60 2024-04-25T01:56:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.777+0000 7f4419600640 1 -- 172.21.15.102:0/414506936 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4428004d20 con 0x5649c3230d60 2024-04-25T01:56:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.777+0000 7f4439a1b740 10 client.25738 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.777+0000 7f4439a1b740 1 -- 172.21.15.102:0/414506936 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649c3235f80 con 0x5649c3230d60 2024-04-25T01:56:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.778+0000 7f4419600640 1 -- 172.21.15.102:0/414506936 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44200c5c10 con 0x5649c3230d60 2024-04-25T01:56:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.778+0000 7f4419600640 1 -- 172.21.15.102:0/414506936 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f442800e600 con 0x5649c3230d60 2024-04-25T01:56:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.779+0000 7f4439a1b740 10 client.25738 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.779+0000 7f4439a1b740 10 client.25738 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.779+0000 7f4439a1b740 20 client.25738 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.779+0000 7f4439a1b740 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5649c32374b0 0x5649c3239910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.779+0000 7f4439a1b740 4 client.25738 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:56:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.779+0000 7f4439a1b740 1 -- 172.21.15.102:0/414506936 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649c318cff0 con 0x5649c32374b0 2024-04-25T01:56:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.779+0000 7f442fe00640 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5649c32374b0 0x5649c3239910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.781+0000 7f442fe00640 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5649c32374b0 0x5649c3239910 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44c261f55851fcf2 server_cookie=35166eb45323176e in_seq=0 out_seq=0 2024-04-25T01:56:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.781+0000 7f4419600640 10 client.25738 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.782+0000 7f4419600640 1 -- 172.21.15.102:0/414506936 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5649c318cff0 con 0x5649c32374b0 2024-04-25T01:56:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.782+0000 7f4419600640 10 client.25738 handle_command_reply: tid=0 2024-04-25T01:56:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.782+0000 7f4439a1b740 1 client.25738 shutdown 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.783+0000 7f4439a1b740 1 -- 172.21.15.102:0/414506936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 msgr2=0x5649c3233150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.783+0000 7f4439a1b740 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 0x5649c3233150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x5649c32194a0 tx=0x7f44280107a0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.783+0000 7f4439a1b740 1 -- 172.21.15.102:0/414506936 shutdown_connections 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.783+0000 7f4439a1b740 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5649c32374b0 0x5649c3239910 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.783+0000 7f4439a1b740 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 0x5649c3233150 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.783+0000 7f4439a1b740 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3224a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.783+0000 7f4439a1b740 1 --2- 172.21.15.102:0/414506936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c32244d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.783+0000 7f4439a1b740 1 -- 172.21.15.102:0/414506936 >> 172.21.15.102:0/414506936 conn(0x5649c3218310 msgr2=0x5649c321e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.783+0000 7f4419600640 0 client.25738 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.784+0000 7f4439a1b740 1 -- 172.21.15.102:0/414506936 shutdown_connections 2024-04-25T01:56:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.784+0000 7f4439a1b740 1 -- 172.21.15.102:0/414506936 wait complete. 2024-04-25T01:56:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.784+0000 7f4439a1b740 20 client.25738 trim_cache size 0 max 16384 2024-04-25T01:56:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.815+0000 7f4439a1b740 1 Processor -- start 2024-04-25T01:56:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.816+0000 7f4439a1b740 1 -- start start 2024-04-25T01:56:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.816+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321c720 0x5649c3223480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.816+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3217300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.816+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3230d60 0x5649c323f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.817+0000 7f4439a1b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b62e0 con 0x5649c3230d60 2024-04-25T01:56:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.817+0000 7f442f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3217300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.817+0000 7f4434c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3230d60 0x5649c323f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.817+0000 7f4434c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3230d60 0x5649c323f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34126/0 (socket says 172.21.15.102:34126) 2024-04-25T01:56:58.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.817+0000 7f4434c00640 1 -- 172.21.15.102:0/751364824 learned_addr learned my addr 172.21.15.102:0/751364824 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:58.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.817+0000 7f4439a1b740 1 -- 172.21.15.102:0/751364824 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b3b50 con 0x5649c3226fe0 2024-04-25T01:56:58.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.817+0000 7f4439a1b740 1 -- 172.21.15.102:0/751364824 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b19c0 con 0x5649c321c720 2024-04-25T01:56:58.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.817+0000 7f442fe00640 1 --2- 172.21.15.102:0/751364824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321c720 0x5649c3223480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.818+0000 7f442f400640 1 -- 172.21.15.102:0/751364824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321c720 msgr2=0x5649c3223480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.818+0000 7f442f400640 1 --2- 172.21.15.102:0/751364824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321c720 0x5649c3223480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.818+0000 7f442f400640 1 -- 172.21.15.102:0/751364824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3230d60 msgr2=0x5649c323f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.818+0000 7f442f400640 1 --2- 172.21.15.102:0/751364824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3230d60 0x5649c323f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.818+0000 7f442f400640 1 -- 172.21.15.102:0/751364824 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c3216890 con 0x5649c3226fe0 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.818+0000 7f442fe00640 1 --2- 172.21.15.102:0/751364824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321c720 0x5649c3223480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.818+0000 7f442f400640 1 --2- 172.21.15.102:0/751364824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3217300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f441c005d00 tx=0x7f441c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aa360414d7dc199 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.819+0000 7f441be00640 1 -- 172.21.15.102:0/751364824 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c01e6e0 con 0x5649c3226fe0 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.819+0000 7f441be00640 1 -- 172.21.15.102:0/751364824 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441c01ed60 con 0x5649c3226fe0 2024-04-25T01:56:58.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.819+0000 7f441be00640 1 -- 172.21.15.102:0/751364824 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c028d70 con 0x5649c3226fe0 2024-04-25T01:56:58.823 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.820+0000 7f4439a1b740 1 -- 172.21.15.102:0/751364824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 msgr2=0x5649c3217300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:58.823 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.820+0000 7f4439a1b740 1 --2- 172.21.15.102:0/751364824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3217300 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f441c005d00 tx=0x7f441c013000 comp rx=0 tx=0).stop 2024-04-25T01:56:58.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.821+0000 7f4439a1b740 1 -- 172.21.15.102:0/751364824 shutdown_connections 2024-04-25T01:56:58.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.821+0000 7f4439a1b740 1 --2- 172.21.15.102:0/751364824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c3230d60 0x5649c323f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.821+0000 7f4439a1b740 1 --2- 172.21.15.102:0/751364824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c3217300 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.821+0000 7f4439a1b740 1 --2- 172.21.15.102:0/751364824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c321c720 0x5649c3223480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.821+0000 7f4439a1b740 1 -- 172.21.15.102:0/751364824 >> 172.21.15.102:0/751364824 conn(0x5649c3218310 msgr2=0x5649c3216bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:58.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.821+0000 7f4439a1b740 1 -- 172.21.15.102:0/751364824 shutdown_connections 2024-04-25T01:56:58.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.821+0000 7f4439a1b740 1 -- 172.21.15.102:0/751364824 wait complete. 2024-04-25T01:56:58.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.822+0000 7f4439a1b740 1 -- start start 2024-04-25T01:56:58.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.822+0000 7f4439a1b740 1 Processor -- start 2024-04-25T01:56:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.823+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c322c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.826 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.823+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c322c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.823+0000 7f4439a1b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 0x5649c329ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.824+0000 7f4439a1b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b2010 con 0x5649c321c720 2024-04-25T01:56:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.824+0000 7f4439a1b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b3480 con 0x5649c3226fe0 2024-04-25T01:56:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.824+0000 7f4439a1b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f44301b26e0 con 0x5649c3230d60 2024-04-25T01:56:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.824+0000 7f442fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c322c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.824+0000 7f442fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c322c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34136/0 (socket says 172.21.15.102:34136) 2024-04-25T01:56:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.824+0000 7f442fe00640 1 -- 172.21.15.102:0/791307391 learned_addr learned my addr 172.21.15.102:0/791307391 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:56:58.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.824+0000 7f4434c00640 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 0x5649c329ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.824+0000 7f4439a1b740 10 client.0 _ll_register_callbacks cb 0x5649c3225240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:56:58.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.824+0000 7f442f400640 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c322c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.825+0000 7f442f400640 1 -- 172.21.15.102:0/791307391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 msgr2=0x5649c329ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.825+0000 7f442f400640 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 0x5649c329ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.825+0000 7f442f400640 1 -- 172.21.15.102:0/791307391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 msgr2=0x5649c322c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:56:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.825+0000 7f442f400640 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c322c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.825+0000 7f442f400640 1 -- 172.21.15.102:0/791307391 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c329cf90 con 0x5649c3226fe0 2024-04-25T01:56:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.825+0000 7f442fe00640 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c322c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:56:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.825+0000 7f442f400640 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c322c5f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x5649c3233800 tx=0x7f441c00beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a12f6a7cdb353003 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:56:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.825+0000 7f4419600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:56:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.826+0000 7f4419600640 1 -- 172.21.15.102:0/791307391 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c038060 con 0x5649c3226fe0 2024-04-25T01:56:58.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.826+0000 7f4419600640 1 -- 172.21.15.102:0/791307391 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f441c028680 con 0x5649c3226fe0 2024-04-25T01:56:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.826+0000 7f4439a1b740 1 -- 172.21.15.102:0/791307391 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f441c01ed60 con 0x5649c3226fe0 2024-04-25T01:56:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.826+0000 7f4419600640 1 -- 172.21.15.102:0/791307391 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f441c0289b0 con 0x5649c3226fe0 2024-04-25T01:56:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.826+0000 7f4439a1b740 10 client.25763 fetch_fsmap learned FSMap version 27 2024-04-25T01:56:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.826+0000 7f4439a1b740 1 -- 172.21.15.102:0/791307391 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649c329e0c0 con 0x5649c3226fe0 2024-04-25T01:56:58.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.827+0000 7f4419600640 1 -- 172.21.15.102:0/791307391 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441c01ed60 con 0x5649c3226fe0 2024-04-25T01:56:58.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.827+0000 7f4419600640 1 -- 172.21.15.102:0/791307391 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f441c026020 con 0x5649c3226fe0 2024-04-25T01:56:58.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.827+0000 7f4439a1b740 10 client.25763 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:56:58.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.827+0000 7f4439a1b740 10 client.25763 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:56:58.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.827+0000 7f4439a1b740 20 client.25763 populate_metadata read hostname 'smithi102' 2024-04-25T01:56:58.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.827+0000 7f4439a1b740 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5649c329e820 0x5649c32a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:56:58.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.827+0000 7f4439a1b740 4 client.25763 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:56:58.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.827+0000 7f4439a1b740 1 -- 172.21.15.102:0/791307391 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649c3187340 con 0x5649c329e820 2024-04-25T01:56:58.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.828+0000 7f442fe00640 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5649c329e820 0x5649c32a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:56:58.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.828+0000 7f442fe00640 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5649c329e820 0x5649c32a0c40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e211304224d3a8c server_cookie=18e002d9d2398e6 in_seq=0 out_seq=0 2024-04-25T01:56:58.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.828+0000 7f4419600640 10 client.25763 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:58.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.842+0000 7f4419600640 1 -- 172.21.15.102:0/791307391 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649c3187340 con 0x5649c329e820 2024-04-25T01:56:58.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.842+0000 7f4419600640 10 client.25763 handle_command_reply: tid=0 2024-04-25T01:56:58.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.842+0000 7f4439a1b740 1 client.25763 shutdown 2024-04-25T01:56:58.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.843+0000 7f4439a1b740 1 -- 172.21.15.102:0/791307391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 msgr2=0x5649c322c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:58.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.843+0000 7f4439a1b740 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c322c5f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x5649c3233800 tx=0x7f441c00beb0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.846+0000 7f4439a1b740 1 -- 172.21.15.102:0/791307391 shutdown_connections 2024-04-25T01:56:58.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.846+0000 7f4439a1b740 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5649c329e820 0x5649c32a0c40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.846+0000 7f4439a1b740 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649c3230d60 0x5649c329ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.846+0000 7f4439a1b740 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5649c3226fe0 0x5649c322c5f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.846+0000 7f4439a1b740 1 --2- 172.21.15.102:0/791307391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5649c321c720 0x5649c322c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.846+0000 7f4439a1b740 1 -- 172.21.15.102:0/791307391 >> 172.21.15.102:0/791307391 conn(0x5649c3218310 msgr2=0x5649c3223300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:58.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.846+0000 7f4419600640 0 client.25763 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:56:58.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.847+0000 7f4439a1b740 1 -- 172.21.15.102:0/791307391 shutdown_connections 2024-04-25T01:56:58.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.847+0000 7f4439a1b740 1 -- 172.21.15.102:0/791307391 wait complete. 2024-04-25T01:56:58.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.848+0000 7f4439a1b740 20 client.25763 trim_cache size 0 max 16384 2024-04-25T01:56:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:56:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:56:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:56:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:56:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 -- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f44140a2fd0 msgr2=0x7f44140a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f44140a2fd0 0x7f44140a5490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4430107040 tx=0x7f441c00b410 comp rx=0 tx=0).stop 2024-04-25T01:56:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 -- 172.21.15.102:0/492901838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 msgr2=0x7f443019e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:56:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 0x7f443019e570 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f442002a960 tx=0x7f4420003b90 comp rx=0 tx=0).stop 2024-04-25T01:56:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 -- 172.21.15.102:0/492901838 shutdown_connections 2024-04-25T01:56:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f44140a2fd0 0x7f44140a5490 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44301a2e30 0x7f44301a32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f443010d070 0x7f443019eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 --2- 172.21.15.102:0/492901838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4430107fa0 0x7f443019e570 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:56:58.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.850+0000 7f4436c00640 1 -- 172.21.15.102:0/492901838 >> 172.21.15.102:0/492901838 conn(0x7f4430075cd0 msgr2=0x7f44300767f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:56:58.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.851+0000 7f4436c00640 1 -- 172.21.15.102:0/492901838 shutdown_connections 2024-04-25T01:56:58.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:56:58.851+0000 7f4436c00640 1 -- 172.21.15.102:0/492901838 wait complete. 2024-04-25T01:56:59.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:56:58 smithi193 ceph-mon[32981]: pgmap v54: 97 pgs: 97 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 976 op/s 2024-04-25T01:56:59.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:56:58 smithi102 ceph-mon[27796]: pgmap v54: 97 pgs: 97 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 976 op/s 2024-04-25T01:56:59.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:56:58 smithi110 ceph-mon[32848]: pgmap v54: 97 pgs: 97 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 976 op/s 2024-04-25T01:57:01.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:00 smithi193 ceph-mon[32981]: pgmap v55: 97 pgs: 97 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.2 MiB/s wr, 639 op/s 2024-04-25T01:57:01.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:00 smithi102 ceph-mon[27796]: pgmap v55: 97 pgs: 97 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.2 MiB/s wr, 639 op/s 2024-04-25T01:57:01.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:00 smithi110 ceph-mon[32848]: pgmap v55: 97 pgs: 97 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.2 MiB/s wr, 639 op/s 2024-04-25T01:57:02.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:57:02.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:57:02.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.703+0000 7ffb3d800640 1 -- 172.21.15.102:0/2350994039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb3810de30 msgr2=0x7ffb38110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.703+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2350994039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb3810de30 0x7ffb38110280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009900 tx=0x7ffb3002a480 comp rx=0 tx=0).stop 2024-04-25T01:57:02.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.703+0000 7ffb3d800640 1 -- 172.21.15.102:0/2350994039 shutdown_connections 2024-04-25T01:57:02.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.703+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2350994039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb3810de30 0x7ffb38110280 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.703+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2350994039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb38074490 0x7ffb3810d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.703+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2350994039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb38073b70 0x7ffb38073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.703+0000 7ffb3d800640 1 -- 172.21.15.102:0/2350994039 >> 172.21.15.102:0/2350994039 conn(0x7ffb380fd0e0 msgr2=0x7ffb380ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:02.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.703+0000 7ffb3d800640 1 -- 172.21.15.102:0/2350994039 shutdown_connections 2024-04-25T01:57:02.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.703+0000 7ffb3d800640 1 -- 172.21.15.102:0/2350994039 wait complete. 2024-04-25T01:57:02.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.704+0000 7ffb3d800640 1 Processor -- start 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.704+0000 7ffb3d800640 1 -- start start 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb38073b70 0x7ffb3819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb38074490 0x7ffb3819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb3d800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb3810de30 0x7ffb3819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb3d800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffb38113c20 con 0x7ffb38074490 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb3d800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffb38113aa0 con 0x7ffb38073b70 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb3d800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb38113da0 con 0x7ffb3810de30 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb37e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb38073b70 0x7ffb3819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb2f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb38074490 0x7ffb3819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb3ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb3810de30 0x7ffb3819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb37e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb38073b70 0x7ffb3819a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48062/0 (socket says 172.21.15.102:48062) 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb37e00640 1 -- 172.21.15.102:0/2255677199 learned_addr learned my addr 172.21.15.102:0/2255677199 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb37e00640 1 -- 172.21.15.102:0/2255677199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb3810de30 msgr2=0x7ffb3819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb37e00640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb3810de30 0x7ffb3819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb37e00640 1 -- 172.21.15.102:0/2255677199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb38074490 msgr2=0x7ffb3819a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb37e00640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb38074490 0x7ffb3819a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.705+0000 7ffb37e00640 1 -- 172.21.15.102:0/2255677199 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3819f0a0 con 0x7ffb38073b70 2024-04-25T01:57:02.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.706+0000 7ffb37e00640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb38073b70 0x7ffb3819a100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800eb90 tx=0x7ffb2800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:02.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.706+0000 7ffb35600640 1 -- 172.21.15.102:0/2255677199 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb28004d20 con 0x7ffb38073b70 2024-04-25T01:57:02.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.706+0000 7ffb3d800640 1 -- 172.21.15.102:0/2255677199 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb3819f390 con 0x7ffb38073b70 2024-04-25T01:57:02.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.706+0000 7ffb35600640 1 -- 172.21.15.102:0/2255677199 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb2800ee90 con 0x7ffb38073b70 2024-04-25T01:57:02.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.706+0000 7ffb3d800640 1 -- 172.21.15.102:0/2255677199 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb381a6c20 con 0x7ffb38073b70 2024-04-25T01:57:02.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.707+0000 7ffb35600640 1 -- 172.21.15.102:0/2255677199 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb28010830 con 0x7ffb38073b70 2024-04-25T01:57:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.710+0000 7ffb41c3a740 1 Processor -- start 2024-04-25T01:57:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.710+0000 7ffb35600640 1 -- 172.21.15.102:0/2255677199 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ffb28005020 con 0x7ffb38073b70 2024-04-25T01:57:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.710+0000 7ffb41c3a740 1 -- start start 2024-04-25T01:57:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a87d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a887c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a88a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557b2a7e1560 con 0x557b2a87d1e0 2024-04-25T01:57:02.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557b2a789930 con 0x557b2a8881d0 2024-04-25T01:57:02.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557b2a7dd520 con 0x557b2a87db00 2024-04-25T01:57:02.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb37e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a887c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb3ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a87d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb2f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a88a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb2f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a88a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48066/0 (socket says 172.21.15.102:48066) 2024-04-25T01:57:02.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb3ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a87d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34148/0 (socket says 172.21.15.102:34148) 2024-04-25T01:57:02.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb2f400640 1 -- 172.21.15.102:0/3685373383 learned_addr learned my addr 172.21.15.102:0/3685373383 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb3ce00640 1 -- 172.21.15.102:0/3685373383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 msgr2=0x557b2a887c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/3685373383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a887c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb3ce00640 1 -- 172.21.15.102:0/3685373383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 msgr2=0x557b2a88a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/3685373383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a88a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb3ce00640 1 -- 172.21.15.102:0/3685373383 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b2a5bf7d0 con 0x557b2a87d1e0 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.711+0000 7ffb2f400640 1 --2- 172.21.15.102:0/3685373383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a88a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.712+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/3685373383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a87d5c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffb3002a9a0 tx=0x7ffb30005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38d8df0d69126a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.712+0000 7ffb35600640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffb1c0a2d00 0x7ffb1c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.712+0000 7ffb2d600640 1 -- 172.21.15.102:0/3685373383 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb30040070 con 0x557b2a87d1e0 2024-04-25T01:57:02.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.712+0000 7ffb2d600640 1 -- 172.21.15.102:0/3685373383 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb30004300 con 0x557b2a87d1e0 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.712+0000 7ffb2d600640 1 -- 172.21.15.102:0/3685373383 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb30035040 con 0x557b2a87d1e0 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.712+0000 7ffb35600640 1 -- 172.21.15.102:0/2255677199 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ffb280cf9c0 con 0x7ffb38073b70 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.712+0000 7ffb37e00640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffb1c0a2d00 0x7ffb1c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb41c3a740 1 -- 172.21.15.102:0/3685373383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 msgr2=0x557b2a87d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/3685373383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a87d5c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffb3002a9a0 tx=0x7ffb30005be0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb37e00640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffb1c0a2d00 0x7ffb1c0a51c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb280cd1f0 tx=0x7ffb280d3500 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb41c3a740 1 -- 172.21.15.102:0/3685373383 shutdown_connections 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/3685373383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a88a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/3685373383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a887c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/3685373383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a87d5c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb41c3a740 1 -- 172.21.15.102:0/3685373383 >> 172.21.15.102:0/3685373383 conn(0x557b2a879500 msgr2=0x557b2a8798e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb41c3a740 1 -- 172.21.15.102:0/3685373383 shutdown_connections 2024-04-25T01:57:02.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.713+0000 7ffb41c3a740 1 -- 172.21.15.102:0/3685373383 wait complete. 2024-04-25T01:57:02.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.714+0000 7ffb41c3a740 1 -- start start 2024-04-25T01:57:02.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.714+0000 7ffb41c3a740 1 Processor -- start 2024-04-25T01:57:02.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.714+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a885120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.714+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a885660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.714+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a885ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.714+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557b2a88af60 con 0x557b2a87d1e0 2024-04-25T01:57:02.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.714+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557b2a79cd90 con 0x557b2a8881d0 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.714+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557b2a88b0e0 con 0x557b2a87db00 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.714+0000 7ffb41c3a740 10 client.0 _ll_register_callbacks cb 0x557b2a79c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb2f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a885660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb3ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a885120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb2f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a885660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:59022/0 (socket says 172.21.15.102:59022) 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb2f400640 1 -- 172.21.15.102:0/293420908 learned_addr learned my addr 172.21.15.102:0/293420908 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a885ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb2f400640 1 -- 172.21.15.102:0/293420908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 msgr2=0x557b2a885ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb2f400640 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a885ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb2f400640 1 -- 172.21.15.102:0/293420908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 msgr2=0x557b2a885120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb2f400640 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a885120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb2f400640 1 -- 172.21.15.102:0/293420908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b2a4f53f0 con 0x557b2a87db00 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb2f400640 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a885660 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ffb200126e0 tx=0x7ffb20012ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2ccd6f86139bfde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:02.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.715+0000 7ffb26000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.716+0000 7ffb26000640 1 -- 172.21.15.102:0/293420908 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb200121d0 con 0x557b2a87db00 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.716+0000 7ffb41c3a740 1 -- 172.21.15.102:0/293420908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb30004300 con 0x557b2a87db00 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.716+0000 7ffb26000640 1 -- 172.21.15.102:0/293420908 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ffb20004050 con 0x557b2a87db00 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.716+0000 7ffb26000640 1 -- 172.21.15.102:0/293420908 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffb20004250 con 0x557b2a87db00 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.716+0000 7ffb41c3a740 10 client.25756 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.716+0000 7ffb41c3a740 1 -- 172.21.15.102:0/293420908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557b2a4b50f0 con 0x557b2a87db00 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.716+0000 7ffb26000640 1 -- 172.21.15.102:0/293420908 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb30004300 con 0x557b2a87db00 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.716+0000 7ffb26000640 1 -- 172.21.15.102:0/293420908 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffb20003af0 con 0x557b2a87db00 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.716+0000 7ffb41c3a740 10 client.25756 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.717+0000 7ffb41c3a740 10 client.25756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.717+0000 7ffb41c3a740 20 client.25756 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.717+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557b2a898560 0x557b2a89a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.717+0000 7ffb41c3a740 4 client.25756 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.717+0000 7ffb41c3a740 1 -- 172.21.15.102:0/293420908 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b2a7f4d90 con 0x557b2a898560 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.717+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557b2a898560 0x557b2a89a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.717+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557b2a898560 0x557b2a89a9c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1095d38df69123b6 server_cookie=db8bca377ee64c3b in_seq=0 out_seq=0 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.717+0000 7ffb26000640 10 client.25756 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb26000640 1 -- 172.21.15.102:0/293420908 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b2a7f4d90 con 0x557b2a898560 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb26000640 10 client.25756 handle_command_reply: tid=0 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb41c3a740 1 client.25756 shutdown 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb41c3a740 1 -- 172.21.15.102:0/293420908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 msgr2=0x557b2a885660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a885660 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ffb200126e0 tx=0x7ffb20012ba0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb41c3a740 1 -- 172.21.15.102:0/293420908 shutdown_connections 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557b2a898560 0x557b2a89a9c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a8881d0 0x557b2a885ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb26000640 0 client.25756 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a87db00 0x557b2a885660 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/293420908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87d1e0 0x557b2a885120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.718+0000 7ffb41c3a740 1 -- 172.21.15.102:0/293420908 >> 172.21.15.102:0/293420908 conn(0x557b2a879500 msgr2=0x557b2a878800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.719+0000 7ffb41c3a740 1 -- 172.21.15.102:0/293420908 shutdown_connections 2024-04-25T01:57:02.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.719+0000 7ffb41c3a740 1 -- 172.21.15.102:0/293420908 wait complete. 2024-04-25T01:57:02.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.719+0000 7ffb41c3a740 20 client.25756 trim_cache size 0 max 16384 2024-04-25T01:57:02.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.739+0000 7ffb41c3a740 1 Processor -- start 2024-04-25T01:57:02.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.739+0000 7ffb41c3a740 1 -- start start 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.739+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a878f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.739+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 0x557b2a893e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.739+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a887060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.739+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffb14010ff0 con 0x557b2a87db00 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.739+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1400e860 con 0x557b2a87d1e0 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.739+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1400c6b0 con 0x557b2a8881d0 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a878f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a878f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:48078/0 (socket says 172.21.15.102:48078) 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 -- 172.21.15.102:0/2193360397 learned_addr learned my addr 172.21.15.102:0/2193360397 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb2f400640 1 --2- 172.21.15.102:0/2193360397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 0x557b2a893e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/2193360397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a887060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 -- 172.21.15.102:0/2193360397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 msgr2=0x557b2a887060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:02.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/2193360397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a887060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 -- 172.21.15.102:0/2193360397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 msgr2=0x557b2a893e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:02.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/2193360397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 0x557b2a893e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 -- 172.21.15.102:0/2193360397 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b2a88a5c0 con 0x557b2a87d1e0 2024-04-25T01:57:02.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.740+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/2193360397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a878f80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffb30000c00 tx=0x7ffb300314b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9395347a02fdfc31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:02.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.741+0000 7ffb2cc00640 1 -- 172.21.15.102:0/2193360397 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb30035040 con 0x557b2a87d1e0 2024-04-25T01:57:02.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.741+0000 7ffb2cc00640 1 -- 172.21.15.102:0/2193360397 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb30005700 con 0x557b2a87d1e0 2024-04-25T01:57:02.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.741+0000 7ffb2cc00640 1 -- 172.21.15.102:0/2193360397 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb30040040 con 0x557b2a87d1e0 2024-04-25T01:57:02.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.741+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2193360397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 msgr2=0x557b2a878f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.741+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2193360397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a878f80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffb30000c00 tx=0x7ffb300314b0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.742+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2193360397 shutdown_connections 2024-04-25T01:57:02.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.742+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2193360397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a887060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.742+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2193360397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 0x557b2a893e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.742+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2193360397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a878f80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.742+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2193360397 >> 172.21.15.102:0/2193360397 conn(0x557b2a879500 msgr2=0x557b2a886950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:02.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.742+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2193360397 shutdown_connections 2024-04-25T01:57:02.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.742+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2193360397 wait complete. 2024-04-25T01:57:02.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.742+0000 7ffb41c3a740 1 -- start start 2024-04-25T01:57:02.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.743+0000 7ffb41c3a740 1 Processor -- start 2024-04-25T01:57:02.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.743+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a890840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.743+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 0x557b2a881c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.743+0000 7ffb41c3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a882180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.743+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1400cd20 con 0x557b2a87db00 2024-04-25T01:57:02.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.743+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1400e190 con 0x557b2a87d1e0 2024-04-25T01:57:02.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.743+0000 7ffb41c3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1400d3f0 con 0x557b2a8881d0 2024-04-25T01:57:02.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.743+0000 7ffb41c3a740 10 client.0 _ll_register_callbacks cb 0x7ffb20009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:02.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.743+0000 7ffb3ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a882180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb3ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a882180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:59046/0 (socket says 172.21.15.102:59046) 2024-04-25T01:57:02.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb3ce00640 1 -- 172.21.15.102:0/2048270648 learned_addr learned my addr 172.21.15.102:0/2048270648 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a890840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb2f400640 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 0x557b2a881c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb3ce00640 1 -- 172.21.15.102:0/2048270648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 msgr2=0x557b2a890840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a890840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb3ce00640 1 -- 172.21.15.102:0/2048270648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 msgr2=0x557b2a881c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 0x557b2a881c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb3ce00640 1 -- 172.21.15.102:0/2048270648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b2a8829b0 con 0x557b2a8881d0 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb3ce00640 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a882180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffb30033000 tx=0x7ffb30005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e510ea565ed17bf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.744+0000 7ffb26000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb26000640 1 -- 172.21.15.102:0/2048270648 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb30040040 con 0x557b2a8881d0 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb26000640 1 -- 172.21.15.102:0/2048270648 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ffb30032c90 con 0x557b2a8881d0 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2048270648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb30005700 con 0x557b2a8881d0 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb26000640 1 -- 172.21.15.102:0/2048270648 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffb30053150 con 0x557b2a8881d0 2024-04-25T01:57:02.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb41c3a740 10 client.25762 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:02.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2048270648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557b2a907400 con 0x557b2a8881d0 2024-04-25T01:57:02.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb26000640 1 -- 172.21.15.102:0/2048270648 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb30005700 con 0x557b2a8881d0 2024-04-25T01:57:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb26000640 1 -- 172.21.15.102:0/2048270648 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffb30003bf0 con 0x557b2a8881d0 2024-04-25T01:57:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb41c3a740 10 client.25762 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb41c3a740 10 client.25762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb41c3a740 20 client.25762 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557b2a898560 0x557b2a909690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb41c3a740 4 client.25762 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.745+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2048270648 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557b2a7eeaa0 con 0x557b2a898560 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.746+0000 7ffb2f400640 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557b2a898560 0x557b2a909690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.746+0000 7ffb2f400640 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557b2a898560 0x557b2a909690 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0bd19a33aca842f server_cookie=6c8275471fbaf312 in_seq=0 out_seq=0 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.746+0000 7ffb26000640 10 client.25762 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb26000640 1 -- 172.21.15.102:0/2048270648 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557b2a7eeaa0 con 0x557b2a898560 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb26000640 10 client.25762 handle_command_reply: tid=0 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 client.25762 shutdown 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2048270648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 msgr2=0x557b2a882180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a882180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ffb30033000 tx=0x7ffb30005fd0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2048270648 shutdown_connections 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557b2a898560 0x557b2a909690 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b2a8881d0 0x557b2a882180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b2a87db00 0x557b2a881c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 --2- 172.21.15.102:0/2048270648 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557b2a87d1e0 0x557b2a890840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2048270648 >> 172.21.15.102:0/2048270648 conn(0x557b2a879500 msgr2=0x557b2a8871e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb26000640 0 client.25762 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2048270648 shutdown_connections 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 1 -- 172.21.15.102:0/2048270648 wait complete. 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.747+0000 7ffb41c3a740 20 client.25762 trim_cache size 0 max 16384 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T01:57:02.753 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:57:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.748+0000 7ffb3d800640 1 -- 172.21.15.102:0/2255677199 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffb1c0a2d00 msgr2=0x7ffb1c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.748+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffb1c0a2d00 0x7ffb1c0a51c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb280cd1f0 tx=0x7ffb280d3500 comp rx=0 tx=0).stop 2024-04-25T01:57:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.748+0000 7ffb3d800640 1 -- 172.21.15.102:0/2255677199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb38073b70 msgr2=0x7ffb3819a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.748+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb38073b70 0x7ffb3819a100 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800eb90 tx=0x7ffb2800c6a0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.749+0000 7ffb3d800640 1 -- 172.21.15.102:0/2255677199 shutdown_connections 2024-04-25T01:57:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.749+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffb1c0a2d00 0x7ffb1c0a51c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.749+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb3810de30 0x7ffb3819e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.749+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb38074490 0x7ffb3819a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.749+0000 7ffb3d800640 1 --2- 172.21.15.102:0/2255677199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb38073b70 0x7ffb3819a100 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.749+0000 7ffb3d800640 1 -- 172.21.15.102:0/2255677199 >> 172.21.15.102:0/2255677199 conn(0x7ffb380fd0e0 msgr2=0x7ffb380fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.749+0000 7ffb3d800640 1 -- 172.21.15.102:0/2255677199 shutdown_connections 2024-04-25T01:57:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.749+0000 7ffb3d800640 1 -- 172.21.15.102:0/2255677199 wait complete. 2024-04-25T01:57:02.771 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:57:02.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:57:02.838 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:02 smithi102 ceph-mon[27796]: pgmap v56: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 13 MiB/s wr, 955 op/s 2024-04-25T01:57:02.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.973+0000 7fb283a00640 1 -- 172.21.15.102:0/1835383854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c10d090 msgr2=0x7fb27c114c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.973+0000 7fb283a00640 1 --2- 172.21.15.102:0/1835383854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c10d090 0x7fb27c114c60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb274009be0 tx=0x7fb27402a6c0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.973+0000 7fb283a00640 1 -- 172.21.15.102:0/1835383854 shutdown_connections 2024-04-25T01:57:02.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.973+0000 7fb283a00640 1 --2- 172.21.15.102:0/1835383854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c10d090 0x7fb27c114c60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.973+0000 7fb283a00640 1 --2- 172.21.15.102:0/1835383854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb27c108970 0x7fb27c10c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.973+0000 7fb283a00640 1 --2- 172.21.15.102:0/1835383854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb27c107fc0 0x7fb27c1083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.973+0000 7fb283a00640 1 -- 172.21.15.102:0/1835383854 >> 172.21.15.102:0/1835383854 conn(0x7fb27c074f40 msgr2=0x7fb27c106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:02.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.973+0000 7fb283a00640 1 -- 172.21.15.102:0/1835383854 shutdown_connections 2024-04-25T01:57:02.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.973+0000 7fb283a00640 1 -- 172.21.15.102:0/1835383854 wait complete. 2024-04-25T01:57:02.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.974+0000 7fb283a00640 1 Processor -- start 2024-04-25T01:57:02.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.974+0000 7fb283a00640 1 -- start start 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb283a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb27c107fc0 0x7fb27c19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb283a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb27c10d090 0x7fb27c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb283a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1a2ef0 0x7fb27c1a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb283a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c118130 con 0x7fb27c107fc0 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb283a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c117fb0 con 0x7fb27c10d090 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb283a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1182b0 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb27be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb27c10d090 0x7fb27c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb281a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1a2ef0 0x7fb27c1a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb27be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb27c10d090 0x7fb27c19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36322/0 (socket says 172.21.15.102:36322) 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb27be00640 1 -- 172.21.15.102:0/2236761601 learned_addr learned my addr 172.21.15.102:0/2236761601 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:02.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.975+0000 7fb281000640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb27c107fc0 0x7fb27c19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb281a00640 1 -- 172.21.15.102:0/2236761601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb27c10d090 msgr2=0x7fb27c19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb281a00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb27c10d090 0x7fb27c19eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb281a00640 1 -- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb27c107fc0 msgr2=0x7fb27c19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:02.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb281a00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb27c107fc0 0x7fb27c19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:02.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb281a00640 1 -- 172.21.15.102:0/2236761601 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c1a39e0 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb27be00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb27c10d090 0x7fb27c19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:02.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb281a00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1a2ef0 0x7fb27c1a33a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb274009b30 tx=0x7fb2740305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:02.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb279600640 1 -- 172.21.15.102:0/2236761601 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274041020 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb279600640 1 -- 172.21.15.102:0/2236761601 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb274003e50 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb279600640 1 -- 172.21.15.102:0/2236761601 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274036040 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.976+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27c1a3c70 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.977+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27c1ab380 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.978+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb27c0689b0 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.980+0000 7fb279600640 1 -- 172.21.15.102:0/2236761601 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb274044020 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.982+0000 7fb279600640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2640a2d80 0x7fb2640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:02.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.982+0000 7fb279600640 1 -- 172.21.15.102:0/2236761601 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb2740eff50 con 0x7fb27c1a2ef0 2024-04-25T01:57:02.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.982+0000 7fb281000640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2640a2d80 0x7fb2640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:02.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.983+0000 7fb281000640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2640a2d80 0x7fb2640a5240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb270009b60 tx=0x7fb270005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:02.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:02.986+0000 7fb279600640 1 -- 172.21.15.102:0/2236761601 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb27404b020 con 0x7fb27c1a2ef0 2024-04-25T01:57:03.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:02 smithi193 ceph-mon[32981]: pgmap v56: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 13 MiB/s wr, 955 op/s 2024-04-25T01:57:03.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 70.0 s, 2107.8 tps, lat 46.821 ms stddev 100.189 2024-04-25T01:57:03.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:02 smithi110 ceph-mon[32848]: pgmap v56: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 13 MiB/s wr, 955 op/s 2024-04-25T01:57:03.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.531+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb27c108970 con 0x7fb27c1a2ef0 2024-04-25T01:57:03.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.533+0000 7fb279600640 1 -- 172.21.15.102:0/2236761601 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11901 (secure 0 0 0) 0x7fb2740bcb40 con 0x7fb27c1a2ef0 2024-04-25T01:57:03.537 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:57:03.537 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:56:02.805319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24541,24587,24533]},"id":1}]} 2024-04-25T01:57:03.537 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-25T01:57:03.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.536+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2640a2d80 msgr2=0x7fb2640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:03.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.536+0000 7fb283a00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2640a2d80 0x7fb2640a5240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb270009b60 tx=0x7fb270005f90 comp rx=0 tx=0).stop 2024-04-25T01:57:03.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1a2ef0 msgr2=0x7fb27c1a33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:03.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1a2ef0 0x7fb27c1a33a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb274009b30 tx=0x7fb2740305d0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 shutdown_connections 2024-04-25T01:57:03.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2640a2d80 0x7fb2640a5240 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1a2ef0 0x7fb27c1a33a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb27c10d090 0x7fb27c19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 --2- 172.21.15.102:0/2236761601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb27c107fc0 0x7fb27c19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 >> 172.21.15.102:0/2236761601 conn(0x7fb27c074f40 msgr2=0x7fb27c075500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 shutdown_connections 2024-04-25T01:57:03.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.537+0000 7fb283a00640 1 -- 172.21.15.102:0/2236761601 wait complete. 2024-04-25T01:57:03.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:57:03.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.917+0000 7f0568a00640 1 -- 172.21.15.102:0/2214758314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056406a3b0 msgr2=0x7f056410e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:03.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.917+0000 7f0568a00640 1 --2- 172.21.15.102:0/2214758314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056406a3b0 0x7f056410e3b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0554009be0 tx=0x7f055402a6d0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.918+0000 7f0568a00640 1 -- 172.21.15.102:0/2214758314 shutdown_connections 2024-04-25T01:57:03.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.918+0000 7f0568a00640 1 --2- 172.21.15.102:0/2214758314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056406a3b0 0x7f056410e3b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.918+0000 7f0568a00640 1 --2- 172.21.15.102:0/2214758314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0564069a10 0x7f0564069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.918+0000 7f0568a00640 1 --2- 172.21.15.102:0/2214758314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05641075c0 0x7f05641079a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.918+0000 7f0568a00640 1 -- 172.21.15.102:0/2214758314 >> 172.21.15.102:0/2214758314 conn(0x7f05640777a0 msgr2=0x7f0564077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:03.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.918+0000 7f0568a00640 1 -- 172.21.15.102:0/2214758314 shutdown_connections 2024-04-25T01:57:03.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.918+0000 7f0568a00640 1 -- 172.21.15.102:0/2214758314 wait complete. 2024-04-25T01:57:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.920+0000 7f0568a00640 1 Processor -- start 2024-04-25T01:57:03.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.921+0000 7f0568a00640 1 -- start start 2024-04-25T01:57:03.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.922+0000 7f0568a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0564069a10 0x7f0564073040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.922+0000 7f0568a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056406a3b0 0x7f056406c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.922+0000 7f0568a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05641075c0 0x7f056406c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.922+0000 7f0568a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f056410f6c0 con 0x7f056406a3b0 2024-04-25T01:57:03.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.922+0000 7f0568a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f056410f540 con 0x7f0564069a10 2024-04-25T01:57:03.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.922+0000 7f0568a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f056410f840 con 0x7f05641075c0 2024-04-25T01:57:03.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.922+0000 7f0561e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05641075c0 0x7f056406c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:03.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.922+0000 7f0561e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05641075c0 0x7f056406c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41414/0 (socket says 172.21.15.102:41414) 2024-04-25T01:57:03.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.922+0000 7f0561e00640 1 -- 172.21.15.102:0/214788615 learned_addr learned my addr 172.21.15.102:0/214788615 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:03.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:03 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2236761601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:57:03.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.923+0000 7f0561e00640 1 -- 172.21.15.102:0/214788615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0564069a10 msgr2=0x7f0564073040 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:57:03.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.923+0000 7f0561e00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0564069a10 0x7f0564073040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.923+0000 7f0561e00640 1 -- 172.21.15.102:0/214788615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056406a3b0 msgr2=0x7f056406c0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:57:03.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.924+0000 7f0561e00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056406a3b0 0x7f056406c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.924+0000 7f0561e00640 1 -- 172.21.15.102:0/214788615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056406cd30 con 0x7f05641075c0 2024-04-25T01:57:03.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.925+0000 7f0561e00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05641075c0 0x7f056406c600 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0554034040 tx=0x7f05540305b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:03.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.927+0000 7f055aa00640 1 -- 172.21.15.102:0/214788615 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554041020 con 0x7f05641075c0 2024-04-25T01:57:03.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.927+0000 7f055aa00640 1 -- 172.21.15.102:0/214788615 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0554037040 con 0x7f05641075c0 2024-04-25T01:57:03.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.927+0000 7f0568a00640 1 -- 172.21.15.102:0/214788615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0564070b60 con 0x7f05641075c0 2024-04-25T01:57:03.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.927+0000 7f0568a00640 1 -- 172.21.15.102:0/214788615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05640710a0 con 0x7f05641075c0 2024-04-25T01:57:03.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.927+0000 7f055aa00640 1 -- 172.21.15.102:0/214788615 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0554036070 con 0x7f05641075c0 2024-04-25T01:57:03.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.934+0000 7f056b764740 1 Processor -- start 2024-04-25T01:57:03.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.934+0000 7f055aa00640 1 -- 172.21.15.102:0/214788615 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0554044020 con 0x7f05641075c0 2024-04-25T01:57:03.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.938+0000 7f056b764740 1 -- start start 2024-04-25T01:57:03.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.938+0000 7f056b764740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0238b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.938+0000 7f056b764740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0243250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.938+0000 7f056b764740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0245b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.938+0000 7f056b764740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5635d019d4e0 con 0x5635d0243790 2024-04-25T01:57:03.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.938+0000 7f056b764740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5635d0145a70 con 0x5635d02387a0 2024-04-25T01:57:03.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.938+0000 7f056b764740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5635d01994a0 con 0x5635d02390c0 2024-04-25T01:57:03.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.939+0000 7f0560a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0238b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:03.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.940+0000 7f0561400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0243250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:03.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.940+0000 7f0561400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0243250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41418/0 (socket says 172.21.15.102:41418) 2024-04-25T01:57:03.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.940+0000 7f0561400640 1 -- 172.21.15.102:0/3568163496 learned_addr learned my addr 172.21.15.102:0/3568163496 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:03.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.940+0000 7f055aa00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f05480a7210 0x7f05480a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.940+0000 7f055aa00640 1 -- 172.21.15.102:0/214788615 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f05540f01d0 con 0x7f05641075c0 2024-04-25T01:57:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.942+0000 7f0561400640 1 -- 172.21.15.102:0/3568163496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 msgr2=0x5635d0238b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.942+0000 7f0561400640 1 --2- 172.21.15.102:0/3568163496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0238b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.942+0000 7f0561400640 1 -- 172.21.15.102:0/3568163496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 msgr2=0x5635d0245b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:03.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.942+0000 7f0561400640 1 --2- 172.21.15.102:0/3568163496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0245b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.942+0000 7f0561400640 1 -- 172.21.15.102:0/3568163496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635cfeb13f0 con 0x5635d02390c0 2024-04-25T01:57:03.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.942+0000 7f0560a00640 1 --2- 172.21.15.102:0/3568163496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0238b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:03.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.943+0000 7f0561400640 1 --2- 172.21.15.102:0/3568163496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0243250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0550009ae0 tx=0x7f0550009670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95ce2d80cee48f7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:03.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.944+0000 7f054f000640 1 -- 172.21.15.102:0/3568163496 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055000ecb0 con 0x5635d02390c0 2024-04-25T01:57:03.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.944+0000 7f054f000640 1 -- 172.21.15.102:0/3568163496 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f055000ee50 con 0x5635d02390c0 2024-04-25T01:57:03.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.944+0000 7f054f000640 1 -- 172.21.15.102:0/3568163496 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550010690 con 0x5635d02390c0 2024-04-25T01:57:03.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.945+0000 7f0561e00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f05480a7210 0x7f05480a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:03.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.946+0000 7f056b764740 1 -- 172.21.15.102:0/3568163496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 msgr2=0x5635d0243250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:03.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.946+0000 7f056b764740 1 --2- 172.21.15.102:0/3568163496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0243250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0550009ae0 tx=0x7f0550009670 comp rx=0 tx=0).stop 2024-04-25T01:57:03.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.946+0000 7f056b764740 1 -- 172.21.15.102:0/3568163496 shutdown_connections 2024-04-25T01:57:03.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.946+0000 7f056b764740 1 --2- 172.21.15.102:0/3568163496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0245b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.946+0000 7f056b764740 1 --2- 172.21.15.102:0/3568163496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0243250 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.946+0000 7f056b764740 1 --2- 172.21.15.102:0/3568163496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0238b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.946+0000 7f056b764740 1 -- 172.21.15.102:0/3568163496 >> 172.21.15.102:0/3568163496 conn(0x5635d0234ac0 msgr2=0x5635d0234ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:03.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.946+0000 7f056b764740 1 -- 172.21.15.102:0/3568163496 shutdown_connections 2024-04-25T01:57:03.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.946+0000 7f056b764740 1 -- 172.21.15.102:0/3568163496 wait complete. 2024-04-25T01:57:03.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.947+0000 7f056b764740 1 -- start start 2024-04-25T01:57:03.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.948+0000 7f056b764740 1 Processor -- start 2024-04-25T01:57:03.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.950+0000 7f056b764740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d02406c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.950+0000 7f056b764740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0240c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.950+0000 7f056b764740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0241140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.950+0000 7f056b764740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5635d0246580 con 0x5635d0243790 2024-04-25T01:57:03.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.950+0000 7f056b764740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5635d0158ec0 con 0x5635d02387a0 2024-04-25T01:57:03.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.950+0000 7f056b764740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5635d0246700 con 0x5635d02390c0 2024-04-25T01:57:03.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.950+0000 7f056b764740 10 client.0 _ll_register_callbacks cb 0x5635d01953f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:03.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.951+0000 7f0561e00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f05480a7210 0x7f05480a96d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f05540041b0 tx=0x7f0554009500 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:03.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.951+0000 7f0560a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d02406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:03.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.951+0000 7f0560a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d02406c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36346/0 (socket says 172.21.15.102:36346) 2024-04-25T01:57:03.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.951+0000 7f0560a00640 1 -- 172.21.15.102:0/36968655 learned_addr learned my addr 172.21.15.102:0/36968655 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:03.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.952+0000 7f0561400640 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0240c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:03.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.952+0000 7f0560a00640 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0241140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:03.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.952+0000 7f0561400640 1 -- 172.21.15.102:0/36968655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 msgr2=0x5635d02406c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:03.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.952+0000 7f0561400640 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d02406c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.953+0000 7f0561400640 1 -- 172.21.15.102:0/36968655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 msgr2=0x5635d0241140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:03.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.953+0000 7f0561400640 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0241140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.953+0000 7f0561400640 1 -- 172.21.15.102:0/36968655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635cff7b7d0 con 0x5635d02390c0 2024-04-25T01:57:03.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.953+0000 7f0560a00640 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0241140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:03.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.953+0000 7f0561400640 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0240c00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0550009ae0 tx=0x7f0550003e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbbe8429e2b7139f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:03.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.953+0000 7f0543400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:03.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.955+0000 7f0543400640 1 -- 172.21.15.102:0/36968655 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055001d040 con 0x5635d02390c0 2024-04-25T01:57:03.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.955+0000 7f0543400640 1 -- 172.21.15.102:0/36968655 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0550004220 con 0x5635d02390c0 2024-04-25T01:57:03.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.955+0000 7f056b764740 1 -- 172.21.15.102:0/36968655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f055000ee50 con 0x5635d02390c0 2024-04-25T01:57:03.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.956+0000 7f0543400640 1 -- 172.21.15.102:0/36968655 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0550005d10 con 0x5635d02390c0 2024-04-25T01:57:03.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.956+0000 7f056b764740 10 client.25780 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:03.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.957+0000 7f056b764740 1 -- 172.21.15.102:0/36968655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5635d025adc0 con 0x5635d02390c0 2024-04-25T01:57:03.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.959+0000 7f0543400640 1 -- 172.21.15.102:0/36968655 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f055000ee50 con 0x5635d02390c0 2024-04-25T01:57:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.959+0000 7f0543400640 1 -- 172.21.15.102:0/36968655 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0550004aa0 con 0x5635d02390c0 2024-04-25T01:57:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.961+0000 7f056b764740 10 client.25780 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.961+0000 7f056b764740 10 client.25780 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:57:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.961+0000 7f056b764740 20 client.25780 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.961+0000 7f056b764740 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x5635d025c110 0x5635d025e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.961+0000 7f056b764740 4 client.25780 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:03.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.961+0000 7f056b764740 1 -- 172.21.15.102:0/36968655 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635d01aaa20 con 0x5635d025c110 2024-04-25T01:57:03.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.961+0000 7f0560a00640 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x5635d025c110 0x5635d025e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.963+0000 7f0560a00640 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x5635d025c110 0x5635d025e500 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac21ea8a1cd527a5 server_cookie=a977711f9b6c8ca0 in_seq=0 out_seq=0 2024-04-25T01:57:03.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.963+0000 7f0543400640 10 client.25780 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:57:03.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.964+0000 7f0543400640 1 -- 172.21.15.102:0/36968655 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635d01aaa20 con 0x5635d025c110 2024-04-25T01:57:03.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.964+0000 7f0543400640 10 client.25780 handle_command_reply: tid=0 2024-04-25T01:57:03.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.965+0000 7f056b764740 1 client.25780 shutdown 2024-04-25T01:57:03.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.966+0000 7f056b764740 1 -- 172.21.15.102:0/36968655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 msgr2=0x5635d0240c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.966+0000 7f056b764740 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0240c00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0550009ae0 tx=0x7f0550003e80 comp rx=0 tx=0).stop 2024-04-25T01:57:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.967+0000 7f056b764740 1 -- 172.21.15.102:0/36968655 shutdown_connections 2024-04-25T01:57:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.967+0000 7f056b764740 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0241140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.967+0000 7f056b764740 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02390c0 0x5635d0240c00 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0550009ae0 tx=0x7f0550003e80 comp rx=0 tx=0).stop 2024-04-25T01:57:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.967+0000 7f056b764740 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x5635d025c110 0x5635d025e500 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.967+0000 7f056b764740 1 --2- 172.21.15.102:0/36968655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d02406c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.967+0000 7f056b764740 1 -- 172.21.15.102:0/36968655 >> 172.21.15.102:0/36968655 conn(0x5635d0234ac0 msgr2=0x5635d023a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:03.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.967+0000 7f0543400640 0 client.25780 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:57:03.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.967+0000 7f056b764740 1 -- 172.21.15.102:0/36968655 shutdown_connections 2024-04-25T01:57:03.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.968+0000 7f056b764740 1 -- 172.21.15.102:0/36968655 wait complete. 2024-04-25T01:57:03.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:03.968+0000 7f056b764740 20 client.25780 trim_cache size 0 max 16384 2024-04-25T01:57:04.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.001+0000 7f056b764740 1 Processor -- start 2024-04-25T01:57:04.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.001+0000 7f056b764740 1 -- start start 2024-04-25T01:57:04.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.002+0000 7f056b764740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0257d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.002+0000 7f056b764740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0240f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.002+0000 7f056b764740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02339c0 0x5635d0233e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.002+0000 7f056b764740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0544011100 con 0x5635d0243790 2024-04-25T01:57:04.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.002+0000 7f056b764740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f054400e970 con 0x5635d02387a0 2024-04-25T01:57:04.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.002+0000 7f056b764740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f054400c7c0 con 0x5635d02339c0 2024-04-25T01:57:04.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.003+0000 7f0560a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0257d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.003+0000 7f0560a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0257d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36356/0 (socket says 172.21.15.102:36356) 2024-04-25T01:57:04.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.003+0000 7f0560a00640 1 -- 172.21.15.102:0/1919673248 learned_addr learned my addr 172.21.15.102:0/1919673248 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.004+0000 7f0560a00640 1 --2- 172.21.15.102:0/1919673248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02339c0 0x5635d0233e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.007+0000 7f0561400640 1 --2- 172.21.15.102:0/1919673248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0240f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.007+0000 7f0560a00640 1 -- 172.21.15.102:0/1919673248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02339c0 msgr2=0x5635d0233e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.007+0000 7f0560a00640 1 --2- 172.21.15.102:0/1919673248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02339c0 0x5635d0233e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.007+0000 7f0560a00640 1 -- 172.21.15.102:0/1919673248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 msgr2=0x5635d0240f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.007+0000 7f0560a00640 1 --2- 172.21.15.102:0/1919673248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0240f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.008+0000 7f0560a00640 1 -- 172.21.15.102:0/1919673248 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d0238ef0 con 0x5635d02387a0 2024-04-25T01:57:04.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.008+0000 7f0560a00640 1 --2- 172.21.15.102:0/1919673248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0257d60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f055c010630 tx=0x7f055c000c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8940ce08d62c4049 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.010+0000 7f054e600640 1 -- 172.21.15.102:0/1919673248 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c003e60 con 0x5635d02387a0 2024-04-25T01:57:04.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.010+0000 7f054e600640 1 -- 172.21.15.102:0/1919673248 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f055c010bc0 con 0x5635d02387a0 2024-04-25T01:57:04.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.010+0000 7f054e600640 1 -- 172.21.15.102:0/1919673248 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c012450 con 0x5635d02387a0 2024-04-25T01:57:04.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.011+0000 7f056b764740 1 -- 172.21.15.102:0/1919673248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 msgr2=0x5635d0257d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.011+0000 7f056b764740 1 --2- 172.21.15.102:0/1919673248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0257d60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f055c010630 tx=0x7f055c000c30 comp rx=0 tx=0).stop 2024-04-25T01:57:04.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.012+0000 7f056b764740 1 -- 172.21.15.102:0/1919673248 shutdown_connections 2024-04-25T01:57:04.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.012+0000 7f056b764740 1 --2- 172.21.15.102:0/1919673248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d02339c0 0x5635d0233e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.012+0000 7f056b764740 1 --2- 172.21.15.102:0/1919673248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d0243790 0x5635d0240f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.012+0000 7f056b764740 1 --2- 172.21.15.102:0/1919673248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02387a0 0x5635d0257d60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.012+0000 7f056b764740 1 -- 172.21.15.102:0/1919673248 >> 172.21.15.102:0/1919673248 conn(0x5635d0234ac0 msgr2=0x5635d02332b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.012+0000 7f056b764740 1 -- 172.21.15.102:0/1919673248 shutdown_connections 2024-04-25T01:57:04.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.012+0000 7f056b764740 1 -- 172.21.15.102:0/1919673248 wait complete. 2024-04-25T01:57:04.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.013+0000 7f056b764740 1 -- start start 2024-04-25T01:57:04.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.013+0000 7f056b764740 1 Processor -- start 2024-04-25T01:57:04.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.017+0000 7f056b764740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02339c0 0x5635d02544c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.017+0000 7f056b764740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d02387a0 0x5635d024f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.018+0000 7f056b764740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d0243790 0x5635d024fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.018+0000 7f056b764740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f054400ce30 con 0x5635d02387a0 2024-04-25T01:57:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.018+0000 7f056b764740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f054400e2a0 con 0x5635d02339c0 2024-04-25T01:57:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.018+0000 7f056b764740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f054400d500 con 0x5635d0243790 2024-04-25T01:57:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.018+0000 7f0560a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02339c0 0x5635d02544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.018+0000 7f0560a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02339c0 0x5635d02544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36368/0 (socket says 172.21.15.102:36368) 2024-04-25T01:57:04.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.018+0000 7f0560a00640 1 -- 172.21.15.102:0/1187591285 learned_addr learned my addr 172.21.15.102:0/1187591285 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.018+0000 7f0560a00640 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d0243790 0x5635d024fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.019+0000 7f056b764740 10 client.0 _ll_register_callbacks cb 0x7f05500099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.019+0000 7f0560a00640 1 -- 172.21.15.102:0/1187591285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d0243790 msgr2=0x5635d024fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.019+0000 7f0560a00640 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d0243790 0x5635d024fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.020+0000 7f0560a00640 1 -- 172.21.15.102:0/1187591285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d02387a0 msgr2=0x5635d024f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.020+0000 7f0560a00640 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d02387a0 0x5635d024f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.020+0000 7f0560a00640 1 -- 172.21.15.102:0/1187591285 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d02503e0 con 0x5635d02339c0 2024-04-25T01:57:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.021+0000 7f0560a00640 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02339c0 0x5635d02544c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f055c000c00 tx=0x7f055c009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e256fe24f135de5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.021+0000 7f0543400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:04.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.022+0000 7f0543400640 1 -- 172.21.15.102:0/1187591285 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c001d90 con 0x5635d02339c0 2024-04-25T01:57:04.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.022+0000 7f056b764740 1 -- 172.21.15.102:0/1187591285 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f055c010bc0 con 0x5635d02339c0 2024-04-25T01:57:04.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.022+0000 7f0543400640 1 -- 172.21.15.102:0/1187591285 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f055c010660 con 0x5635d02339c0 2024-04-25T01:57:04.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.023+0000 7f0543400640 1 -- 172.21.15.102:0/1187591285 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f055c010860 con 0x5635d02339c0 2024-04-25T01:57:04.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.023+0000 7f056b764740 10 client.25799 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:04.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.023+0000 7f056b764740 1 -- 172.21.15.102:0/1187591285 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5635d0246be0 con 0x5635d02339c0 2024-04-25T01:57:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.023+0000 7f0543400640 1 -- 172.21.15.102:0/1187591285 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f055c010bc0 con 0x5635d02339c0 2024-04-25T01:57:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.024+0000 7f0543400640 1 -- 172.21.15.102:0/1187591285 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f055c02b7e0 con 0x5635d02339c0 2024-04-25T01:57:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.024+0000 7f056b764740 10 client.25799 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:04.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.024+0000 7f056b764740 10 client.25799 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:57:04.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.024+0000 7f056b764740 20 client.25799 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:04.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.024+0000 7f056b764740 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x5635d02472a0 0x5635d02496c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.050 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.024+0000 7f056b764740 4 client.25799 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.025+0000 7f056b764740 1 -- 172.21.15.102:0/1187591285 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5635d01a4d70 con 0x5635d02472a0 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.025+0000 7f0561400640 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x5635d02472a0 0x5635d02496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.026+0000 7f0543400640 10 client.25799 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.027+0000 7f0561400640 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x5635d02472a0 0x5635d02496c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2cd9e07c74db4acb server_cookie=10339c0a17c48b29 in_seq=0 out_seq=0 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.028+0000 7f0543400640 1 -- 172.21.15.102:0/1187591285 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5635d01a4d70 con 0x5635d02472a0 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.028+0000 7f0543400640 10 client.25799 handle_command_reply: tid=0 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.028+0000 7f056b764740 1 client.25799 shutdown 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.031+0000 7f056b764740 1 -- 172.21.15.102:0/1187591285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02339c0 msgr2=0x5635d02544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.031+0000 7f056b764740 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02339c0 0x5635d02544c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f055c000c00 tx=0x7f055c009f70 comp rx=0 tx=0).stop 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.031+0000 7f056b764740 1 -- 172.21.15.102:0/1187591285 shutdown_connections 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.031+0000 7f056b764740 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5635d0243790 0x5635d024fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.031+0000 7f056b764740 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5635d02387a0 0x5635d024f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.031+0000 7f056b764740 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x5635d02472a0 0x5635d02496c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.031+0000 7f056b764740 1 --2- 172.21.15.102:0/1187591285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5635d02339c0 0x5635d02544c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.031+0000 7f056b764740 1 -- 172.21.15.102:0/1187591285 >> 172.21.15.102:0/1187591285 conn(0x5635d0234ac0 msgr2=0x5635d023a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.031+0000 7f0543400640 0 client.25799 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.032+0000 7f056b764740 1 -- 172.21.15.102:0/1187591285 shutdown_connections 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.032+0000 7f056b764740 1 -- 172.21.15.102:0/1187591285 wait complete. 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.033+0000 7f056b764740 20 client.25799 trim_cache size 0 max 16384 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.034+0000 7f0568a00640 1 -- 172.21.15.102:0/214788615 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f05480a7210 msgr2=0x7f05480a96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.034+0000 7f0568a00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f05480a7210 0x7f05480a96d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f05540041b0 tx=0x7f0554009500 comp rx=0 tx=0).stop 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.034+0000 7f0568a00640 1 -- 172.21.15.102:0/214788615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05641075c0 msgr2=0x7f056406c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.034+0000 7f0568a00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05641075c0 0x7f056406c600 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0554034040 tx=0x7f05540305b0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.035+0000 7f0568a00640 1 -- 172.21.15.102:0/214788615 shutdown_connections 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.035+0000 7f0568a00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f05480a7210 0x7f05480a96d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.035+0000 7f0568a00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05641075c0 0x7f056406c600 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.035+0000 7f0568a00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f056406a3b0 0x7f056406c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.035+0000 7f0568a00640 1 --2- 172.21.15.102:0/214788615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0564069a10 0x7f0564073040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.035+0000 7f0568a00640 1 -- 172.21.15.102:0/214788615 >> 172.21.15.102:0/214788615 conn(0x7f05640777a0 msgr2=0x7f05640ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.035+0000 7f0568a00640 1 -- 172.21.15.102:0/214788615 shutdown_connections 2024-04-25T01:57:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.035+0000 7f0568a00640 1 -- 172.21.15.102:0/214788615 wait complete. 2024-04-25T01:57:04.068 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:57:04.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:03 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2236761601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:57:04.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:03 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2236761601' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:57:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.284+0000 7f0028a00640 1 -- 172.21.15.102:0/811576075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 msgr2=0x7f0024100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.284+0000 7f0028a00640 1 --2- 172.21.15.102:0/811576075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 0x7f0024100970 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0010009860 tx=0x7f001002a340 comp rx=0 tx=0).stop 2024-04-25T01:57:04.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.287+0000 7f0028a00640 1 -- 172.21.15.102:0/811576075 shutdown_connections 2024-04-25T01:57:04.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.287+0000 7f0028a00640 1 --2- 172.21.15.102:0/811576075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0024100eb0 0x7f00240688d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.287+0000 7f0028a00640 1 --2- 172.21.15.102:0/811576075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 0x7f0024100970 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.287+0000 7f0028a00640 1 --2- 172.21.15.102:0/811576075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00241065b0 0x7f0024106990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.287+0000 7f0028a00640 1 -- 172.21.15.102:0/811576075 >> 172.21.15.102:0/811576075 conn(0x7f00240fbfc0 msgr2=0x7f00240fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.288+0000 7f0028a00640 1 -- 172.21.15.102:0/811576075 shutdown_connections 2024-04-25T01:57:04.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.288+0000 7f0028a00640 1 -- 172.21.15.102:0/811576075 wait complete. 2024-04-25T01:57:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.289+0000 7f0028a00640 1 Processor -- start 2024-04-25T01:57:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.289+0000 7f0028a00640 1 -- start start 2024-04-25T01:57:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0028a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 0x7f002419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0028a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0024100eb0 0x7f002419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0028a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00241065b0 0x7f002419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0028a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0024113c00 con 0x7f0024100eb0 2024-04-25T01:57:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0028a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0024113a80 con 0x7f00241004f0 2024-04-25T01:57:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0028a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0024113d80 con 0x7f00241065b0 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0022800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 0x7f002419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0022800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 0x7f002419a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36392/0 (socket says 172.21.15.102:36392) 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0022800640 1 -- 172.21.15.102:0/172018237 learned_addr learned my addr 172.21.15.102:0/172018237 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0021e00640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0024100eb0 0x7f002419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.290+0000 7f0023200640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00241065b0 0x7f002419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.291+0000 7f0022800640 1 -- 172.21.15.102:0/172018237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00241065b0 msgr2=0x7f002419e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.291+0000 7f0022800640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00241065b0 0x7f002419e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.291+0000 7f0022800640 1 -- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0024100eb0 msgr2=0x7f002419a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.291+0000 7f0022800640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0024100eb0 0x7f002419a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.291+0000 7f0022800640 1 -- 172.21.15.102:0/172018237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002419f040 con 0x7f00241004f0 2024-04-25T01:57:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.291+0000 7f0023200640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00241065b0 0x7f002419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:04.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.291+0000 7f0021e00640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0024100eb0 0x7f002419a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:04.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.291+0000 7f0022800640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 0x7f002419a050 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f001400d950 tx=0x7f001400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.292+0000 7f001b400640 1 -- 172.21.15.102:0/172018237 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014017030 con 0x7f00241004f0 2024-04-25T01:57:04.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.292+0000 7f001b400640 1 -- 172.21.15.102:0/172018237 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0014004020 con 0x7f00241004f0 2024-04-25T01:57:04.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.292+0000 7f001b400640 1 -- 172.21.15.102:0/172018237 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0014003d60 con 0x7f00241004f0 2024-04-25T01:57:04.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.292+0000 7f0028a00640 1 -- 172.21.15.102:0/172018237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f002419f330 con 0x7f00241004f0 2024-04-25T01:57:04.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.292+0000 7f0028a00640 1 -- 172.21.15.102:0/172018237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00241a6bc0 con 0x7f00241004f0 2024-04-25T01:57:04.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.298+0000 7f001b400640 1 -- 172.21.15.102:0/172018237 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f001401a070 con 0x7f00241004f0 2024-04-25T01:57:04.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.301+0000 7f002c24b740 1 Processor -- start 2024-04-25T01:57:04.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.302+0000 7f001b400640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00080a2d80 0x7f00080a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.302+0000 7f001b400640 1 -- 172.21.15.102:0/172018237 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f00140cebe0 con 0x7f00241004f0 2024-04-25T01:57:04.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.303+0000 7f002c24b740 1 -- start start 2024-04-25T01:57:04.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.303+0000 7f002c24b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 0x55b037057b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.303+0000 7f002c24b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370580c0 0x55b037062250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.303+0000 7f002c24b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037062790 0x55b037064b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.303+0000 7f002c24b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b036fbc4e0 con 0x55b037062790 2024-04-25T01:57:04.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.303+0000 7f002c24b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b036f64a70 con 0x55b0370580c0 2024-04-25T01:57:04.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.303+0000 7f002c24b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b036fb84a0 con 0x55b0370577a0 2024-04-25T01:57:04.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0023200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037062790 0x55b037064b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0023200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037062790 0x55b037064b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37092/0 (socket says 172.21.15.102:37092) 2024-04-25T01:57:04.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0023200640 1 -- 172.21.15.102:0/1111206404 learned_addr learned my addr 172.21.15.102:0/1111206404 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0022800640 1 --2- 172.21.15.102:0/1111206404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 0x55b037057b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0021e00640 1 --2- 172.21.15.102:0/1111206404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370580c0 0x55b037062250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0023200640 1 -- 172.21.15.102:0/1111206404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 msgr2=0x55b037057b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0023200640 1 --2- 172.21.15.102:0/1111206404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 0x55b037057b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0023200640 1 -- 172.21.15.102:0/1111206404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370580c0 msgr2=0x55b037062250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0023200640 1 --2- 172.21.15.102:0/1111206404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370580c0 0x55b037062250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.304+0000 7f0023200640 1 -- 172.21.15.102:0/1111206404 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b036cd03f0 con 0x55b037062790 2024-04-25T01:57:04.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.305+0000 7f0023200640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00080a2d80 0x7f00080a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.305+0000 7f0023200640 1 --2- 172.21.15.102:0/1111206404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037062790 0x55b037064b80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f001c012600 tx=0x7f001c012ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27971c7dc54320af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.306+0000 7f000f400640 1 -- 172.21.15.102:0/1111206404 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c010960 con 0x55b037062790 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.306+0000 7f000f400640 1 -- 172.21.15.102:0/1111206404 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f001c0243b0 con 0x55b037062790 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.306+0000 7f000f400640 1 -- 172.21.15.102:0/1111206404 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c00e5b0 con 0x55b037062790 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.307+0000 7f002c24b740 1 -- 172.21.15.102:0/1111206404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037062790 msgr2=0x55b037064b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.307+0000 7f002c24b740 1 --2- 172.21.15.102:0/1111206404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037062790 0x55b037064b80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f001c012600 tx=0x7f001c012ac0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.308+0000 7f0023200640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00080a2d80 0x7f00080a5240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f001c01e000 tx=0x7f001c01b000 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.308+0000 7f002c24b740 1 -- 172.21.15.102:0/1111206404 shutdown_connections 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.308+0000 7f002c24b740 1 --2- 172.21.15.102:0/1111206404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037062790 0x55b037064b80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.308+0000 7f002c24b740 1 --2- 172.21.15.102:0/1111206404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370580c0 0x55b037062250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.308+0000 7f002c24b740 1 --2- 172.21.15.102:0/1111206404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 0x55b037057b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.308+0000 7f002c24b740 1 -- 172.21.15.102:0/1111206404 >> 172.21.15.102:0/1111206404 conn(0x55b037053ac0 msgr2=0x55b037053ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.308+0000 7f002c24b740 1 -- 172.21.15.102:0/1111206404 shutdown_connections 2024-04-25T01:57:04.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.308+0000 7f002c24b740 1 -- 172.21.15.102:0/1111206404 wait complete. 2024-04-25T01:57:04.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.309+0000 7f002c24b740 1 -- start start 2024-04-25T01:57:04.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.309+0000 7f002c24b740 1 Processor -- start 2024-04-25T01:57:04.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f002c24b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370577a0 0x55b03705f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f002c24b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370580c0 0x55b03705fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f002c24b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037062790 0x55b037060160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f002c24b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b037065520 con 0x55b0370580c0 2024-04-25T01:57:04.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f002c24b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b036f77ec0 con 0x55b0370577a0 2024-04-25T01:57:04.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f002c24b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b0370656a0 con 0x55b037062790 2024-04-25T01:57:04.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f002c24b740 10 client.0 _ll_register_callbacks cb 0x55b036fb43f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:04.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f0021e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370580c0 0x55b03705fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f0023200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037062790 0x55b037060160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f0023200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037062790 0x55b037060160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41504/0 (socket says 172.21.15.102:41504) 2024-04-25T01:57:04.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f0021e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370580c0 0x55b03705fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37098/0 (socket says 172.21.15.102:37098) 2024-04-25T01:57:04.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.310+0000 7f0023200640 1 -- 172.21.15.102:0/1353702847 learned_addr learned my addr 172.21.15.102:0/1353702847 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.311+0000 7f0021e00640 1 -- 172.21.15.102:0/1353702847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037062790 msgr2=0x55b037060160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.311+0000 7f0021e00640 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037062790 0x55b037060160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.311+0000 7f0022800640 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370577a0 0x55b03705f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.311+0000 7f0021e00640 1 -- 172.21.15.102:0/1353702847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370577a0 msgr2=0x55b03705f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.311+0000 7f0021e00640 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370577a0 0x55b03705f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.311+0000 7f0021e00640 1 -- 172.21.15.102:0/1353702847 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b036d9a7d0 con 0x55b0370580c0 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.311+0000 7f0022800640 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370577a0 0x55b03705f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.311+0000 7f0021e00640 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370580c0 0x55b03705fc20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0010009830 tx=0x7f0010004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad67b6c63287fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.311+0000 7f0003e00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.312+0000 7f0003e00640 1 -- 172.21.15.102:0/1353702847 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010037040 con 0x55b0370580c0 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.312+0000 7f0003e00640 1 -- 172.21.15.102:0/1353702847 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0010030550 con 0x55b0370580c0 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.312+0000 7f002c24b740 1 -- 172.21.15.102:0/1353702847 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f001c0243b0 con 0x55b0370580c0 2024-04-25T01:57:04.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.313+0000 7f0003e00640 1 -- 172.21.15.102:0/1353702847 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0010030750 con 0x55b0370580c0 2024-04-25T01:57:04.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.316+0000 7f002c24b740 10 client.15636 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:04.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.316+0000 7f002c24b740 1 -- 172.21.15.102:0/1353702847 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b0370717d0 con 0x55b0370580c0 2024-04-25T01:57:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.317+0000 7f0003e00640 1 -- 172.21.15.102:0/1353702847 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f001c0243b0 con 0x55b0370580c0 2024-04-25T01:57:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.317+0000 7f0003e00640 1 -- 172.21.15.102:0/1353702847 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0010045050 con 0x55b0370580c0 2024-04-25T01:57:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.319+0000 7f002c24b740 10 client.15636 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:04.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.319+0000 7f002c24b740 10 client.15636 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:57:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.319+0000 7f002c24b740 20 client.15636 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.319+0000 7f002c24b740 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55b037072c30 0x55b037075080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.319+0000 7f002c24b740 4 client.15636 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:04.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.319+0000 7f002c24b740 1 -- 172.21.15.102:0/1353702847 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b036fc9a20 con 0x55b037072c30 2024-04-25T01:57:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.321+0000 7f0022800640 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55b037072c30 0x55b037075080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.323+0000 7f0003e00640 10 client.15636 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:57:04.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.322+0000 7f0022800640 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55b037072c30 0x55b037075080 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b574fa5bcff039ea server_cookie=1b451c57bf7b9710 in_seq=0 out_seq=0 2024-04-25T01:57:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.324+0000 7f0003e00640 1 -- 172.21.15.102:0/1353702847 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b036fc9a20 con 0x55b037072c30 2024-04-25T01:57:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.324+0000 7f0003e00640 10 client.15636 handle_command_reply: tid=0 2024-04-25T01:57:04.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.324+0000 7f002c24b740 1 client.15636 shutdown 2024-04-25T01:57:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.325+0000 7f002c24b740 1 -- 172.21.15.102:0/1353702847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370580c0 msgr2=0x55b03705fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.325+0000 7f002c24b740 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370580c0 0x55b03705fc20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0010009830 tx=0x7f0010004040 comp rx=0 tx=0).stop 2024-04-25T01:57:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.325+0000 7f002c24b740 1 -- 172.21.15.102:0/1353702847 shutdown_connections 2024-04-25T01:57:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.325+0000 7f002c24b740 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55b037072c30 0x55b037075080 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.325+0000 7f002c24b740 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b037062790 0x55b037060160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.325+0000 7f0003e00640 0 client.15636 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:57:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.326+0000 7f002c24b740 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370580c0 0x55b03705fc20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.326+0000 7f002c24b740 1 --2- 172.21.15.102:0/1353702847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370577a0 0x55b03705f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.326+0000 7f002c24b740 1 -- 172.21.15.102:0/1353702847 >> 172.21.15.102:0/1353702847 conn(0x55b037053ac0 msgr2=0x55b0370597c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.326+0000 7f002c24b740 1 -- 172.21.15.102:0/1353702847 shutdown_connections 2024-04-25T01:57:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.326+0000 7f002c24b740 1 -- 172.21.15.102:0/1353702847 wait complete. 2024-04-25T01:57:04.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.326+0000 7f002c24b740 20 client.15636 trim_cache size 0 max 16384 2024-04-25T01:57:04.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.349+0000 7f002c24b740 1 Processor -- start 2024-04-25T01:57:04.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.349+0000 7f002c24b740 1 -- start start 2024-04-25T01:57:04.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.350+0000 7f002c24b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037053ac0 0x55b03706e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.350+0000 7f002c24b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 0x55b03705fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.350+0000 7f002c24b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370580c0 0x55b03707a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.350+0000 7f002c24b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0004011030 con 0x55b037053ac0 2024-04-25T01:57:04.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.350+0000 7f002c24b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f000400e8a0 con 0x55b0370580c0 2024-04-25T01:57:04.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.350+0000 7f002c24b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f000400c6f0 con 0x55b0370577a0 2024-04-25T01:57:04.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0022800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037053ac0 0x55b03706e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0022800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037053ac0 0x55b03706e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37100/0 (socket says 172.21.15.102:37100) 2024-04-25T01:57:04.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0022800640 1 -- 172.21.15.102:0/1137498330 learned_addr learned my addr 172.21.15.102:0/1137498330 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0021e00640 1 --2- 172.21.15.102:0/1137498330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 0x55b03705fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0021e00640 1 -- 172.21.15.102:0/1137498330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370580c0 msgr2=0x55b03707a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0021e00640 1 --2- 172.21.15.102:0/1137498330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370580c0 0x55b03707a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0021e00640 1 -- 172.21.15.102:0/1137498330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037053ac0 msgr2=0x55b03706e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0021e00640 1 --2- 172.21.15.102:0/1137498330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037053ac0 0x55b03706e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0021e00640 1 -- 172.21.15.102:0/1137498330 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b03705dc30 con 0x55b0370577a0 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.351+0000 7f0022800640 1 --2- 172.21.15.102:0/1137498330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037053ac0 0x55b03706e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.352+0000 7f0021e00640 1 --2- 172.21.15.102:0/1137498330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 0x55b03705fdb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0010004da0 tx=0x7f001002a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef98beb46bc14594 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.352+0000 7f000ea00640 1 -- 172.21.15.102:0/1137498330 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010037070 con 0x55b0370577a0 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.352+0000 7f000ea00640 1 -- 172.21.15.102:0/1137498330 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0010030550 con 0x55b0370577a0 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.353+0000 7f000ea00640 1 -- 172.21.15.102:0/1137498330 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00100046c0 con 0x55b0370577a0 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.354+0000 7f002c24b740 1 -- 172.21.15.102:0/1137498330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 msgr2=0x55b03705fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.354+0000 7f002c24b740 1 --2- 172.21.15.102:0/1137498330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 0x55b03705fdb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0010004da0 tx=0x7f001002a340 comp rx=0 tx=0).stop 2024-04-25T01:57:04.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.354+0000 7f002c24b740 1 -- 172.21.15.102:0/1137498330 shutdown_connections 2024-04-25T01:57:04.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.354+0000 7f002c24b740 1 --2- 172.21.15.102:0/1137498330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b0370580c0 0x55b03707a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.354+0000 7f002c24b740 1 --2- 172.21.15.102:0/1137498330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370577a0 0x55b03705fdb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.354+0000 7f002c24b740 1 --2- 172.21.15.102:0/1137498330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b037053ac0 0x55b03706e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.354+0000 7f002c24b740 1 -- 172.21.15.102:0/1137498330 >> 172.21.15.102:0/1137498330 conn(0x55b03705b0b0 msgr2=0x55b03705f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.357+0000 7f002c24b740 1 -- 172.21.15.102:0/1137498330 shutdown_connections 2024-04-25T01:57:04.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.357+0000 7f002c24b740 1 -- 172.21.15.102:0/1137498330 wait complete. 2024-04-25T01:57:04.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.359+0000 7f002c24b740 1 -- start start 2024-04-25T01:57:04.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.359+0000 7f002c24b740 1 Processor -- start 2024-04-25T01:57:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.363+0000 7f002c24b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b037053ac0 0x55b037066b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.364+0000 7f002c24b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370577a0 0x55b0370670b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.364+0000 7f002c24b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370580c0 0x55b0370e0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.365+0000 7f0022800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b037053ac0 0x55b037066b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.365+0000 7f0022800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b037053ac0 0x55b037066b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36430/0 (socket says 172.21.15.102:36430) 2024-04-25T01:57:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.365+0000 7f002c24b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f000400cd60 con 0x55b0370577a0 2024-04-25T01:57:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.365+0000 7f002c24b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f000400e1d0 con 0x55b037053ac0 2024-04-25T01:57:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.365+0000 7f002c24b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f000400d430 con 0x55b0370580c0 2024-04-25T01:57:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.365+0000 7f0022800640 1 -- 172.21.15.102:0/3489127469 learned_addr learned my addr 172.21.15.102:0/3489127469 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.365+0000 7f002c24b740 10 client.0 _ll_register_callbacks cb 0x7f0010035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:04.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.365+0000 7f0023200640 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370580c0 0x55b0370e0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.366+0000 7f0021e00640 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370577a0 0x55b0370670b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.367+0000 7f0022800640 1 -- 172.21.15.102:0/3489127469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370580c0 msgr2=0x55b0370e0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.367+0000 7f0022800640 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370580c0 0x55b0370e0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.367+0000 7f0022800640 1 -- 172.21.15.102:0/3489127469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370577a0 msgr2=0x55b0370670b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.367+0000 7f0022800640 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370577a0 0x55b0370670b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.367+0000 7f0022800640 1 -- 172.21.15.102:0/3489127469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0370e1020 con 0x55b037053ac0 2024-04-25T01:57:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.369+0000 7f0022800640 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b037053ac0 0x55b037066b70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0014003910 tx=0x7f0014009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bf47943aeba3e94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.369+0000 7f0003e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:04.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.370+0000 7f0003e00640 1 -- 172.21.15.102:0/3489127469 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00140035f0 con 0x55b037053ac0 2024-04-25T01:57:04.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.371+0000 7f0003e00640 1 -- 172.21.15.102:0/3489127469 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f00140bb0b0 con 0x55b037053ac0 2024-04-25T01:57:04.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.371+0000 7f002c24b740 1 -- 172.21.15.102:0/3489127469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0010030550 con 0x55b037053ac0 2024-04-25T01:57:04.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.372+0000 7f0003e00640 1 -- 172.21.15.102:0/3489127469 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0014009a30 con 0x55b037053ac0 2024-04-25T01:57:04.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.372+0000 7f002c24b740 10 client.25811 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:04.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.372+0000 7f002c24b740 1 -- 172.21.15.102:0/3489127469 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b0370e2380 con 0x55b037053ac0 2024-04-25T01:57:04.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.373+0000 7f0003e00640 1 -- 172.21.15.102:0/3489127469 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0010030550 con 0x55b037053ac0 2024-04-25T01:57:04.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.373+0000 7f0003e00640 1 -- 172.21.15.102:0/3489127469 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0014081080 con 0x55b037053ac0 2024-04-25T01:57:04.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.375+0000 7f002c24b740 10 client.25811 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:04.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.375+0000 7f002c24b740 10 client.25811 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:57:04.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.375+0000 7f002c24b740 20 client.25811 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:04.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.375+0000 7f002c24b740 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55b037062790 0x55b0370e4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.375+0000 7f002c24b740 4 client.25811 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:57:04.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.375+0000 7f002c24b740 1 -- 172.21.15.102:0/3489127469 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b036fc3d70 con 0x55b037062790 2024-04-25T01:57:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.375+0000 7f0021e00640 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55b037062790 0x55b0370e4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.376+0000 7f0021e00640 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55b037062790 0x55b0370e4aa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=449f02a0b45030e1 server_cookie=547ef3e5e3828fc1 in_seq=0 out_seq=0 2024-04-25T01:57:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.376+0000 7f0003e00640 10 client.25811 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:57:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.376+0000 7f0003e00640 1 -- 172.21.15.102:0/3489127469 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b036fc3d70 con 0x55b037062790 2024-04-25T01:57:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.376+0000 7f0003e00640 10 client.25811 handle_command_reply: tid=0 2024-04-25T01:57:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.377+0000 7f002c24b740 1 client.25811 shutdown 2024-04-25T01:57:04.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.378+0000 7f002c24b740 1 -- 172.21.15.102:0/3489127469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b037053ac0 msgr2=0x55b037066b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.378+0000 7f002c24b740 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b037053ac0 0x55b037066b70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0014003910 tx=0x7f0014009f90 comp rx=0 tx=0).stop 2024-04-25T01:57:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.378+0000 7f002c24b740 1 -- 172.21.15.102:0/3489127469 shutdown_connections 2024-04-25T01:57:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.378+0000 7f002c24b740 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55b037062790 0x55b0370e4aa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.378+0000 7f002c24b740 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0370580c0 0x55b0370e0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.378+0000 7f002c24b740 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0370577a0 0x55b0370670b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.378+0000 7f002c24b740 1 --2- 172.21.15.102:0/3489127469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b037053ac0 0x55b037066b70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.378+0000 7f002c24b740 1 -- 172.21.15.102:0/3489127469 >> 172.21.15.102:0/3489127469 conn(0x55b03705b0b0 msgr2=0x55b03705ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.379+0000 7f0003e00640 0 client.25811 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:57:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.380+0000 7f002c24b740 1 -- 172.21.15.102:0/3489127469 shutdown_connections 2024-04-25T01:57:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.380+0000 7f002c24b740 1 -- 172.21.15.102:0/3489127469 wait complete. 2024-04-25T01:57:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.380+0000 7f002c24b740 20 client.25811 trim_cache size 0 max 16384 2024-04-25T01:57:04.385 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:57:04.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.398+0000 7f0028a00640 1 -- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00080a2d80 msgr2=0x7f00080a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.398+0000 7f0028a00640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00080a2d80 0x7f00080a5240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f001c01e000 tx=0x7f001c01b000 comp rx=0 tx=0).stop 2024-04-25T01:57:04.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.398+0000 7f0028a00640 1 -- 172.21.15.102:0/172018237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 msgr2=0x7f002419a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.398+0000 7f0028a00640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 0x7f002419a050 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f001400d950 tx=0x7f001400de10 comp rx=0 tx=0).stop 2024-04-25T01:57:04.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.399+0000 7f0028a00640 1 -- 172.21.15.102:0/172018237 shutdown_connections 2024-04-25T01:57:04.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.399+0000 7f0028a00640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00080a2d80 0x7f00080a5240 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.399+0000 7f0028a00640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00241065b0 0x7f002419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.399+0000 7f0028a00640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0024100eb0 0x7f002419a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.399+0000 7f0028a00640 1 --2- 172.21.15.102:0/172018237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00241004f0 0x7f002419a050 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.399+0000 7f0028a00640 1 -- 172.21.15.102:0/172018237 >> 172.21.15.102:0/172018237 conn(0x7f00240fbfc0 msgr2=0x7f002410e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.400+0000 7f0028a00640 1 -- 172.21.15.102:0/172018237 shutdown_connections 2024-04-25T01:57:04.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.400+0000 7f0028a00640 1 -- 172.21.15.102:0/172018237 wait complete. 2024-04-25T01:57:04.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:57:04.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.720+0000 7f5760c00640 1 -- 172.21.15.102:0/3039626246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c10d040 msgr2=0x7f575c114c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.720+0000 7f5760c00640 1 --2- 172.21.15.102:0/3039626246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c10d040 0x7f575c114c10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5748009900 tx=0x7f574802a440 comp rx=0 tx=0).stop 2024-04-25T01:57:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.723+0000 7f5760c00640 1 -- 172.21.15.102:0/3039626246 shutdown_connections 2024-04-25T01:57:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.723+0000 7f5760c00640 1 --2- 172.21.15.102:0/3039626246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c10d040 0x7f575c114c10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.723+0000 7f5760c00640 1 --2- 172.21.15.102:0/3039626246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f575c108970 0x7f575c10c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.723+0000 7f5760c00640 1 --2- 172.21.15.102:0/3039626246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f575c107fc0 0x7f575c1083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.723+0000 7f5760c00640 1 -- 172.21.15.102:0/3039626246 >> 172.21.15.102:0/3039626246 conn(0x7f575c075c80 msgr2=0x7f575c0780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.723+0000 7f5760c00640 1 -- 172.21.15.102:0/3039626246 shutdown_connections 2024-04-25T01:57:04.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.723+0000 7f5760c00640 1 -- 172.21.15.102:0/3039626246 wait complete. 2024-04-25T01:57:04.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.724+0000 7f5760c00640 1 Processor -- start 2024-04-25T01:57:04.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.725+0000 7f5760c00640 1 -- start start 2024-04-25T01:57:04.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.728+0000 7f5760c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f575c107fc0 0x7f575c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.729+0000 7f5760c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f575c10d040 0x7f575c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.729+0000 7f5759e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f575c107fc0 0x7f575c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.729+0000 7f5760c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1a2ea0 0x7f575c1a3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.729+0000 7f5759e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f575c107fc0 0x7f575c19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41546/0 (socket says 172.21.15.102:41546) 2024-04-25T01:57:04.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.730+0000 7f575a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1a2ea0 0x7f575c1a3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.730+0000 7f575a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1a2ea0 0x7f575c1a3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36456/0 (socket says 172.21.15.102:36456) 2024-04-25T01:57:04.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.730+0000 7f575a800640 1 -- 172.21.15.102:0/1871798668 learned_addr learned my addr 172.21.15.102:0/1871798668 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.730+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f575c1181a0 con 0x7f575c10d040 2024-04-25T01:57:04.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.730+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f575c118020 con 0x7f575c1a2ea0 2024-04-25T01:57:04.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.730+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f575c118320 con 0x7f575c107fc0 2024-04-25T01:57:04.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.730+0000 7f5759400640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f575c10d040 0x7f575c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.730+0000 7f575a800640 1 -- 172.21.15.102:0/1871798668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f575c107fc0 msgr2=0x7f575c19e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.731+0000 7f575a800640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f575c107fc0 0x7f575c19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.731+0000 7f575a800640 1 -- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f575c10d040 msgr2=0x7f575c19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.731+0000 7f575a800640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f575c10d040 0x7f575c19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.731+0000 7f575a800640 1 -- 172.21.15.102:0/1871798668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575c1a3990 con 0x7f575c1a2ea0 2024-04-25T01:57:04.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.733+0000 7f5759400640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f575c10d040 0x7f575c19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:04.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.733+0000 7f5759e00640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f575c107fc0 0x7f575c19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-25T01:57:04.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.734+0000 7f575a800640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1a2ea0 0x7f575c1a3350 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f574802a960 tx=0x7f5748003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.735+0000 7f5752a00640 1 -- 172.21.15.102:0/1871798668 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574803f070 con 0x7f575c1a2ea0 2024-04-25T01:57:04.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.735+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f575c1a3c20 con 0x7f575c1a2ea0 2024-04-25T01:57:04.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.735+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f575c1ab440 con 0x7f575c1a2ea0 2024-04-25T01:57:04.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.736+0000 7f5752a00640 1 -- 172.21.15.102:0/1871798668 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5748031c20 con 0x7f575c1a2ea0 2024-04-25T01:57:04.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.736+0000 7f5752a00640 1 -- 172.21.15.102:0/1871798668 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748035030 con 0x7f575c1a2ea0 2024-04-25T01:57:04.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.748+0000 7f5752a00640 1 -- 172.21.15.102:0/1871798668 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5748042070 con 0x7f575c1a2ea0 2024-04-25T01:57:04.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.750+0000 7f5763bbf740 1 Processor -- start 2024-04-25T01:57:04.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.750+0000 7f5763bbf740 1 -- start start 2024-04-25T01:57:04.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.751+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 0x56523c0579a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.752+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c062070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.752+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c0649a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.752+0000 7f5763bbf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56523bfbc4e0 con 0x56523c057ee0 2024-04-25T01:57:04.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.752+0000 7f5763bbf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56523bf64a70 con 0x56523c0625b0 2024-04-25T01:57:04.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.752+0000 7f5763bbf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56523bfb84a0 con 0x56523c0575c0 2024-04-25T01:57:04.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.753+0000 7f5752a00640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57400a2d80 0x7f57400a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.754+0000 7f5759e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c062070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.754+0000 7f5759e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c062070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37142/0 (socket says 172.21.15.102:37142) 2024-04-25T01:57:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.754+0000 7f5759e00640 1 -- 172.21.15.102:0/1072727072 learned_addr learned my addr 172.21.15.102:0/1072727072 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.754+0000 7f5752a00640 1 -- 172.21.15.102:0/1871798668 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f57480efb10 con 0x7f575c1a2ea0 2024-04-25T01:57:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.754+0000 7f5759400640 1 --2- 172.21.15.102:0/1072727072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c0649a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.755+0000 7f575a800640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57400a2d80 0x7f57400a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.755+0000 7f5759e00640 1 -- 172.21.15.102:0/1072727072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 msgr2=0x56523c0579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.755+0000 7f5759e00640 1 --2- 172.21.15.102:0/1072727072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 0x56523c0579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.755+0000 7f5759e00640 1 -- 172.21.15.102:0/1072727072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 msgr2=0x56523c0649a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.755+0000 7f5759e00640 1 --2- 172.21.15.102:0/1072727072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c0649a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.755+0000 7f5759e00640 1 -- 172.21.15.102:0/1072727072 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56523bcd03f0 con 0x56523c057ee0 2024-04-25T01:57:04.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.755+0000 7f5759400640 1 --2- 172.21.15.102:0/1072727072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c0649a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:04.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.757+0000 7f575a800640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57400a2d80 0x7f57400a5240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f57480b84c0 tx=0x7f5748005c90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.757+0000 7f5759e00640 1 --2- 172.21.15.102:0/1072727072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c062070 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f574c007fb0 tx=0x7f574c00c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269a1586018e75a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.758+0000 7f5747000640 1 -- 172.21.15.102:0/1072727072 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c014070 con 0x56523c057ee0 2024-04-25T01:57:04.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.758+0000 7f5747000640 1 -- 172.21.15.102:0/1072727072 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f574c00fd30 con 0x56523c057ee0 2024-04-25T01:57:04.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.758+0000 7f5747000640 1 -- 172.21.15.102:0/1072727072 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c00edd0 con 0x56523c057ee0 2024-04-25T01:57:04.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.759+0000 7f5763bbf740 1 -- 172.21.15.102:0/1072727072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 msgr2=0x56523c062070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.759+0000 7f5763bbf740 1 --2- 172.21.15.102:0/1072727072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c062070 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f574c007fb0 tx=0x7f574c00c750 comp rx=0 tx=0).stop 2024-04-25T01:57:04.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.760+0000 7f5763bbf740 1 -- 172.21.15.102:0/1072727072 shutdown_connections 2024-04-25T01:57:04.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.760+0000 7f5763bbf740 1 --2- 172.21.15.102:0/1072727072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c0649a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.760+0000 7f5763bbf740 1 --2- 172.21.15.102:0/1072727072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c062070 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.760+0000 7f5763bbf740 1 --2- 172.21.15.102:0/1072727072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 0x56523c0579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.760+0000 7f5763bbf740 1 -- 172.21.15.102:0/1072727072 >> 172.21.15.102:0/1072727072 conn(0x56523c0538e0 msgr2=0x56523c053cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.762+0000 7f5763bbf740 1 -- 172.21.15.102:0/1072727072 shutdown_connections 2024-04-25T01:57:04.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.762+0000 7f5763bbf740 1 -- 172.21.15.102:0/1072727072 wait complete. 2024-04-25T01:57:04.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.763+0000 7f5763bbf740 1 -- start start 2024-04-25T01:57:04.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.763+0000 7f5763bbf740 1 Processor -- start 2024-04-25T01:57:04.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c0575c0 0x56523c05f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c057ee0 0x56523c05fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c05ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5763bbf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56523c0653a0 con 0x56523c0575c0 2024-04-25T01:57:04.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5763bbf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56523bf77ec0 con 0x56523c0625b0 2024-04-25T01:57:04.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5763bbf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56523c065520 con 0x56523c057ee0 2024-04-25T01:57:04.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5763bbf740 10 client.0 _ll_register_callbacks cb 0x56523bfb43f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:04.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5759e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c057ee0 0x56523c05fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5759e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c057ee0 0x56523c05fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41570/0 (socket says 172.21.15.102:41570) 2024-04-25T01:57:04.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.765+0000 7f5759e00640 1 -- 172.21.15.102:0/3565399744 learned_addr learned my addr 172.21.15.102:0/3565399744 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.766+0000 7f5759400640 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c0575c0 0x56523c05f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.766+0000 7f5759400640 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c05ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.766+0000 7f5759e00640 1 -- 172.21.15.102:0/3565399744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 msgr2=0x56523c05ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.766+0000 7f5759e00640 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c05ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.767+0000 7f5759e00640 1 -- 172.21.15.102:0/3565399744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c0575c0 msgr2=0x56523c05f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.767+0000 7f5759e00640 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c0575c0 0x56523c05f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.767+0000 7f5759e00640 1 -- 172.21.15.102:0/3565399744 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56523bd9a7d0 con 0x56523c057ee0 2024-04-25T01:57:04.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.768+0000 7f5759e00640 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c057ee0 0x56523c05fa60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f574c004b90 tx=0x7f574c007620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad9ab41158dfc0d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.768+0000 7f573b400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:04.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.771+0000 7f573b400640 1 -- 172.21.15.102:0/3565399744 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c014040 con 0x56523c057ee0 2024-04-25T01:57:04.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.771+0000 7f5763bbf740 1 -- 172.21.15.102:0/3565399744 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f574c00fd30 con 0x56523c057ee0 2024-04-25T01:57:04.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.771+0000 7f573b400640 1 -- 172.21.15.102:0/3565399744 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f574c00f850 con 0x56523c057ee0 2024-04-25T01:57:04.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.772+0000 7f573b400640 1 -- 172.21.15.102:0/3565399744 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f574c02f050 con 0x56523c057ee0 2024-04-25T01:57:04.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.772+0000 7f5763bbf740 10 client.25807 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:04.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.772+0000 7f5763bbf740 1 -- 172.21.15.102:0/3565399744 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56523c071620 con 0x56523c057ee0 2024-04-25T01:57:04.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.774+0000 7f573b400640 1 -- 172.21.15.102:0/3565399744 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f574c00fd30 con 0x56523c057ee0 2024-04-25T01:57:04.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.775+0000 7f573b400640 1 -- 172.21.15.102:0/3565399744 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f574c021420 con 0x56523c057ee0 2024-04-25T01:57:04.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.775+0000 7f5763bbf740 10 client.25807 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:04.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.775+0000 7f5763bbf740 10 client.25807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:04.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.775+0000 7f5763bbf740 20 client.25807 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:04.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.775+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56523c072a20 0x56523c074e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.776+0000 7f5763bbf740 4 client.25807 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:04.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.776+0000 7f5763bbf740 1 -- 172.21.15.102:0/3565399744 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56523bfc9a20 con 0x56523c072a20 2024-04-25T01:57:04.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.777+0000 7f5759400640 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56523c072a20 0x56523c074e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.781+0000 7f573b400640 10 client.25807 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:04.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.781+0000 7f5759400640 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56523c072a20 0x56523c074e10 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60fc51eb26dc9aa8 server_cookie=daeada49759a26ca in_seq=0 out_seq=0 2024-04-25T01:57:04.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.783+0000 7f573b400640 1 -- 172.21.15.102:0/3565399744 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56523bfc9a20 con 0x56523c072a20 2024-04-25T01:57:04.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.783+0000 7f573b400640 10 client.25807 handle_command_reply: tid=0 2024-04-25T01:57:04.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.783+0000 7f5763bbf740 1 client.25807 shutdown 2024-04-25T01:57:04.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.784+0000 7f5763bbf740 1 -- 172.21.15.102:0/3565399744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c057ee0 msgr2=0x56523c05fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.785+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c057ee0 0x56523c05fa60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f574c004b90 tx=0x7f574c007620 comp rx=0 tx=0).stop 2024-04-25T01:57:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.786+0000 7f5763bbf740 1 -- 172.21.15.102:0/3565399744 shutdown_connections 2024-04-25T01:57:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.786+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56523c072a20 0x56523c074e10 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.786+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c05ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.786+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c057ee0 0x56523c05fa60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.786+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3565399744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c0575c0 0x56523c05f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.786+0000 7f573b400640 0 client.25807 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:04.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.786+0000 7f5763bbf740 1 -- 172.21.15.102:0/3565399744 >> 172.21.15.102:0/3565399744 conn(0x56523c0538e0 msgr2=0x56523c052b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.786+0000 7f5763bbf740 1 -- 172.21.15.102:0/3565399744 shutdown_connections 2024-04-25T01:57:04.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.786+0000 7f5763bbf740 1 -- 172.21.15.102:0/3565399744 wait complete. 2024-04-25T01:57:04.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.787+0000 7f5763bbf740 20 client.25807 trim_cache size 0 max 16384 2024-04-25T01:57:04.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.811+0000 7f5763bbf740 1 Processor -- start 2024-04-25T01:57:04.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.811+0000 7f5763bbf740 1 -- start start 2024-04-25T01:57:04.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.811+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 0x56523c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.811+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c05fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.811+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c07a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.811+0000 7f5763bbf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f573c010fa0 con 0x56523c057ee0 2024-04-25T01:57:04.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.811+0000 7f5763bbf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f573c00e810 con 0x56523c0625b0 2024-04-25T01:57:04.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.811+0000 7f5763bbf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f573c00c660 con 0x56523c0575c0 2024-04-25T01:57:04.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 0x56523c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c05fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 0x56523c06e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41572/0 (socket says 172.21.15.102:41572) 2024-04-25T01:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c05fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37152/0 (socket says 172.21.15.102:37152) 2024-04-25T01:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759400640 1 -- 172.21.15.102:0/3786348175 learned_addr learned my addr 172.21.15.102:0/3786348175 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759400640 1 --2- 172.21.15.102:0/3786348175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c07a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759e00640 1 -- 172.21.15.102:0/3786348175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 msgr2=0x56523c06e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759e00640 1 --2- 172.21.15.102:0/3786348175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 0x56523c06e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759e00640 1 -- 172.21.15.102:0/3786348175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 msgr2=0x56523c07a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.812+0000 7f5759e00640 1 --2- 172.21.15.102:0/3786348175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c07a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.813+0000 7f5759e00640 1 -- 172.21.15.102:0/3786348175 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56523c04f230 con 0x56523c057ee0 2024-04-25T01:57:04.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.813+0000 7f5759e00640 1 --2- 172.21.15.102:0/3786348175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c05fbf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f574c002230 tx=0x7f574c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62919b150074822 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.814+0000 7f5746600640 1 -- 172.21.15.102:0/3786348175 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c037070 con 0x56523c057ee0 2024-04-25T01:57:04.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.814+0000 7f5746600640 1 -- 172.21.15.102:0/3786348175 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f574c026070 con 0x56523c057ee0 2024-04-25T01:57:04.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.814+0000 7f5746600640 1 -- 172.21.15.102:0/3786348175 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c00fb30 con 0x56523c057ee0 2024-04-25T01:57:04.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.815+0000 7f5763bbf740 1 -- 172.21.15.102:0/3786348175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 msgr2=0x56523c05fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.815+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3786348175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c05fbf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f574c002230 tx=0x7f574c00cf80 comp rx=0 tx=0).stop 2024-04-25T01:57:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.815+0000 7f5763bbf740 1 -- 172.21.15.102:0/3786348175 shutdown_connections 2024-04-25T01:57:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.816+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3786348175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0625b0 0x56523c07a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.816+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3786348175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c05fbf0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.816+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3786348175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0575c0 0x56523c06e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.816+0000 7f5763bbf740 1 -- 172.21.15.102:0/3786348175 >> 172.21.15.102:0/3786348175 conn(0x56523c0538e0 msgr2=0x56523c05f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.816+0000 7f5763bbf740 1 -- 172.21.15.102:0/3786348175 shutdown_connections 2024-04-25T01:57:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.816+0000 7f5763bbf740 1 -- 172.21.15.102:0/3786348175 wait complete. 2024-04-25T01:57:04.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.816+0000 7f5763bbf740 1 -- start start 2024-04-25T01:57:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.817+0000 7f5763bbf740 1 Processor -- start 2024-04-25T01:57:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.817+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0575c0 0x56523c06a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5763bbf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0625b0 0x56523c0d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5763bbf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f573c00ccd0 con 0x56523c057ee0 2024-04-25T01:57:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5763bbf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f573c00e140 con 0x56523c0575c0 2024-04-25T01:57:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5763bbf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f573c00d3a0 con 0x56523c0625b0 2024-04-25T01:57:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5763bbf740 10 client.0 _ll_register_callbacks cb 0x7f574c007e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5759e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5759400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0575c0 0x56523c06a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5759e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c06ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37158/0 (socket says 172.21.15.102:37158) 2024-04-25T01:57:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5759e00640 1 -- 172.21.15.102:0/3631458778 learned_addr learned my addr 172.21.15.102:0/3631458778 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:04.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.818+0000 7f5759400640 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0625b0 0x56523c0d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.819+0000 7f5759400640 1 -- 172.21.15.102:0/3631458778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0625b0 msgr2=0x56523c0d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.819+0000 7f5759400640 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0625b0 0x56523c0d1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.819+0000 7f5759400640 1 -- 172.21.15.102:0/3631458778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 msgr2=0x56523c06ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:04.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.819+0000 7f5759400640 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c06ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.819+0000 7f5759400640 1 -- 172.21.15.102:0/3631458778 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56523c0d2230 con 0x56523c0575c0 2024-04-25T01:57:04.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.819+0000 7f5759e00640 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c06ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:04.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.819+0000 7f5759400640 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0575c0 0x56523c06a800 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f575400bc90 tx=0x7f574c00ea10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c46655812359143c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:04.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.819+0000 7f573b400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.819+0000 7f573b400640 1 -- 172.21.15.102:0/3631458778 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575401c040 con 0x56523c0575c0 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.820+0000 7f573b400640 1 -- 172.21.15.102:0/3631458778 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f5754005220 con 0x56523c0575c0 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.820+0000 7f5763bbf740 1 -- 172.21.15.102:0/3631458778 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f574c026070 con 0x56523c0575c0 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.820+0000 7f573b400640 1 -- 172.21.15.102:0/3631458778 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5754005420 con 0x56523c0575c0 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.820+0000 7f5763bbf740 10 client.25826 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.820+0000 7f5763bbf740 1 -- 172.21.15.102:0/3631458778 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56523c0ea520 con 0x56523c0575c0 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.820+0000 7f573b400640 1 -- 172.21.15.102:0/3631458778 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f574c026070 con 0x56523c0575c0 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.821+0000 7f573b400640 1 -- 172.21.15.102:0/3631458778 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5754004aa0 con 0x56523c0575c0 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.821+0000 7f5763bbf740 10 client.25826 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.821+0000 7f5763bbf740 10 client.25826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:04.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.821+0000 7f5763bbf740 20 client.25826 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.821+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56523c0ead00 0x56523c0ed120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.821+0000 7f5763bbf740 4 client.25826 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.821+0000 7f5763bbf740 1 -- 172.21.15.102:0/3631458778 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56523bfc3d70 con 0x56523c0ead00 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.821+0000 7f5759e00640 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56523c0ead00 0x56523c0ed120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.822+0000 7f5759e00640 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56523c0ead00 0x56523c0ed120 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=557f8bbcfb6179ee server_cookie=7f4791a5ea39d6c9 in_seq=0 out_seq=0 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.822+0000 7f573b400640 10 client.25826 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.822+0000 7f573b400640 1 -- 172.21.15.102:0/3631458778 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56523bfc3d70 con 0x56523c0ead00 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.822+0000 7f573b400640 10 client.25826 handle_command_reply: tid=0 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f5763bbf740 1 client.25826 shutdown 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f5763bbf740 1 -- 172.21.15.102:0/3631458778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0575c0 msgr2=0x56523c06a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0575c0 0x56523c06a800 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f575400bc90 tx=0x7f574c00ea10 comp rx=0 tx=0).stop 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f5763bbf740 1 -- 172.21.15.102:0/3631458778 shutdown_connections 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56523c0ead00 0x56523c0ed120 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56523c0625b0 0x56523c0d1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56523c057ee0 0x56523c06ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f5763bbf740 1 --2- 172.21.15.102:0/3631458778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56523c0575c0 0x56523c06a800 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f5763bbf740 1 -- 172.21.15.102:0/3631458778 >> 172.21.15.102:0/3631458778 conn(0x56523c0538e0 msgr2=0x56523c05fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.823+0000 7f573b400640 0 client.25826 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.824+0000 7f5763bbf740 1 -- 172.21.15.102:0/3631458778 shutdown_connections 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.824+0000 7f5763bbf740 1 -- 172.21.15.102:0/3631458778 wait complete. 2024-04-25T01:57:04.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.824+0000 7f5763bbf740 20 client.25826 trim_cache size 0 max 16384 2024-04-25T01:57:04.829 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:57:04.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57400a2d80 msgr2=0x7f57400a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57400a2d80 0x7f57400a5240 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f57480b84c0 tx=0x7f5748005c90 comp rx=0 tx=0).stop 2024-04-25T01:57:04.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1a2ea0 msgr2=0x7f575c1a3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:04.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1a2ea0 0x7f575c1a3350 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f574802a960 tx=0x7f5748003b90 comp rx=0 tx=0).stop 2024-04-25T01:57:04.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 shutdown_connections 2024-04-25T01:57:04.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f57400a2d80 0x7f57400a5240 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f575c1a2ea0 0x7f575c1a3350 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f575c10d040 0x7f575c19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 --2- 172.21.15.102:0/1871798668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f575c107fc0 0x7f575c19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:04.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 >> 172.21.15.102:0/1871798668 conn(0x7f575c075c80 msgr2=0x7f575c076810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:04.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.825+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 shutdown_connections 2024-04-25T01:57:04.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:04.826+0000 7f5760c00640 1 -- 172.21.15.102:0/1871798668 wait complete. 2024-04-25T01:57:05.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:04 smithi102 ceph-mon[27796]: pgmap v57: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.6 MiB/s wr, 635 op/s 2024-04-25T01:57:05.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:04 smithi110 ceph-mon[32848]: pgmap v57: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.6 MiB/s wr, 635 op/s 2024-04-25T01:57:05.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:04 smithi193 ceph-mon[32981]: pgmap v57: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.6 MiB/s wr, 635 op/s 2024-04-25T01:57:05.853 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:57:05.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:57:05.854 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:57:06.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.110+0000 7f00cee00640 1 -- 172.21.15.102:0/1454871573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c810bbc0 msgr2=0x7f00c810e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.110+0000 7f00cee00640 1 --2- 172.21.15.102:0/1454871573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c810bbc0 0x7f00c810e060 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009ca0 tx=0x7f00c002a950 comp rx=0 tx=0).stop 2024-04-25T01:57:06.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.110+0000 7f00cee00640 1 -- 172.21.15.102:0/1454871573 shutdown_connections 2024-04-25T01:57:06.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.110+0000 7f00cee00640 1 --2- 172.21.15.102:0/1454871573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c810bbc0 0x7f00c810e060 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.110+0000 7f00cee00640 1 --2- 172.21.15.102:0/1454871573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00c81030c0 0x7f00c810b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.110+0000 7f00cee00640 1 --2- 172.21.15.102:0/1454871573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c81027a0 0x7f00c8102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.110+0000 7f00cee00640 1 -- 172.21.15.102:0/1454871573 >> 172.21.15.102:0/1454871573 conn(0x7f00c80fbfc0 msgr2=0x7f00c80fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.110+0000 7f00cee00640 1 -- 172.21.15.102:0/1454871573 shutdown_connections 2024-04-25T01:57:06.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.110+0000 7f00cee00640 1 -- 172.21.15.102:0/1454871573 wait complete. 2024-04-25T01:57:06.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.112+0000 7f00cee00640 1 Processor -- start 2024-04-25T01:57:06.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.112+0000 7f00cee00640 1 -- start start 2024-04-25T01:57:06.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cee00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c81027a0 0x7f00c819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cee00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00c81030c0 0x7f00c819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c810bbc0 0x7f00c819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00c81030c0 0x7f00c819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00c81030c0 0x7f00c819a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41588/0 (socket says 172.21.15.102:41588) 2024-04-25T01:57:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cda00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c81027a0 0x7f00c819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cda00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c81027a0 0x7f00c819a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36512/0 (socket says 172.21.15.102:36512) 2024-04-25T01:57:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f00c8111a10 con 0x7f00c810bbc0 2024-04-25T01:57:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cd000640 1 -- 172.21.15.102:0/863812068 learned_addr learned my addr 172.21.15.102:0/863812068 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f00c8111890 con 0x7f00c81027a0 2024-04-25T01:57:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.113+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f00c8111b90 con 0x7f00c81030c0 2024-04-25T01:57:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.114+0000 7f00ce400640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c810bbc0 0x7f00c819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.114+0000 7f00cda00640 1 -- 172.21.15.102:0/863812068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00c81030c0 msgr2=0x7f00c819a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.114+0000 7f00cda00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00c81030c0 0x7f00c819a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.114+0000 7f00cda00640 1 -- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c810bbc0 msgr2=0x7f00c819e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.114+0000 7f00cda00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c810bbc0 0x7f00c819e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.114+0000 7f00cda00640 1 -- 172.21.15.102:0/863812068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c819f020 con 0x7f00c81027a0 2024-04-25T01:57:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.114+0000 7f00ce400640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c810bbc0 0x7f00c819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.114+0000 7f00cda00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c81027a0 0x7f00c819a030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00c970 tx=0x7f00bc00ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.115+0000 7f00c6000640 1 -- 172.21.15.102:0/863812068 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc007bb0 con 0x7f00c81027a0 2024-04-25T01:57:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.115+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c819f310 con 0x7f00c81027a0 2024-04-25T01:57:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.115+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c81a6ba0 con 0x7f00c81027a0 2024-04-25T01:57:06.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.115+0000 7f00c6000640 1 -- 172.21.15.102:0/863812068 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00bc007d50 con 0x7f00c81027a0 2024-04-25T01:57:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.115+0000 7f00c6000640 1 -- 172.21.15.102:0/863812068 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00bc018820 con 0x7f00c81027a0 2024-04-25T01:57:06.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.120+0000 7f00d3374740 1 Processor -- start 2024-04-25T01:57:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.120+0000 7f00d3374740 1 -- start start 2024-04-25T01:57:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.121+0000 7f00d3374740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb52300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.121+0000 7f00d3374740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb5c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.121+0000 7f00d3374740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.121+0000 7f00d3374740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f8ecab0f40 con 0x55f8ecb5cf10 2024-04-25T01:57:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.121+0000 7f00d3374740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f8ecab6190 con 0x55f8ecb51f20 2024-04-25T01:57:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.121+0000 7f00d3374740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f8eca5ea10 con 0x55f8ecb52840 2024-04-25T01:57:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.121+0000 7f00cd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.121+0000 7f00cd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37188/0 (socket says 172.21.15.102:37188) 2024-04-25T01:57:06.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.121+0000 7f00cd000640 1 -- 172.21.15.102:0/1722960237 learned_addr learned my addr 172.21.15.102:0/1722960237 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.122+0000 7f00ce400640 1 --2- 172.21.15.102:0/1722960237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb52300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.122+0000 7f00ce400640 1 -- 172.21.15.102:0/1722960237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 msgr2=0x55f8ecb5c9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:06.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.122+0000 7f00ce400640 1 --2- 172.21.15.102:0/1722960237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb5c9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.122+0000 7f00ce400640 1 -- 172.21.15.102:0/1722960237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 msgr2=0x55f8ecb5f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.122+0000 7f00c6000640 1 -- 172.21.15.102:0/863812068 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f00bc0189c0 con 0x7f00c81027a0 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00ce400640 1 --2- 172.21.15.102:0/1722960237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00ce400640 1 -- 172.21.15.102:0/1722960237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8ec8927e0 con 0x55f8ecb51f20 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00c6000640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00b00a2e50 0x7f00b00a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00c6000640 1 -- 172.21.15.102:0/863812068 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f00bc0cf2e0 con 0x7f00c81027a0 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00ce400640 1 --2- 172.21.15.102:0/1722960237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb52300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009ca0 tx=0x7f00c0009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba955ab85c9d3b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00cd000640 1 --2- 172.21.15.102:0/1722960237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00baa00640 1 -- 172.21.15.102:0/1722960237 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0034050 con 0x55f8ecb51f20 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00baa00640 1 -- 172.21.15.102:0/1722960237 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00c0003e50 con 0x55f8ecb51f20 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00baa00640 1 -- 172.21.15.102:0/1722960237 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c00042c0 con 0x55f8ecb51f20 2024-04-25T01:57:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.124+0000 7f00cda00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00b00a2e50 0x7f00b00a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.126+0000 7f00d3374740 1 -- 172.21.15.102:0/1722960237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 msgr2=0x55f8ecb52300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.126+0000 7f00d3374740 1 --2- 172.21.15.102:0/1722960237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb52300 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f00c0009ca0 tx=0x7f00c0009a20 comp rx=0 tx=0).stop 2024-04-25T01:57:06.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.126+0000 7f00d3374740 1 -- 172.21.15.102:0/1722960237 shutdown_connections 2024-04-25T01:57:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.126+0000 7f00d3374740 1 --2- 172.21.15.102:0/1722960237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.126+0000 7f00d3374740 1 --2- 172.21.15.102:0/1722960237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb5c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.126+0000 7f00d3374740 1 --2- 172.21.15.102:0/1722960237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb52300 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.126+0000 7f00d3374740 1 -- 172.21.15.102:0/1722960237 >> 172.21.15.102:0/1722960237 conn(0x55f8ecb4e240 msgr2=0x55f8ecb4e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.126+0000 7f00cda00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00b00a2e50 0x7f00b00a5310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55f8ecb534e0 tx=0x7f00bc007f00 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.127+0000 7f00d3374740 1 -- 172.21.15.102:0/1722960237 shutdown_connections 2024-04-25T01:57:06.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.127+0000 7f00d3374740 1 -- 172.21.15.102:0/1722960237 wait complete. 2024-04-25T01:57:06.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.127+0000 7f00d3374740 1 -- start start 2024-04-25T01:57:06.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.128+0000 7f00d3374740 1 Processor -- start 2024-04-25T01:57:06.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.129+0000 7f00d3374740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb51f20 0x55f8ecb59f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.129+0000 7f00d3374740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb5a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.129+0000 7f00d3374740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.129+0000 7f00ce400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb51f20 0x55f8ecb59f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.129+0000 7f00ce400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb51f20 0x55f8ecb59f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37204/0 (socket says 172.21.15.102:37204) 2024-04-25T01:57:06.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.129+0000 7f00cd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb5a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.129+0000 7f00cd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb5a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41604/0 (socket says 172.21.15.102:41604) 2024-04-25T01:57:06.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.129+0000 7f00ce400640 1 -- 172.21.15.102:0/921573924 learned_addr learned my addr 172.21.15.102:0/921573924 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.129+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f8eca71e70 con 0x55f8ecb51f20 2024-04-25T01:57:06.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.130+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f8ecab2150 con 0x55f8ecb5cf10 2024-04-25T01:57:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.130+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f8ecb5fd40 con 0x55f8ecb52840 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.130+0000 7f00ce400640 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.130+0000 7f00d3374740 10 client.0 _ll_register_callbacks cb 0x55f8ecaad1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.130+0000 7f00ce400640 1 -- 172.21.15.102:0/921573924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 msgr2=0x55f8ecb5a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.130+0000 7f00ce400640 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb5a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.130+0000 7f00ce400640 1 -- 172.21.15.102:0/921573924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 msgr2=0x55f8ecb5a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.130+0000 7f00ce400640 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.131+0000 7f00ce400640 1 -- 172.21.15.102:0/921573924 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8ec8f2f70 con 0x55f8ecb51f20 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.131+0000 7f00cd000640 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb5a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.131+0000 7f00ce400640 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb51f20 0x55f8ecb59f00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f00c002ffb0 tx=0x7f00c00095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cda7b4524757fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.131+0000 7f00af400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:57:06.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.132+0000 7f00af400640 1 -- 172.21.15.102:0/921573924 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0048040 con 0x55f8ecb51f20 2024-04-25T01:57:06.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.132+0000 7f00af400640 1 -- 172.21.15.102:0/921573924 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f00c0056d10 con 0x55f8ecb51f20 2024-04-25T01:57:06.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.132+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00c0003e50 con 0x55f8ecb51f20 2024-04-25T01:57:06.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.132+0000 7f00af400640 1 -- 172.21.15.102:0/921573924 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f00c0065050 con 0x55f8ecb51f20 2024-04-25T01:57:06.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.132+0000 7f00d3374740 10 client.15663 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:06.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.133+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8ec78a170 con 0x55f8ecb51f20 2024-04-25T01:57:06.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.134+0000 7f00af400640 1 -- 172.21.15.102:0/921573924 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00c0003e50 con 0x55f8ecb51f20 2024-04-25T01:57:06.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.135+0000 7f00af400640 1 -- 172.21.15.102:0/921573924 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f00c0041070 con 0x55f8ecb51f20 2024-04-25T01:57:06.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.135+0000 7f00d3374740 10 client.15663 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:06.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.135+0000 7f00d3374740 10 client.15663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:06.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.135+0000 7f00d3374740 20 client.15663 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:06.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.135+0000 7f00d3374740 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f8ecb6d070 0x55f8ecb6f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.136+0000 7f00d3374740 4 client.15663 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:06.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.136+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8ecac36d0 con 0x55f8ecb6d070 2024-04-25T01:57:06.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.136+0000 7f00cd000640 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f8ecb6d070 0x55f8ecb6f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.136+0000 7f00cd000640 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f8ecb6d070 0x55f8ecb6f460 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c943a94fcd48338 server_cookie=a9277f409b8b84d8 in_seq=0 out_seq=0 2024-04-25T01:57:06.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.137+0000 7f00af400640 10 client.15663 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:06.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.137+0000 7f00af400640 1 -- 172.21.15.102:0/921573924 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f8ecac36d0 con 0x55f8ecb6d070 2024-04-25T01:57:06.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.137+0000 7f00af400640 10 client.15663 handle_command_reply: tid=0 2024-04-25T01:57:06.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.137+0000 7f00d3374740 1 client.15663 shutdown 2024-04-25T01:57:06.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.138+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb51f20 msgr2=0x55f8ecb59f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.138+0000 7f00d3374740 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb51f20 0x55f8ecb59f00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f00c002ffb0 tx=0x7f00c00095a0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.139+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 shutdown_connections 2024-04-25T01:57:06.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.139+0000 7f00d3374740 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f8ecb6d070 0x55f8ecb6f460 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.139+0000 7f00d3374740 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb5a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.139+0000 7f00af400640 0 client.15663 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:06.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.139+0000 7f00d3374740 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb5a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.139+0000 7f00d3374740 1 --2- 172.21.15.102:0/921573924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb51f20 0x55f8ecb59f00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.139+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 >> 172.21.15.102:0/921573924 conn(0x55f8ecb4e240 msgr2=0x55f8ecb53f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.140+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 shutdown_connections 2024-04-25T01:57:06.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.140+0000 7f00d3374740 1 -- 172.21.15.102:0/921573924 wait complete. 2024-04-25T01:57:06.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.140+0000 7f00d3374740 20 client.15663 trim_cache size 0 max 16384 2024-04-25T01:57:06.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00d3374740 1 Processor -- start 2024-04-25T01:57:06.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00d3374740 1 -- start start 2024-04-25T01:57:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00d3374740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb68b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00d3374740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb74a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00d3374740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb74f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00d3374740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f00a8011070 con 0x55f8ecb5cf10 2024-04-25T01:57:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00d3374740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f00a800e8e0 con 0x55f8ecb51f20 2024-04-25T01:57:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00d3374740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f00a800c730 con 0x55f8ecb52840 2024-04-25T01:57:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00cd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb74a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00cd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb74a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41610/0 (socket says 172.21.15.102:41610) 2024-04-25T01:57:06.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.175+0000 7f00cd000640 1 -- 172.21.15.102:0/1081784362 learned_addr learned my addr 172.21.15.102:0/1081784362 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.176+0000 7f00ce400640 1 --2- 172.21.15.102:0/1081784362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb68b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.176+0000 7f00cd000640 1 -- 172.21.15.102:0/1081784362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 msgr2=0x55f8ecb68b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.177+0000 7f00cd000640 1 --2- 172.21.15.102:0/1081784362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb68b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.177+0000 7f00cd000640 1 -- 172.21.15.102:0/1081784362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 msgr2=0x55f8ecb74f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.177+0000 7f00cd000640 1 --2- 172.21.15.102:0/1081784362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb74f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.177+0000 7f00cd000640 1 -- 172.21.15.102:0/1081784362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8ecb4c750 con 0x55f8ecb52840 2024-04-25T01:57:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.177+0000 7f00ce400640 1 --2- 172.21.15.102:0/1081784362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb68b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.178+0000 7f00cd000640 1 --2- 172.21.15.102:0/1081784362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb74a30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009fd0 tx=0x7f00b4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4200e29d88396454 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.178+0000 7f00ba000640 1 -- 172.21.15.102:0/1081784362 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4005220 con 0x55f8ecb52840 2024-04-25T01:57:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.178+0000 7f00ba000640 1 -- 172.21.15.102:0/1081784362 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b4004c80 con 0x55f8ecb52840 2024-04-25T01:57:06.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.178+0000 7f00ba000640 1 -- 172.21.15.102:0/1081784362 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b40206a0 con 0x55f8ecb52840 2024-04-25T01:57:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.179+0000 7f00d3374740 1 -- 172.21.15.102:0/1081784362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 msgr2=0x55f8ecb74a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.179+0000 7f00d3374740 1 --2- 172.21.15.102:0/1081784362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb74a30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009fd0 tx=0x7f00b4005f90 comp rx=0 tx=0).stop 2024-04-25T01:57:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.179+0000 7f00d3374740 1 -- 172.21.15.102:0/1081784362 shutdown_connections 2024-04-25T01:57:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.179+0000 7f00d3374740 1 --2- 172.21.15.102:0/1081784362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb74f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.179+0000 7f00d3374740 1 --2- 172.21.15.102:0/1081784362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb52840 0x55f8ecb74a30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.179+0000 7f00d3374740 1 --2- 172.21.15.102:0/1081784362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb51f20 0x55f8ecb68b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.179+0000 7f00d3374740 1 -- 172.21.15.102:0/1081784362 >> 172.21.15.102:0/1081784362 conn(0x55f8ecb4e240 msgr2=0x55f8ecb4d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.179+0000 7f00d3374740 1 -- 172.21.15.102:0/1081784362 shutdown_connections 2024-04-25T01:57:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.179+0000 7f00d3374740 1 -- 172.21.15.102:0/1081784362 wait complete. 2024-04-25T01:57:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.180+0000 7f00d3374740 1 -- start start 2024-04-25T01:57:06.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.180+0000 7f00d3374740 1 Processor -- start 2024-04-25T01:57:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.180+0000 7f00d3374740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb51f20 0x55f8ecb65210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.180+0000 7f00d3374740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb52840 0x55f8ecb61ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.180+0000 7f00d3374740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb62530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.180+0000 7f00d3374740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f00a800cda0 con 0x55f8ecb52840 2024-04-25T01:57:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.180+0000 7f00d3374740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f00a800e210 con 0x55f8ecb5cf10 2024-04-25T01:57:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.180+0000 7f00d3374740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f00a800d470 con 0x55f8ecb51f20 2024-04-25T01:57:06.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.181+0000 7f00d3374740 10 client.0 _ll_register_callbacks cb 0x7f00b4009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.181+0000 7f00ce400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb51f20 0x55f8ecb65210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.181+0000 7f00cd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb52840 0x55f8ecb61ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.182+0000 7f00ce400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb51f20 0x55f8ecb65210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41614/0 (socket says 172.21.15.102:41614) 2024-04-25T01:57:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.182+0000 7f00ce400640 1 -- 172.21.15.102:0/1896974989 learned_addr learned my addr 172.21.15.102:0/1896974989 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.182+0000 7f00ce400640 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb62530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.182+0000 7f00ce400640 1 -- 172.21.15.102:0/1896974989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb51f20 msgr2=0x55f8ecb65210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.182+0000 7f00ce400640 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb51f20 0x55f8ecb65210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.182+0000 7f00ce400640 1 -- 172.21.15.102:0/1896974989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb52840 msgr2=0x55f8ecb61ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.182+0000 7f00ce400640 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb52840 0x55f8ecb61ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.183+0000 7f00ce400640 1 -- 172.21.15.102:0/1896974989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8ecb62a70 con 0x55f8ecb5cf10 2024-04-25T01:57:06.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.183+0000 7f00cd000640 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb52840 0x55f8ecb61ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.183+0000 7f00ce400640 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb62530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f00c0002230 tx=0x7f00c0009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d784d377c49ef63f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.183+0000 7f00af400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.183+0000 7f00af400640 1 -- 172.21.15.102:0/1896974989 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0055060 con 0x55f8ecb5cf10 2024-04-25T01:57:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.183+0000 7f00af400640 1 -- 172.21.15.102:0/1896974989 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f00c0043850 con 0x55f8ecb5cf10 2024-04-25T01:57:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.184+0000 7f00d3374740 1 -- 172.21.15.102:0/1896974989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00b4004c80 con 0x55f8ecb5cf10 2024-04-25T01:57:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.184+0000 7f00af400640 1 -- 172.21.15.102:0/1896974989 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f00c0043a50 con 0x55f8ecb5cf10 2024-04-25T01:57:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.184+0000 7f00d3374740 10 client.25841 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.184+0000 7f00d3374740 1 -- 172.21.15.102:0/1896974989 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8ecbe4fc0 con 0x55f8ecb5cf10 2024-04-25T01:57:06.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.184+0000 7f00af400640 1 -- 172.21.15.102:0/1896974989 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b4004c80 con 0x55f8ecb5cf10 2024-04-25T01:57:06.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.185+0000 7f00af400640 1 -- 172.21.15.102:0/1896974989 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f00c0041050 con 0x55f8ecb5cf10 2024-04-25T01:57:06.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.186+0000 7f00d3374740 10 client.25841 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:06.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.186+0000 7f00d3374740 10 client.25841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:06.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.186+0000 7f00d3374740 20 client.25841 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:06.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.186+0000 7f00d3374740 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f8ecbe5640 0x55f8ecbe7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.186+0000 7f00d3374740 4 client.25841 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:57:06.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.186+0000 7f00d3374740 1 -- 172.21.15.102:0/1896974989 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f8ecabda20 con 0x55f8ecbe5640 2024-04-25T01:57:06.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.186+0000 7f00cd000640 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f8ecbe5640 0x55f8ecbe7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.187+0000 7f00cd000640 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f8ecbe5640 0x55f8ecbe7a60 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce7239002034683 server_cookie=d5c4c161f9d87817 in_seq=0 out_seq=0 2024-04-25T01:57:06.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.187+0000 7f00af400640 10 client.25841 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:06.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.188+0000 7f00af400640 1 -- 172.21.15.102:0/1896974989 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f8ecabda20 con 0x55f8ecbe5640 2024-04-25T01:57:06.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.188+0000 7f00af400640 10 client.25841 handle_command_reply: tid=0 2024-04-25T01:57:06.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.188+0000 7f00d3374740 1 client.25841 shutdown 2024-04-25T01:57:06.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.189+0000 7f00d3374740 1 -- 172.21.15.102:0/1896974989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 msgr2=0x55f8ecb62530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.189+0000 7f00d3374740 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb62530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f00c0002230 tx=0x7f00c0009a20 comp rx=0 tx=0).stop 2024-04-25T01:57:06.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.190+0000 7f00d3374740 1 -- 172.21.15.102:0/1896974989 shutdown_connections 2024-04-25T01:57:06.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.190+0000 7f00d3374740 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f8ecbe5640 0x55f8ecbe7a60 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.190+0000 7f00d3374740 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f8ecb5cf10 0x55f8ecb62530 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.190+0000 7f00d3374740 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f8ecb52840 0x55f8ecb61ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.190+0000 7f00af400640 0 client.25841 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:06.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.190+0000 7f00d3374740 1 --2- 172.21.15.102:0/1896974989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f8ecb51f20 0x55f8ecb65210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.190+0000 7f00d3374740 1 -- 172.21.15.102:0/1896974989 >> 172.21.15.102:0/1896974989 conn(0x55f8ecb4e240 msgr2=0x55f8ecb59ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.191+0000 7f00d3374740 1 -- 172.21.15.102:0/1896974989 shutdown_connections 2024-04-25T01:57:06.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.191+0000 7f00d3374740 1 -- 172.21.15.102:0/1896974989 wait complete. 2024-04-25T01:57:06.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.191+0000 7f00d3374740 20 client.25841 trim_cache size 0 max 16384 2024-04-25T01:57:06.195 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:57:06.195 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T01:57:06.195 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "d77a5dde-12ff-41d0-a109-d03404ea4f91", 2024-04-25T01:57:06.195 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T01:57:06.195 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:57:06.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00b00a2e50 msgr2=0x7f00b00a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00b00a2e50 0x7f00b00a5310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55f8ecb534e0 tx=0x7f00bc007f00 comp rx=0 tx=0).stop 2024-04-25T01:57:06.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c81027a0 msgr2=0x7f00c819a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c81027a0 0x7f00c819a030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00c970 tx=0x7f00bc00ce30 comp rx=0 tx=0).stop 2024-04-25T01:57:06.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 shutdown_connections 2024-04-25T01:57:06.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00b00a2e50 0x7f00b00a5310 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f00c810bbc0 0x7f00c819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00c81030c0 0x7f00c819a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 --2- 172.21.15.102:0/863812068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00c81027a0 0x7f00c819a030 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.193+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 >> 172.21.15.102:0/863812068 conn(0x7f00c80fbfc0 msgr2=0x7f00c810d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.194+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 shutdown_connections 2024-04-25T01:57:06.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.195+0000 7f00cee00640 1 -- 172.21.15.102:0/863812068 wait complete. 2024-04-25T01:57:06.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:57:06.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:57:06.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.528+0000 7f74ad800640 1 -- 172.21.15.102:0/1225179254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a806da70 msgr2=0x7f74a811a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.528+0000 7f74ad800640 1 --2- 172.21.15.102:0/1225179254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a806da70 0x7f74a811a8c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7494009900 tx=0x7f749402a480 comp rx=0 tx=0).stop 2024-04-25T01:57:06.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.529+0000 7f74ad800640 1 -- 172.21.15.102:0/1225179254 shutdown_connections 2024-04-25T01:57:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.529+0000 7f74ad800640 1 --2- 172.21.15.102:0/1225179254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a811ae00 0x7f74a811d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.529+0000 7f74ad800640 1 --2- 172.21.15.102:0/1225179254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a806da70 0x7f74a811a8c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.529+0000 7f74ad800640 1 --2- 172.21.15.102:0/1225179254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74a806d150 0x7f74a806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.529+0000 7f74ad800640 1 -- 172.21.15.102:0/1225179254 >> 172.21.15.102:0/1225179254 conn(0x7f74a806c7b0 msgr2=0x7f74a810a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.530+0000 7f74ad800640 1 -- 172.21.15.102:0/1225179254 shutdown_connections 2024-04-25T01:57:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.530+0000 7f74ad800640 1 -- 172.21.15.102:0/1225179254 wait complete. 2024-04-25T01:57:06.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.532+0000 7f74ad800640 1 Processor -- start 2024-04-25T01:57:06.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.533+0000 7f74ad800640 1 -- start start 2024-04-25T01:57:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.534+0000 7f74ad800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a806d150 0x7f74a811a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.534+0000 7f74ad800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74a806da70 0x7f74a81134a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.534+0000 7f74ad800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a811ae00 0x7f74a81139e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.534+0000 7f74ad800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f74a8120940 con 0x7f74a806da70 2024-04-25T01:57:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.534+0000 7f74ad800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f74a81207c0 con 0x7f74a811ae00 2024-04-25T01:57:06.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.534+0000 7f74ad800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74a8120ac0 con 0x7f74a806d150 2024-04-25T01:57:06.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.534+0000 7f74a7400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a811ae00 0x7f74a81139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.534+0000 7f74a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a806d150 0x7f74a811a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.534+0000 7f74a7400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a811ae00 0x7f74a81139e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36564/0 (socket says 172.21.15.102:36564) 2024-04-25T01:57:06.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.535+0000 7f74a7400640 1 -- 172.21.15.102:0/147071160 learned_addr learned my addr 172.21.15.102:0/147071160 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.535+0000 7f74a6000640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74a806da70 0x7f74a81134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.535+0000 7f74a7400640 1 -- 172.21.15.102:0/147071160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a806d150 msgr2=0x7f74a811a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.535+0000 7f74a7400640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a806d150 0x7f74a811a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.535+0000 7f74a7400640 1 -- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74a806da70 msgr2=0x7f74a81134a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.535+0000 7f74a7400640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74a806da70 0x7f74a81134a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.535+0000 7f74a7400640 1 -- 172.21.15.102:0/147071160 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a8113fa0 con 0x7f74a811ae00 2024-04-25T01:57:06.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.535+0000 7f74a6000640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74a806da70 0x7f74a81134a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:06.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.536+0000 7f74a7400640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a811ae00 0x7f74a81139e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f74a000b970 tx=0x7f74a000be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.537+0000 7f749f400640 1 -- 172.21.15.102:0/147071160 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0019070 con 0x7f74a811ae00 2024-04-25T01:57:06.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.537+0000 7f749f400640 1 -- 172.21.15.102:0/147071160 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a000cb40 con 0x7f74a811ae00 2024-04-25T01:57:06.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.537+0000 7f749f400640 1 -- 172.21.15.102:0/147071160 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0007650 con 0x7f74a811ae00 2024-04-25T01:57:06.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.537+0000 7f74ad800640 1 -- 172.21.15.102:0/147071160 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a8114290 con 0x7f74a811ae00 2024-04-25T01:57:06.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.538+0000 7f74ad800640 1 -- 172.21.15.102:0/147071160 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a8118220 con 0x7f74a811ae00 2024-04-25T01:57:06.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.542+0000 7f74b05b7740 1 Processor -- start 2024-04-25T01:57:06.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.542+0000 7f749f400640 1 -- 172.21.15.102:0/147071160 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f74a001c070 con 0x7f74a811ae00 2024-04-25T01:57:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.543+0000 7f74b05b7740 1 -- start start 2024-04-25T01:57:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.543+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.543+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85ce920 0x5643e85d8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.543+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85d8ff0 0x5643e85db3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.543+0000 7f74b05b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5643e8532560 con 0x5643e85ce920 2024-04-25T01:57:06.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.543+0000 7f74b05b7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5643e84da930 con 0x5643e85d8ff0 2024-04-25T01:57:06.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.543+0000 7f74b05b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5643e852e520 con 0x5643e85ce000 2024-04-25T01:57:06.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.544+0000 7f74a6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85d8ff0 0x5643e85db3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.544+0000 7f74a6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.544+0000 7f74a6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85ce3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41650/0 (socket says 172.21.15.102:41650) 2024-04-25T01:57:06.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.544+0000 7f74a6000640 1 -- 172.21.15.102:0/2788300214 learned_addr learned my addr 172.21.15.102:0/2788300214 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.544+0000 7f74a6a00640 1 --2- 172.21.15.102:0/2788300214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85ce920 0x5643e85d8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.545+0000 7f74a6000640 1 -- 172.21.15.102:0/2788300214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 msgr2=0x5643e85ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.545+0000 7f74a6000640 1 --2- 172.21.15.102:0/2788300214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.545+0000 7f74a6000640 1 -- 172.21.15.102:0/2788300214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85ce920 msgr2=0x5643e85d8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.545+0000 7f74a6000640 1 --2- 172.21.15.102:0/2788300214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85ce920 0x5643e85d8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.545+0000 7f74a6000640 1 -- 172.21.15.102:0/2788300214 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643e83107d0 con 0x5643e85d8ff0 2024-04-25T01:57:06.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.547+0000 7f749f400640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f748c0a2e50 0x7f748c0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.547+0000 7f74a6000640 1 --2- 172.21.15.102:0/2788300214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85d8ff0 0x5643e85db3e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7494005ec0 tx=0x7f7494000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46160a9009c80c1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.548+0000 7f7493e00640 1 -- 172.21.15.102:0/2788300214 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494035040 con 0x5643e85d8ff0 2024-04-25T01:57:06.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.548+0000 7f7493e00640 1 -- 172.21.15.102:0/2788300214 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7494005020 con 0x5643e85d8ff0 2024-04-25T01:57:06.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.548+0000 7f7493e00640 1 -- 172.21.15.102:0/2788300214 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494004920 con 0x5643e85d8ff0 2024-04-25T01:57:06.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.548+0000 7f749f400640 1 -- 172.21.15.102:0/147071160 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f74a00ce730 con 0x7f74a811ae00 2024-04-25T01:57:06.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.548+0000 7f74a6a00640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f748c0a2e50 0x7f748c0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.549+0000 7f74b05b7740 1 -- 172.21.15.102:0/2788300214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85d8ff0 msgr2=0x5643e85db3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.549+0000 7f74b05b7740 1 --2- 172.21.15.102:0/2788300214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85d8ff0 0x5643e85db3e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7494005ec0 tx=0x7f7494000bc0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.549+0000 7f74b05b7740 1 -- 172.21.15.102:0/2788300214 shutdown_connections 2024-04-25T01:57:06.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.549+0000 7f74b05b7740 1 --2- 172.21.15.102:0/2788300214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85d8ff0 0x5643e85db3e0 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7494005ec0 tx=0x7f7494000bc0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.550+0000 7f74b05b7740 1 --2- 172.21.15.102:0/2788300214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85ce920 0x5643e85d8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.550+0000 7f74b05b7740 1 --2- 172.21.15.102:0/2788300214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.550+0000 7f74b05b7740 1 -- 172.21.15.102:0/2788300214 >> 172.21.15.102:0/2788300214 conn(0x5643e85ca320 msgr2=0x5643e85ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.551+0000 7f74a6a00640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f748c0a2e50 0x7f748c0a5310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5643e85cf5c0 tx=0x7f749800b390 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.551+0000 7f74b05b7740 1 -- 172.21.15.102:0/2788300214 shutdown_connections 2024-04-25T01:57:06.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.552+0000 7f74b05b7740 1 -- 172.21.15.102:0/2788300214 wait complete. 2024-04-25T01:57:06.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.552+0000 7f74b05b7740 1 -- start start 2024-04-25T01:57:06.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.553+0000 7f74b05b7740 1 Processor -- start 2024-04-25T01:57:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.553+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.553+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce920 0x5643e85d6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.553+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85d8ff0 0x5643e85d6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.553+0000 7f74b05b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5643e85dbd80 con 0x5643e85d8ff0 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.553+0000 7f74b05b7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5643e84edd90 con 0x5643e85ce920 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.553+0000 7f74b05b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5643e85dbf00 con 0x5643e85ce000 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.554+0000 7f74b05b7740 10 client.0 _ll_register_callbacks cb 0x5643e84ed780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.554+0000 7f74a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.554+0000 7f74a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41660/0 (socket says 172.21.15.102:41660) 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.554+0000 7f74a6a00640 1 -- 172.21.15.102:0/3191729672 learned_addr learned my addr 172.21.15.102:0/3191729672 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.554+0000 7f74a6a00640 1 -- 172.21.15.102:0/3191729672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce920 msgr2=0x5643e85d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.554+0000 7f74a6a00640 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce920 0x5643e85d6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.554+0000 7f74a6a00640 1 -- 172.21.15.102:0/3191729672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85d8ff0 msgr2=0x5643e85d6990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.554+0000 7f74a6a00640 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85d8ff0 0x5643e85d6990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.554+0000 7f74a6a00640 1 -- 172.21.15.102:0/3191729672 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643e82463f0 con 0x5643e85ce000 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.555+0000 7f74a6a00640 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d5f10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7498003dd0 tx=0x7f7498004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b117a4a7ca1a5368 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.555+0000 7f7490c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.555+0000 7f7490c00640 1 -- 172.21.15.102:0/3191729672 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749802c060 con 0x5643e85ce000 2024-04-25T01:57:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.556+0000 7f74b05b7740 1 -- 172.21.15.102:0/3191729672 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7494005020 con 0x5643e85ce000 2024-04-25T01:57:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.555+0000 7f7490c00640 1 -- 172.21.15.102:0/3191729672 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7498017030 con 0x5643e85ce000 2024-04-25T01:57:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.556+0000 7f7490c00640 1 -- 172.21.15.102:0/3191729672 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7498027410 con 0x5643e85ce000 2024-04-25T01:57:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.556+0000 7f74b05b7740 10 client.25837 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.556+0000 7f74b05b7740 1 -- 172.21.15.102:0/3191729672 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643e82060f0 con 0x5643e85ce000 2024-04-25T01:57:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.558+0000 7f7490c00640 1 -- 172.21.15.102:0/3191729672 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7494005020 con 0x5643e85ce000 2024-04-25T01:57:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.558+0000 7f7490c00640 1 -- 172.21.15.102:0/3191729672 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7498011680 con 0x5643e85ce000 2024-04-25T01:57:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.558+0000 7f74b05b7740 10 client.25837 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.558+0000 7f74b05b7740 10 client.25837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.558+0000 7f74b05b7740 20 client.25837 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.558+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5643e85e9480 0x5643e85eb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.558+0000 7f74b05b7740 4 client.25837 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.558+0000 7f74b05b7740 1 -- 172.21.15.102:0/3191729672 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643e8545d90 con 0x5643e85e9480 2024-04-25T01:57:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.559+0000 7f74a6000640 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5643e85e9480 0x5643e85eb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.560+0000 7f7490c00640 10 client.25837 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.560+0000 7f74a6000640 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5643e85e9480 0x5643e85eb940 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=409bfabf7e67c2ba server_cookie=27e0755362ec50d7 in_seq=0 out_seq=0 2024-04-25T01:57:06.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.561+0000 7f7490c00640 1 -- 172.21.15.102:0/3191729672 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5643e8545d90 con 0x5643e85e9480 2024-04-25T01:57:06.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.561+0000 7f7490c00640 10 client.25837 handle_command_reply: tid=0 2024-04-25T01:57:06.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.561+0000 7f74b05b7740 1 client.25837 shutdown 2024-04-25T01:57:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.562+0000 7f74b05b7740 1 -- 172.21.15.102:0/3191729672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 msgr2=0x5643e85d5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.562+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d5f10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7498003dd0 tx=0x7f7498004730 comp rx=0 tx=0).stop 2024-04-25T01:57:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.562+0000 7f74b05b7740 1 -- 172.21.15.102:0/3191729672 shutdown_connections 2024-04-25T01:57:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.562+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5643e85e9480 0x5643e85eb940 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.562+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85d8ff0 0x5643e85d6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.562+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce920 0x5643e85d6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.562+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3191729672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d5f10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.562+0000 7f7490c00640 0 client.25837 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.562+0000 7f74b05b7740 1 -- 172.21.15.102:0/3191729672 >> 172.21.15.102:0/3191729672 conn(0x5643e85ca320 msgr2=0x5643e85c9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.563+0000 7f74b05b7740 1 -- 172.21.15.102:0/3191729672 shutdown_connections 2024-04-25T01:57:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.563+0000 7f74b05b7740 1 -- 172.21.15.102:0/3191729672 wait complete. 2024-04-25T01:57:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.564+0000 7f74b05b7740 20 client.25837 trim_cache size 0 max 16384 2024-04-25T01:57:06.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.590+0000 7f74b05b7740 1 Processor -- start 2024-04-25T01:57:06.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.590+0000 7f74b05b7740 1 -- start start 2024-04-25T01:57:06.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce920 0x5643e85e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85d8ff0 0x5643e85d1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74b05b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f74a81c7410 con 0x5643e85d8ff0 2024-04-25T01:57:06.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74b05b7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f74a81c4c80 con 0x5643e85ce920 2024-04-25T01:57:06.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74b05b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74a81c2a90 con 0x5643e85ce000 2024-04-25T01:57:06.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41662/0 (socket says 172.21.15.102:41662) 2024-04-25T01:57:06.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74a6a00640 1 -- 172.21.15.102:0/3133495770 learned_addr learned my addr 172.21.15.102:0/3133495770 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.591+0000 7f74a6000640 1 --2- 172.21.15.102:0/3133495770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce920 0x5643e85e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.592+0000 7f74a6a00640 1 -- 172.21.15.102:0/3133495770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce920 msgr2=0x5643e85e4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.592+0000 7f74a6a00640 1 --2- 172.21.15.102:0/3133495770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce920 0x5643e85e4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.592+0000 7f74a6a00640 1 -- 172.21.15.102:0/3133495770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85d8ff0 msgr2=0x5643e85d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:06.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.592+0000 7f74a6a00640 1 --2- 172.21.15.102:0/3133495770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85d8ff0 0x5643e85d1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.592+0000 7f74a6a00640 1 -- 172.21.15.102:0/3133495770 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643e85c48c0 con 0x5643e85ce000 2024-04-25T01:57:06.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.592+0000 7f74a6a00640 1 --2- 172.21.15.102:0/3133495770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d4e70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7498018040 tx=0x7f7498011fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf81c62be3ef320f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.592+0000 7f7493400640 1 -- 172.21.15.102:0/3133495770 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498027410 con 0x5643e85ce000 2024-04-25T01:57:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.593+0000 7f7493400640 1 -- 172.21.15.102:0/3133495770 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74980275b0 con 0x5643e85ce000 2024-04-25T01:57:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.593+0000 7f7493400640 1 -- 172.21.15.102:0/3133495770 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749801c6f0 con 0x5643e85ce000 2024-04-25T01:57:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.593+0000 7f74b05b7740 1 -- 172.21.15.102:0/3133495770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 msgr2=0x5643e85d4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.594+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3133495770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d4e70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7498018040 tx=0x7f7498011fa0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.594+0000 7f74b05b7740 1 -- 172.21.15.102:0/3133495770 shutdown_connections 2024-04-25T01:57:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.594+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3133495770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85d8ff0 0x5643e85d1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.594+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3133495770 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce920 0x5643e85e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.594+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3133495770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85ce000 0x5643e85d4e70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.594+0000 7f74b05b7740 1 -- 172.21.15.102:0/3133495770 >> 172.21.15.102:0/3133495770 conn(0x5643e85ca320 msgr2=0x5643e85c9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.594+0000 7f74b05b7740 1 -- 172.21.15.102:0/3133495770 shutdown_connections 2024-04-25T01:57:06.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.594+0000 7f74b05b7740 1 -- 172.21.15.102:0/3133495770 wait complete. 2024-04-25T01:57:06.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.595+0000 7f74b05b7740 1 -- start start 2024-04-25T01:57:06.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.595+0000 7f74b05b7740 1 Processor -- start 2024-04-25T01:57:06.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.595+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce000 0x5643e85d3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.595+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85ce920 0x5643e85d38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.595+0000 7f74b05b7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85d8ff0 0x5643e864eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.595+0000 7f74b05b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f74a81c3140 con 0x5643e85ce920 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.595+0000 7f74b05b7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f74a81c45b0 con 0x5643e85ce000 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.595+0000 7f74b05b7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74a81c3810 con 0x5643e85d8ff0 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.596+0000 7f74b05b7740 10 client.0 _ll_register_callbacks cb 0x7f7498003ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.596+0000 7f74a6a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce000 0x5643e85d3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.596+0000 7f74a6a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce000 0x5643e85d3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36602/0 (socket says 172.21.15.102:36602) 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.596+0000 7f74a6a00640 1 -- 172.21.15.102:0/3432508835 learned_addr learned my addr 172.21.15.102:0/3432508835 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.596+0000 7f74a6a00640 1 -- 172.21.15.102:0/3432508835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85d8ff0 msgr2=0x5643e864eac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.596+0000 7f74a6a00640 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85d8ff0 0x5643e864eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.596+0000 7f74a6a00640 1 -- 172.21.15.102:0/3432508835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85ce920 msgr2=0x5643e85d38a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:06.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.596+0000 7f74a6a00640 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85ce920 0x5643e85d38a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.596+0000 7f74a6a00640 1 -- 172.21.15.102:0/3432508835 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643e864f000 con 0x5643e85ce000 2024-04-25T01:57:06.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.597+0000 7f74a6a00640 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce000 0x5643e85d3360 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7498005d00 tx=0x7f7498011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebe10650554bcf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:06.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.597+0000 7f7490c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.597+0000 7f7490c00640 1 -- 172.21.15.102:0/3432508835 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f749803e030 con 0x5643e85ce000 2024-04-25T01:57:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.597+0000 7f74b05b7740 1 -- 172.21.15.102:0/3432508835 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74980275b0 con 0x5643e85ce000 2024-04-25T01:57:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.597+0000 7f7490c00640 1 -- 172.21.15.102:0/3432508835 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7498036040 con 0x5643e85ce000 2024-04-25T01:57:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.597+0000 7f7490c00640 1 -- 172.21.15.102:0/3432508835 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f74980273e0 con 0x5643e85ce000 2024-04-25T01:57:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.597+0000 7f74b05b7740 10 client.25853 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.597+0000 7f74b05b7740 1 -- 172.21.15.102:0/3432508835 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643e8650310 con 0x5643e85ce000 2024-04-25T01:57:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.598+0000 7f7490c00640 1 -- 172.21.15.102:0/3432508835 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74980275b0 con 0x5643e85ce000 2024-04-25T01:57:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.598+0000 7f7490c00640 1 -- 172.21.15.102:0/3432508835 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7498004870 con 0x5643e85ce000 2024-04-25T01:57:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.599+0000 7f74b05b7740 10 client.25853 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.599+0000 7f74b05b7740 10 client.25853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.599+0000 7f74b05b7740 20 client.25853 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.599+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5643e8650940 0x5643e8652d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.599+0000 7f74b05b7740 4 client.25853 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:57:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.599+0000 7f74b05b7740 1 -- 172.21.15.102:0/3432508835 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5643e853faa0 con 0x5643e8650940 2024-04-25T01:57:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.599+0000 7f74a6000640 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5643e8650940 0x5643e8652d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.600+0000 7f74a6000640 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5643e8650940 0x5643e8652d60 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1d5c2500a3c82da server_cookie=9a14b60d23276ec in_seq=0 out_seq=0 2024-04-25T01:57:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.600+0000 7f7490c00640 10 client.25853 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.601+0000 7f7490c00640 1 -- 172.21.15.102:0/3432508835 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5643e853faa0 con 0x5643e8650940 2024-04-25T01:57:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.601+0000 7f7490c00640 10 client.25853 handle_command_reply: tid=0 2024-04-25T01:57:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.601+0000 7f74b05b7740 1 client.25853 shutdown 2024-04-25T01:57:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.601+0000 7f74b05b7740 1 -- 172.21.15.102:0/3432508835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce000 msgr2=0x5643e85d3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.601+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce000 0x5643e85d3360 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7498005d00 tx=0x7f7498011930 comp rx=0 tx=0).stop 2024-04-25T01:57:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.602+0000 7f74b05b7740 1 -- 172.21.15.102:0/3432508835 shutdown_connections 2024-04-25T01:57:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.602+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5643e8650940 0x5643e8652d60 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.602+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5643e85d8ff0 0x5643e864eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.602+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5643e85ce920 0x5643e85d38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.602+0000 7f74b05b7740 1 --2- 172.21.15.102:0/3432508835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5643e85ce000 0x5643e85d3360 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.602+0000 7f74b05b7740 1 -- 172.21.15.102:0/3432508835 >> 172.21.15.102:0/3432508835 conn(0x5643e85ca320 msgr2=0x5643e85d5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.602+0000 7f7490c00640 0 client.25853 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:06.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.603+0000 7f74b05b7740 1 -- 172.21.15.102:0/3432508835 shutdown_connections 2024-04-25T01:57:06.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.603+0000 7f74b05b7740 1 -- 172.21.15.102:0/3432508835 wait complete. 2024-04-25T01:57:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.603+0000 7f74b05b7740 20 client.25853 trim_cache size 0 max 16384 2024-04-25T01:57:06.607 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:57:06.607 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:57:06.607 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T01:57:06.608 INFO:teuthology.orchestra.run.smithi102.stdout: "d77a5dde-12ff-41d0-a109-d03404ea4f91": { 2024-04-25T01:57:06.608 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T01:57:06.608 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "d77a5dde-12ff-41d0-a109-d03404ea4f91", 2024-04-25T01:57:06.608 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T01:57:06.608 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:57:06.608 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:57:06.608 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:57:06.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 -- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f748c0a2e50 msgr2=0x7f748c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f748c0a2e50 0x7f748c0a5310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5643e85cf5c0 tx=0x7f749800b390 comp rx=0 tx=0).stop 2024-04-25T01:57:06.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 -- 172.21.15.102:0/147071160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a811ae00 msgr2=0x7f74a81139e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:06.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a811ae00 0x7f74a81139e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f74a000b970 tx=0x7f74a000be30 comp rx=0 tx=0).stop 2024-04-25T01:57:06.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 -- 172.21.15.102:0/147071160 shutdown_connections 2024-04-25T01:57:06.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f748c0a2e50 0x7f748c0a5310 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a811ae00 0x7f74a81139e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74a806da70 0x7f74a81134a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 --2- 172.21.15.102:0/147071160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74a806d150 0x7f74a811a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:06.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.605+0000 7f74ad800640 1 -- 172.21.15.102:0/147071160 >> 172.21.15.102:0/147071160 conn(0x7f74a806c7b0 msgr2=0x7f74a811c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:06.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.606+0000 7f74ad800640 1 -- 172.21.15.102:0/147071160 shutdown_connections 2024-04-25T01:57:06.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:06.606+0000 7f74ad800640 1 -- 172.21.15.102:0/147071160 wait complete. 2024-04-25T01:57:06.637 INFO:tasks.cephfs.filesystem:scrub status for tag:d77a5dde-12ff-41d0-a109-d03404ea4f91 - {'path': '/', 'tag': 'd77a5dde-12ff-41d0-a109-d03404ea4f91', 'options': 'recursive,force'} 2024-04-25T01:57:06.925 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:06 smithi102 ceph-mon[27796]: pgmap v58: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.6 MiB/s wr, 635 op/s 2024-04-25T01:57:07.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:06 smithi110 ceph-mon[32848]: pgmap v58: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.6 MiB/s wr, 635 op/s 2024-04-25T01:57:07.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:06 smithi193 ceph-mon[32981]: pgmap v58: 97 pgs: 97 active+clean; 1.2 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.6 MiB/s wr, 635 op/s 2024-04-25T01:57:08.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 75.0 s, 2351.0 tps, lat 43.353 ms stddev 111.011 2024-04-25T01:57:08.140 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.141 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:07 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:07 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T01:57:08.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:07 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:09.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:08 smithi102 ceph-mon[27796]: pgmap v59: 97 pgs: 97 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 42 MiB/s wr, 1.65k op/s 2024-04-25T01:57:09.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:08 smithi110 ceph-mon[32848]: pgmap v59: 97 pgs: 97 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 42 MiB/s wr, 1.65k op/s 2024-04-25T01:57:09.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:08 smithi193 ceph-mon[32981]: pgmap v59: 97 pgs: 97 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 42 MiB/s wr, 1.65k op/s 2024-04-25T01:57:11.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:10 smithi102 ceph-mon[27796]: pgmap v60: 97 pgs: 97 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 39 MiB/s wr, 1.33k op/s 2024-04-25T01:57:11.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:10 smithi110 ceph-mon[32848]: pgmap v60: 97 pgs: 97 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 39 MiB/s wr, 1.33k op/s 2024-04-25T01:57:11.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:10 smithi193 ceph-mon[32981]: pgmap v60: 97 pgs: 97 active+clean; 1.5 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 39 MiB/s wr, 1.33k op/s 2024-04-25T01:57:12.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.057+0000 7f0cf1c00640 1 -- 172.21.15.102:0/2409148756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec1042f0 msgr2=0x7f0cec110800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:12.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.057+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/2409148756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec1042f0 0x7f0cec110800 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009ca0 tx=0x7f0ce402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.057+0000 7f0cf1c00640 1 -- 172.21.15.102:0/2409148756 shutdown_connections 2024-04-25T01:57:12.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.057+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/2409148756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec1042f0 0x7f0cec110800 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.057+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/2409148756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cec103930 0x7f0cec103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.057+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/2409148756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cec109940 0x7f0cec109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.057+0000 7f0cf1c00640 1 -- 172.21.15.102:0/2409148756 >> 172.21.15.102:0/2409148756 conn(0x7f0cec0ff400 msgr2=0x7f0cec101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:12.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.058+0000 7f0cf1c00640 1 -- 172.21.15.102:0/2409148756 shutdown_connections 2024-04-25T01:57:12.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.058+0000 7f0cf1c00640 1 -- 172.21.15.102:0/2409148756 wait complete. 2024-04-25T01:57:12.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.059+0000 7f0cf1c00640 1 Processor -- start 2024-04-25T01:57:12.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.060+0000 7f0cf1c00640 1 -- start start 2024-04-25T01:57:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.060+0000 7f0cf1c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cec103930 0x7f0cec19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.060+0000 7f0cf1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cec1042f0 0x7f0cec19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.060+0000 7f0cf1c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec109940 0x7f0cec19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.060+0000 7f0cf1c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec113dc0 con 0x7f0cec1042f0 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.060+0000 7f0cf1c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec113c40 con 0x7f0cec109940 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.060+0000 7f0cf1c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec113f40 con 0x7f0cec103930 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.061+0000 7f0ceb400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec109940 0x7f0cec19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.061+0000 7f0ceb400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec109940 0x7f0cec19ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36632/0 (socket says 172.21.15.102:36632) 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.061+0000 7f0ceb400640 1 -- 172.21.15.102:0/3037978503 learned_addr learned my addr 172.21.15.102:0/3037978503 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.061+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cec103930 0x7f0cec19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.061+0000 7f0cea000640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cec1042f0 0x7f0cec19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.061+0000 7f0ceb400640 1 -- 172.21.15.102:0/3037978503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cec103930 msgr2=0x7f0cec19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.061+0000 7f0ceb400640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cec103930 0x7f0cec19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.062+0000 7f0ceb400640 1 -- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cec1042f0 msgr2=0x7f0cec19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.062+0000 7f0ceb400640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cec1042f0 0x7f0cec19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.062+0000 7f0ceb400640 1 -- 172.21.15.102:0/3037978503 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cec19f1a0 con 0x7f0cec109940 2024-04-25T01:57:12.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.062+0000 7f0ceb400640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec109940 0x7f0cec19ea70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4032fa0 tx=0x7f0ce4032fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:12.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.062+0000 7f0ce3400640 1 -- 172.21.15.102:0/3037978503 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4003cb0 con 0x7f0cec109940 2024-04-25T01:57:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.062+0000 7f0ce3400640 1 -- 172.21.15.102:0/3037978503 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ce4003e50 con 0x7f0cec109940 2024-04-25T01:57:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.062+0000 7f0ce3400640 1 -- 172.21.15.102:0/3037978503 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce403a400 con 0x7f0cec109940 2024-04-25T01:57:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.062+0000 7f0cf1c00640 1 -- 172.21.15.102:0/3037978503 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cec19f430 con 0x7f0cec109940 2024-04-25T01:57:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.062+0000 7f0cf1c00640 1 -- 172.21.15.102:0/3037978503 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cec1a6d20 con 0x7f0cec109940 2024-04-25T01:57:12.069 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.067+0000 7f0ce3400640 1 -- 172.21.15.102:0/3037978503 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0ce4004040 con 0x7f0cec109940 2024-04-25T01:57:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.068+0000 7f0cf4be5740 1 Processor -- start 2024-04-25T01:57:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.069+0000 7f0cf4be5740 1 -- start start 2024-04-25T01:57:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.069+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638eff9a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.069+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.069+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effa73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.069+0000 7f0cf4be5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5638efefdab0 con 0x5638eff9a720 2024-04-25T01:57:12.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.069+0000 7f0cf4be5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5638efea6640 con 0x5638eff99e00 2024-04-25T01:57:12.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.069+0000 7f0cf4be5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5638efef9a70 con 0x5638effa4fe0 2024-04-25T01:57:12.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.069+0000 7f0ce3400640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0cd00a2e20 0x7f0cd00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ceaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ceaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37308/0 (socket says 172.21.15.102:37308) 2024-04-25T01:57:12.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ceaa00640 1 -- 172.21.15.102:0/1069395711 learned_addr learned my addr 172.21.15.102:0/1069395711 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:12.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ce3400640 1 -- 172.21.15.102:0/3037978503 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0ce40f0450 con 0x7f0cec109940 2024-04-25T01:57:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ceb400640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0cd00a2e20 0x7f0cd00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0cea000640 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638eff9a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0cea000640 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effa73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ceaa00640 1 -- 172.21.15.102:0/1069395711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 msgr2=0x5638effa73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effa73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ceaa00640 1 -- 172.21.15.102:0/1069395711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 msgr2=0x5638eff9a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638eff9a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.070+0000 7f0ceaa00640 1 -- 172.21.15.102:0/1069395711 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638efc12210 con 0x5638eff9a720 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.071+0000 7f0ceb400640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0cd00a2e20 0x7f0cd00a52e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40bc840 tx=0x7f0ce402a8e0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.071+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa4aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009a40 tx=0x7f0cdc009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ff39217e050147 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.071+0000 7f0cea000640 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638eff9a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.071+0000 7f0cd7e00640 1 -- 172.21.15.102:0/1069395711 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc024070 con 0x5638eff9a720 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.071+0000 7f0cd7e00640 1 -- 172.21.15.102:0/1069395711 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cdc017d20 con 0x5638eff9a720 2024-04-25T01:57:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.071+0000 7f0cd7e00640 1 -- 172.21.15.102:0/1069395711 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc01fc50 con 0x5638eff9a720 2024-04-25T01:57:12.075 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.072+0000 7f0cf4be5740 1 -- 172.21.15.102:0/1069395711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 msgr2=0x5638effa4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:12.075 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.072+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa4aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009a40 tx=0x7f0cdc009f00 comp rx=0 tx=0).stop 2024-04-25T01:57:12.075 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.072+0000 7f0cf4be5740 1 -- 172.21.15.102:0/1069395711 shutdown_connections 2024-04-25T01:57:12.075 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.072+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effa73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.075 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.072+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa4aa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.076 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.072+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/1069395711 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638eff9a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.076 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.072+0000 7f0cf4be5740 1 -- 172.21.15.102:0/1069395711 >> 172.21.15.102:0/1069395711 conn(0x5638eff96310 msgr2=0x5638eff966f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:12.076 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.073+0000 7f0cf4be5740 1 -- 172.21.15.102:0/1069395711 shutdown_connections 2024-04-25T01:57:12.076 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.073+0000 7f0cf4be5740 1 -- 172.21.15.102:0/1069395711 wait complete. 2024-04-25T01:57:12.076 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.073+0000 7f0cf4be5740 1 -- start start 2024-04-25T01:57:12.076 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.074+0000 7f0cf4be5740 1 Processor -- start 2024-04-25T01:57:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.075+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effa1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.075+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.075+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effa2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.075+0000 7f0cf4be5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5638effa7c10 con 0x5638eff9a720 2024-04-25T01:57:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.075+0000 7f0cf4be5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5638efeb9990 con 0x5638eff99e00 2024-04-25T01:57:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.075+0000 7f0cf4be5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5638effa7d90 con 0x5638effa4fe0 2024-04-25T01:57:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.075+0000 7f0cf4be5740 10 client.0 _ll_register_callbacks cb 0x5638efebb950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.075+0000 7f0cea000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effa2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.075+0000 7f0cea000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effa2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41690/0 (socket says 172.21.15.102:41690) 2024-04-25T01:57:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.076+0000 7f0ceaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.076+0000 7f0ceaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37318/0 (socket says 172.21.15.102:37318) 2024-04-25T01:57:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.076+0000 7f0ceaa00640 1 -- 172.21.15.102:0/446866481 learned_addr learned my addr 172.21.15.102:0/446866481 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:12.079 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.076+0000 7f0cea000640 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effa1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.079 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.076+0000 7f0ceaa00640 1 -- 172.21.15.102:0/446866481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 msgr2=0x5638effa2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.076+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effa2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.076+0000 7f0ceaa00640 1 -- 172.21.15.102:0/446866481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 msgr2=0x5638effa1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.076+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effa1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.076+0000 7f0ceaa00640 1 -- 172.21.15.102:0/446866481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638efcde620 con 0x5638eff9a720 2024-04-25T01:57:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.077+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa2500 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc000c00 tx=0x7f0cdc01f4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67235f39e9f5e6ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.077+0000 7f0cd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:57:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.077+0000 7f0cd4c00640 1 -- 172.21.15.102:0/446866481 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc02f060 con 0x5638eff9a720 2024-04-25T01:57:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.077+0000 7f0cf4be5740 1 -- 172.21.15.102:0/446866481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cdc017d20 con 0x5638eff9a720 2024-04-25T01:57:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.077+0000 7f0cd4c00640 1 -- 172.21.15.102:0/446866481 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0cdc00ea80 con 0x5638eff9a720 2024-04-25T01:57:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.078+0000 7f0cd4c00640 1 -- 172.21.15.102:0/446866481 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cdc024070 con 0x5638eff9a720 2024-04-25T01:57:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.078+0000 7f0cf4be5740 10 client.15684 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.078+0000 7f0cf4be5740 1 -- 172.21.15.102:0/446866481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5638effc4b80 con 0x5638eff9a720 2024-04-25T01:57:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.078+0000 7f0cd4c00640 1 -- 172.21.15.102:0/446866481 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cdc017d20 con 0x5638eff9a720 2024-04-25T01:57:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.079+0000 7f0cd4c00640 1 -- 172.21.15.102:0/446866481 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0cdc02aca0 con 0x5638eff9a720 2024-04-25T01:57:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.079+0000 7f0cf4be5740 10 client.15684 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.079+0000 7f0cf4be5740 10 client.15684 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.079+0000 7f0cf4be5740 20 client.15684 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.079+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638effc5f90 0x5638effc8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.079+0000 7f0cf4be5740 4 client.15684 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.079+0000 7f0cf4be5740 1 -- 172.21.15.102:0/446866481 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638eff0aff0 con 0x5638effc5f90 2024-04-25T01:57:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.079+0000 7f0cea000640 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638effc5f90 0x5638effc8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.080+0000 7f0cea000640 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638effc5f90 0x5638effc8380 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ec728a5ef5f6544 server_cookie=d1fbdb56140dd0bd in_seq=0 out_seq=0 2024-04-25T01:57:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.080+0000 7f0cd4c00640 10 client.15684 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.080+0000 7f0cd4c00640 1 -- 172.21.15.102:0/446866481 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638eff0aff0 con 0x5638effc5f90 2024-04-25T01:57:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cd4c00640 10 client.15684 handle_command_reply: tid=0 2024-04-25T01:57:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cf4be5740 1 client.15684 shutdown 2024-04-25T01:57:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cf4be5740 1 -- 172.21.15.102:0/446866481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 msgr2=0x5638effa2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa2500 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc000c00 tx=0x7f0cdc01f4e0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cf4be5740 1 -- 172.21.15.102:0/446866481 shutdown_connections 2024-04-25T01:57:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638effc5f90 0x5638effc8380 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effa2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effa2500 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/446866481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effa1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cf4be5740 1 -- 172.21.15.102:0/446866481 >> 172.21.15.102:0/446866481 conn(0x5638eff96310 msgr2=0x5638eff9c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.081+0000 7f0cd4c00640 0 client.15684 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.082+0000 7f0cf4be5740 1 -- 172.21.15.102:0/446866481 shutdown_connections 2024-04-25T01:57:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.082+0000 7f0cf4be5740 1 -- 172.21.15.102:0/446866481 wait complete. 2024-04-25T01:57:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.082+0000 7f0cf4be5740 20 client.15684 trim_cache size 0 max 16384 2024-04-25T01:57:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.109+0000 7f0cf4be5740 1 Processor -- start 2024-04-25T01:57:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.110+0000 7f0cf4be5740 1 -- start start 2024-04-25T01:57:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.110+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effb0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.110+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effcd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.110+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effcde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.110+0000 7f0cf4be5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec071dd0 con 0x5638eff9a720 2024-04-25T01:57:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.110+0000 7f0cf4be5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec06f640 con 0x5638eff99e00 2024-04-25T01:57:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.110+0000 7f0cf4be5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec06d4d0 con 0x5638effa4fe0 2024-04-25T01:57:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0cea000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effb0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0cea000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effb0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36654/0 (socket says 172.21.15.102:36654) 2024-04-25T01:57:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0cea000640 1 -- 172.21.15.102:0/3336583558 learned_addr learned my addr 172.21.15.102:0/3336583558 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/3336583558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effcd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0cea000640 1 --2- 172.21.15.102:0/3336583558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effcde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0cea000640 1 -- 172.21.15.102:0/3336583558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 msgr2=0x5638effcde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0cea000640 1 --2- 172.21.15.102:0/3336583558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effcde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0cea000640 1 -- 172.21.15.102:0/3336583558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 msgr2=0x5638effcd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0cea000640 1 --2- 172.21.15.102:0/3336583558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effcd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.111+0000 7f0cea000640 1 -- 172.21.15.102:0/3336583558 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638eff96c90 con 0x5638eff99e00 2024-04-25T01:57:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.112+0000 7f0cea000640 1 --2- 172.21.15.102:0/3336583558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effb0ad0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0cd800c010 tx=0x7f0cd8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a65e0aa1b6a26f06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.112+0000 7f0cd7400640 1 -- 172.21.15.102:0/3336583558 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd801e730 con 0x5638eff99e00 2024-04-25T01:57:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.112+0000 7f0cd7400640 1 -- 172.21.15.102:0/3336583558 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd801e8d0 con 0x5638eff99e00 2024-04-25T01:57:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.112+0000 7f0cd7400640 1 -- 172.21.15.102:0/3336583558 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8004a90 con 0x5638eff99e00 2024-04-25T01:57:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.113+0000 7f0cf4be5740 1 -- 172.21.15.102:0/3336583558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 msgr2=0x5638effb0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.113+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/3336583558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effb0ad0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0cd800c010 tx=0x7f0cd8004730 comp rx=0 tx=0).stop 2024-04-25T01:57:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.113+0000 7f0cf4be5740 1 -- 172.21.15.102:0/3336583558 shutdown_connections 2024-04-25T01:57:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.113+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/3336583558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638effa4fe0 0x5638effcde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.113+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/3336583558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638eff9a720 0x5638effcd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.113+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/3336583558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effb0ad0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.113+0000 7f0cf4be5740 1 -- 172.21.15.102:0/3336583558 >> 172.21.15.102:0/3336583558 conn(0x5638eff96310 msgr2=0x5638effa2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.114+0000 7f0cf4be5740 1 -- 172.21.15.102:0/3336583558 shutdown_connections 2024-04-25T01:57:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.114+0000 7f0cf4be5740 1 -- 172.21.15.102:0/3336583558 wait complete. 2024-04-25T01:57:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.114+0000 7f0cf4be5740 1 -- start start 2024-04-25T01:57:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.115+0000 7f0cf4be5740 1 Processor -- start 2024-04-25T01:57:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.115+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effad4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.116+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638eff9a720 0x5638eff9ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.116+0000 7f0cf4be5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638effa4fe0 0x5638eff9f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.116+0000 7f0cea000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effad4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.116+0000 7f0cea000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effad4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36658/0 (socket says 172.21.15.102:36658) 2024-04-25T01:57:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0cea000640 1 -- 172.21.15.102:0/4278269715 learned_addr learned my addr 172.21.15.102:0/4278269715 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec06db20 con 0x5638effa4fe0 2024-04-25T01:57:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec06ef70 con 0x5638eff99e00 2024-04-25T01:57:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0cec06e1d0 con 0x5638eff9a720 2024-04-25T01:57:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0cf4be5740 10 client.0 _ll_register_callbacks cb 0x7f0cdc022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.116+0000 7f0ceaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638eff9a720 0x5638eff9ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0ceaa00640 1 -- 172.21.15.102:0/4278269715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 msgr2=0x5638effad4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effad4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0ceaa00640 1 -- 172.21.15.102:0/4278269715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638effa4fe0 msgr2=0x5638eff9f1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638effa4fe0 0x5638eff9f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.117+0000 7f0ceaa00640 1 -- 172.21.15.102:0/4278269715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638f002b1c0 con 0x5638eff9a720 2024-04-25T01:57:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.118+0000 7f0ceaa00640 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638eff9a720 0x5638eff9ea90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009300 tx=0x7f0cdc009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc3b3f88baa646df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.118+0000 7f0cd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.118+0000 7f0cd4c00640 1 -- 172.21.15.102:0/4278269715 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc02f040 con 0x5638eff9a720 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.118+0000 7f0cd4c00640 1 -- 172.21.15.102:0/4278269715 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0cdc01fce0 con 0x5638eff9a720 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.118+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cd801e8d0 con 0x5638eff9a720 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.119+0000 7f0cd4c00640 1 -- 172.21.15.102:0/4278269715 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cdc017d20 con 0x5638eff9a720 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.119+0000 7f0cf4be5740 10 client.25855 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.119+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5638f002c2b0 con 0x5638eff9a720 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.119+0000 7f0cd4c00640 1 -- 172.21.15.102:0/4278269715 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd801e8d0 con 0x5638eff9a720 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.119+0000 7f0cd4c00640 1 -- 172.21.15.102:0/4278269715 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0cdc03c030 con 0x5638eff9a720 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.119+0000 7f0cf4be5740 10 client.25855 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.119+0000 7f0cf4be5740 10 client.25855 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.119+0000 7f0cf4be5740 20 client.25855 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.119+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638f002c9e0 0x5638f002ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.120+0000 7f0cf4be5740 4 client.25855 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:57:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.120+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5638eff05340 con 0x5638f002c9e0 2024-04-25T01:57:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.120+0000 7f0cea000640 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638f002c9e0 0x5638f002ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.121+0000 7f0cd4c00640 10 client.25855 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.121+0000 7f0cea000640 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638f002c9e0 0x5638f002ee00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ee1fb6110f5f004 server_cookie=c2954a44242b9bbd in_seq=0 out_seq=0 2024-04-25T01:57:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.133+0000 7f0cd4c00640 1 -- 172.21.15.102:0/4278269715 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5638eff05340 con 0x5638f002c9e0 2024-04-25T01:57:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.133+0000 7f0cd4c00640 10 client.25855 handle_command_reply: tid=0 2024-04-25T01:57:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.133+0000 7f0cf4be5740 1 client.25855 shutdown 2024-04-25T01:57:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.134+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638eff9a720 msgr2=0x5638eff9ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.134+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638eff9a720 0x5638eff9ea90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009300 tx=0x7f0cdc009490 comp rx=0 tx=0).stop 2024-04-25T01:57:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 shutdown_connections 2024-04-25T01:57:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638f002c9e0 0x5638f002ee00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638effa4fe0 0x5638eff9f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638eff9a720 0x5638eff9ea90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cf4be5740 1 --2- 172.21.15.102:0/4278269715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638eff99e00 0x5638effad4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cd4c00640 0 client.25855 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 >> 172.21.15.102:0/4278269715 conn(0x5638eff96310 msgr2=0x5638effa1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 shutdown_connections 2024-04-25T01:57:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cf4be5740 1 -- 172.21.15.102:0/4278269715 wait complete. 2024-04-25T01:57:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.135+0000 7f0cf4be5740 20 client.25855 trim_cache size 0 max 16384 2024-04-25T01:57:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:57:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:57:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:57:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:57:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 -- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0cd00a2e20 msgr2=0x7f0cd00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0cd00a2e20 0x7f0cd00a52e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40bc840 tx=0x7f0ce402a8e0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 -- 172.21.15.102:0/3037978503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec109940 msgr2=0x7f0cec19ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec109940 0x7f0cec19ea70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4032fa0 tx=0x7f0ce4032fd0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 -- 172.21.15.102:0/3037978503 shutdown_connections 2024-04-25T01:57:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0cd00a2e20 0x7f0cd00a52e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0cec109940 0x7f0cec19ea70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cec1042f0 0x7f0cec19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 --2- 172.21.15.102:0/3037978503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cec103930 0x7f0cec19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 -- 172.21.15.102:0/3037978503 >> 172.21.15.102:0/3037978503 conn(0x7f0cec0ff400 msgr2=0x7f0cec0ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.136+0000 7f0cf1c00640 1 -- 172.21.15.102:0/3037978503 shutdown_connections 2024-04-25T01:57:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:12.137+0000 7f0cf1c00640 1 -- 172.21.15.102:0/3037978503 wait complete. 2024-04-25T01:57:13.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 80.0 s, 3646.6 tps, lat 27.315 ms stddev 31.830 2024-04-25T01:57:13.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:12 smithi102 ceph-mon[27796]: pgmap v61: 97 pgs: 97 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 44 MiB/s wr, 1.69k op/s 2024-04-25T01:57:13.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:12 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:57:13.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:12 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:57:13.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:12 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:57:13.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:12 smithi110 ceph-mon[32848]: pgmap v61: 97 pgs: 97 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 44 MiB/s wr, 1.69k op/s 2024-04-25T01:57:13.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:12 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:57:13.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:12 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:57:13.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:12 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:57:13.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:12 smithi193 ceph-mon[32981]: pgmap v61: 97 pgs: 97 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 44 MiB/s wr, 1.69k op/s 2024-04-25T01:57:13.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:12 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:57:13.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:12 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:57:13.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:12 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:57:15.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:14 smithi102 ceph-mon[27796]: pgmap v62: 97 pgs: 97 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 39 MiB/s wr, 1.37k op/s 2024-04-25T01:57:15.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:14 smithi110 ceph-mon[32848]: pgmap v62: 97 pgs: 97 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 39 MiB/s wr, 1.37k op/s 2024-04-25T01:57:15.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:14 smithi193 ceph-mon[32981]: pgmap v62: 97 pgs: 97 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 39 MiB/s wr, 1.37k op/s 2024-04-25T01:57:17.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:16 smithi102 ceph-mon[27796]: pgmap v63: 97 pgs: 97 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 39 MiB/s wr, 1.37k op/s 2024-04-25T01:57:17.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:16 smithi110 ceph-mon[32848]: pgmap v63: 97 pgs: 97 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 39 MiB/s wr, 1.37k op/s 2024-04-25T01:57:17.256 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:16 smithi193 ceph-mon[32981]: pgmap v63: 97 pgs: 97 active+clean; 1.5 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 39 MiB/s wr, 1.37k op/s 2024-04-25T01:57:18.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 85.0 s, 3748.8 tps, lat 26.694 ms stddev 31.063 2024-04-25T01:57:19.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:18 smithi102 ceph-mon[27796]: pgmap v64: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 43 MiB/s wr, 1.78k op/s 2024-04-25T01:57:19.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:18 smithi110 ceph-mon[32848]: pgmap v64: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 43 MiB/s wr, 1.78k op/s 2024-04-25T01:57:19.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:18 smithi193 ceph-mon[32981]: pgmap v64: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 43 MiB/s wr, 1.78k op/s 2024-04-25T01:57:21.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:20 smithi102 ceph-mon[27796]: pgmap v65: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 9.1 MiB/s wr, 757 op/s 2024-04-25T01:57:21.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:20 smithi110 ceph-mon[32848]: pgmap v65: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 9.1 MiB/s wr, 757 op/s 2024-04-25T01:57:21.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:20 smithi193 ceph-mon[32981]: pgmap v65: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 9.1 MiB/s wr, 757 op/s 2024-04-25T01:57:23.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 90.0 s, 3626.6 tps, lat 27.503 ms stddev 33.458 2024-04-25T01:57:23.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:22 smithi102 ceph-mon[27796]: pgmap v66: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 1.09k op/s 2024-04-25T01:57:23.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:22 smithi110 ceph-mon[32848]: pgmap v66: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 1.09k op/s 2024-04-25T01:57:23.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:22 smithi193 ceph-mon[32981]: pgmap v66: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 1.09k op/s 2024-04-25T01:57:25.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:24 smithi102 ceph-mon[27796]: pgmap v67: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.9 MiB/s wr, 740 op/s 2024-04-25T01:57:25.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:24 smithi110 ceph-mon[32848]: pgmap v67: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.9 MiB/s wr, 740 op/s 2024-04-25T01:57:25.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:24 smithi193 ceph-mon[32981]: pgmap v67: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.9 MiB/s wr, 740 op/s 2024-04-25T01:57:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.320+0000 7f6434600640 1 -- 172.21.15.102:0/2817554398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f642c075c70 msgr2=0x7f642c076050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.320+0000 7f6434600640 1 --2- 172.21.15.102:0/2817554398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f642c075c70 0x7f642c076050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6420009930 tx=0x7f642002a440 comp rx=0 tx=0).stop 2024-04-25T01:57:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.320+0000 7f6434600640 1 -- 172.21.15.102:0/2817554398 shutdown_connections 2024-04-25T01:57:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.320+0000 7f6434600640 1 --2- 172.21.15.102:0/2817554398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c1125d0 0x7f642c1149c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.320+0000 7f6434600640 1 --2- 172.21.15.102:0/2817554398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f642c076590 0x7f642c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.320+0000 7f6434600640 1 --2- 172.21.15.102:0/2817554398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f642c075c70 0x7f642c076050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.320+0000 7f6434600640 1 -- 172.21.15.102:0/2817554398 >> 172.21.15.102:0/2817554398 conn(0x7f642c0ff440 msgr2=0x7f642c101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.321+0000 7f6434600640 1 -- 172.21.15.102:0/2817554398 shutdown_connections 2024-04-25T01:57:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.321+0000 7f6434600640 1 -- 172.21.15.102:0/2817554398 wait complete. 2024-04-25T01:57:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.322+0000 7f6434600640 1 Processor -- start 2024-04-25T01:57:25.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.322+0000 7f6434600640 1 -- start start 2024-04-25T01:57:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.323+0000 7f6434600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c075c70 0x7f642c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.325 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.323+0000 7f6434600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f642c076590 0x7f642c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.324+0000 7f6434600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f642c1125d0 0x7f642c1a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.324+0000 7f6434600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f642c1181e0 con 0x7f642c075c70 2024-04-25T01:57:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.324+0000 7f6434600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f642c118060 con 0x7f642c076590 2024-04-25T01:57:25.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.324+0000 7f6434600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f642c118360 con 0x7f642c1125d0 2024-04-25T01:57:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.324+0000 7f6431200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f642c076590 0x7f642c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.324+0000 7f6431200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f642c076590 0x7f642c19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:55388/0 (socket says 172.21.15.102:55388) 2024-04-25T01:57:25.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.324+0000 7f6431200640 1 -- 172.21.15.102:0/2414519749 learned_addr learned my addr 172.21.15.102:0/2414519749 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.324+0000 7f6432600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f642c1125d0 0x7f642c1a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.325+0000 7f6431c00640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c075c70 0x7f642c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.325+0000 7f6431200640 1 -- 172.21.15.102:0/2414519749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f642c1125d0 msgr2=0x7f642c1a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.325+0000 7f6431200640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f642c1125d0 0x7f642c1a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.325+0000 7f6431200640 1 -- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c075c70 msgr2=0x7f642c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.325+0000 7f6431200640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c075c70 0x7f642c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.325+0000 7f6431200640 1 -- 172.21.15.102:0/2414519749 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f642c1a3630 con 0x7f642c076590 2024-04-25T01:57:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.325+0000 7f6432600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f642c1125d0 0x7f642c1a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:25.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.325+0000 7f6431200640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f642c076590 0x7f642c19eb80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f641800d950 tx=0x7f641800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.326+0000 7f642a000640 1 -- 172.21.15.102:0/2414519749 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418017030 con 0x7f642c076590 2024-04-25T01:57:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.326+0000 7f642a000640 1 -- 172.21.15.102:0/2414519749 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6418004020 con 0x7f642c076590 2024-04-25T01:57:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.326+0000 7f642a000640 1 -- 172.21.15.102:0/2414519749 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418003d60 con 0x7f642c076590 2024-04-25T01:57:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.326+0000 7f6434600640 1 -- 172.21.15.102:0/2414519749 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f642c1a3920 con 0x7f642c076590 2024-04-25T01:57:25.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.326+0000 7f6434600640 1 -- 172.21.15.102:0/2414519749 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f642c1ab1b0 con 0x7f642c076590 2024-04-25T01:57:25.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.334+0000 7f642a000640 1 -- 172.21.15.102:0/2414519749 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f641801a070 con 0x7f642c076590 2024-04-25T01:57:25.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.334+0000 7f6437594740 1 Processor -- start 2024-04-25T01:57:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.335+0000 7f6437594740 1 -- start start 2024-04-25T01:57:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.335+0000 7f6437594740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f251e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.335+0000 7f6437594740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 0x556884f2faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.335+0000 7f6437594740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884f323d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.335+0000 7f6437594740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x556884e88ab0 con 0x556884f25720 2024-04-25T01:57:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.335+0000 7f6437594740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556884e31640 con 0x556884f2ffe0 2024-04-25T01:57:25.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.335+0000 7f6437594740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556884e84a70 con 0x556884f24e00 2024-04-25T01:57:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.336+0000 7f6432600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.336+0000 7f6432600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f251e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55976/0 (socket says 172.21.15.102:55976) 2024-04-25T01:57:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.336+0000 7f6432600640 1 -- 172.21.15.102:0/471185528 learned_addr learned my addr 172.21.15.102:0/471185528 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.336+0000 7f6431c00640 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 0x556884f2faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.336+0000 7f6431200640 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884f323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.337+0000 7f6431200640 1 -- 172.21.15.102:0/471185528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 msgr2=0x556884f251e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.337+0000 7f6431200640 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f251e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.337+0000 7f6431200640 1 -- 172.21.15.102:0/471185528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 msgr2=0x556884f2faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.337+0000 7f6431200640 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 0x556884f2faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.337+0000 7f6431200640 1 -- 172.21.15.102:0/471185528 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556884b9d210 con 0x556884f2ffe0 2024-04-25T01:57:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.337+0000 7f642a000640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f64140a2f80 0x7f64140a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.338+0000 7f642a000640 1 -- 172.21.15.102:0/2414519749 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f64180cea40 con 0x7f642c076590 2024-04-25T01:57:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.338+0000 7f6431c00640 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 0x556884f2faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.338+0000 7f6432600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f64140a2f80 0x7f64140a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.338+0000 7f6431200640 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884f323d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f64180cc160 tx=0x7f64180d6610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=349414c6f9708fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:25.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.339+0000 7f641ea00640 1 -- 172.21.15.102:0/471185528 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180971d0 con 0x556884f2ffe0 2024-04-25T01:57:25.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.339+0000 7f641ea00640 1 -- 172.21.15.102:0/471185528 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f641809b630 con 0x556884f2ffe0 2024-04-25T01:57:25.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.339+0000 7f641ea00640 1 -- 172.21.15.102:0/471185528 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641809a890 con 0x556884f2ffe0 2024-04-25T01:57:25.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.340+0000 7f6437594740 1 -- 172.21.15.102:0/471185528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 msgr2=0x556884f323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:25.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.340+0000 7f6437594740 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884f323d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f64180cc160 tx=0x7f64180d6610 comp rx=0 tx=0).stop 2024-04-25T01:57:25.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.340+0000 7f6432600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f64140a2f80 0x7f64140a5440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6424007fd0 tx=0x7f64240077d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:25.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.341+0000 7f6437594740 1 -- 172.21.15.102:0/471185528 shutdown_connections 2024-04-25T01:57:25.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.341+0000 7f6437594740 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884f323d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.341+0000 7f6437594740 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 0x556884f2faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.341+0000 7f6437594740 1 --2- 172.21.15.102:0/471185528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f251e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.341+0000 7f6437594740 1 -- 172.21.15.102:0/471185528 >> 172.21.15.102:0/471185528 conn(0x556884f21310 msgr2=0x556884f216f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:25.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.341+0000 7f6437594740 1 -- 172.21.15.102:0/471185528 shutdown_connections 2024-04-25T01:57:25.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.341+0000 7f6437594740 1 -- 172.21.15.102:0/471185528 wait complete. 2024-04-25T01:57:25.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.342+0000 7f6437594740 1 -- start start 2024-04-25T01:57:25.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.342+0000 7f6437594740 1 Processor -- start 2024-04-25T01:57:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.343+0000 7f6437594740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f24e00 0x556884f2d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.343+0000 7f6437594740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f25720 0x556884f2d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.343+0000 7f6437594740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f2ffe0 0x556884f2daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.343+0000 7f6437594740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x556884f32c10 con 0x556884f2ffe0 2024-04-25T01:57:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.343+0000 7f6437594740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556884e44990 con 0x556884f24e00 2024-04-25T01:57:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.343+0000 7f6437594740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556884f32d90 con 0x556884f25720 2024-04-25T01:57:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6437594740 10 client.0 _ll_register_callbacks cb 0x556884e46950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6431200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f25720 0x556884f2d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6431200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f25720 0x556884f2d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55984/0 (socket says 172.21.15.102:55984) 2024-04-25T01:57:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6431200640 1 -- 172.21.15.102:0/4123276504 learned_addr learned my addr 172.21.15.102:0/4123276504 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6431c00640 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f24e00 0x556884f2d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6431200640 1 -- 172.21.15.102:0/4123276504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f24e00 msgr2=0x556884f2d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6431200640 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f24e00 0x556884f2d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6431200640 1 -- 172.21.15.102:0/4123276504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f2ffe0 msgr2=0x556884f2daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6431200640 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f2ffe0 0x556884f2daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.344+0000 7f6431200640 1 -- 172.21.15.102:0/4123276504 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556884c69620 con 0x556884f25720 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.345+0000 7f6431200640 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f25720 0x556884f2d560 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f64180ced70 tx=0x7f6418005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99595c150df5c5e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.345+0000 7f6413400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.345+0000 7f6413400640 1 -- 172.21.15.102:0/4123276504 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641809aa70 con 0x556884f25720 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.346+0000 7f6413400640 1 -- 172.21.15.102:0/4123276504 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f64180d46d0 con 0x556884f25720 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.346+0000 7f6437594740 1 -- 172.21.15.102:0/4123276504 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f641809b630 con 0x556884f25720 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.346+0000 7f6413400640 1 -- 172.21.15.102:0/4123276504 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f641809ad00 con 0x556884f25720 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.346+0000 7f6437594740 10 client.25864 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.346+0000 7f6437594740 1 -- 172.21.15.102:0/4123276504 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556884f32c10 con 0x556884f25720 2024-04-25T01:57:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.347+0000 7f6413400640 1 -- 172.21.15.102:0/4123276504 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f641809b630 con 0x556884f25720 2024-04-25T01:57:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.347+0000 7f6413400640 1 -- 172.21.15.102:0/4123276504 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f64180975a0 con 0x556884f25720 2024-04-25T01:57:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.347+0000 7f6437594740 10 client.25864 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.347+0000 7f6437594740 10 client.25864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.348+0000 7f6437594740 20 client.25864 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.348+0000 7f6437594740 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556884f3fd20 0x556884f42110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.348+0000 7f6437594740 4 client.25864 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.348+0000 7f6437594740 1 -- 172.21.15.102:0/4123276504 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556884e95ff0 con 0x556884f3fd20 2024-04-25T01:57:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.349+0000 7f6431c00640 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556884f3fd20 0x556884f42110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.350+0000 7f6413400640 10 client.25864 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.350+0000 7f6431c00640 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556884f3fd20 0x556884f42110 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3487fe3246640b48 server_cookie=bab33cafafe274d in_seq=0 out_seq=0 2024-04-25T01:57:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.351+0000 7f6413400640 1 -- 172.21.15.102:0/4123276504 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556884e95ff0 con 0x556884f3fd20 2024-04-25T01:57:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.351+0000 7f6413400640 10 client.25864 handle_command_reply: tid=0 2024-04-25T01:57:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.351+0000 7f6437594740 1 client.25864 shutdown 2024-04-25T01:57:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.352+0000 7f6437594740 1 -- 172.21.15.102:0/4123276504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f25720 msgr2=0x556884f2d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.352+0000 7f6437594740 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f25720 0x556884f2d560 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f64180ced70 tx=0x7f6418005db0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.353+0000 7f6437594740 1 -- 172.21.15.102:0/4123276504 shutdown_connections 2024-04-25T01:57:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.353+0000 7f6437594740 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556884f3fd20 0x556884f42110 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.353+0000 7f6437594740 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f2ffe0 0x556884f2daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.353+0000 7f6413400640 0 client.25864 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.353+0000 7f6437594740 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f25720 0x556884f2d560 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.353+0000 7f6437594740 1 --2- 172.21.15.102:0/4123276504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f24e00 0x556884f2d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.353+0000 7f6437594740 1 -- 172.21.15.102:0/4123276504 >> 172.21.15.102:0/4123276504 conn(0x556884f21310 msgr2=0x556884f27090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.354+0000 7f6437594740 1 -- 172.21.15.102:0/4123276504 shutdown_connections 2024-04-25T01:57:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.354+0000 7f6437594740 1 -- 172.21.15.102:0/4123276504 wait complete. 2024-04-25T01:57:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.354+0000 7f6437594740 20 client.25864 trim_cache size 0 max 16384 2024-04-25T01:57:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.384+0000 7f6437594740 1 Processor -- start 2024-04-25T01:57:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.385+0000 7f6437594740 1 -- start start 2024-04-25T01:57:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6437594740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f24e00 0x556884f27730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6437594740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f25720 0x556884f2d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6437594740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f2ffe0 0x556884f477a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6437594740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f642c1b6400 con 0x556884f24e00 2024-04-25T01:57:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6437594740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f642c1b3c70 con 0x556884f25720 2024-04-25T01:57:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6437594740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f642c1b1ac0 con 0x556884f2ffe0 2024-04-25T01:57:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6431200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f25720 0x556884f2d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6431200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f25720 0x556884f2d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:55414/0 (socket says 172.21.15.102:55414) 2024-04-25T01:57:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6431200640 1 -- 172.21.15.102:0/1558333757 learned_addr learned my addr 172.21.15.102:0/1558333757 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.386+0000 7f6431c00640 1 --2- 172.21.15.102:0/1558333757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f24e00 0x556884f27730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.387+0000 7f6431200640 1 -- 172.21.15.102:0/1558333757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f2ffe0 msgr2=0x556884f477a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:57:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.387+0000 7f6431200640 1 --2- 172.21.15.102:0/1558333757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f2ffe0 0x556884f477a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.387+0000 7f6431200640 1 -- 172.21.15.102:0/1558333757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f24e00 msgr2=0x556884f27730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.387+0000 7f6431200640 1 --2- 172.21.15.102:0/1558333757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f24e00 0x556884f27730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.387+0000 7f6431200640 1 -- 172.21.15.102:0/1558333757 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556884f42570 con 0x556884f25720 2024-04-25T01:57:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.387+0000 7f6431c00640 1 --2- 172.21.15.102:0/1558333757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f24e00 0x556884f27730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.387+0000 7f6431200640 1 --2- 172.21.15.102:0/1558333757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f25720 0x556884f2d6f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f64180a3670 tx=0x7f64180a3cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8feb2c82f1c44609 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.387+0000 7f641e000640 1 -- 172.21.15.102:0/1558333757 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180a8080 con 0x556884f25720 2024-04-25T01:57:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.388+0000 7f641e000640 1 -- 172.21.15.102:0/1558333757 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64180d42a0 con 0x556884f25720 2024-04-25T01:57:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.388+0000 7f641e000640 1 -- 172.21.15.102:0/1558333757 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180971d0 con 0x556884f25720 2024-04-25T01:57:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.389+0000 7f6437594740 1 -- 172.21.15.102:0/1558333757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f25720 msgr2=0x556884f2d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.389+0000 7f6437594740 1 --2- 172.21.15.102:0/1558333757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f25720 0x556884f2d6f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f64180a3670 tx=0x7f64180a3cc0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.389+0000 7f6437594740 1 -- 172.21.15.102:0/1558333757 shutdown_connections 2024-04-25T01:57:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.389+0000 7f6437594740 1 --2- 172.21.15.102:0/1558333757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f2ffe0 0x556884f477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.390+0000 7f6437594740 1 --2- 172.21.15.102:0/1558333757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f25720 0x556884f2d6f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.390+0000 7f6437594740 1 --2- 172.21.15.102:0/1558333757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f24e00 0x556884f27730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.390+0000 7f6437594740 1 -- 172.21.15.102:0/1558333757 >> 172.21.15.102:0/1558333757 conn(0x556884f21310 msgr2=0x556884f2cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.390+0000 7f6437594740 1 -- 172.21.15.102:0/1558333757 shutdown_connections 2024-04-25T01:57:25.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.390+0000 7f6437594740 1 -- 172.21.15.102:0/1558333757 wait complete. 2024-04-25T01:57:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.391+0000 7f6437594740 1 -- start start 2024-04-25T01:57:25.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.391+0000 7f6437594740 1 Processor -- start 2024-04-25T01:57:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6437594740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f2a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6437594740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 0x556884f2a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6437594740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884fa5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6437594740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f642c1b2130 con 0x556884f25720 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6437594740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f642c1b35a0 con 0x556884f2ffe0 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6437594740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f642c1b2800 con 0x556884f24e00 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6437594740 10 client.0 _ll_register_callbacks cb 0x7f6424007670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6432600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884fa5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6431c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f2a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6431c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f2a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56002/0 (socket says 172.21.15.102:56002) 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6431c00640 1 -- 172.21.15.102:0/550065827 learned_addr learned my addr 172.21.15.102:0/550065827 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6432600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884fa5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:55422/0 (socket says 172.21.15.102:55422) 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6432600640 1 -- 172.21.15.102:0/550065827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 msgr2=0x556884f2a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6432600640 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f2a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6432600640 1 -- 172.21.15.102:0/550065827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 msgr2=0x556884f2a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6431200640 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 0x556884f2a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6432600640 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 0x556884f2a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.392+0000 7f6432600640 1 -- 172.21.15.102:0/550065827 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556884fb6560 con 0x556884f2ffe0 2024-04-25T01:57:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.393+0000 7f6432600640 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884fa5010 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x556884f24cd0 tx=0x7f6424004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a26da47d8ee07894 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.393+0000 7f6413400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.394+0000 7f6413400640 1 -- 172.21.15.102:0/550065827 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6424026680 con 0x556884f2ffe0 2024-04-25T01:57:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.394+0000 7f6413400640 1 -- 172.21.15.102:0/550065827 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6424026820 con 0x556884f2ffe0 2024-04-25T01:57:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.394+0000 7f6437594740 1 -- 172.21.15.102:0/550065827 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64180d42a0 con 0x556884f2ffe0 2024-04-25T01:57:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.396+0000 7f6413400640 1 -- 172.21.15.102:0/550065827 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6424026a20 con 0x556884f2ffe0 2024-04-25T01:57:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.396+0000 7f6437594740 10 client.25886 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.396+0000 7f6437594740 1 -- 172.21.15.102:0/550065827 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556884fb7750 con 0x556884f2ffe0 2024-04-25T01:57:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.397+0000 7f6413400640 1 -- 172.21.15.102:0/550065827 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64180d42a0 con 0x556884f2ffe0 2024-04-25T01:57:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.397+0000 7f6413400640 1 -- 172.21.15.102:0/550065827 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f642401d420 con 0x556884f2ffe0 2024-04-25T01:57:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.399+0000 7f6437594740 10 client.25886 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.399+0000 7f6437594740 10 client.25886 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.399+0000 7f6437594740 20 client.25886 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.399+0000 7f6437594740 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556884f3fd20 0x556884fb9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.399+0000 7f6437594740 4 client.25886 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:57:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.399+0000 7f6437594740 1 -- 172.21.15.102:0/550065827 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556884e90340 con 0x556884f3fd20 2024-04-25T01:57:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.399+0000 7f6431c00640 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556884f3fd20 0x556884fb9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:25.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.401+0000 7f6413400640 10 client.25886 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:25.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.401+0000 7f6431c00640 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556884f3fd20 0x556884fb9ef0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b6c02fd72c25c9f server_cookie=a3546ddb6aa20a6 in_seq=0 out_seq=0 2024-04-25T01:57:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.414+0000 7f6413400640 1 -- 172.21.15.102:0/550065827 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556884e90340 con 0x556884f3fd20 2024-04-25T01:57:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.414+0000 7f6413400640 10 client.25886 handle_command_reply: tid=0 2024-04-25T01:57:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.415+0000 7f6437594740 1 client.25886 shutdown 2024-04-25T01:57:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.415+0000 7f6437594740 1 -- 172.21.15.102:0/550065827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 msgr2=0x556884fa5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.415+0000 7f6437594740 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884fa5010 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x556884f24cd0 tx=0x7f6424004a40 comp rx=0 tx=0).stop 2024-04-25T01:57:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.415+0000 7f6437594740 1 -- 172.21.15.102:0/550065827 shutdown_connections 2024-04-25T01:57:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.415+0000 7f6437594740 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556884f3fd20 0x556884fb9ef0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.415+0000 7f6413400640 0 client.25886 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.415+0000 7f6437594740 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556884f2ffe0 0x556884fa5010 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.416+0000 7f6437594740 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556884f25720 0x556884f2a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.416+0000 7f6437594740 1 --2- 172.21.15.102:0/550065827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556884f24e00 0x556884f2a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.416+0000 7f6437594740 1 -- 172.21.15.102:0/550065827 >> 172.21.15.102:0/550065827 conn(0x556884f21310 msgr2=0x556884f2d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.416+0000 7f6437594740 1 -- 172.21.15.102:0/550065827 shutdown_connections 2024-04-25T01:57:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.416+0000 7f6437594740 1 -- 172.21.15.102:0/550065827 wait complete. 2024-04-25T01:57:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.416+0000 7f6437594740 20 client.25886 trim_cache size 0 max 16384 2024-04-25T01:57:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:57:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:57:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:57:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:57:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.417+0000 7f6434600640 1 -- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f64140a2f80 msgr2=0x7f64140a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.417+0000 7f6434600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f64140a2f80 0x7f64140a5440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6424007fd0 tx=0x7f64240077d0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.417+0000 7f6434600640 1 -- 172.21.15.102:0/2414519749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f642c076590 msgr2=0x7f642c19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.417+0000 7f6434600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f642c076590 0x7f642c19eb80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f641800d950 tx=0x7f641800de10 comp rx=0 tx=0).stop 2024-04-25T01:57:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.418+0000 7f6434600640 1 -- 172.21.15.102:0/2414519749 shutdown_connections 2024-04-25T01:57:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.418+0000 7f6434600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f64140a2f80 0x7f64140a5440 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.418+0000 7f6434600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f642c1125d0 0x7f642c1a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.418+0000 7f6434600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f642c076590 0x7f642c19eb80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.418+0000 7f6434600640 1 --2- 172.21.15.102:0/2414519749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f642c075c70 0x7f642c19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.418+0000 7f6434600640 1 -- 172.21.15.102:0/2414519749 >> 172.21.15.102:0/2414519749 conn(0x7f642c0ff440 msgr2=0x7f642c113c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.418+0000 7f6434600640 1 -- 172.21.15.102:0/2414519749 shutdown_connections 2024-04-25T01:57:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:25.418+0000 7f6434600640 1 -- 172.21.15.102:0/2414519749 wait complete. 2024-04-25T01:57:27.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:26 smithi102 ceph-mon[27796]: pgmap v68: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.9 MiB/s wr, 740 op/s 2024-04-25T01:57:27.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:26 smithi110 ceph-mon[32848]: pgmap v68: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.9 MiB/s wr, 740 op/s 2024-04-25T01:57:27.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:26 smithi193 ceph-mon[32981]: pgmap v68: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.9 MiB/s wr, 740 op/s 2024-04-25T01:57:28.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 95.0 s, 2972.6 tps, lat 33.666 ms stddev 38.910 2024-04-25T01:57:28.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:27 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:57:28.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:27 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:57:28.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:27 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:57:29.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:28 smithi102 ceph-mon[27796]: pgmap v69: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-25T01:57:29.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:28 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:57:29.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:28 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:57:29.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:28 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:57:29.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:28 smithi110 ceph-mon[32848]: pgmap v69: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-25T01:57:29.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:28 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:57:29.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:28 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:57:29.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:28 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:57:29.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:28 smithi193 ceph-mon[32981]: pgmap v69: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-25T01:57:29.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:28 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:57:29.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:28 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:57:29.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:28 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:57:31.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:30 smithi102 ceph-mon[27796]: pgmap v70: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.1 MiB/s wr, 694 op/s 2024-04-25T01:57:31.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:30 smithi110 ceph-mon[32848]: pgmap v70: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.1 MiB/s wr, 694 op/s 2024-04-25T01:57:31.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:30 smithi193 ceph-mon[32981]: pgmap v70: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.1 MiB/s wr, 694 op/s 2024-04-25T01:57:33.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 100.0 s, 3516.5 tps, lat 28.413 ms stddev 33.486 2024-04-25T01:57:33.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:32 smithi102 ceph-mon[27796]: pgmap v71: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-25T01:57:33.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:32 smithi110 ceph-mon[32848]: pgmap v71: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-25T01:57:33.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:32 smithi193 ceph-mon[32981]: pgmap v71: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-25T01:57:35.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:34 smithi102 ceph-mon[27796]: pgmap v72: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.2 MiB/s wr, 691 op/s 2024-04-25T01:57:35.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:34 smithi110 ceph-mon[32848]: pgmap v72: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.2 MiB/s wr, 691 op/s 2024-04-25T01:57:35.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:34 smithi193 ceph-mon[32981]: pgmap v72: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.2 MiB/s wr, 691 op/s 2024-04-25T01:57:36.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:57:36.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:57:36.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.928+0000 7f420c400640 1 -- 172.21.15.102:0/3176735768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4204106630 msgr2=0x7f4204106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:36.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.928+0000 7f420c400640 1 --2- 172.21.15.102:0/3176735768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4204106630 0x7f4204106a10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009900 tx=0x7f41f802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.929+0000 7f420c400640 1 -- 172.21.15.102:0/3176735768 shutdown_connections 2024-04-25T01:57:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.929+0000 7f420c400640 1 --2- 172.21.15.102:0/3176735768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4204100fe0 0x7f420410d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.929+0000 7f420c400640 1 --2- 172.21.15.102:0/3176735768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4204100620 0x7f4204100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.929+0000 7f420c400640 1 --2- 172.21.15.102:0/3176735768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4204106630 0x7f4204106a10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.929+0000 7f420c400640 1 -- 172.21.15.102:0/3176735768 >> 172.21.15.102:0/3176735768 conn(0x7f42040776a0 msgr2=0x7f4204077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.930+0000 7f420c400640 1 -- 172.21.15.102:0/3176735768 shutdown_connections 2024-04-25T01:57:36.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.930+0000 7f420c400640 1 -- 172.21.15.102:0/3176735768 wait complete. 2024-04-25T01:57:36.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.931+0000 7f420c400640 1 Processor -- start 2024-04-25T01:57:36.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.932+0000 7f420c400640 1 -- start start 2024-04-25T01:57:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.932+0000 7f420c400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4204100620 0x7f420410f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.932+0000 7f420c400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4204100fe0 0x7f42041a8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.932+0000 7f420c400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4204106630 0x7f42041a85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.932+0000 7f420c400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4204111bb0 con 0x7f4204100620 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.932+0000 7f420c400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4204111a30 con 0x7f4204106630 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.932+0000 7f420c400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4204111d30 con 0x7f4204100fe0 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.933+0000 7f4209000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4204100fe0 0x7f42041a8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.933+0000 7f4209000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4204100fe0 0x7f42041a8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44328/0 (socket says 172.21.15.102:44328) 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.933+0000 7f4209000640 1 -- 172.21.15.102:0/3094228581 learned_addr learned my addr 172.21.15.102:0/3094228581 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.933+0000 7f4209000640 1 -- 172.21.15.102:0/3094228581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4204106630 msgr2=0x7f42041a85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.933+0000 7f420a400640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4204106630 0x7f42041a85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.933+0000 7f4209000640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4204106630 0x7f42041a85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.933+0000 7f4209000640 1 -- 172.21.15.102:0/3094228581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4204100620 msgr2=0x7f420410f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.933+0000 7f4209000640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4204100620 0x7f420410f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.933+0000 7f4209000640 1 -- 172.21.15.102:0/3094228581 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4204073140 con 0x7f4204100fe0 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.934+0000 7f4209000640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4204100fe0 0x7f42041a8090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f41f400e9f0 tx=0x7f41f400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.934+0000 7f420a400640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4204106630 0x7f42041a85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.934+0000 7f4202000640 1 -- 172.21.15.102:0/3094228581 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f400cd80 con 0x7f4204100fe0 2024-04-25T01:57:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.934+0000 7f4202000640 1 -- 172.21.15.102:0/3094228581 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f4005500 con 0x7f4204100fe0 2024-04-25T01:57:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.934+0000 7f4202000640 1 -- 172.21.15.102:0/3094228581 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4010690 con 0x7f4204100fe0 2024-04-25T01:57:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.934+0000 7f420c400640 1 -- 172.21.15.102:0/3094228581 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42041a8c30 con 0x7f4204100fe0 2024-04-25T01:57:36.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.934+0000 7f420c400640 1 -- 172.21.15.102:0/3094228581 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42041a9130 con 0x7f4204100fe0 2024-04-25T01:57:36.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.942+0000 7f4202000640 1 -- 172.21.15.102:0/3094228581 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f41f4004f50 con 0x7f4204100fe0 2024-04-25T01:57:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.944+0000 7f4202000640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f41ec0a2d80 0x7f41ec0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.944+0000 7f420f1d6740 1 Processor -- start 2024-04-25T01:57:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.944+0000 7f4202000640 1 -- 172.21.15.102:0/3094228581 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f41f40cf8a0 con 0x7f4204100fe0 2024-04-25T01:57:36.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.945+0000 7f4209a00640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f41ec0a2d80 0x7f41ec0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.946+0000 7f420f1d6740 1 -- start start 2024-04-25T01:57:36.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.947+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c76863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.947+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c7690ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.947+0000 7f4209a00640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f41ec0a2d80 0x7f41ec0a5240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41f80096c0 tx=0x7f41f8000bc0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.947+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76933e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.947+0000 7f420f1d6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5648c75ea560 con 0x5648c7686920 2024-04-25T01:57:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.947+0000 7f420f1d6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5648c7592930 con 0x5648c7686000 2024-04-25T01:57:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.947+0000 7f420f1d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648c75e6520 con 0x5648c7690ff0 2024-04-25T01:57:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.948+0000 7f420a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76933e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.948+0000 7f4209000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c7690ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.948+0000 7f4209000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c7690ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57078/0 (socket says 172.21.15.102:57078) 2024-04-25T01:57:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.948+0000 7f4209000640 1 -- 172.21.15.102:0/1236225446 learned_addr learned my addr 172.21.15.102:0/1236225446 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.948+0000 7f420a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76933e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44338/0 (socket says 172.21.15.102:44338) 2024-04-25T01:57:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.948+0000 7f4209a00640 1 --2- 172.21.15.102:0/1236225446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c76863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.949+0000 7f4209000640 1 -- 172.21.15.102:0/1236225446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 msgr2=0x5648c76933e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.949+0000 7f4209000640 1 --2- 172.21.15.102:0/1236225446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76933e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.949+0000 7f4209000640 1 -- 172.21.15.102:0/1236225446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 msgr2=0x5648c76863e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.949+0000 7f4209000640 1 --2- 172.21.15.102:0/1236225446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c76863e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.949+0000 7f4209000640 1 -- 172.21.15.102:0/1236225446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648c73c87d0 con 0x5648c7686920 2024-04-25T01:57:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.949+0000 7f420a400640 1 --2- 172.21.15.102:0/1236225446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76933e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:36.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.949+0000 7f4209000640 1 --2- 172.21.15.102:0/1236225446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c7690ab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f41f4098250 tx=0x7f41f409b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87a439543ef8213 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:36.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.950+0000 7f41f2a00640 1 -- 172.21.15.102:0/1236225446 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40a01a0 con 0x5648c7686920 2024-04-25T01:57:36.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.950+0000 7f41f2a00640 1 -- 172.21.15.102:0/1236225446 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f409f070 con 0x5648c7686920 2024-04-25T01:57:36.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.951+0000 7f420f1d6740 1 -- 172.21.15.102:0/1236225446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 msgr2=0x5648c7690ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:36.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.951+0000 7f420f1d6740 1 --2- 172.21.15.102:0/1236225446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c7690ab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f41f4098250 tx=0x7f41f409b750 comp rx=0 tx=0).stop 2024-04-25T01:57:36.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.951+0000 7f41f2a00640 1 -- 172.21.15.102:0/1236225446 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4098460 con 0x5648c7686920 2024-04-25T01:57:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.952+0000 7f420f1d6740 1 -- 172.21.15.102:0/1236225446 shutdown_connections 2024-04-25T01:57:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.952+0000 7f420f1d6740 1 --2- 172.21.15.102:0/1236225446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76933e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.952+0000 7f420f1d6740 1 --2- 172.21.15.102:0/1236225446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c7690ab0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.952+0000 7f420f1d6740 1 --2- 172.21.15.102:0/1236225446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c76863e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.952+0000 7f420f1d6740 1 -- 172.21.15.102:0/1236225446 >> 172.21.15.102:0/1236225446 conn(0x5648c7682320 msgr2=0x5648c7682700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:36.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.953+0000 7f420f1d6740 1 -- 172.21.15.102:0/1236225446 shutdown_connections 2024-04-25T01:57:36.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.953+0000 7f420f1d6740 1 -- 172.21.15.102:0/1236225446 wait complete. 2024-04-25T01:57:36.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.954+0000 7f420f1d6740 1 -- start start 2024-04-25T01:57:36.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.954+0000 7f420f1d6740 1 Processor -- start 2024-04-25T01:57:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c768df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7686920 0x5648c768e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7690ff0 0x5648c768e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f420f1d6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5648c7693dd0 con 0x5648c7690ff0 2024-04-25T01:57:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f420f1d6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5648c75a5d90 con 0x5648c7686000 2024-04-25T01:57:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f420f1d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648c7693f50 con 0x5648c7686920 2024-04-25T01:57:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f420f1d6740 10 client.0 _ll_register_callbacks cb 0x5648c75a5780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:36.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f4209000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7686920 0x5648c768e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:36.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f4209000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7686920 0x5648c768e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44340/0 (socket says 172.21.15.102:44340) 2024-04-25T01:57:36.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.956+0000 7f4209000640 1 -- 172.21.15.102:0/4121704038 learned_addr learned my addr 172.21.15.102:0/4121704038 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:36.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.957+0000 7f4209000640 1 -- 172.21.15.102:0/4121704038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 msgr2=0x5648c768df70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:36.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.957+0000 7f4209000640 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c768df70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.957+0000 7f4209000640 1 -- 172.21.15.102:0/4121704038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7690ff0 msgr2=0x5648c768e9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:36.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.957+0000 7f4209000640 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7690ff0 0x5648c768e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.957+0000 7f4209000640 1 -- 172.21.15.102:0/4121704038 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648c72fe3f0 con 0x5648c7686920 2024-04-25T01:57:36.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.958+0000 7f4209000640 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7686920 0x5648c768e4b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f41f40bc060 tx=0x7f41f4004830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5ec1017027bc728 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:36.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.958+0000 7f41e7400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:36.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.958+0000 7f41e7400640 1 -- 172.21.15.102:0/4121704038 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f40a8070 con 0x5648c7686920 2024-04-25T01:57:36.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.958+0000 7f41e7400640 1 -- 172.21.15.102:0/4121704038 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f41f4098460 con 0x5648c7686920 2024-04-25T01:57:36.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.959+0000 7f420f1d6740 1 -- 172.21.15.102:0/4121704038 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f409f070 con 0x5648c7686920 2024-04-25T01:57:36.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.960+0000 7f41e7400640 1 -- 172.21.15.102:0/4121704038 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f41f40a01a0 con 0x5648c7686920 2024-04-25T01:57:36.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.960+0000 7f420f1d6740 10 client.25876 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.960+0000 7f420f1d6740 1 -- 172.21.15.102:0/4121704038 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5648c72be0f0 con 0x5648c7686920 2024-04-25T01:57:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.961+0000 7f41e7400640 1 -- 172.21.15.102:0/4121704038 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f409f070 con 0x5648c7686920 2024-04-25T01:57:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.961+0000 7f41e7400640 1 -- 172.21.15.102:0/4121704038 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f41f40ab030 con 0x5648c7686920 2024-04-25T01:57:36.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.961+0000 7f420f1d6740 10 client.25876 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.961+0000 7f420f1d6740 10 client.25876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.961+0000 7f420f1d6740 20 client.25876 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.961+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5648c76a1190 0x5648c76a3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.961+0000 7f420f1d6740 4 client.25876 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.961+0000 7f420f1d6740 1 -- 172.21.15.102:0/4121704038 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648c75fdd90 con 0x5648c76a1190 2024-04-25T01:57:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.962+0000 7f4209a00640 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5648c76a1190 0x5648c76a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:36.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.963+0000 7f4209a00640 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5648c76a1190 0x5648c76a3580 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e5091f09dfc495 server_cookie=632ed545a2ade492 in_seq=0 out_seq=0 2024-04-25T01:57:36.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.963+0000 7f41e7400640 10 client.25876 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:36.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.964+0000 7f41e7400640 1 -- 172.21.15.102:0/4121704038 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5648c75fdd90 con 0x5648c76a1190 2024-04-25T01:57:36.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.964+0000 7f41e7400640 10 client.25876 handle_command_reply: tid=0 2024-04-25T01:57:36.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.964+0000 7f420f1d6740 1 client.25876 shutdown 2024-04-25T01:57:36.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.966+0000 7f420f1d6740 1 -- 172.21.15.102:0/4121704038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7686920 msgr2=0x5648c768e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:36.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.966+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7686920 0x5648c768e4b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f41f40bc060 tx=0x7f41f4004830 comp rx=0 tx=0).stop 2024-04-25T01:57:36.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.967+0000 7f420f1d6740 1 -- 172.21.15.102:0/4121704038 shutdown_connections 2024-04-25T01:57:36.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.967+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5648c76a1190 0x5648c76a3580 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.967+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7690ff0 0x5648c768e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.967+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7686920 0x5648c768e4b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.967+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4121704038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c768df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:36.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.967+0000 7f41e7400640 0 client.25876 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:36.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.967+0000 7f420f1d6740 1 -- 172.21.15.102:0/4121704038 >> 172.21.15.102:0/4121704038 conn(0x5648c7682320 msgr2=0x5648c7688120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:36.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.967+0000 7f420f1d6740 1 -- 172.21.15.102:0/4121704038 shutdown_connections 2024-04-25T01:57:36.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.968+0000 7f420f1d6740 1 -- 172.21.15.102:0/4121704038 wait complete. 2024-04-25T01:57:36.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:36.968+0000 7f420f1d6740 20 client.25876 trim_cache size 0 max 16384 2024-04-25T01:57:37.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.009+0000 7f420f1d6740 1 Processor -- start 2024-04-25T01:57:37.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.009+0000 7f420f1d6740 1 -- start start 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.009+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686000 0x5648c769cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.009+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686920 0x5648c7688870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.009+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76a8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.009+0000 7f420f1d6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f42041c7420 con 0x5648c7686000 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.009+0000 7f420f1d6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f42041c4c90 con 0x5648c7686920 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.009+0000 7f420f1d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f42041afab0 con 0x5648c7690ff0 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.010+0000 7f4209000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686920 0x5648c7688870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.010+0000 7f420a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76a8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.010+0000 7f4209000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686920 0x5648c7688870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:44876/0 (socket says 172.21.15.102:44876) 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.010+0000 7f4209000640 1 -- 172.21.15.102:0/2869730123 learned_addr learned my addr 172.21.15.102:0/2869730123 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.010+0000 7f420a400640 1 --2- 172.21.15.102:0/2869730123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76a8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44348/0 (socket says 172.21.15.102:44348) 2024-04-25T01:57:37.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.010+0000 7f420a400640 1 -- 172.21.15.102:0/2869730123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686920 msgr2=0x5648c7688870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:37.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.010+0000 7f420a400640 1 --2- 172.21.15.102:0/2869730123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686920 0x5648c7688870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.010+0000 7f420a400640 1 -- 172.21.15.102:0/2869730123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686000 msgr2=0x5648c769cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:37.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.010+0000 7f420a400640 1 --2- 172.21.15.102:0/2869730123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686000 0x5648c769cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.011+0000 7f420a400640 1 -- 172.21.15.102:0/2869730123 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648c767dc80 con 0x5648c7690ff0 2024-04-25T01:57:37.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.011+0000 7f420a400640 1 --2- 172.21.15.102:0/2869730123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76a8c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f41fc007d80 tx=0x7f41fc00d700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dcae9eb325ce9da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:37.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.011+0000 7f41f2000640 1 -- 172.21.15.102:0/2869730123 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc00f820 con 0x5648c7690ff0 2024-04-25T01:57:37.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.011+0000 7f41f2000640 1 -- 172.21.15.102:0/2869730123 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41fc00de10 con 0x5648c7690ff0 2024-04-25T01:57:37.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.011+0000 7f41f2000640 1 -- 172.21.15.102:0/2869730123 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc00f820 con 0x5648c7690ff0 2024-04-25T01:57:37.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.013+0000 7f420f1d6740 1 -- 172.21.15.102:0/2869730123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 msgr2=0x5648c76a8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:37.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.013+0000 7f420f1d6740 1 --2- 172.21.15.102:0/2869730123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76a8c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f41fc007d80 tx=0x7f41fc00d700 comp rx=0 tx=0).stop 2024-04-25T01:57:37.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.013+0000 7f420f1d6740 1 -- 172.21.15.102:0/2869730123 shutdown_connections 2024-04-25T01:57:37.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.013+0000 7f420f1d6740 1 --2- 172.21.15.102:0/2869730123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c76a8c30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.013+0000 7f420f1d6740 1 --2- 172.21.15.102:0/2869730123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686920 0x5648c7688870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.013+0000 7f420f1d6740 1 --2- 172.21.15.102:0/2869730123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686000 0x5648c769cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.013+0000 7f420f1d6740 1 -- 172.21.15.102:0/2869730123 >> 172.21.15.102:0/2869730123 conn(0x5648c7682320 msgr2=0x5648c7682700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:37.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.014+0000 7f420f1d6740 1 -- 172.21.15.102:0/2869730123 shutdown_connections 2024-04-25T01:57:37.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.014+0000 7f420f1d6740 1 -- 172.21.15.102:0/2869730123 wait complete. 2024-04-25T01:57:37.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.015+0000 7f420f1d6740 1 -- start start 2024-04-25T01:57:37.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.015+0000 7f420f1d6740 1 Processor -- start 2024-04-25T01:57:37.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.016+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c7699380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.016+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c76947b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.016+0000 7f420f1d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c7694cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.016+0000 7f420f1d6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f42041b0120 con 0x5648c7686920 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.016+0000 7f420f1d6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f42041b1590 con 0x5648c7686000 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.016+0000 7f420f1d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f42041b07f0 con 0x5648c7690ff0 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.016+0000 7f420f1d6740 10 client.0 _ll_register_callbacks cb 0x7f41f4004b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.017+0000 7f420a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c7694cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.017+0000 7f4209a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c7699380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.017+0000 7f420a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c7694cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44362/0 (socket says 172.21.15.102:44362) 2024-04-25T01:57:37.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.017+0000 7f420a400640 1 -- 172.21.15.102:0/4108071974 learned_addr learned my addr 172.21.15.102:0/4108071974 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:37.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.018+0000 7f4209000640 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c76947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.018+0000 7f420a400640 1 -- 172.21.15.102:0/4108071974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 msgr2=0x5648c7699380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:37.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.018+0000 7f420a400640 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c7699380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.018+0000 7f420a400640 1 -- 172.21.15.102:0/4108071974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 msgr2=0x5648c76947b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:37.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.018+0000 7f420a400640 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c76947b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.018+0000 7f420a400640 1 -- 172.21.15.102:0/4108071974 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648c76952b0 con 0x5648c7690ff0 2024-04-25T01:57:37.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.019+0000 7f420a400640 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c7694cf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00f880 tx=0x7f41fc007280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0da8d99f5a685b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:37.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.019+0000 7f41e7400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:37.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.019+0000 7f41e7400640 1 -- 172.21.15.102:0/4108071974 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc007980 con 0x5648c7690ff0 2024-04-25T01:57:37.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.019+0000 7f420f1d6740 1 -- 172.21.15.102:0/4108071974 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41fc00de10 con 0x5648c7690ff0 2024-04-25T01:57:37.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.019+0000 7f41e7400640 1 -- 172.21.15.102:0/4108071974 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f41fc00fbf0 con 0x5648c7690ff0 2024-04-25T01:57:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.020+0000 7f41e7400640 1 -- 172.21.15.102:0/4108071974 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f41fc030050 con 0x5648c7690ff0 2024-04-25T01:57:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.020+0000 7f420f1d6740 10 client.25882 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.020+0000 7f420f1d6740 1 -- 172.21.15.102:0/4108071974 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5648c7707280 con 0x5648c7690ff0 2024-04-25T01:57:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.020+0000 7f41e7400640 1 -- 172.21.15.102:0/4108071974 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41fc00de10 con 0x5648c7690ff0 2024-04-25T01:57:37.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.021+0000 7f41e7400640 1 -- 172.21.15.102:0/4108071974 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f41fc00b070 con 0x5648c7690ff0 2024-04-25T01:57:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.021+0000 7f420f1d6740 10 client.25882 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.021+0000 7f420f1d6740 10 client.25882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.021+0000 7f420f1d6740 20 client.25882 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.021+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5648c77078f0 0x5648c7709d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.021+0000 7f420f1d6740 4 client.25882 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:57:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.021+0000 7f420f1d6740 1 -- 172.21.15.102:0/4108071974 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5648c75f7aa0 con 0x5648c77078f0 2024-04-25T01:57:37.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.022+0000 7f4209a00640 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5648c77078f0 0x5648c7709d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.024+0000 7f4209a00640 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5648c77078f0 0x5648c7709d10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecb5b8d6ea213817 server_cookie=4b689cdfd4854cb3 in_seq=0 out_seq=0 2024-04-25T01:57:37.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.024+0000 7f41e7400640 10 client.25882 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:37.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.025+0000 7f41e7400640 1 -- 172.21.15.102:0/4108071974 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5648c75f7aa0 con 0x5648c77078f0 2024-04-25T01:57:37.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.025+0000 7f41e7400640 10 client.25882 handle_command_reply: tid=0 2024-04-25T01:57:37.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.025+0000 7f420f1d6740 1 client.25882 shutdown 2024-04-25T01:57:37.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.026+0000 7f420f1d6740 1 -- 172.21.15.102:0/4108071974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 msgr2=0x5648c7694cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:37.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.026+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c7694cf0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00f880 tx=0x7f41fc007280 comp rx=0 tx=0).stop 2024-04-25T01:57:37.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.027+0000 7f420f1d6740 1 -- 172.21.15.102:0/4108071974 shutdown_connections 2024-04-25T01:57:37.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.027+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5648c77078f0 0x5648c7709d10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.027+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648c7690ff0 0x5648c7694cf0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.028+0000 7f41e7400640 0 client.25882 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:37.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.028+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5648c7686920 0x5648c76947b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.028+0000 7f420f1d6740 1 --2- 172.21.15.102:0/4108071974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5648c7686000 0x5648c7699380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.028+0000 7f420f1d6740 1 -- 172.21.15.102:0/4108071974 >> 172.21.15.102:0/4108071974 conn(0x5648c7682320 msgr2=0x5648c768df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:37.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.029+0000 7f420f1d6740 1 -- 172.21.15.102:0/4108071974 shutdown_connections 2024-04-25T01:57:37.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.029+0000 7f420f1d6740 1 -- 172.21.15.102:0/4108071974 wait complete. 2024-04-25T01:57:37.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.029+0000 7f420f1d6740 20 client.25882 trim_cache size 0 max 16384 2024-04-25T01:57:37.034 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:57:37.034 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T01:57:37.034 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T01:57:37.034 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:57:37.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.031+0000 7f420c400640 1 -- 172.21.15.102:0/3094228581 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f41ec0a2d80 msgr2=0x7f41ec0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:37.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.031+0000 7f420c400640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f41ec0a2d80 0x7f41ec0a5240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41f80096c0 tx=0x7f41f8000bc0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.031+0000 7f420c400640 1 -- 172.21.15.102:0/3094228581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4204100fe0 msgr2=0x7f42041a8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:37.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.031+0000 7f420c400640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4204100fe0 0x7f42041a8090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f41f400e9f0 tx=0x7f41f400edf0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.032+0000 7f420c400640 1 -- 172.21.15.102:0/3094228581 shutdown_connections 2024-04-25T01:57:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.032+0000 7f420c400640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f41ec0a2d80 0x7f41ec0a5240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.032+0000 7f420c400640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4204106630 0x7f42041a85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.032+0000 7f420c400640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4204100fe0 0x7f42041a8090 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.032+0000 7f420c400640 1 --2- 172.21.15.102:0/3094228581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4204100620 0x7f420410f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.032+0000 7f420c400640 1 -- 172.21.15.102:0/3094228581 >> 172.21.15.102:0/3094228581 conn(0x7f42040776a0 msgr2=0x7f4204103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.032+0000 7f420c400640 1 -- 172.21.15.102:0/3094228581 shutdown_connections 2024-04-25T01:57:37.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.033+0000 7f420c400640 1 -- 172.21.15.102:0/3094228581 wait complete. 2024-04-25T01:57:37.062 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:36 smithi102 ceph-mon[27796]: pgmap v73: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.2 MiB/s wr, 691 op/s 2024-04-25T01:57:37.067 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:57:37.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:57:37.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:36 smithi110 ceph-mon[32848]: pgmap v73: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.2 MiB/s wr, 691 op/s 2024-04-25T01:57:37.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:36 smithi193 ceph-mon[32981]: pgmap v73: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.2 MiB/s wr, 691 op/s 2024-04-25T01:57:37.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.351+0000 7f471b800640 1 -- 172.21.15.102:0/3389465822 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471410d090 msgr2=0x7f4714114c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.351+0000 7f471b800640 1 --2- 172.21.15.102:0/3389465822 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471410d090 0x7f4714114c60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f470c009be0 tx=0x7f470c02a6c0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.351+0000 7f471b800640 1 -- 172.21.15.102:0/3389465822 shutdown_connections 2024-04-25T01:57:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.351+0000 7f471b800640 1 --2- 172.21.15.102:0/3389465822 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471410d090 0x7f4714114c60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.351+0000 7f471b800640 1 --2- 172.21.15.102:0/3389465822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4714108970 0x7f471410c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.351+0000 7f471b800640 1 --2- 172.21.15.102:0/3389465822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4714107fc0 0x7f47141083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.351+0000 7f471b800640 1 -- 172.21.15.102:0/3389465822 >> 172.21.15.102:0/3389465822 conn(0x7f4714074f40 msgr2=0x7f4714106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:37.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.351+0000 7f471b800640 1 -- 172.21.15.102:0/3389465822 shutdown_connections 2024-04-25T01:57:37.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.352+0000 7f471b800640 1 -- 172.21.15.102:0/3389465822 wait complete. 2024-04-25T01:57:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.353+0000 7f471b800640 1 Processor -- start 2024-04-25T01:57:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.353+0000 7f471b800640 1 -- start start 2024-04-25T01:57:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.353+0000 7f471b800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4714107fc0 0x7f4714072ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.353+0000 7f471b800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d090 0x7f471406c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.353+0000 7f471b800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471406c5b0 0x7f471406ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.353+0000 7f471b800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4714118130 con 0x7f471410d090 2024-04-25T01:57:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.353+0000 7f471b800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4714117fb0 con 0x7f471406c5b0 2024-04-25T01:57:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.353+0000 7f471b800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47141182b0 con 0x7f4714107fc0 2024-04-25T01:57:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.354+0000 7f4718e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4714107fc0 0x7f4714072ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.353+0000 7f4719800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471406c5b0 0x7f471406ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.354+0000 7f4718e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4714107fc0 0x7f4714072ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44372/0 (socket says 172.21.15.102:44372) 2024-04-25T01:57:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.354+0000 7f4718e00640 1 -- 172.21.15.102:0/3493899839 learned_addr learned my addr 172.21.15.102:0/3493899839 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.354+0000 7f4713e00640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d090 0x7f471406c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.354+0000 7f4719800640 1 -- 172.21.15.102:0/3493899839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4714107fc0 msgr2=0x7f4714072ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.354+0000 7f4719800640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4714107fc0 0x7f4714072ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.354+0000 7f4719800640 1 -- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d090 msgr2=0x7f471406c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.354+0000 7f4719800640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d090 0x7f471406c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.354+0000 7f4719800640 1 -- 172.21.15.102:0/3493899839 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4714070b10 con 0x7f471406c5b0 2024-04-25T01:57:37.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.355+0000 7f4719800640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471406c5b0 0x7f471406ca60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f470c034040 tx=0x7f470c030710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:37.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.355+0000 7f4711600640 1 -- 172.21.15.102:0/3493899839 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c041030 con 0x7f471406c5b0 2024-04-25T01:57:37.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.356+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4714070da0 con 0x7f471406c5b0 2024-04-25T01:57:37.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.356+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47140712e0 con 0x7f471406c5b0 2024-04-25T01:57:37.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.356+0000 7f4711600640 1 -- 172.21.15.102:0/3493899839 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f470c003e50 con 0x7f471406c5b0 2024-04-25T01:57:37.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.356+0000 7f4711600640 1 -- 172.21.15.102:0/3493899839 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c036040 con 0x7f471406c5b0 2024-04-25T01:57:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.359+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f4005290 con 0x7f471406c5b0 2024-04-25T01:57:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.361+0000 7f4711600640 1 -- 172.21.15.102:0/3493899839 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f470c044070 con 0x7f471406c5b0 2024-04-25T01:57:37.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.364+0000 7f4711600640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e40 0x7f46fc0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:37.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.364+0000 7f4711600640 1 -- 172.21.15.102:0/3493899839 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f470c0efed0 con 0x7f471406c5b0 2024-04-25T01:57:37.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.364+0000 7f4718e00640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e40 0x7f46fc0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:37.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.367+0000 7f4718e00640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e40 0x7f46fc0a5300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4708005fd0 tx=0x7f470800b480 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:37.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.367+0000 7f4711600640 1 -- 172.21.15.102:0/3493899839 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f470c0bcac0 con 0x7f471406c5b0 2024-04-25T01:57:37.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.818+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46f4005850 con 0x7f471406c5b0 2024-04-25T01:57:37.823 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:57:37.824 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:56:02.805319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24541,24587,24533]},"id":1}]} 2024-04-25T01:57:37.824 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-25T01:57:37.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.819+0000 7f4711600640 1 -- 172.21.15.102:0/3493899839 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11901 (secure 0 0 0) 0x7f470c0bc3c0 con 0x7f471406c5b0 2024-04-25T01:57:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.822+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e40 msgr2=0x7f46fc0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.822+0000 7f471b800640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e40 0x7f46fc0a5300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4708005fd0 tx=0x7f470800b480 comp rx=0 tx=0).stop 2024-04-25T01:57:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471406c5b0 msgr2=0x7f471406ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471406c5b0 0x7f471406ca60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f470c034040 tx=0x7f470c030710 comp rx=0 tx=0).stop 2024-04-25T01:57:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 shutdown_connections 2024-04-25T01:57:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e40 0x7f46fc0a5300 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4708005fd0 tx=0x7f470800b480 comp rx=0 tx=0).stop 2024-04-25T01:57:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471406c5b0 0x7f471406ca60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d090 0x7f471406c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 --2- 172.21.15.102:0/3493899839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4714107fc0 0x7f4714072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 >> 172.21.15.102:0/3493899839 conn(0x7f4714074f40 msgr2=0x7f4714076b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 shutdown_connections 2024-04-25T01:57:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:37.823+0000 7f471b800640 1 -- 172.21.15.102:0/3493899839 wait complete. 2024-04-25T01:57:37.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:57:38.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 105.0 s, 3552.2 tps, lat 28.160 ms stddev 30.862 2024-04-25T01:57:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.110+0000 7f3f8d000640 1 -- 172.21.15.102:0/2753599124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f88102150 msgr2=0x7f3f8810e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.110+0000 7f3f8d000640 1 --2- 172.21.15.102:0/2753599124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f88102150 0x7f3f8810e660 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3f80009cd0 tx=0x7f3f8002a8e0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.110+0000 7f3f8d000640 1 -- 172.21.15.102:0/2753599124 shutdown_connections 2024-04-25T01:57:38.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.111+0000 7f3f8d000640 1 --2- 172.21.15.102:0/2753599124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f88102150 0x7f3f8810e660 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.111+0000 7f3f8d000640 1 --2- 172.21.15.102:0/2753599124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f88101790 0x7f3f88101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.111+0000 7f3f8d000640 1 --2- 172.21.15.102:0/2753599124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f881077a0 0x7f3f88107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.111+0000 7f3f8d000640 1 -- 172.21.15.102:0/2753599124 >> 172.21.15.102:0/2753599124 conn(0x7f3f880fd2a0 msgr2=0x7f3f880ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.111+0000 7f3f8d000640 1 -- 172.21.15.102:0/2753599124 shutdown_connections 2024-04-25T01:57:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.112+0000 7f3f8d000640 1 -- 172.21.15.102:0/2753599124 wait complete. 2024-04-25T01:57:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.114+0000 7f3f8d000640 1 Processor -- start 2024-04-25T01:57:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.114+0000 7f3f8d000640 1 -- start start 2024-04-25T01:57:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.115+0000 7f3f8d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f88101790 0x7f3f88072f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.115+0000 7f3f8d000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f88102150 0x7f3f8806bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.115+0000 7f3f8d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f881077a0 0x7f3f8806c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.115+0000 7f3f8d000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3f88113e00 con 0x7f3f881077a0 2024-04-25T01:57:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.115+0000 7f3f8d000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3f88113c80 con 0x7f3f88102150 2024-04-25T01:57:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.115+0000 7f3f8d000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3f88113f80 con 0x7f3f88101790 2024-04-25T01:57:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.115+0000 7f3f85400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f88102150 0x7f3f8806bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.116+0000 7f3f85400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f88102150 0x7f3f8806bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:44928/0 (socket says 172.21.15.102:44928) 2024-04-25T01:57:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.116+0000 7f3f85400640 1 -- 172.21.15.102:0/4287975090 learned_addr learned my addr 172.21.15.102:0/4287975090 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.116+0000 7f3f85400640 1 -- 172.21.15.102:0/4287975090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f88101790 msgr2=0x7f3f88072f10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:57:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.116+0000 7f3f85400640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f88101790 0x7f3f88072f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.116+0000 7f3f85400640 1 -- 172.21.15.102:0/4287975090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f881077a0 msgr2=0x7f3f8806c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.116+0000 7f3f85400640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f881077a0 0x7f3f8806c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.116+0000 7f3f85400640 1 -- 172.21.15.102:0/4287975090 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8806ca90 con 0x7f3f88102150 2024-04-25T01:57:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.116+0000 7f3f85400640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f88102150 0x7f3f8806bf90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3f7400b4a0 tx=0x7f3f7400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.117+0000 7f3f7ea00640 1 -- 172.21.15.102:0/4287975090 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74011830 con 0x7f3f88102150 2024-04-25T01:57:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.117+0000 7f3f7ea00640 1 -- 172.21.15.102:0/4287975090 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f74016070 con 0x7f3f88102150 2024-04-25T01:57:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.117+0000 7f3f8d000640 1 -- 172.21.15.102:0/4287975090 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8806cd80 con 0x7f3f88102150 2024-04-25T01:57:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.117+0000 7f3f7ea00640 1 -- 172.21.15.102:0/4287975090 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7400f2e0 con 0x7f3f88102150 2024-04-25T01:57:38.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.118+0000 7f3f8d000640 1 -- 172.21.15.102:0/4287975090 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f88070d60 con 0x7f3f88102150 2024-04-25T01:57:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.123+0000 7f3f7ea00640 1 -- 172.21.15.102:0/4287975090 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3f740119d0 con 0x7f3f88102150 2024-04-25T01:57:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.123+0000 7f3f8fe12740 1 Processor -- start 2024-04-25T01:57:38.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.123+0000 7f3f8fe12740 1 -- start start 2024-04-25T01:57:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c74a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c74acc0 0x55628c75d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c75ddc0 0x55628c7601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55628c6af4e0 con 0x55628c74acc0 2024-04-25T01:57:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55628c657a70 con 0x55628c75ddc0 2024-04-25T01:57:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55628c6ab4a0 con 0x55628c74a310 2024-04-25T01:57:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f86800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c74a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f86800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c74a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44396/0 (socket says 172.21.15.102:44396) 2024-04-25T01:57:38.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f86800640 1 -- 172.21.15.102:0/1399277682 learned_addr learned my addr 172.21.15.102:0/1399277682 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.124+0000 7f3f85400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c75ddc0 0x55628c7601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.125+0000 7f3f86800640 1 -- 172.21.15.102:0/1399277682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c75ddc0 msgr2=0x55628c7601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.125+0000 7f3f86800640 1 --2- 172.21.15.102:0/1399277682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c75ddc0 0x55628c7601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.125+0000 7f3f86800640 1 -- 172.21.15.102:0/1399277682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c74acc0 msgr2=0x55628c75d880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.125+0000 7f3f86800640 1 --2- 172.21.15.102:0/1399277682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c74acc0 0x55628c75d880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.125+0000 7f3f86800640 1 -- 172.21.15.102:0/1399277682 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55628c3c33f0 con 0x55628c74a310 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.125+0000 7f3f85400640 1 --2- 172.21.15.102:0/1399277682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c75ddc0 0x55628c7601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.125+0000 7f3f86800640 1 --2- 172.21.15.102:0/1399277682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c74a6f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3f8002ae50 tx=0x7f3f80009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86268f8182971515 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.125+0000 7f3f7ea00640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f6c0a2dc0 0x7f3f6c0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.126+0000 7f3f73400640 1 -- 172.21.15.102:0/1399277682 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8003f070 con 0x55628c74a310 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.126+0000 7f3f73400640 1 -- 172.21.15.102:0/1399277682 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f80036040 con 0x55628c74a310 2024-04-25T01:57:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.126+0000 7f3f73400640 1 -- 172.21.15.102:0/1399277682 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f80004280 con 0x55628c74a310 2024-04-25T01:57:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.126+0000 7f3f7ea00640 1 -- 172.21.15.102:0/4287975090 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3f740cea20 con 0x7f3f88102150 2024-04-25T01:57:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.126+0000 7f3f8fe12740 1 -- 172.21.15.102:0/1399277682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 msgr2=0x55628c74a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.126+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/1399277682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c74a6f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3f8002ae50 tx=0x7f3f80009a20 comp rx=0 tx=0).stop 2024-04-25T01:57:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.126+0000 7f3f85e00640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f6c0a2dc0 0x7f3f6c0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.127+0000 7f3f8fe12740 1 -- 172.21.15.102:0/1399277682 shutdown_connections 2024-04-25T01:57:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.127+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/1399277682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c75ddc0 0x55628c7601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.127+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/1399277682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c74acc0 0x55628c75d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.127+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/1399277682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c74a6f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.127+0000 7f3f8fe12740 1 -- 172.21.15.102:0/1399277682 >> 172.21.15.102:0/1399277682 conn(0x55628c7468e0 msgr2=0x55628c746cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.130+0000 7f3f8fe12740 1 -- 172.21.15.102:0/1399277682 shutdown_connections 2024-04-25T01:57:38.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.130+0000 7f3f85e00640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f6c0a2dc0 0x7f3f6c0a5280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3f78005fd0 tx=0x7f3f78005850 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.130+0000 7f3f8fe12740 1 -- 172.21.15.102:0/1399277682 wait complete. 2024-04-25T01:57:38.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.131+0000 7f3f8fe12740 1 -- start start 2024-04-25T01:57:38.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.131+0000 7f3f8fe12740 1 Processor -- start 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c7579f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c75d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c757f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55628c74d3b0 con 0x55628c75ddc0 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55628c66aec0 con 0x55628c74acc0 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55628c74d530 con 0x55628c74a310 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f8fe12740 10 client.0 _ll_register_callbacks cb 0x55628c6a73f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f85e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c7579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f86800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c757f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.132+0000 7f3f85e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c7579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44400/0 (socket says 172.21.15.102:44400) 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.133+0000 7f3f85e00640 1 -- 172.21.15.102:0/3084367859 learned_addr learned my addr 172.21.15.102:0/3084367859 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.133+0000 7f3f85400640 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c75d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.133+0000 7f3f85e00640 1 -- 172.21.15.102:0/3084367859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 msgr2=0x55628c75d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.133+0000 7f3f85e00640 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c75d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.133+0000 7f3f85e00640 1 -- 172.21.15.102:0/3084367859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 msgr2=0x55628c757f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.133+0000 7f3f85e00640 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c757f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.133+0000 7f3f85e00640 1 -- 172.21.15.102:0/3084367859 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55628c48d7d0 con 0x55628c74a310 2024-04-25T01:57:38.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.133+0000 7f3f85e00640 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c7579f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800dc80 tx=0x7f3f78004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1952b2fda380130d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.133+0000 7f3f67e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.134+0000 7f3f67e00640 1 -- 172.21.15.102:0/3084367859 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78021070 con 0x55628c74a310 2024-04-25T01:57:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.134+0000 7f3f67e00640 1 -- 172.21.15.102:0/3084367859 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3f78005220 con 0x55628c74a310 2024-04-25T01:57:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.134+0000 7f3f8fe12740 1 -- 172.21.15.102:0/3084367859 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f80036040 con 0x55628c74a310 2024-04-25T01:57:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.134+0000 7f3f67e00640 1 -- 172.21.15.102:0/3084367859 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f78005420 con 0x55628c74a310 2024-04-25T01:57:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.134+0000 7f3f8fe12740 10 client.25897 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.134+0000 7f3f8fe12740 1 -- 172.21.15.102:0/3084367859 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55628c775680 con 0x55628c74a310 2024-04-25T01:57:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.135+0000 7f3f67e00640 1 -- 172.21.15.102:0/3084367859 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f80036040 con 0x55628c74a310 2024-04-25T01:57:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.136+0000 7f3f67e00640 1 -- 172.21.15.102:0/3084367859 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3f7801c070 con 0x55628c74a310 2024-04-25T01:57:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.136+0000 7f3f8fe12740 10 client.25897 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.136+0000 7f3f8fe12740 10 client.25897 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:57:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.136+0000 7f3f8fe12740 20 client.25897 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.136+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55628c776ae0 0x55628c778f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.136+0000 7f3f8fe12740 4 client.25897 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.136+0000 7f3f8fe12740 1 -- 172.21.15.102:0/3084367859 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55628c6bca20 con 0x55628c776ae0 2024-04-25T01:57:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.136+0000 7f3f85400640 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55628c776ae0 0x55628c778f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.137+0000 7f3f85400640 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55628c776ae0 0x55628c778f10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b60d274c5aec498b server_cookie=7055c794ddeeab8b in_seq=0 out_seq=0 2024-04-25T01:57:38.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.137+0000 7f3f67e00640 10 client.25897 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:57:38.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.138+0000 7f3f67e00640 1 -- 172.21.15.102:0/3084367859 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55628c6bca20 con 0x55628c776ae0 2024-04-25T01:57:38.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.138+0000 7f3f67e00640 10 client.25897 handle_command_reply: tid=0 2024-04-25T01:57:38.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.138+0000 7f3f8fe12740 1 client.25897 shutdown 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 -- 172.21.15.102:0/3084367859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 msgr2=0x55628c7579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c7579f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800dc80 tx=0x7f3f78004730 comp rx=0 tx=0).stop 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 -- 172.21.15.102:0/3084367859 shutdown_connections 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c757f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55628c776ae0 0x55628c778f10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c75d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/3084367859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c7579f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 -- 172.21.15.102:0/3084367859 >> 172.21.15.102:0/3084367859 conn(0x55628c7468e0 msgr2=0x55628c74c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f67e00640 0 client.25897 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 -- 172.21.15.102:0/3084367859 shutdown_connections 2024-04-25T01:57:38.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.139+0000 7f3f8fe12740 1 -- 172.21.15.102:0/3084367859 wait complete. 2024-04-25T01:57:38.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.140+0000 7f3f8fe12740 20 client.25897 trim_cache size 0 max 16384 2024-04-25T01:57:38.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.164+0000 7f3f8fe12740 1 Processor -- start 2024-04-25T01:57:38.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.164+0000 7f3f8fe12740 1 -- start start 2024-04-25T01:57:38.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.164+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74a310 0x55628c75d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.164+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74acc0 0x55628c77e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.164+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c77ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.164+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3f881b63d0 con 0x55628c75ddc0 2024-04-25T01:57:38.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.164+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3f881b3c40 con 0x55628c74a310 2024-04-25T01:57:38.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.164+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3f881b1a90 con 0x55628c74acc0 2024-04-25T01:57:38.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.165+0000 7f3f86800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c77ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.165+0000 7f3f85e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74a310 0x55628c75d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.165+0000 7f3f86800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c77ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57168/0 (socket says 172.21.15.102:57168) 2024-04-25T01:57:38.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.165+0000 7f3f86800640 1 -- 172.21.15.102:0/4173797168 learned_addr learned my addr 172.21.15.102:0/4173797168 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.165+0000 7f3f85400640 1 --2- 172.21.15.102:0/4173797168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74acc0 0x55628c77e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.165+0000 7f3f85400640 1 -- 172.21.15.102:0/4173797168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74a310 msgr2=0x55628c75d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.165+0000 7f3f85400640 1 --2- 172.21.15.102:0/4173797168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74a310 0x55628c75d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.165+0000 7f3f85400640 1 -- 172.21.15.102:0/4173797168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 msgr2=0x55628c77ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.166+0000 7f3f85400640 1 --2- 172.21.15.102:0/4173797168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c77ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.166+0000 7f3f85400640 1 -- 172.21.15.102:0/4173797168 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55628c751d00 con 0x55628c74acc0 2024-04-25T01:57:38.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.167+0000 7f3f86800640 1 --2- 172.21.15.102:0/4173797168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c77ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:38.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.167+0000 7f3f85400640 1 --2- 172.21.15.102:0/4173797168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74acc0 0x55628c77e7e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3f740cc250 tx=0x7f3f74010480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f80e441f0121fd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.167+0000 7f3f72a00640 1 -- 172.21.15.102:0/4173797168 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f740a3c90 con 0x55628c74acc0 2024-04-25T01:57:38.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.167+0000 7f3f72a00640 1 -- 172.21.15.102:0/4173797168 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f74010e80 con 0x55628c74acc0 2024-04-25T01:57:38.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.167+0000 7f3f72a00640 1 -- 172.21.15.102:0/4173797168 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74097690 con 0x55628c74acc0 2024-04-25T01:57:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.169+0000 7f3f8fe12740 1 -- 172.21.15.102:0/4173797168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74acc0 msgr2=0x55628c77e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.169+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/4173797168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74acc0 0x55628c77e7e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3f740cc250 tx=0x7f3f74010480 comp rx=0 tx=0).stop 2024-04-25T01:57:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.169+0000 7f3f8fe12740 1 -- 172.21.15.102:0/4173797168 shutdown_connections 2024-04-25T01:57:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.169+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/4173797168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c77ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.169+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/4173797168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74acc0 0x55628c77e7e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.169+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/4173797168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74a310 0x55628c75d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.169+0000 7f3f8fe12740 1 -- 172.21.15.102:0/4173797168 >> 172.21.15.102:0/4173797168 conn(0x55628c7468e0 msgr2=0x55628c745b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.170+0000 7f3f8fe12740 1 -- 172.21.15.102:0/4173797168 shutdown_connections 2024-04-25T01:57:38.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.170+0000 7f3f8fe12740 1 -- 172.21.15.102:0/4173797168 wait complete. 2024-04-25T01:57:38.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.170+0000 7f3f8fe12740 1 -- start start 2024-04-25T01:57:38.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.171+0000 7f3f8fe12740 1 Processor -- start 2024-04-25T01:57:38.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:37 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3493899839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:57:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.171+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c789a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.171+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c789f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.171+0000 7f3f8fe12740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c785ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.171+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3f881b2100 con 0x55628c75ddc0 2024-04-25T01:57:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.171+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3f881b3570 con 0x55628c74acc0 2024-04-25T01:57:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.171+0000 7f3f8fe12740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3f881b27d0 con 0x55628c74a310 2024-04-25T01:57:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.171+0000 7f3f8fe12740 10 client.0 _ll_register_callbacks cb 0x7f3f78005e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c789f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c789f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:44962/0 (socket says 172.21.15.102:44962) 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85400640 1 -- 172.21.15.102:0/371418498 learned_addr learned my addr 172.21.15.102:0/371418498 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f86800640 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c785ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85e00640 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c789a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85400640 1 -- 172.21.15.102:0/371418498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 msgr2=0x55628c789a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85400640 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c789a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85400640 1 -- 172.21.15.102:0/371418498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 msgr2=0x55628c785ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85400640 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c785ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85400640 1 -- 172.21.15.102:0/371418498 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55628c786630 con 0x55628c74acc0 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f85e00640 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c789a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.172+0000 7f3f86800640 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c785ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.173+0000 7f3f85400640 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c789f40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f740093e0 tx=0x7f3f740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d67fcf99e8313d07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.173+0000 7f3f67e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.173+0000 7f3f67e00640 1 -- 172.21.15.102:0/371418498 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74011cc0 con 0x55628c74acc0 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.173+0000 7f3f67e00640 1 -- 172.21.15.102:0/371418498 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3f7400ed40 con 0x55628c74acc0 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.173+0000 7f3f8fe12740 1 -- 172.21.15.102:0/371418498 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f74010e80 con 0x55628c74acc0 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f67e00640 1 -- 172.21.15.102:0/371418498 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3f740b4020 con 0x55628c74acc0 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f8fe12740 10 client.25919 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f8fe12740 1 -- 172.21.15.102:0/371418498 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55628c7dd540 con 0x55628c74acc0 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f67e00640 1 -- 172.21.15.102:0/371418498 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f74010e80 con 0x55628c74acc0 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f67e00640 1 -- 172.21.15.102:0/371418498 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3f74095b90 con 0x55628c74acc0 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f8fe12740 10 client.25919 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f8fe12740 10 client.25919 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:57:38.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f8fe12740 20 client.25919 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55628c7ddbe0 0x55628c7e0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f8fe12740 4 client.25919 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:57:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.174+0000 7f3f8fe12740 1 -- 172.21.15.102:0/371418498 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55628c6b6d70 con 0x55628c7ddbe0 2024-04-25T01:57:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.175+0000 7f3f85e00640 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55628c7ddbe0 0x55628c7e0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.175+0000 7f3f85e00640 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55628c7ddbe0 0x55628c7e0000 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=828b02205e474eb2 server_cookie=6cfe016f7c2a30c6 in_seq=0 out_seq=0 2024-04-25T01:57:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.175+0000 7f3f67e00640 10 client.25919 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:57:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f67e00640 1 -- 172.21.15.102:0/371418498 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55628c6b6d70 con 0x55628c7ddbe0 2024-04-25T01:57:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f67e00640 10 client.25919 handle_command_reply: tid=0 2024-04-25T01:57:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f8fe12740 1 client.25919 shutdown 2024-04-25T01:57:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f8fe12740 1 -- 172.21.15.102:0/371418498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 msgr2=0x55628c789f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c789f40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f740093e0 tx=0x7f3f740046c0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f8fe12740 1 -- 172.21.15.102:0/371418498 shutdown_connections 2024-04-25T01:57:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55628c75ddc0 0x55628c785ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55628c7ddbe0 0x55628c7e0000 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55628c74acc0 0x55628c789f40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f8fe12740 1 --2- 172.21.15.102:0/371418498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55628c74a310 0x55628c789a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f67e00640 0 client.25919 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:57:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.176+0000 7f3f8fe12740 1 -- 172.21.15.102:0/371418498 >> 172.21.15.102:0/371418498 conn(0x55628c7468e0 msgr2=0x55628c75c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.177+0000 7f3f8fe12740 1 -- 172.21.15.102:0/371418498 shutdown_connections 2024-04-25T01:57:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.177+0000 7f3f8fe12740 1 -- 172.21.15.102:0/371418498 wait complete. 2024-04-25T01:57:38.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.177+0000 7f3f8fe12740 20 client.25919 trim_cache size 0 max 16384 2024-04-25T01:57:38.182 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.178+0000 7f3f8d000640 1 -- 172.21.15.102:0/4287975090 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f6c0a2dc0 msgr2=0x7f3f6c0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.178+0000 7f3f8d000640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f6c0a2dc0 0x7f3f6c0a5280 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3f78005fd0 tx=0x7f3f78005850 comp rx=0 tx=0).stop 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.178+0000 7f3f8d000640 1 -- 172.21.15.102:0/4287975090 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f88102150 msgr2=0x7f3f8806bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.178+0000 7f3f8d000640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f88102150 0x7f3f8806bf90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3f7400b4a0 tx=0x7f3f7400b960 comp rx=0 tx=0).stop 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.179+0000 7f3f8d000640 1 -- 172.21.15.102:0/4287975090 shutdown_connections 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.179+0000 7f3f8d000640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f6c0a2dc0 0x7f3f6c0a5280 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.179+0000 7f3f8d000640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f881077a0 0x7f3f8806c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.179+0000 7f3f8d000640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f88102150 0x7f3f8806bf90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.179+0000 7f3f8d000640 1 --2- 172.21.15.102:0/4287975090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f88101790 0x7f3f88072f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.179+0000 7f3f8d000640 1 -- 172.21.15.102:0/4287975090 >> 172.21.15.102:0/4287975090 conn(0x7f3f880fd2a0 msgr2=0x7f3f880fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.180+0000 7f3f8d000640 1 -- 172.21.15.102:0/4287975090 shutdown_connections 2024-04-25T01:57:38.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.180+0000 7f3f8d000640 1 -- 172.21.15.102:0/4287975090 wait complete. 2024-04-25T01:57:38.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:57:38.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:37 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3493899839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:57:38.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:37 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3493899839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:57:38.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.481+0000 7f1133800640 1 -- 172.21.15.102:0/1479272488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f112c1032e0 msgr2=0x7f112c10b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.481+0000 7f1133800640 1 --2- 172.21.15.102:0/1479272488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f112c1032e0 0x7f112c10b7c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f111c009860 tx=0x7f111c02a480 comp rx=0 tx=0).stop 2024-04-25T01:57:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.481+0000 7f1133800640 1 -- 172.21.15.102:0/1479272488 shutdown_connections 2024-04-25T01:57:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.481+0000 7f1133800640 1 --2- 172.21.15.102:0/1479272488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f112c10bd00 0x7f112c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.481+0000 7f1133800640 1 --2- 172.21.15.102:0/1479272488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f112c1032e0 0x7f112c10b7c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.481+0000 7f1133800640 1 --2- 172.21.15.102:0/1479272488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f112c1029c0 0x7f112c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.481+0000 7f1133800640 1 -- 172.21.15.102:0/1479272488 >> 172.21.15.102:0/1479272488 conn(0x7f112c077740 msgr2=0x7f112c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.482+0000 7f1133800640 1 -- 172.21.15.102:0/1479272488 shutdown_connections 2024-04-25T01:57:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.483+0000 7f1133800640 1 -- 172.21.15.102:0/1479272488 wait complete. 2024-04-25T01:57:38.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.484+0000 7f1133800640 1 Processor -- start 2024-04-25T01:57:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.485+0000 7f1133800640 1 -- start start 2024-04-25T01:57:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.485+0000 7f1133800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f112c1029c0 0x7f112c103830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.485+0000 7f1133800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f112c1032e0 0x7f112c103d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.485+0000 7f1133800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f112c10bd00 0x7f112c1042b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.485+0000 7f1133800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f112c111b50 con 0x7f112c10bd00 2024-04-25T01:57:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.485+0000 7f1133800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f112c1119d0 con 0x7f112c1032e0 2024-04-25T01:57:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.485+0000 7f1133800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f112c111cd0 con 0x7f112c1029c0 2024-04-25T01:57:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.486+0000 7f1130e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f112c1029c0 0x7f112c103830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.486+0000 7f112be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f112c1032e0 0x7f112c103d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.486+0000 7f112be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f112c1032e0 0x7f112c103d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:44982/0 (socket says 172.21.15.102:44982) 2024-04-25T01:57:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.486+0000 7f1130e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f112c1029c0 0x7f112c103830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44432/0 (socket says 172.21.15.102:44432) 2024-04-25T01:57:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.486+0000 7f1130e00640 1 -- 172.21.15.102:0/3908587068 learned_addr learned my addr 172.21.15.102:0/3908587068 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.486+0000 7f1131800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f112c10bd00 0x7f112c1042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.487+0000 7f1131800640 1 -- 172.21.15.102:0/3908587068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f112c1029c0 msgr2=0x7f112c103830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.487+0000 7f1131800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f112c1029c0 0x7f112c103830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.487+0000 7f1131800640 1 -- 172.21.15.102:0/3908587068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f112c1032e0 msgr2=0x7f112c103d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.487+0000 7f1131800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f112c1032e0 0x7f112c103d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.487+0000 7f1131800640 1 -- 172.21.15.102:0/3908587068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c1aa8d0 con 0x7f112c10bd00 2024-04-25T01:57:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.488+0000 7f1131800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f112c10bd00 0x7f112c1042b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f111800e710 tx=0x7f111800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.488+0000 7f1129600640 1 -- 172.21.15.102:0/3908587068 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118004220 con 0x7f112c10bd00 2024-04-25T01:57:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.489+0000 7f1133800640 1 -- 172.21.15.102:0/3908587068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f112c1aabc0 con 0x7f112c10bd00 2024-04-25T01:57:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.489+0000 7f1129600640 1 -- 172.21.15.102:0/3908587068 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11180043c0 con 0x7f112c10bd00 2024-04-25T01:57:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.489+0000 7f1129600640 1 -- 172.21.15.102:0/3908587068 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118004a00 con 0x7f112c10bd00 2024-04-25T01:57:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.489+0000 7f1133800640 1 -- 172.21.15.102:0/3908587068 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f112c1ab090 con 0x7f112c10bd00 2024-04-25T01:57:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.493+0000 7f1129600640 1 -- 172.21.15.102:0/3908587068 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1118004ba0 con 0x7f112c10bd00 2024-04-25T01:57:38.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.497+0000 7f1129600640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11100a2d80 0x7f11100a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.497+0000 7f1136663740 1 Processor -- start 2024-04-25T01:57:38.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.497+0000 7f1136663740 1 -- start start 2024-04-25T01:57:38.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.498+0000 7f1136663740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb289a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.498+0000 7f1136663740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb28ee0 0x56200eb33070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.498+0000 7f1136663740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 0x56200eb359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.498+0000 7f1136663740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56200ea8d4e0 con 0x56200eb335b0 2024-04-25T01:57:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.498+0000 7f1136663740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56200ea35a70 con 0x56200eb28ee0 2024-04-25T01:57:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.498+0000 7f1136663740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56200ea894a0 con 0x56200eb285c0 2024-04-25T01:57:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.498+0000 7f1130e00640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11100a2d80 0x7f11100a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.498+0000 7f112be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb28ee0 0x56200eb33070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.499+0000 7f112be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb28ee0 0x56200eb33070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:44984/0 (socket says 172.21.15.102:44984) 2024-04-25T01:57:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.499+0000 7f112be00640 1 -- 172.21.15.102:0/715993455 learned_addr learned my addr 172.21.15.102:0/715993455 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.500+0000 7f112be00640 1 -- 172.21.15.102:0/715993455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 msgr2=0x56200eb289a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:38.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.500+0000 7f112be00640 1 --2- 172.21.15.102:0/715993455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb289a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.500+0000 7f112be00640 1 -- 172.21.15.102:0/715993455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 msgr2=0x56200eb359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.500+0000 7f112be00640 1 --2- 172.21.15.102:0/715993455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 0x56200eb359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.500+0000 7f112be00640 1 -- 172.21.15.102:0/715993455 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56200e7a13f0 con 0x56200eb28ee0 2024-04-25T01:57:38.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.499+0000 7f1129600640 1 -- 172.21.15.102:0/3908587068 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f11180cf230 con 0x7f112c10bd00 2024-04-25T01:57:38.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.500+0000 7f112be00640 1 --2- 172.21.15.102:0/715993455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb28ee0 0x56200eb33070 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f111c02a9a0 tx=0x7f111c030bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d08b05296be32e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.501+0000 7f1122000640 1 -- 172.21.15.102:0/715993455 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c004230 con 0x56200eb28ee0 2024-04-25T01:57:38.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.501+0000 7f1122000640 1 -- 172.21.15.102:0/715993455 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f111c030e60 con 0x56200eb28ee0 2024-04-25T01:57:38.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.501+0000 7f1122000640 1 -- 172.21.15.102:0/715993455 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c005630 con 0x56200eb28ee0 2024-04-25T01:57:38.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.502+0000 7f1136663740 1 -- 172.21.15.102:0/715993455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb28ee0 msgr2=0x56200eb33070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.502+0000 7f1136663740 1 --2- 172.21.15.102:0/715993455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb28ee0 0x56200eb33070 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f111c02a9a0 tx=0x7f111c030bb0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.502+0000 7f1130e00640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11100a2d80 0x7f11100a5240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1124007fb0 tx=0x7f1124007670 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.502+0000 7f1136663740 1 -- 172.21.15.102:0/715993455 shutdown_connections 2024-04-25T01:57:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.502+0000 7f1136663740 1 --2- 172.21.15.102:0/715993455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 0x56200eb359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.502+0000 7f1136663740 1 --2- 172.21.15.102:0/715993455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb28ee0 0x56200eb33070 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.502+0000 7f1136663740 1 --2- 172.21.15.102:0/715993455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb289a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.502+0000 7f1136663740 1 -- 172.21.15.102:0/715993455 >> 172.21.15.102:0/715993455 conn(0x56200eb248e0 msgr2=0x56200eb24cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.503+0000 7f1136663740 1 -- 172.21.15.102:0/715993455 shutdown_connections 2024-04-25T01:57:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.503+0000 7f1136663740 1 -- 172.21.15.102:0/715993455 wait complete. 2024-04-25T01:57:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.503+0000 7f1136663740 1 -- start start 2024-04-25T01:57:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.504+0000 7f1136663740 1 Processor -- start 2024-04-25T01:57:38.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.504+0000 7f1136663740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb285c0 0x56200eb30610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.504+0000 7f1136663740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb28ee0 0x56200eb30b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.504+0000 7f1136663740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 0x56200eb31090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.504+0000 7f1136663740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56200eb36340 con 0x56200eb335b0 2024-04-25T01:57:38.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.504+0000 7f1136663740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56200ea48ec0 con 0x56200eb285c0 2024-04-25T01:57:38.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.504+0000 7f1136663740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56200eb364c0 con 0x56200eb28ee0 2024-04-25T01:57:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.504+0000 7f1136663740 10 client.0 _ll_register_callbacks cb 0x56200ea853f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.505+0000 7f1130e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb285c0 0x56200eb30610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.505+0000 7f112be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb28ee0 0x56200eb30b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.505+0000 7f1130e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb285c0 0x56200eb30610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:44994/0 (socket says 172.21.15.102:44994) 2024-04-25T01:57:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.505+0000 7f1130e00640 1 -- 172.21.15.102:0/4269184732 learned_addr learned my addr 172.21.15.102:0/4269184732 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.505+0000 7f1131800640 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 0x56200eb31090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.505+0000 7f1130e00640 1 -- 172.21.15.102:0/4269184732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb28ee0 msgr2=0x56200eb30b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.505+0000 7f1130e00640 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb28ee0 0x56200eb30b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.505+0000 7f1130e00640 1 -- 172.21.15.102:0/4269184732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 msgr2=0x56200eb31090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.507+0000 7f1130e00640 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 0x56200eb31090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.507+0000 7f1130e00640 1 -- 172.21.15.102:0/4269184732 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56200e86b7d0 con 0x56200eb285c0 2024-04-25T01:57:38.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.507+0000 7f1131800640 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 0x56200eb31090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.507+0000 7f1130e00640 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb285c0 0x56200eb30610 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1124010f70 tx=0x7f1124004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65a5ab6ca89dc35f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.507+0000 7f1116a00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.508+0000 7f1116a00640 1 -- 172.21.15.102:0/4269184732 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112401d830 con 0x56200eb285c0 2024-04-25T01:57:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.508+0000 7f1116a00640 1 -- 172.21.15.102:0/4269184732 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f112401d9d0 con 0x56200eb285c0 2024-04-25T01:57:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.508+0000 7f1136663740 1 -- 172.21.15.102:0/4269184732 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f111c030e60 con 0x56200eb285c0 2024-04-25T01:57:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.508+0000 7f1116a00640 1 -- 172.21.15.102:0/4269184732 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f112401be10 con 0x56200eb285c0 2024-04-25T01:57:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.509+0000 7f1136663740 10 client.25931 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.509+0000 7f1136663740 1 -- 172.21.15.102:0/4269184732 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56200eb42560 con 0x56200eb285c0 2024-04-25T01:57:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.509+0000 7f1116a00640 1 -- 172.21.15.102:0/4269184732 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f111c030e60 con 0x56200eb285c0 2024-04-25T01:57:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.510+0000 7f1116a00640 1 -- 172.21.15.102:0/4269184732 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f112402f450 con 0x56200eb285c0 2024-04-25T01:57:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.510+0000 7f1136663740 10 client.25931 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.510+0000 7f1136663740 10 client.25931 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:57:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.510+0000 7f1136663740 20 client.25931 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.510+0000 7f1136663740 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x56200eb43970 0x56200eb45d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.510+0000 7f1136663740 4 client.25931 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.510+0000 7f1136663740 1 -- 172.21.15.102:0/4269184732 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56200ea9aa20 con 0x56200eb43970 2024-04-25T01:57:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.511+0000 7f112be00640 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x56200eb43970 0x56200eb45d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.512+0000 7f112be00640 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x56200eb43970 0x56200eb45d80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b44674e7aa330786 server_cookie=2dea1a6e9132e1d6 in_seq=0 out_seq=0 2024-04-25T01:57:38.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.512+0000 7f1116a00640 10 client.25931 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1116a00640 1 -- 172.21.15.102:0/4269184732 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56200ea9aa20 con 0x56200eb43970 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1116a00640 10 client.25931 handle_command_reply: tid=0 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 client.25931 shutdown 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 -- 172.21.15.102:0/4269184732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb285c0 msgr2=0x56200eb30610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb285c0 0x56200eb30610 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1124010f70 tx=0x7f1124004ab0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 -- 172.21.15.102:0/4269184732 shutdown_connections 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x56200eb43970 0x56200eb45d80 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb335b0 0x56200eb31090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb28ee0 0x56200eb30b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 --2- 172.21.15.102:0/4269184732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb285c0 0x56200eb30610 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 -- 172.21.15.102:0/4269184732 >> 172.21.15.102:0/4269184732 conn(0x56200eb248e0 msgr2=0x56200eb2a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1116a00640 0 client.25931 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:57:38.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.513+0000 7f1136663740 1 -- 172.21.15.102:0/4269184732 shutdown_connections 2024-04-25T01:57:38.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.514+0000 7f1136663740 1 -- 172.21.15.102:0/4269184732 wait complete. 2024-04-25T01:57:38.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.514+0000 7f1136663740 20 client.25931 trim_cache size 0 max 16384 2024-04-25T01:57:38.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.577+0000 7f1136663740 1 Processor -- start 2024-04-25T01:57:38.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.577+0000 7f1136663740 1 -- start start 2024-04-25T01:57:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.577+0000 7f1136663740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.577+0000 7f1136663740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 0x56200eb4b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.577+0000 7f1136663740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 0x56200eb4bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.577+0000 7f1136663740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f112c1b62a0 con 0x56200eb28ee0 2024-04-25T01:57:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.577+0000 7f1136663740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f112c1b3b10 con 0x56200eb335b0 2024-04-25T01:57:38.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.577+0000 7f1136663740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f112c1b1960 con 0x56200eb285c0 2024-04-25T01:57:38.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.578+0000 7f1131800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 0x56200eb4bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.578+0000 7f1131800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 0x56200eb4bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45010/0 (socket says 172.21.15.102:45010) 2024-04-25T01:57:38.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.578+0000 7f1131800640 1 -- 172.21.15.102:0/1430569026 learned_addr learned my addr 172.21.15.102:0/1430569026 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.578+0000 7f1130e00640 1 --2- 172.21.15.102:0/1430569026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.578+0000 7f112be00640 1 --2- 172.21.15.102:0/1430569026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 0x56200eb4b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.579+0000 7f1131800640 1 -- 172.21.15.102:0/1430569026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 msgr2=0x56200eb2ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.579+0000 7f1131800640 1 --2- 172.21.15.102:0/1430569026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.579+0000 7f1131800640 1 -- 172.21.15.102:0/1430569026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 msgr2=0x56200eb4b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.579+0000 7f1131800640 1 --2- 172.21.15.102:0/1430569026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 0x56200eb4b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.579+0000 7f1131800640 1 -- 172.21.15.102:0/1430569026 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56200eb35cd0 con 0x56200eb335b0 2024-04-25T01:57:38.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.579+0000 7f1131800640 1 --2- 172.21.15.102:0/1430569026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 0x56200eb4bbd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f111809b420 tx=0x7f1118004f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d02e62b7ea08323 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.579+0000 7f1121600640 1 -- 172.21.15.102:0/1430569026 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111801c040 con 0x56200eb335b0 2024-04-25T01:57:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.580+0000 7f1121600640 1 -- 172.21.15.102:0/1430569026 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f111809be20 con 0x56200eb335b0 2024-04-25T01:57:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.580+0000 7f1121600640 1 -- 172.21.15.102:0/1430569026 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11180ccbd0 con 0x56200eb335b0 2024-04-25T01:57:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.581+0000 7f1136663740 1 -- 172.21.15.102:0/1430569026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 msgr2=0x56200eb4bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.581+0000 7f1136663740 1 --2- 172.21.15.102:0/1430569026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 0x56200eb4bbd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f111809b420 tx=0x7f1118004f30 comp rx=0 tx=0).stop 2024-04-25T01:57:38.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.581+0000 7f1136663740 1 -- 172.21.15.102:0/1430569026 shutdown_connections 2024-04-25T01:57:38.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.581+0000 7f1136663740 1 --2- 172.21.15.102:0/1430569026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 0x56200eb4bbd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.581+0000 7f1136663740 1 --2- 172.21.15.102:0/1430569026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 0x56200eb4b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.581+0000 7f1136663740 1 --2- 172.21.15.102:0/1430569026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.581+0000 7f1136663740 1 -- 172.21.15.102:0/1430569026 >> 172.21.15.102:0/1430569026 conn(0x56200eb248e0 msgr2=0x56200eb23b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.582+0000 7f1136663740 1 -- 172.21.15.102:0/1430569026 shutdown_connections 2024-04-25T01:57:38.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.582+0000 7f1136663740 1 -- 172.21.15.102:0/1430569026 wait complete. 2024-04-25T01:57:38.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.582+0000 7f1136663740 1 -- start start 2024-04-25T01:57:38.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.582+0000 7f1136663740 1 Processor -- start 2024-04-25T01:57:38.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1136663740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1136663740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 0x56200eb2d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1136663740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 0x56200eb2dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1130e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1130e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44450/0 (socket says 172.21.15.102:44450) 2024-04-25T01:57:38.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1130e00640 1 -- 172.21.15.102:0/3320329485 learned_addr learned my addr 172.21.15.102:0/3320329485 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f112c1b1fd0 con 0x56200eb28ee0 2024-04-25T01:57:38.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f112c1b3440 con 0x56200eb335b0 2024-04-25T01:57:38.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f112c1b26a0 con 0x56200eb285c0 2024-04-25T01:57:38.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f1136663740 10 client.0 _ll_register_callbacks cb 0x7f1124010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.584+0000 7f112be00640 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 0x56200eb2d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.585+0000 7f1130e00640 1 -- 172.21.15.102:0/3320329485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 msgr2=0x56200eb2dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.585+0000 7f1130e00640 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 0x56200eb2dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.585+0000 7f1130e00640 1 -- 172.21.15.102:0/3320329485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 msgr2=0x56200eb2d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.585+0000 7f1130e00640 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 0x56200eb2d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.585+0000 7f1130e00640 1 -- 172.21.15.102:0/3320329485 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56200eba8f00 con 0x56200eb285c0 2024-04-25T01:57:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.585+0000 7f112be00640 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 0x56200eb2d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.585+0000 7f1130e00640 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2d2c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1124000f80 tx=0x7f112400fcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c4623b5d08e3c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.585+0000 7f1116a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:38.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.586+0000 7f1116a00640 1 -- 172.21.15.102:0/3320329485 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112403f060 con 0x56200eb285c0 2024-04-25T01:57:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.586+0000 7f1116a00640 1 -- 172.21.15.102:0/3320329485 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f112402fc40 con 0x56200eb285c0 2024-04-25T01:57:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.586+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f111809be20 con 0x56200eb285c0 2024-04-25T01:57:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.586+0000 7f1116a00640 1 -- 172.21.15.102:0/3320329485 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f112402f660 con 0x56200eb285c0 2024-04-25T01:57:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.587+0000 7f1136663740 10 client.25915 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.587+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56200ebaa190 con 0x56200eb285c0 2024-04-25T01:57:38.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.587+0000 7f1116a00640 1 -- 172.21.15.102:0/3320329485 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f111809be20 con 0x56200eb285c0 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.587+0000 7f1116a00640 1 -- 172.21.15.102:0/3320329485 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1124039070 con 0x56200eb285c0 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.588+0000 7f1136663740 10 client.25915 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.588+0000 7f1136663740 10 client.25915 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.588+0000 7f1136663740 20 client.25915 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.588+0000 7f1136663740 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x56200ebaa850 0x56200ebaccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.588+0000 7f1136663740 4 client.25915 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.588+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56200ea94d70 con 0x56200ebaa850 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.588+0000 7f112be00640 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x56200ebaa850 0x56200ebaccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.589+0000 7f1116a00640 10 client.25915 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.589+0000 7f112be00640 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x56200ebaa850 0x56200ebaccb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=30e82d027225ffb0 server_cookie=8d994904deaf073c in_seq=0 out_seq=0 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.591+0000 7f1116a00640 1 -- 172.21.15.102:0/3320329485 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56200ea94d70 con 0x56200ebaa850 2024-04-25T01:57:38.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.591+0000 7f1116a00640 10 client.25915 handle_command_reply: tid=0 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.591+0000 7f1136663740 1 client.25915 shutdown 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.592+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 msgr2=0x56200eb2d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.592+0000 7f1136663740 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2d2c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1124000f80 tx=0x7f112400fcf0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.592+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 shutdown_connections 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.592+0000 7f1136663740 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x56200ebaa850 0x56200ebaccb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.592+0000 7f1136663740 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56200eb335b0 0x56200eb2dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.592+0000 7f1136663740 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56200eb28ee0 0x56200eb2d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.592+0000 7f1136663740 1 --2- 172.21.15.102:0/3320329485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56200eb285c0 0x56200eb2d2c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.592+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 >> 172.21.15.102:0/3320329485 conn(0x56200eb248e0 msgr2=0x56200eb31e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.592+0000 7f1116a00640 0 client.25915 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.593+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 shutdown_connections 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.593+0000 7f1136663740 1 -- 172.21.15.102:0/3320329485 wait complete. 2024-04-25T01:57:38.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.593+0000 7f1136663740 20 client.25915 trim_cache size 0 max 16384 2024-04-25T01:57:38.603 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:57:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.595+0000 7f1133800640 1 -- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11100a2d80 msgr2=0x7f11100a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.595+0000 7f1133800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11100a2d80 0x7f11100a5240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1124007fb0 tx=0x7f1124007670 comp rx=0 tx=0).stop 2024-04-25T01:57:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.595+0000 7f1133800640 1 -- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f112c10bd00 msgr2=0x7f112c1042b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.595+0000 7f1133800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f112c10bd00 0x7f112c1042b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f111800e710 tx=0x7f111800ebd0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.596+0000 7f1133800640 1 -- 172.21.15.102:0/3908587068 shutdown_connections 2024-04-25T01:57:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.596+0000 7f1133800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11100a2d80 0x7f11100a5240 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.596+0000 7f1133800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f112c10bd00 0x7f112c1042b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.596+0000 7f1133800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f112c1032e0 0x7f112c103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.596+0000 7f1133800640 1 --2- 172.21.15.102:0/3908587068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f112c1029c0 0x7f112c103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.596+0000 7f1133800640 1 -- 172.21.15.102:0/3908587068 >> 172.21.15.102:0/3908587068 conn(0x7f112c077740 msgr2=0x7f112c107bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.596+0000 7f1133800640 1 -- 172.21.15.102:0/3908587068 shutdown_connections 2024-04-25T01:57:38.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.596+0000 7f1133800640 1 -- 172.21.15.102:0/3908587068 wait complete. 2024-04-25T01:57:38.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:57:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.632+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2064883373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8103940 msgr2=0x7fb0c8103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.632+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2064883373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8103940 0x7fb0c8103dc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009900 tx=0x7fb0b402a3f0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.632+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2064883373 shutdown_connections 2024-04-25T01:57:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.632+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2064883373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c8104300 0x7fb0c8110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.632+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2064883373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8103940 0x7fb0c8103dc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.632+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2064883373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c8109950 0x7fb0c8109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.632+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2064883373 >> 172.21.15.102:0/2064883373 conn(0x7fb0c80ff410 msgr2=0x7fb0c8101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.632+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2064883373 shutdown_connections 2024-04-25T01:57:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.632+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2064883373 wait complete. 2024-04-25T01:57:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.634+0000 7fb0ccc00640 1 Processor -- start 2024-04-25T01:57:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.635+0000 7fb0ccc00640 1 -- start start 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0ccc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c8103940 0x7fb0c819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0ccc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c8104300 0x7fb0c819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0ccc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8109950 0x7fb0c819ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0ccc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c8113d80 con 0x7fb0c8104300 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0ccc00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c8113c00 con 0x7fb0c8109950 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0ccc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c8113f00 con 0x7fb0c8103940 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0c6800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8109950 0x7fb0c819ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0c6800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8109950 0x7fb0c819ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45024/0 (socket says 172.21.15.102:45024) 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0c6800640 1 -- 172.21.15.102:0/2231607962 learned_addr learned my addr 172.21.15.102:0/2231607962 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.636+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c8103940 0x7fb0c819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.637+0000 7fb0c6800640 1 -- 172.21.15.102:0/2231607962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c8103940 msgr2=0x7fb0c819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.637+0000 7fb0c6800640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c8103940 0x7fb0c819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.637+0000 7fb0c6800640 1 -- 172.21.15.102:0/2231607962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c8104300 msgr2=0x7fb0c819a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.637+0000 7fb0c6800640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c8104300 0x7fb0c819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.637+0000 7fb0c6800640 1 -- 172.21.15.102:0/2231607962 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c819f120 con 0x7fb0c8109950 2024-04-25T01:57:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.637+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c8103940 0x7fb0c819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.638+0000 7fb0c6800640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8109950 0x7fb0c819ea20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000eb40 tx=0x7fb0c000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.638+0000 7fb0bea00640 1 -- 172.21.15.102:0/2231607962 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0004b20 con 0x7fb0c8109950 2024-04-25T01:57:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.638+0000 7fb0bea00640 1 -- 172.21.15.102:0/2231607962 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0c0004cc0 con 0x7fb0c8109950 2024-04-25T01:57:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.638+0000 7fb0bea00640 1 -- 172.21.15.102:0/2231607962 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0010760 con 0x7fb0c8109950 2024-04-25T01:57:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.638+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2231607962 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c819f410 con 0x7fb0c8109950 2024-04-25T01:57:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.638+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2231607962 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c806b160 con 0x7fb0c8109950 2024-04-25T01:57:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.645+0000 7fb0cfa1f740 1 Processor -- start 2024-04-25T01:57:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.647+0000 7fb0bea00640 1 -- 172.21.15.102:0/2231607962 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb0c0010900 con 0x7fb0c8109950 2024-04-25T01:57:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.647+0000 7fb0cfa1f740 1 -- start start 2024-04-25T01:57:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.648+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601610 0x5650596019f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.648+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x56505960c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.648+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 0x56505960ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.648+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56505955cab0 con 0x565059601610 2024-04-25T01:57:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.648+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x565059505640 con 0x56505960c7f0 2024-04-25T01:57:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.648+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x565059558a70 con 0x565059601f30 2024-04-25T01:57:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.649+0000 7fb0c5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x56505960c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.649+0000 7fb0c5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x56505960c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44486/0 (socket says 172.21.15.102:44486) 2024-04-25T01:57:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.649+0000 7fb0c5e00640 1 -- 172.21.15.102:0/2155718100 learned_addr learned my addr 172.21.15.102:0/2155718100 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.649+0000 7fb0c5400640 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 0x56505960ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.649+0000 7fb0c5400640 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601610 0x5650596019f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.650+0000 7fb0c5e00640 1 -- 172.21.15.102:0/2155718100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 msgr2=0x56505960ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.650+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 0x56505960ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.650+0000 7fb0c5e00640 1 -- 172.21.15.102:0/2155718100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601610 msgr2=0x5650596019f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.650+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601610 0x5650596019f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.650+0000 7fb0c5e00640 1 -- 172.21.15.102:0/2155718100 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565059271210 con 0x565059601f30 2024-04-25T01:57:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.650+0000 7fb0c5400640 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601610 0x5650596019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.650+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x56505960c2b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009cb0 tx=0x7fb0b80117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d1d3ce6e09e2f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.651+0000 7fb0bea00640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb0ac0a2e20 0x7fb0ac0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.651+0000 7fb0b3000640 1 -- 172.21.15.102:0/2155718100 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8012450 con 0x565059601f30 2024-04-25T01:57:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.651+0000 7fb0bea00640 1 -- 172.21.15.102:0/2231607962 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb0c00cfb40 con 0x7fb0c8109950 2024-04-25T01:57:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.651+0000 7fb0b3000640 1 -- 172.21.15.102:0/2155718100 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b8012ad0 con 0x565059601f30 2024-04-25T01:57:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.651+0000 7fb0b3000640 1 -- 172.21.15.102:0/2155718100 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b800fc30 con 0x565059601f30 2024-04-25T01:57:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.651+0000 7fb0c5400640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb0ac0a2e20 0x7fb0ac0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.653+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2155718100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 msgr2=0x56505960c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.654+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x56505960c2b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009cb0 tx=0x7fb0b80117b0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.654+0000 7fb0c5400640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb0ac0a2e20 0x7fb0ac0a52e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009230 tx=0x7fb0b4000bc0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.654+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2155718100 shutdown_connections 2024-04-25T01:57:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.654+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 0x56505960ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.654+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x56505960c2b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.654+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2155718100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601610 0x5650596019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.654+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2155718100 >> 172.21.15.102:0/2155718100 conn(0x5650595fdb20 msgr2=0x5650595fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.655+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2155718100 shutdown_connections 2024-04-25T01:57:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.655+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2155718100 wait complete. 2024-04-25T01:57:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.655+0000 7fb0cfa1f740 1 -- start start 2024-04-25T01:57:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.656+0000 7fb0cfa1f740 1 Processor -- start 2024-04-25T01:57:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.658+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601610 0x565059609720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.658+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601f30 0x565059609c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.658+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 0x56505960a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.659+0000 7fb0c6800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 0x56505960a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.659+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56505960f420 con 0x565059601f30 2024-04-25T01:57:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.659+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x565059518990 con 0x56505960c7f0 2024-04-25T01:57:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.659+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56505960f5a0 con 0x565059601610 2024-04-25T01:57:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.659+0000 7fb0c5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601f30 0x565059609c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.659+0000 7fb0cfa1f740 10 client.0 _ll_register_callbacks cb 0x56505951a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.659+0000 7fb0c5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601f30 0x565059609c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57258/0 (socket says 172.21.15.102:57258) 2024-04-25T01:57:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.659+0000 7fb0c5400640 1 -- 172.21.15.102:0/4248867116 learned_addr learned my addr 172.21.15.102:0/4248867116 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.659+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601610 0x565059609720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.660+0000 7fb0c6800640 1 -- 172.21.15.102:0/4248867116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601610 msgr2=0x565059609720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.660+0000 7fb0c6800640 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601610 0x565059609720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.660+0000 7fb0c6800640 1 -- 172.21.15.102:0/4248867116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601f30 msgr2=0x565059609c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.660+0000 7fb0c6800640 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601f30 0x565059609c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.660+0000 7fb0c6800640 1 -- 172.21.15.102:0/4248867116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56505933d620 con 0x56505960c7f0 2024-04-25T01:57:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.660+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601610 0x565059609720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.660+0000 7fb0c6800640 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 0x56505960a1a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00cd260 tx=0x7fb0c009d890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a80c1cdec921fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.661+0000 7fb0a7400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.661+0000 7fb0a7400640 1 -- 172.21.15.102:0/4248867116 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c00a5e00 con 0x56505960c7f0 2024-04-25T01:57:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.661+0000 7fb0a7400640 1 -- 172.21.15.102:0/4248867116 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb0c00a1170 con 0x56505960c7f0 2024-04-25T01:57:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.661+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/4248867116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b8012ad0 con 0x56505960c7f0 2024-04-25T01:57:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.662+0000 7fb0a7400640 1 -- 172.21.15.102:0/4248867116 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0c00a5970 con 0x56505960c7f0 2024-04-25T01:57:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.662+0000 7fb0cfa1f740 10 client.25946 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.662+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/4248867116 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56505961b2a0 con 0x56505960c7f0 2024-04-25T01:57:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.663+0000 7fb0a7400640 1 -- 172.21.15.102:0/4248867116 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b8012ad0 con 0x56505960c7f0 2024-04-25T01:57:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.663+0000 7fb0a7400640 1 -- 172.21.15.102:0/4248867116 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb0c009c700 con 0x56505960c7f0 2024-04-25T01:57:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.663+0000 7fb0cfa1f740 10 client.25946 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.663+0000 7fb0cfa1f740 10 client.25946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.663+0000 7fb0cfa1f740 20 client.25946 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.663+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56505961c900 0x56505961ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.663+0000 7fb0cfa1f740 4 client.25946 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.663+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/4248867116 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565059569ff0 con 0x56505961c900 2024-04-25T01:57:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.664+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56505961c900 0x56505961ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.668 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.665+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56505961c900 0x56505961ecf0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa81ac4ba4a92c69 server_cookie=ab416443f738849 in_seq=0 out_seq=0 2024-04-25T01:57:38.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.665+0000 7fb0a7400640 10 client.25946 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:38.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.666+0000 7fb0a7400640 1 -- 172.21.15.102:0/4248867116 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565059569ff0 con 0x56505961c900 2024-04-25T01:57:38.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.666+0000 7fb0a7400640 10 client.25946 handle_command_reply: tid=0 2024-04-25T01:57:38.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.666+0000 7fb0cfa1f740 1 client.25946 shutdown 2024-04-25T01:57:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.666+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/4248867116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 msgr2=0x56505960a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.666+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 0x56505960a1a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00cd260 tx=0x7fb0c009d890 comp rx=0 tx=0).stop 2024-04-25T01:57:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.667+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/4248867116 shutdown_connections 2024-04-25T01:57:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.667+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56505961c900 0x56505961ecf0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.667+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56505960c7f0 0x56505960a1a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.667+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x565059601f30 0x565059609c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.667+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/4248867116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601610 0x565059609720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.667+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/4248867116 >> 172.21.15.102:0/4248867116 conn(0x5650595fdb20 msgr2=0x5650595f4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.667+0000 7fb0a7400640 0 client.25946 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.669+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/4248867116 shutdown_connections 2024-04-25T01:57:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.669+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/4248867116 wait complete. 2024-04-25T01:57:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.669+0000 7fb0cfa1f740 20 client.25946 trim_cache size 0 max 16384 2024-04-25T01:57:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.702+0000 7fb0cfa1f740 1 Processor -- start 2024-04-25T01:57:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.703+0000 7fb0cfa1f740 1 -- start start 2024-04-25T01:57:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.703+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059618310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.703+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x565059624570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.703+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 0x565059624ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.703+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a8010ff0 con 0x56505960c7f0 2024-04-25T01:57:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.703+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800e860 con 0x565059601610 2024-04-25T01:57:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.703+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800c6b0 con 0x565059601f30 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c5e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059618310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c5e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059618310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45056/0 (socket says 172.21.15.102:45056) 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c5e00640 1 -- 172.21.15.102:0/1750160068 learned_addr learned my addr 172.21.15.102:0/1750160068 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c6800640 1 --2- 172.21.15.102:0/1750160068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 0x565059624ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c5400640 1 --2- 172.21.15.102:0/1750160068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x565059624570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c5e00640 1 -- 172.21.15.102:0/1750160068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 msgr2=0x565059624570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/1750160068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x565059624570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c5e00640 1 -- 172.21.15.102:0/1750160068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 msgr2=0x565059624ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/1750160068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 0x565059624ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.704+0000 7fb0c5e00640 1 -- 172.21.15.102:0/1750160068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650596051e0 con 0x565059601610 2024-04-25T01:57:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.705+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/1750160068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059618310 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8012fd0 tx=0x7fb0b800f7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47957b72af62969f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.705+0000 7fb0b2600640 1 -- 172.21.15.102:0/1750160068 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8004020 con 0x565059601610 2024-04-25T01:57:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.705+0000 7fb0b2600640 1 -- 172.21.15.102:0/1750160068 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b800fd70 con 0x565059601610 2024-04-25T01:57:38.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.705+0000 7fb0b2600640 1 -- 172.21.15.102:0/1750160068 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8012740 con 0x565059601610 2024-04-25T01:57:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.706+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/1750160068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 msgr2=0x565059618310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.706+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/1750160068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059618310 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8012fd0 tx=0x7fb0b800f7d0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.706+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/1750160068 shutdown_connections 2024-04-25T01:57:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.707+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/1750160068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 0x565059624ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.707+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/1750160068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x565059624570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.707+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/1750160068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059618310 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.707+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/1750160068 >> 172.21.15.102:0/1750160068 conn(0x5650595fdb20 msgr2=0x565059604ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.707+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/1750160068 shutdown_connections 2024-04-25T01:57:38.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.707+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/1750160068 wait complete. 2024-04-25T01:57:38.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.708+0000 7fb0cfa1f740 1 -- start start 2024-04-25T01:57:38.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.709+0000 7fb0cfa1f740 1 Processor -- start 2024-04-25T01:57:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059614820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x565059614d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0cfa1f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 0x565059605f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800cd20 con 0x56505960c7f0 2024-04-25T01:57:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800e190 con 0x565059601610 2024-04-25T01:57:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0cfa1f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a800d3f0 con 0x565059601f30 2024-04-25T01:57:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0cfa1f740 10 client.0 _ll_register_callbacks cb 0x7fb0c0004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0c6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 0x565059605f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0c6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 0x565059605f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57280/0 (socket says 172.21.15.102:57280) 2024-04-25T01:57:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.710+0000 7fb0c6800640 1 -- 172.21.15.102:0/2686828621 learned_addr learned my addr 172.21.15.102:0/2686828621 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.711+0000 7fb0c5400640 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x565059614d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.711+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059614820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.711+0000 7fb0c5e00640 1 -- 172.21.15.102:0/2686828621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 msgr2=0x565059614d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.711+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x565059614d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.711+0000 7fb0c5e00640 1 -- 172.21.15.102:0/2686828621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 msgr2=0x565059605f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.711+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 0x565059605f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.711+0000 7fb0c5e00640 1 -- 172.21.15.102:0/2686828621 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565059606710 con 0x565059601610 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.711+0000 7fb0c5e00640 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059614820 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80095f0 tx=0x7fb0b8009320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea51305ac73a0c25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.711+0000 7fb0a7400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.712+0000 7fb0a7400640 1 -- 172.21.15.102:0/2686828621 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8004020 con 0x565059601610 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.712+0000 7fb0a7400640 1 -- 172.21.15.102:0/2686828621 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb0b80041c0 con 0x565059601610 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.712+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2686828621 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b800fd70 con 0x565059601610 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.712+0000 7fb0a7400640 1 -- 172.21.15.102:0/2686828621 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0b800f840 con 0x565059601610 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.712+0000 7fb0cfa1f740 10 client.25952 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.712+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2686828621 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565059682a10 con 0x565059601610 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.713+0000 7fb0a7400640 1 -- 172.21.15.102:0/2686828621 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b800fd70 con 0x565059601610 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.713+0000 7fb0a7400640 1 -- 172.21.15.102:0/2686828621 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb0b802d070 con 0x565059601610 2024-04-25T01:57:38.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.713+0000 7fb0cfa1f740 10 client.25952 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.713+0000 7fb0cfa1f740 10 client.25952 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.713+0000 7fb0cfa1f740 20 client.25952 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.713+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x565059682e80 0x565059685270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.713+0000 7fb0cfa1f740 4 client.25952 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:57:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.714+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2686828621 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565059564340 con 0x565059682e80 2024-04-25T01:57:38.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.714+0000 7fb0c5400640 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x565059682e80 0x565059685270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.714+0000 7fb0a7400640 10 client.25952 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:38.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.715+0000 7fb0c5400640 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x565059682e80 0x565059685270 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1e7b07cfc4cc567 server_cookie=34aad2aea330eb43 in_seq=0 out_seq=0 2024-04-25T01:57:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.730+0000 7fb0a7400640 1 -- 172.21.15.102:0/2686828621 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565059564340 con 0x565059682e80 2024-04-25T01:57:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.730+0000 7fb0a7400640 10 client.25952 handle_command_reply: tid=0 2024-04-25T01:57:38.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.730+0000 7fb0cfa1f740 1 client.25952 shutdown 2024-04-25T01:57:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.730+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2686828621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 msgr2=0x565059614820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.730+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059614820 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80095f0 tx=0x7fb0b8009320 comp rx=0 tx=0).stop 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.731+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2686828621 shutdown_connections 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.731+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x565059682e80 0x565059685270 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.731+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56505960c7f0 0x565059605f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.731+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565059601f30 0x565059614d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.731+0000 7fb0cfa1f740 1 --2- 172.21.15.102:0/2686828621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x565059601610 0x565059614820 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.731+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2686828621 >> 172.21.15.102:0/2686828621 conn(0x5650595fdb20 msgr2=0x5650595f3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.731+0000 7fb0a7400640 0 client.25952 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.732+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2686828621 shutdown_connections 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.732+0000 7fb0cfa1f740 1 -- 172.21.15.102:0/2686828621 wait complete. 2024-04-25T01:57:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.732+0000 7fb0cfa1f740 20 client.25952 trim_cache size 0 max 16384 2024-04-25T01:57:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:57:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:57:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:57:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.733+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2231607962 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb0ac0a2e20 msgr2=0x7fb0ac0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.733+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb0ac0a2e20 0x7fb0ac0a52e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009230 tx=0x7fb0b4000bc0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.733+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2231607962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8109950 msgr2=0x7fb0c819ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.733+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8109950 0x7fb0c819ea20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb0c000eb40 tx=0x7fb0c000c2d0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.734+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2231607962 shutdown_connections 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.734+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb0ac0a2e20 0x7fb0ac0a52e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.734+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0c8109950 0x7fb0c819ea20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.734+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c8104300 0x7fb0c819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.734+0000 7fb0ccc00640 1 --2- 172.21.15.102:0/2231607962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0c8103940 0x7fb0c819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.734+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2231607962 >> 172.21.15.102:0/2231607962 conn(0x7fb0c80ff410 msgr2=0x7fb0c80ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.734+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2231607962 shutdown_connections 2024-04-25T01:57:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:38.734+0000 7fb0ccc00640 1 -- 172.21.15.102:0/2231607962 wait complete. 2024-04-25T01:57:38.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.913+0000 7f718fa00640 1 -- 172.21.15.102:0/4243862581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 msgr2=0x7f7188100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.913+0000 7f718fa00640 1 --2- 172.21.15.102:0/4243862581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 0x7f7188100b90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7178009900 tx=0x7f717802a510 comp rx=0 tx=0).stop 2024-04-25T01:57:38.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.914+0000 7f718fa00640 1 -- 172.21.15.102:0/4243862581 shutdown_connections 2024-04-25T01:57:38.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.914+0000 7f718fa00640 1 --2- 172.21.15.102:0/4243862581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7188105ad0 0x7f7188105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.914+0000 7f718fa00640 1 --2- 172.21.15.102:0/4243862581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71881010d0 0x7f7188105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.914+0000 7f718fa00640 1 --2- 172.21.15.102:0/4243862581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 0x7f7188100b90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.914+0000 7f718fa00640 1 -- 172.21.15.102:0/4243862581 >> 172.21.15.102:0/4243862581 conn(0x7f7188077740 msgr2=0x7f7188077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.914+0000 7f718fa00640 1 -- 172.21.15.102:0/4243862581 shutdown_connections 2024-04-25T01:57:38.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.914+0000 7f718fa00640 1 -- 172.21.15.102:0/4243862581 wait complete. 2024-04-25T01:57:38.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.915+0000 7f718fa00640 1 Processor -- start 2024-04-25T01:57:38.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.915+0000 7f718fa00640 1 -- start start 2024-04-25T01:57:38.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.916+0000 7f718fa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 0x7f7188114900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.916+0000 7f718fa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71881010d0 0x7f7188114e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.916+0000 7f718fa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7188105ad0 0x7f718810db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.916+0000 7f718fa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7188118140 con 0x7f7188105ad0 2024-04-25T01:57:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.916+0000 7f718fa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7188117fc0 con 0x7f71881007b0 2024-04-25T01:57:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.916+0000 7f718fa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f71881182c0 con 0x7f71881010d0 2024-04-25T01:57:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.917+0000 7f718d000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 0x7f7188114900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.917+0000 7f718d000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 0x7f7188114900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45086/0 (socket says 172.21.15.102:45086) 2024-04-25T01:57:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.917+0000 7f718d000640 1 -- 172.21.15.102:0/1146478105 learned_addr learned my addr 172.21.15.102:0/1146478105 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.917+0000 7f717fe00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71881010d0 0x7f7188114e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.917+0000 7f718d000640 1 -- 172.21.15.102:0/1146478105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71881010d0 msgr2=0x7f7188114e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.917+0000 7f718d000640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71881010d0 0x7f7188114e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.917+0000 7f718d000640 1 -- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7188105ad0 msgr2=0x7f718810db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.917+0000 7f718da00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7188105ad0 0x7f718810db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.918+0000 7f718d000640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7188105ad0 0x7f718810db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.918+0000 7f718d000640 1 -- 172.21.15.102:0/1146478105 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718810e400 con 0x7f71881007b0 2024-04-25T01:57:38.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.918+0000 7f717fe00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71881010d0 0x7f7188114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:38.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.918+0000 7f718da00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7188105ad0 0x7f718810db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:38.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.918+0000 7f718d000640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 0x7f7188114900 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7178004770 tx=0x7f7178003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.919+0000 7f7186000640 1 -- 172.21.15.102:0/1146478105 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717803f070 con 0x7f71881007b0 2024-04-25T01:57:38.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.919+0000 7f7186000640 1 -- 172.21.15.102:0/1146478105 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7178031ca0 con 0x7f71881007b0 2024-04-25T01:57:38.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.919+0000 7f7186000640 1 -- 172.21.15.102:0/1146478105 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178035030 con 0x7f71881007b0 2024-04-25T01:57:38.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.919+0000 7f718fa00640 1 -- 172.21.15.102:0/1146478105 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718810e690 con 0x7f71881007b0 2024-04-25T01:57:38.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.920+0000 7f718fa00640 1 -- 172.21.15.102:0/1146478105 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7188112600 con 0x7f71881007b0 2024-04-25T01:57:38.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.925+0000 7f7186000640 1 -- 172.21.15.102:0/1146478105 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7178042070 con 0x7f71881007b0 2024-04-25T01:57:38.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.929+0000 7f7186000640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f716c0a2d80 0x7f716c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.929+0000 7f719281d740 1 Processor -- start 2024-04-25T01:57:38.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.929+0000 7f719281d740 1 -- start start 2024-04-25T01:57:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.930+0000 7f717fe00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f716c0a2d80 0x7f716c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.930+0000 7f719281d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8dd5c0 0x562f0a8dd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.930+0000 7f719281d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 0x562f0a8e8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.930+0000 7f719281d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8e85b0 0x562f0a8ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.930+0000 7f718d000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8dd5c0 0x562f0a8dd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.930+0000 7f718d000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8dd5c0 0x562f0a8dd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45094/0 (socket says 172.21.15.102:45094) 2024-04-25T01:57:38.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f718d000640 1 -- 172.21.15.102:0/1724482462 learned_addr learned my addr 172.21.15.102:0/1724482462 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f719281d740 1 -- 172.21.15.102:0/1724482462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x562f0a8424e0 con 0x562f0a8ddee0 2024-04-25T01:57:38.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f719281d740 1 -- 172.21.15.102:0/1724482462 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562f0a7eaa70 con 0x562f0a8dd5c0 2024-04-25T01:57:38.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f719281d740 1 -- 172.21.15.102:0/1724482462 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562f0a83e4a0 con 0x562f0a8e85b0 2024-04-25T01:57:38.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f718da00640 1 --2- 172.21.15.102:0/1724482462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8e85b0 0x562f0a8ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f718d000640 1 -- 172.21.15.102:0/1724482462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8e85b0 msgr2=0x562f0a8ea9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f718d000640 1 --2- 172.21.15.102:0/1724482462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8e85b0 0x562f0a8ea9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f718d000640 1 -- 172.21.15.102:0/1724482462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 msgr2=0x562f0a8e8070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:38.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f718d000640 1 --2- 172.21.15.102:0/1724482462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 0x562f0a8e8070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f718d000640 1 -- 172.21.15.102:0/1724482462 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f0a5563f0 con 0x562f0a8dd5c0 2024-04-25T01:57:38.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.931+0000 7f718da00640 1 --2- 172.21.15.102:0/1724482462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8e85b0 0x562f0a8ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:38.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.932+0000 7f718d000640 1 --2- 172.21.15.102:0/1724482462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8dd5c0 0x562f0a8dd9a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f71780bfaf0 tx=0x7f71780e66d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fb7430f724cc05a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.932+0000 7f7186000640 1 -- 172.21.15.102:0/1146478105 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f71780ef5f0 con 0x7f71881007b0 2024-04-25T01:57:38.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.932+0000 7f717e000640 1 -- 172.21.15.102:0/1724482462 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178005d50 con 0x562f0a8dd5c0 2024-04-25T01:57:38.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.933+0000 7f717e000640 1 -- 172.21.15.102:0/1724482462 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71780bc300 con 0x562f0a8dd5c0 2024-04-25T01:57:38.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.933+0000 7f717e000640 1 -- 172.21.15.102:0/1724482462 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71780c4870 con 0x562f0a8dd5c0 2024-04-25T01:57:38.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f719281d740 1 -- 172.21.15.102:0/1724482462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8dd5c0 msgr2=0x562f0a8dd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f717fe00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f716c0a2d80 0x7f716c0a5240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f718810f140 tx=0x7f7170005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f719281d740 1 --2- 172.21.15.102:0/1724482462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8dd5c0 0x562f0a8dd9a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f71780bfaf0 tx=0x7f71780e66d0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f719281d740 1 -- 172.21.15.102:0/1724482462 shutdown_connections 2024-04-25T01:57:38.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f719281d740 1 --2- 172.21.15.102:0/1724482462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8e85b0 0x562f0a8ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f719281d740 1 --2- 172.21.15.102:0/1724482462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 0x562f0a8e8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f719281d740 1 --2- 172.21.15.102:0/1724482462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8dd5c0 0x562f0a8dd9a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f719281d740 1 -- 172.21.15.102:0/1724482462 >> 172.21.15.102:0/1724482462 conn(0x562f0a8d98e0 msgr2=0x562f0a8d9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f719281d740 1 -- 172.21.15.102:0/1724482462 shutdown_connections 2024-04-25T01:57:38.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.934+0000 7f719281d740 1 -- 172.21.15.102:0/1724482462 wait complete. 2024-04-25T01:57:38.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.935+0000 7f719281d740 1 -- start start 2024-04-25T01:57:38.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.935+0000 7f719281d740 1 Processor -- start 2024-04-25T01:57:38.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f719281d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8e5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f719281d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 0x562f0a8e5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f719281d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a8e5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f719281d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x562f0a8eb160 con 0x562f0a8ddee0 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f719281d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562f0a7fdec0 con 0x562f0a8e85b0 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f719281d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562f0a8eb2e0 con 0x562f0a8dd5c0 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f719281d740 10 client.0 _ll_register_callbacks cb 0x562f0a83a3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f718da00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a8e5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f718da00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a8e5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45108/0 (socket says 172.21.15.102:45108) 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f718da00640 1 -- 172.21.15.102:0/2211815582 learned_addr learned my addr 172.21.15.102:0/2211815582 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.936+0000 7f718d000640 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8e5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.937+0000 7f718da00640 1 -- 172.21.15.102:0/2211815582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 msgr2=0x562f0a8e5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.937+0000 7f718da00640 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8e5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.937+0000 7f718da00640 1 -- 172.21.15.102:0/2211815582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 msgr2=0x562f0a8e5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.937+0000 7f718da00640 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 0x562f0a8e5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.937+0000 7f718da00640 1 -- 172.21.15.102:0/2211815582 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f0a6207d0 con 0x562f0a8e85b0 2024-04-25T01:57:38.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.937+0000 7f718d000640 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8e5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:38.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.937+0000 7f718da00640 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a8e5fd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7180012700 tx=0x7f7180012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e984efa0872ce03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.937+0000 7f7176a00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:38.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.938+0000 7f7176a00640 1 -- 172.21.15.102:0/2211815582 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180010cc0 con 0x562f0a8e85b0 2024-04-25T01:57:38.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.938+0000 7f7176a00640 1 -- 172.21.15.102:0/2211815582 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7180003970 con 0x562f0a8e85b0 2024-04-25T01:57:38.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.938+0000 7f719281d740 1 -- 172.21.15.102:0/2211815582 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71780bc300 con 0x562f0a8e85b0 2024-04-25T01:57:38.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.939+0000 7f7176a00640 1 -- 172.21.15.102:0/2211815582 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7180010e60 con 0x562f0a8e85b0 2024-04-25T01:57:38.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.939+0000 7f719281d740 10 client.25964 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:38.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.939+0000 7f719281d740 1 -- 172.21.15.102:0/2211815582 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562f0a8f71e0 con 0x562f0a8e85b0 2024-04-25T01:57:38.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.939+0000 7f7176a00640 1 -- 172.21.15.102:0/2211815582 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71780bc300 con 0x562f0a8e85b0 2024-04-25T01:57:38.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.940+0000 7f7176a00640 1 -- 172.21.15.102:0/2211815582 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7180019070 con 0x562f0a8e85b0 2024-04-25T01:57:38.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.940+0000 7f719281d740 10 client.25964 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:38.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.940+0000 7f719281d740 10 client.25964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:38.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.940+0000 7f719281d740 20 client.25964 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:38.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.940+0000 7f719281d740 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562f0a8f8740 0x562f0a8faba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.940+0000 7f719281d740 4 client.25964 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:38.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.940+0000 7f719281d740 1 -- 172.21.15.102:0/2211815582 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f0a84fa20 con 0x562f0a8f8740 2024-04-25T01:57:38.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.940+0000 7f718d000640 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562f0a8f8740 0x562f0a8faba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.941+0000 7f718d000640 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562f0a8f8740 0x562f0a8faba0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53bd0967f0764c8e server_cookie=bd9ab219f7dd9931 in_seq=0 out_seq=0 2024-04-25T01:57:38.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.941+0000 7f7176a00640 10 client.25964 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:38.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.942+0000 7f7176a00640 1 -- 172.21.15.102:0/2211815582 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f0a84fa20 con 0x562f0a8f8740 2024-04-25T01:57:38.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.942+0000 7f7176a00640 10 client.25964 handle_command_reply: tid=0 2024-04-25T01:57:38.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.942+0000 7f719281d740 1 client.25964 shutdown 2024-04-25T01:57:38.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.943+0000 7f719281d740 1 -- 172.21.15.102:0/2211815582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 msgr2=0x562f0a8e5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.943+0000 7f719281d740 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a8e5fd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7180012700 tx=0x7f7180012bc0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f719281d740 1 -- 172.21.15.102:0/2211815582 shutdown_connections 2024-04-25T01:57:38.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f719281d740 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562f0a8f8740 0x562f0a8faba0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f719281d740 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a8e5fd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f719281d740 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 0x562f0a8e5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f719281d740 1 --2- 172.21.15.102:0/2211815582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8e5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f719281d740 1 -- 172.21.15.102:0/2211815582 >> 172.21.15.102:0/2211815582 conn(0x562f0a8d98e0 msgr2=0x562f0a8d8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f7176a00640 0 client.25964 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:38.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f719281d740 1 -- 172.21.15.102:0/2211815582 shutdown_connections 2024-04-25T01:57:38.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f719281d740 1 -- 172.21.15.102:0/2211815582 wait complete. 2024-04-25T01:57:38.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.944+0000 7f719281d740 20 client.25964 trim_cache size 0 max 16384 2024-04-25T01:57:38.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.972+0000 7f719281d740 1 Processor -- start 2024-04-25T01:57:38.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.973+0000 7f719281d740 1 -- start start 2024-04-25T01:57:38.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.974+0000 7f719281d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.974+0000 7f719281d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 0x562f0a900470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.974+0000 7f719281d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a9009b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.974+0000 7f719281d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f71881bebb0 con 0x562f0a8ddee0 2024-04-25T01:57:38.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.974+0000 7f719281d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f71881bc420 con 0x562f0a8e85b0 2024-04-25T01:57:38.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.974+0000 7f719281d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f71881ba270 con 0x562f0a8dd5c0 2024-04-25T01:57:38.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.975+0000 7f718d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.975+0000 7f718d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44588/0 (socket says 172.21.15.102:44588) 2024-04-25T01:57:38.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.975+0000 7f718d000640 1 -- 172.21.15.102:0/1198206933 learned_addr learned my addr 172.21.15.102:0/1198206933 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.976+0000 7f718d000640 1 -- 172.21.15.102:0/1198206933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 msgr2=0x562f0a9009b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.976+0000 7f718da00640 1 --2- 172.21.15.102:0/1198206933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a9009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.976+0000 7f718d000640 1 --2- 172.21.15.102:0/1198206933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a9009b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.976+0000 7f718d000640 1 -- 172.21.15.102:0/1198206933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 msgr2=0x562f0a900470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.977+0000 7f718d000640 1 --2- 172.21.15.102:0/1198206933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 0x562f0a900470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.977+0000 7f718d000640 1 -- 172.21.15.102:0/1198206933 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f0a8de3e0 con 0x562f0a8dd5c0 2024-04-25T01:57:38.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.977+0000 7f718da00640 1 --2- 172.21.15.102:0/1198206933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a9009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:38.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.978+0000 7f718d000640 1 --2- 172.21.15.102:0/1198206933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f3f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f71780bbb90 tx=0x7f71780c26d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88574dc4160a5fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.979+0000 7f717d600640 1 -- 172.21.15.102:0/1198206933 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178049040 con 0x562f0a8dd5c0 2024-04-25T01:57:38.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.979+0000 7f717d600640 1 -- 172.21.15.102:0/1198206933 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7178031e50 con 0x562f0a8dd5c0 2024-04-25T01:57:38.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.979+0000 7f717d600640 1 -- 172.21.15.102:0/1198206933 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71780bffb0 con 0x562f0a8dd5c0 2024-04-25T01:57:38.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.981+0000 7f719281d740 1 -- 172.21.15.102:0/1198206933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 msgr2=0x562f0a8f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.981+0000 7f719281d740 1 --2- 172.21.15.102:0/1198206933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f3f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f71780bbb90 tx=0x7f71780c26d0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.981+0000 7f719281d740 1 -- 172.21.15.102:0/1198206933 shutdown_connections 2024-04-25T01:57:38.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.981+0000 7f719281d740 1 --2- 172.21.15.102:0/1198206933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8e85b0 0x562f0a9009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.981+0000 7f719281d740 1 --2- 172.21.15.102:0/1198206933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8ddee0 0x562f0a900470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.981+0000 7f719281d740 1 --2- 172.21.15.102:0/1198206933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f3f90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.981+0000 7f719281d740 1 -- 172.21.15.102:0/1198206933 >> 172.21.15.102:0/1198206933 conn(0x562f0a8d98e0 msgr2=0x562f0a8df5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:38.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.984+0000 7f719281d740 1 -- 172.21.15.102:0/1198206933 shutdown_connections 2024-04-25T01:57:38.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.984+0000 7f719281d740 1 -- 172.21.15.102:0/1198206933 wait complete. 2024-04-25T01:57:38.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.984+0000 7f719281d740 1 -- start start 2024-04-25T01:57:38.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.984+0000 7f719281d740 1 Processor -- start 2024-04-25T01:57:38.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.985+0000 7f719281d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.986+0000 7f719281d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8ddee0 0x562f0a8ebc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.986+0000 7f719281d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8e85b0 0x562f0a8ec160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.986+0000 7f718d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.986+0000 7f717fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8ddee0 0x562f0a8ebc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.986+0000 7f718d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44600/0 (socket says 172.21.15.102:44600) 2024-04-25T01:57:38.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.986+0000 7f717fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8ddee0 0x562f0a8ebc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45120/0 (socket says 172.21.15.102:45120) 2024-04-25T01:57:38.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.986+0000 7f718d000640 1 -- 172.21.15.102:0/2334987070 learned_addr learned my addr 172.21.15.102:0/2334987070 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:38.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.987+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f71881ba8e0 con 0x562f0a8e85b0 2024-04-25T01:57:38.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.987+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f71881bbd50 con 0x562f0a8ddee0 2024-04-25T01:57:38.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.987+0000 7f718da00640 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8e85b0 0x562f0a8ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.987+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f71881bafb0 con 0x562f0a8dd5c0 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.988+0000 7f718d000640 1 -- 172.21.15.102:0/2334987070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8ddee0 msgr2=0x562f0a8ebc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.988+0000 7f718d000640 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8ddee0 0x562f0a8ebc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.988+0000 7f718d000640 1 -- 172.21.15.102:0/2334987070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8e85b0 msgr2=0x562f0a8ec160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.988+0000 7f718d000640 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8e85b0 0x562f0a8ec160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.988+0000 7f718d000640 1 -- 172.21.15.102:0/2334987070 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f0a8ec940 con 0x562f0a8dd5c0 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.988+0000 7f717fe00640 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8ddee0 0x562f0a8ebc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.988+0000 7f718da00640 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8e85b0 0x562f0a8ec160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.989+0000 7f718d000640 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f0860 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f71780ed370 tx=0x7f717802a510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c78d1d31e4be63a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.989+0000 7f7176a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:38.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.989+0000 7f7176a00640 1 -- 172.21.15.102:0/2334987070 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178049070 con 0x562f0a8dd5c0 2024-04-25T01:57:38.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.989+0000 7f7176a00640 1 -- 172.21.15.102:0/2334987070 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f71780b8270 con 0x562f0a8dd5c0 2024-04-25T01:57:38.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.989+0000 7f7176a00640 10 client.0 inactive, discarding osd_map(90..90 src has 1..90) 2024-04-25T01:57:38.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.989+0000 7f719281d740 10 client.0 _ll_register_callbacks cb 0x7f7180009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:38.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.989+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7178031e50 con 0x562f0a8dd5c0 2024-04-25T01:57:38.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.990+0000 7f7176a00640 1 -- 172.21.15.102:0/2334987070 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f717808f230 con 0x562f0a8dd5c0 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.990+0000 7f719281d740 10 client.25948 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.990+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=27}) -- 0x562f0a95ea10 con 0x562f0a8dd5c0 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.991+0000 7f7176a00640 1 -- 172.21.15.102:0/2334987070 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f717803b4f0 con 0x562f0a8dd5c0 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.991+0000 7f719281d740 10 client.25948 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.991+0000 7f719281d740 10 client.25948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.991+0000 7f719281d740 20 client.25948 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.991+0000 7f719281d740 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562f0a95ed40 0x562f0a9611a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.991+0000 7f719281d740 4 client.25948 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.991+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562f0a849d70 con 0x562f0a95ed40 2024-04-25T01:57:38.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.991+0000 7f717fe00640 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562f0a95ed40 0x562f0a9611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:38.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.992+0000 7f7176a00640 10 client.25948 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:38.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.992+0000 7f717fe00640 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562f0a95ed40 0x562f0a9611a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ff47d3c78d9f1e server_cookie=ca833c3331c0dff5 in_seq=0 out_seq=0 2024-04-25T01:57:38.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.993+0000 7f7176a00640 1 -- 172.21.15.102:0/2334987070 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562f0a849d70 con 0x562f0a95ed40 2024-04-25T01:57:38.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.993+0000 7f7176a00640 10 client.25948 handle_command_reply: tid=0 2024-04-25T01:57:38.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.993+0000 7f719281d740 1 client.25948 shutdown 2024-04-25T01:57:38.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.994+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 msgr2=0x562f0a8f0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:38.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.994+0000 7f719281d740 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f0860 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f71780ed370 tx=0x7f717802a510 comp rx=0 tx=0).stop 2024-04-25T01:57:38.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.995+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 shutdown_connections 2024-04-25T01:57:38.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.995+0000 7f719281d740 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562f0a95ed40 0x562f0a9611a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.995+0000 7f719281d740 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562f0a8e85b0 0x562f0a8ec160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.995+0000 7f719281d740 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562f0a8ddee0 0x562f0a8ebc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.995+0000 7f719281d740 1 --2- 172.21.15.102:0/2334987070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f0a8dd5c0 0x562f0a8f0860 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:38.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.995+0000 7f7176a00640 0 client.25948 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:38.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.995+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 >> 172.21.15.102:0/2334987070 conn(0x562f0a8d98e0 msgr2=0x562f0a8e5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:39.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.996+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 shutdown_connections 2024-04-25T01:57:39.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.996+0000 7f719281d740 1 -- 172.21.15.102:0/2334987070 wait complete. 2024-04-25T01:57:39.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.997+0000 7f719281d740 20 client.25948 trim_cache size 0 max 16384 2024-04-25T01:57:39.001 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 -- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f716c0a2d80 msgr2=0x7f716c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f716c0a2d80 0x7f716c0a5240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f718810f140 tx=0x7f7170005f90 comp rx=0 tx=0).stop 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 -- 172.21.15.102:0/1146478105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 msgr2=0x7f7188114900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 0x7f7188114900 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7178004770 tx=0x7f7178003c10 comp rx=0 tx=0).stop 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 -- 172.21.15.102:0/1146478105 shutdown_connections 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f716c0a2d80 0x7f716c0a5240 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7188105ad0 0x7f718810db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71881010d0 0x7f7188114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 --2- 172.21.15.102:0/1146478105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71881007b0 0x7f7188114900 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:39.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.998+0000 7f718fa00640 1 -- 172.21.15.102:0/1146478105 >> 172.21.15.102:0/1146478105 conn(0x7f7188077740 msgr2=0x7f71880fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.999+0000 7f718fa00640 1 -- 172.21.15.102:0/1146478105 shutdown_connections 2024-04-25T01:57:39.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:38.999+0000 7f718fa00640 1 -- 172.21.15.102:0/1146478105 wait complete. 2024-04-25T01:57:39.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:38 smithi102 ceph-mon[27796]: pgmap v74: 97 pgs: 97 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 12 MiB/s wr, 1.01k op/s 2024-04-25T01:57:39.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:38 smithi110 ceph-mon[32848]: pgmap v74: 97 pgs: 97 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 12 MiB/s wr, 1.01k op/s 2024-04-25T01:57:39.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:38 smithi193 ceph-mon[32981]: pgmap v74: 97 pgs: 97 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 12 MiB/s wr, 1.01k op/s 2024-04-25T01:57:40.031 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:57:40.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:57:40.031 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:57:40.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.320+0000 7f17cbe00640 1 -- 172.21.15.102:0/1745208555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c4107fc0 msgr2=0x7f17c41083a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.320+0000 7f17cbe00640 1 --2- 172.21.15.102:0/1745208555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c4107fc0 0x7f17c41083a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009900 tx=0x7f17b802a400 comp rx=0 tx=0).stop 2024-04-25T01:57:40.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.321+0000 7f17cbe00640 1 -- 172.21.15.102:0/1745208555 shutdown_connections 2024-04-25T01:57:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.321+0000 7f17cbe00640 1 --2- 172.21.15.102:0/1745208555 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17c410d040 0x7f17c4114c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.321+0000 7f17cbe00640 1 --2- 172.21.15.102:0/1745208555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17c4108970 0x7f17c410c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.321+0000 7f17cbe00640 1 --2- 172.21.15.102:0/1745208555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c4107fc0 0x7f17c41083a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.321+0000 7f17cbe00640 1 -- 172.21.15.102:0/1745208555 >> 172.21.15.102:0/1745208555 conn(0x7f17c4075c80 msgr2=0x7f17c40780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.321+0000 7f17cbe00640 1 -- 172.21.15.102:0/1745208555 shutdown_connections 2024-04-25T01:57:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.322+0000 7f17cbe00640 1 -- 172.21.15.102:0/1745208555 wait complete. 2024-04-25T01:57:40.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.323+0000 7f17cbe00640 1 Processor -- start 2024-04-25T01:57:40.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.323+0000 7f17cbe00640 1 -- start start 2024-04-25T01:57:40.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.324+0000 7f17cbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17c4107fc0 0x7f17c419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.324+0000 7f17cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17c410d040 0x7f17c419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.324+0000 7f17cbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c41a2ee0 0x7f17c41a3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.324+0000 7f17cbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4118120 con 0x7f17c410d040 2024-04-25T01:57:40.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.324+0000 7f17cbe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4117fa0 con 0x7f17c4107fc0 2024-04-25T01:57:40.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.324+0000 7f17cbe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17c41182a0 con 0x7f17c41a2ee0 2024-04-25T01:57:40.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.325+0000 7f17c9400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17c4107fc0 0x7f17c419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.325+0000 7f17c9400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17c4107fc0 0x7f17c419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45132/0 (socket says 172.21.15.102:45132) 2024-04-25T01:57:40.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.325+0000 7f17c9400640 1 -- 172.21.15.102:0/2596449717 learned_addr learned my addr 172.21.15.102:0/2596449717 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.326+0000 7f17c9e00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c41a2ee0 0x7f17c41a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.326+0000 7f17c8a00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17c410d040 0x7f17c419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.326+0000 7f17c9400640 1 -- 172.21.15.102:0/2596449717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c41a2ee0 msgr2=0x7f17c41a3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.326+0000 7f17c9400640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c41a2ee0 0x7f17c41a3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.326+0000 7f17c9400640 1 -- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17c410d040 msgr2=0x7f17c419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.326+0000 7f17c9400640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17c410d040 0x7f17c419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.326+0000 7f17c9400640 1 -- 172.21.15.102:0/2596449717 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c41a39d0 con 0x7f17c4107fc0 2024-04-25T01:57:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.327+0000 7f17c9400640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17c4107fc0 0x7f17c419e620 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f17b8004770 tx=0x7f17b8003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.328+0000 7f17c2000640 1 -- 172.21.15.102:0/2596449717 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b803f070 con 0x7f17c4107fc0 2024-04-25T01:57:40.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.328+0000 7f17c2000640 1 -- 172.21.15.102:0/2596449717 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b8031ca0 con 0x7f17c4107fc0 2024-04-25T01:57:40.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.328+0000 7f17c2000640 1 -- 172.21.15.102:0/2596449717 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8035030 con 0x7f17c4107fc0 2024-04-25T01:57:40.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.328+0000 7f17cbe00640 1 -- 172.21.15.102:0/2596449717 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c41a3c60 con 0x7f17c4107fc0 2024-04-25T01:57:40.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.328+0000 7f17cbe00640 1 -- 172.21.15.102:0/2596449717 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c41ab3d0 con 0x7f17c4107fc0 2024-04-25T01:57:40.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.335+0000 7f17c2000640 1 -- 172.21.15.102:0/2596449717 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f17b8042070 con 0x7f17c4107fc0 2024-04-25T01:57:40.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.335+0000 7f17ceba3740 1 Processor -- start 2024-04-25T01:57:40.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.337+0000 7f17c2000640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17ac0a2e50 0x7f17ac0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.338+0000 7f17ceba3740 1 -- start start 2024-04-25T01:57:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.338+0000 7f17c2000640 1 -- 172.21.15.102:0/2596449717 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f17b80ba020 con 0x7f17c4107fc0 2024-04-25T01:57:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.338+0000 7f17c9e00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17ac0a2e50 0x7f17ac0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.338+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292df140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.338+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 0x5638292e9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.338+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292ec140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.338+0000 7f17ceba3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56382923df40 con 0x5638292df680 2024-04-25T01:57:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.338+0000 7f17ceba3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563829243190 con 0x5638292e9d50 2024-04-25T01:57:40.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.338+0000 7f17ceba3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5638291eba10 con 0x5638292ded60 2024-04-25T01:57:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.339+0000 7f17c9e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292ec140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.339+0000 7f17c9e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292ec140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45148/0 (socket says 172.21.15.102:45148) 2024-04-25T01:57:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.339+0000 7f17c9e00640 1 -- 172.21.15.102:0/1953241806 learned_addr learned my addr 172.21.15.102:0/1953241806 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.339+0000 7f17c8a00640 1 --2- 172.21.15.102:0/1953241806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 0x5638292e9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.340+0000 7f17c9400640 1 --2- 172.21.15.102:0/1953241806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292df140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.340+0000 7f17c8a00640 1 -- 172.21.15.102:0/1953241806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 msgr2=0x5638292df140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.340+0000 7f17c8a00640 1 --2- 172.21.15.102:0/1953241806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292df140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.340+0000 7f17c8a00640 1 -- 172.21.15.102:0/1953241806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 msgr2=0x5638292ec140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.340+0000 7f17c8a00640 1 --2- 172.21.15.102:0/1953241806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292ec140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.340+0000 7f17c8a00640 1 -- 172.21.15.102:0/1953241806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56382901f7e0 con 0x5638292df680 2024-04-25T01:57:40.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.340+0000 7f17c8a00640 1 --2- 172.21.15.102:0/1953241806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 0x5638292e9810 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f17c406adc0 tx=0x7f17b000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc41eebaa83b815a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.341+0000 7f17c9e00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17ac0a2e50 0x7f17ac0a5310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f17bc013930 tx=0x7f17bc009e00 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.341+0000 7f17b6600640 1 -- 172.21.15.102:0/1953241806 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b0015070 con 0x5638292df680 2024-04-25T01:57:40.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.341+0000 7f17b6600640 1 -- 172.21.15.102:0/1953241806 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b00035f0 con 0x5638292df680 2024-04-25T01:57:40.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.341+0000 7f17b6600640 1 -- 172.21.15.102:0/1953241806 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b00116a0 con 0x5638292df680 2024-04-25T01:57:40.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.342+0000 7f17ceba3740 1 -- 172.21.15.102:0/1953241806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 msgr2=0x5638292e9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.342+0000 7f17ceba3740 1 --2- 172.21.15.102:0/1953241806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 0x5638292e9810 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f17c406adc0 tx=0x7f17b000f730 comp rx=0 tx=0).stop 2024-04-25T01:57:40.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.342+0000 7f17ceba3740 1 -- 172.21.15.102:0/1953241806 shutdown_connections 2024-04-25T01:57:40.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.342+0000 7f17ceba3740 1 --2- 172.21.15.102:0/1953241806 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292ec140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.342+0000 7f17ceba3740 1 --2- 172.21.15.102:0/1953241806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 0x5638292e9810 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.342+0000 7f17ceba3740 1 --2- 172.21.15.102:0/1953241806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292df140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.342+0000 7f17ceba3740 1 -- 172.21.15.102:0/1953241806 >> 172.21.15.102:0/1953241806 conn(0x5638292db080 msgr2=0x5638292db460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.344+0000 7f17ceba3740 1 -- 172.21.15.102:0/1953241806 shutdown_connections 2024-04-25T01:57:40.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.344+0000 7f17ceba3740 1 -- 172.21.15.102:0/1953241806 wait complete. 2024-04-25T01:57:40.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.345+0000 7f17ceba3740 1 -- start start 2024-04-25T01:57:40.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.345+0000 7f17ceba3740 1 Processor -- start 2024-04-25T01:57:40.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.346+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292e6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.346+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 0x5638292e7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.346+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292e76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.346+0000 7f17ceba3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5638291fee70 con 0x5638292df680 2024-04-25T01:57:40.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.346+0000 7f17ceba3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56382923f150 con 0x5638292e9d50 2024-04-25T01:57:40.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.346+0000 7f17ceba3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5638292fdbf0 con 0x5638292ded60 2024-04-25T01:57:40.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.346+0000 7f17ceba3740 10 client.0 _ll_register_callbacks cb 0x56382923a1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.347+0000 7f17c9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292e6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.347+0000 7f17c9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292e6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44632/0 (socket says 172.21.15.102:44632) 2024-04-25T01:57:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.347+0000 7f17c9400640 1 -- 172.21.15.102:0/3894158278 learned_addr learned my addr 172.21.15.102:0/3894158278 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.347+0000 7f17c8a00640 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 0x5638292e7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.347+0000 7f17c9e00640 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292e76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.348+0000 7f17c9400640 1 -- 172.21.15.102:0/3894158278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 msgr2=0x5638292e76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.348+0000 7f17c9400640 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292e76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.348+0000 7f17c9400640 1 -- 172.21.15.102:0/3894158278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 msgr2=0x5638292e7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.348+0000 7f17c9400640 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 0x5638292e7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.348+0000 7f17c9400640 1 -- 172.21.15.102:0/3894158278 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56382907ff70 con 0x5638292ded60 2024-04-25T01:57:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.348+0000 7f17c9e00640 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292e76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.348+0000 7f17aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.348+0000 7f17c9400640 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292e6c50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f17b80b8e90 tx=0x7f17b80c46f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5e01091fb42ec13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.349+0000 7f17aaa00640 1 -- 172.21.15.102:0/3894158278 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b80b89b0 con 0x5638292ded60 2024-04-25T01:57:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.349+0000 7f17aaa00640 1 -- 172.21.15.102:0/3894158278 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f17b80b85f0 con 0x5638292ded60 2024-04-25T01:57:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.349+0000 7f17ceba3740 1 -- 172.21.15.102:0/3894158278 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17b00035f0 con 0x5638292ded60 2024-04-25T01:57:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.349+0000 7f17aaa00640 1 -- 172.21.15.102:0/3894158278 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f17b80b89b0 con 0x5638292ded60 2024-04-25T01:57:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.349+0000 7f17ceba3740 10 client.25960 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.349+0000 7f17ceba3740 1 -- 172.21.15.102:0/3894158278 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563828f17170 con 0x5638292ded60 2024-04-25T01:57:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.349+0000 7f17aaa00640 1 -- 172.21.15.102:0/3894158278 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b80d6020 con 0x5638292ded60 2024-04-25T01:57:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.350+0000 7f17aaa00640 1 -- 172.21.15.102:0/3894158278 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f17b80b8050 con 0x5638292ded60 2024-04-25T01:57:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.350+0000 7f17ceba3740 10 client.25960 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.350+0000 7f17ceba3740 10 client.25960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.350+0000 7f17ceba3740 20 client.25960 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.350+0000 7f17ceba3740 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56382930aee0 0x56382930d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.350+0000 7f17ceba3740 4 client.25960 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.350+0000 7f17ceba3740 1 -- 172.21.15.102:0/3894158278 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f17b00035f0 con 0x56382930aee0 2024-04-25T01:57:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.350+0000 7f17c8a00640 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56382930aee0 0x56382930d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.352+0000 7f17c8a00640 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56382930aee0 0x56382930d350 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e83ac4cb33c9cce server_cookie=7ac0b1b716a60bb4 in_seq=0 out_seq=0 2024-04-25T01:57:40.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.352+0000 7f17aaa00640 10 client.25960 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:40.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.352+0000 7f17aaa00640 1 -- 172.21.15.102:0/3894158278 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f17b00035f0 con 0x56382930aee0 2024-04-25T01:57:40.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.352+0000 7f17aaa00640 10 client.25960 handle_command_reply: tid=0 2024-04-25T01:57:40.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17ceba3740 1 client.25960 shutdown 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17ceba3740 1 -- 172.21.15.102:0/3894158278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 msgr2=0x5638292e6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17ceba3740 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292e6c50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f17b80b8e90 tx=0x7f17b80c46f0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17ceba3740 1 -- 172.21.15.102:0/3894158278 shutdown_connections 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17ceba3740 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56382930aee0 0x56382930d350 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17ceba3740 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292e9d50 0x5638292e76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17ceba3740 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292df680 0x5638292e7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17ceba3740 1 --2- 172.21.15.102:0/3894158278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x5638292e6c50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17aaa00640 0 client.25960 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.353+0000 7f17ceba3740 1 -- 172.21.15.102:0/3894158278 >> 172.21.15.102:0/3894158278 conn(0x5638292db080 msgr2=0x5638292e0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.354+0000 7f17ceba3740 1 -- 172.21.15.102:0/3894158278 shutdown_connections 2024-04-25T01:57:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.355+0000 7f17ceba3740 1 -- 172.21.15.102:0/3894158278 wait complete. 2024-04-25T01:57:40.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.355+0000 7f17ceba3740 20 client.25960 trim_cache size 0 max 16384 2024-04-25T01:57:40.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.387+0000 7f17ceba3740 1 Processor -- start 2024-04-25T01:57:40.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.387+0000 7f17ceba3740 1 -- start start 2024-04-25T01:57:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x563829306960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292df680 0x563829312c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292e9d50 0x563829313160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17ceba3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f17c41adc00 con 0x5638292e9d50 2024-04-25T01:57:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17ceba3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4082550 con 0x5638292df680 2024-04-25T01:57:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17ceba3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4080380 con 0x5638292ded60 2024-04-25T01:57:40.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17c9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x563829306960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17c9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x563829306960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44642/0 (socket says 172.21.15.102:44642) 2024-04-25T01:57:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17c9400640 1 -- 172.21.15.102:0/2704502055 learned_addr learned my addr 172.21.15.102:0/2704502055 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.388+0000 7f17c8a00640 1 --2- 172.21.15.102:0/2704502055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292df680 0x563829312c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.389+0000 7f17c9e00640 1 --2- 172.21.15.102:0/2704502055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292e9d50 0x563829313160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.389+0000 7f17c9400640 1 -- 172.21.15.102:0/2704502055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292df680 msgr2=0x563829312c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.389+0000 7f17c9400640 1 --2- 172.21.15.102:0/2704502055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292df680 0x563829312c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.389+0000 7f17c9400640 1 -- 172.21.15.102:0/2704502055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292e9d50 msgr2=0x563829313160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.389+0000 7f17c9400640 1 --2- 172.21.15.102:0/2704502055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292e9d50 0x563829313160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.389+0000 7f17c9400640 1 -- 172.21.15.102:0/2704502055 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638292e51e0 con 0x5638292ded60 2024-04-25T01:57:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.389+0000 7f17c9400640 1 --2- 172.21.15.102:0/2704502055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x563829306960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f17b80b9200 tx=0x7f17b803b460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b8c03a85c590896 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.389+0000 7f17b5c00640 1 -- 172.21.15.102:0/2704502055 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b80b89b0 con 0x5638292ded60 2024-04-25T01:57:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.390+0000 7f17b5c00640 1 -- 172.21.15.102:0/2704502055 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b80d6040 con 0x5638292ded60 2024-04-25T01:57:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.390+0000 7f17b5c00640 1 -- 172.21.15.102:0/2704502055 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b80b8b50 con 0x5638292ded60 2024-04-25T01:57:40.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.390+0000 7f17ceba3740 1 -- 172.21.15.102:0/2704502055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 msgr2=0x563829306960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.390+0000 7f17ceba3740 1 --2- 172.21.15.102:0/2704502055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x563829306960 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f17b80b9200 tx=0x7f17b803b460 comp rx=0 tx=0).stop 2024-04-25T01:57:40.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.391+0000 7f17ceba3740 1 -- 172.21.15.102:0/2704502055 shutdown_connections 2024-04-25T01:57:40.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.391+0000 7f17ceba3740 1 --2- 172.21.15.102:0/2704502055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292e9d50 0x563829313160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.391+0000 7f17ceba3740 1 --2- 172.21.15.102:0/2704502055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292df680 0x563829312c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.391+0000 7f17ceba3740 1 --2- 172.21.15.102:0/2704502055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292ded60 0x563829306960 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.391+0000 7f17ceba3740 1 -- 172.21.15.102:0/2704502055 >> 172.21.15.102:0/2704502055 conn(0x5638292db080 msgr2=0x5638292e04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.391+0000 7f17ceba3740 1 -- 172.21.15.102:0/2704502055 shutdown_connections 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.391+0000 7f17ceba3740 1 -- 172.21.15.102:0/2704502055 wait complete. 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.391+0000 7f17ceba3740 1 -- start start 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.392+0000 7f17ceba3740 1 Processor -- start 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.392+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292ded60 0x5638293030e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.392+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292df680 0x563829301480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.392+0000 7f17ceba3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292e9d50 0x5638293019c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.392+0000 7f17ceba3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4080a10 con 0x5638292e9d50 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.392+0000 7f17ceba3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17c4081e80 con 0x5638292ded60 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.392+0000 7f17ceba3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17c40810e0 con 0x5638292df680 2024-04-25T01:57:40.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.392+0000 7f17ceba3740 10 client.0 _ll_register_callbacks cb 0x7f17bc0098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.393+0000 7f17c9400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292ded60 0x5638293030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.393+0000 7f17c9400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292ded60 0x5638293030e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45180/0 (socket says 172.21.15.102:45180) 2024-04-25T01:57:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.393+0000 7f17c9400640 1 -- 172.21.15.102:0/764863287 learned_addr learned my addr 172.21.15.102:0/764863287 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.393+0000 7f17c8a00640 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292df680 0x563829301480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.393+0000 7f17c9400640 1 -- 172.21.15.102:0/764863287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292df680 msgr2=0x563829301480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.393+0000 7f17c9400640 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292df680 0x563829301480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.393+0000 7f17c9400640 1 -- 172.21.15.102:0/764863287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292e9d50 msgr2=0x5638293019c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.393+0000 7f17c9400640 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292e9d50 0x5638293019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.393+0000 7f17c9400640 1 -- 172.21.15.102:0/764863287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563829302030 con 0x5638292ded60 2024-04-25T01:57:40.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.394+0000 7f17c9400640 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292ded60 0x5638293030e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f17b8031550 tx=0x7f17b8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3ea38c2dcc2f71b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.394+0000 7f17aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.394+0000 7f17aaa00640 1 -- 172.21.15.102:0/764863287 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b80b8cf0 con 0x5638292ded60 2024-04-25T01:57:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.394+0000 7f17aaa00640 1 -- 172.21.15.102:0/764863287 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f17b80dadc0 con 0x5638292ded60 2024-04-25T01:57:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.394+0000 7f17ceba3740 1 -- 172.21.15.102:0/764863287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17b80d6040 con 0x5638292ded60 2024-04-25T01:57:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.395+0000 7f17aaa00640 1 -- 172.21.15.102:0/764863287 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f17b80b8ec0 con 0x5638292ded60 2024-04-25T01:57:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.395+0000 7f17ceba3740 10 client.25985 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.395+0000 7f17ceba3740 1 -- 172.21.15.102:0/764863287 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563829302030 con 0x5638292ded60 2024-04-25T01:57:40.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.395+0000 7f17aaa00640 1 -- 172.21.15.102:0/764863287 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b80d6040 con 0x5638292ded60 2024-04-25T01:57:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.396+0000 7f17aaa00640 1 -- 172.21.15.102:0/764863287 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f17b80bbd50 con 0x5638292ded60 2024-04-25T01:57:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.396+0000 7f17ceba3740 10 client.25985 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.396+0000 7f17ceba3740 10 client.25985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.396+0000 7f17ceba3740 20 client.25985 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.396+0000 7f17ceba3740 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563829372680 0x563829374aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.396+0000 7f17ceba3740 4 client.25985 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:57:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.396+0000 7f17ceba3740 1 -- 172.21.15.102:0/764863287 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5638292506d0 con 0x563829372680 2024-04-25T01:57:40.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.397+0000 7f17c8a00640 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563829372680 0x563829374aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.397+0000 7f17c8a00640 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563829372680 0x563829374aa0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6945841100ae9424 server_cookie=c2e642992aa6e3a2 in_seq=0 out_seq=0 2024-04-25T01:57:40.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.397+0000 7f17aaa00640 10 client.25985 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:40.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.398+0000 7f17aaa00640 1 -- 172.21.15.102:0/764863287 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5638292506d0 con 0x563829372680 2024-04-25T01:57:40.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.398+0000 7f17aaa00640 10 client.25985 handle_command_reply: tid=0 2024-04-25T01:57:40.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.398+0000 7f17ceba3740 1 client.25985 shutdown 2024-04-25T01:57:40.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.399+0000 7f17ceba3740 1 -- 172.21.15.102:0/764863287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292ded60 msgr2=0x5638293030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.399+0000 7f17ceba3740 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292ded60 0x5638293030e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f17b8031550 tx=0x7f17b8009260 comp rx=0 tx=0).stop 2024-04-25T01:57:40.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.399+0000 7f17ceba3740 1 -- 172.21.15.102:0/764863287 shutdown_connections 2024-04-25T01:57:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.399+0000 7f17ceba3740 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563829372680 0x563829374aa0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.399+0000 7f17aaa00640 0 client.25985 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.399+0000 7f17ceba3740 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638292e9d50 0x5638293019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.399+0000 7f17ceba3740 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638292df680 0x563829301480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.399+0000 7f17ceba3740 1 --2- 172.21.15.102:0/764863287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638292ded60 0x5638293030e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.399+0000 7f17ceba3740 1 -- 172.21.15.102:0/764863287 >> 172.21.15.102:0/764863287 conn(0x5638292db080 msgr2=0x5638292e0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.400+0000 7f17ceba3740 1 -- 172.21.15.102:0/764863287 shutdown_connections 2024-04-25T01:57:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.400+0000 7f17ceba3740 1 -- 172.21.15.102:0/764863287 wait complete. 2024-04-25T01:57:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.400+0000 7f17ceba3740 20 client.25985 trim_cache size 0 max 16384 2024-04-25T01:57:40.405 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:57:40.405 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T01:57:40.405 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "8dfce396-555d-477d-adef-3963ee7e3880", 2024-04-25T01:57:40.405 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T01:57:40.405 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 -- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17ac0a2e50 msgr2=0x7f17ac0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17ac0a2e50 0x7f17ac0a5310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f17bc013930 tx=0x7f17bc009e00 comp rx=0 tx=0).stop 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 -- 172.21.15.102:0/2596449717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17c4107fc0 msgr2=0x7f17c419e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17c4107fc0 0x7f17c419e620 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f17b8004770 tx=0x7f17b8003c10 comp rx=0 tx=0).stop 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 -- 172.21.15.102:0/2596449717 shutdown_connections 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17ac0a2e50 0x7f17ac0a5310 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c41a2ee0 0x7f17c41a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17c410d040 0x7f17c419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 --2- 172.21.15.102:0/2596449717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17c4107fc0 0x7f17c419e620 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.401+0000 7f17cbe00640 1 -- 172.21.15.102:0/2596449717 >> 172.21.15.102:0/2596449717 conn(0x7f17c4075c80 msgr2=0x7f17c4077f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.402+0000 7f17cbe00640 1 -- 172.21.15.102:0/2596449717 shutdown_connections 2024-04-25T01:57:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.402+0000 7f17cbe00640 1 -- 172.21.15.102:0/2596449717 wait complete. 2024-04-25T01:57:40.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:57:40.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:57:40.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.710+0000 7fb3a4200640 1 -- 172.21.15.102:0/2392527525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c1042e0 msgr2=0x7fb39c1107f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.710+0000 7fb3a4200640 1 --2- 172.21.15.102:0/2392527525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c1042e0 0x7fb39c1107f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb394009ca0 tx=0x7fb39402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.710+0000 7fb3a4200640 1 -- 172.21.15.102:0/2392527525 shutdown_connections 2024-04-25T01:57:40.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.711+0000 7fb3a4200640 1 --2- 172.21.15.102:0/2392527525 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c1042e0 0x7fb39c1107f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.711+0000 7fb3a4200640 1 --2- 172.21.15.102:0/2392527525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb39c103920 0x7fb39c103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.711+0000 7fb3a4200640 1 --2- 172.21.15.102:0/2392527525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb39c109930 0x7fb39c109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.711+0000 7fb3a4200640 1 -- 172.21.15.102:0/2392527525 >> 172.21.15.102:0/2392527525 conn(0x7fb39c0ff3b0 msgr2=0x7fb39c1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.711+0000 7fb3a4200640 1 -- 172.21.15.102:0/2392527525 shutdown_connections 2024-04-25T01:57:40.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.712+0000 7fb3a4200640 1 -- 172.21.15.102:0/2392527525 wait complete. 2024-04-25T01:57:40.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.713+0000 7fb3a4200640 1 Processor -- start 2024-04-25T01:57:40.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.714+0000 7fb3a4200640 1 -- start start 2024-04-25T01:57:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a4200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c103920 0x7fb39c1106d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a4200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb39c1042e0 0x7fb39c10d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a4200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb39c109930 0x7fb39c10dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a4200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c113d40 con 0x7fb39c1042e0 2024-04-25T01:57:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a4200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c113bc0 con 0x7fb39c103920 2024-04-25T01:57:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a4200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c113ec0 con 0x7fb39c109930 2024-04-25T01:57:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a1800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c103920 0x7fb39c1106d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a1800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c103920 0x7fb39c1106d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45198/0 (socket says 172.21.15.102:45198) 2024-04-25T01:57:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a1800640 1 -- 172.21.15.102:0/164179142 learned_addr learned my addr 172.21.15.102:0/164179142 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a1800640 1 -- 172.21.15.102:0/164179142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb39c109930 msgr2=0x7fb39c10dd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:57:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a1800640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb39c109930 0x7fb39c10dd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a1800640 1 -- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb39c1042e0 msgr2=0x7fb39c10d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.715+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb39c1042e0 0x7fb39c10d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.716+0000 7fb3a1800640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb39c1042e0 0x7fb39c10d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.716+0000 7fb3a1800640 1 -- 172.21.15.102:0/164179142 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39c10e650 con 0x7fb39c103920 2024-04-25T01:57:40.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.716+0000 7fb3a1800640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c103920 0x7fb39c1106d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb38c01f980 tx=0x7fb38c01fe40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.717+0000 7fb39a000640 1 -- 172.21.15.102:0/164179142 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c01dcd0 con 0x7fb39c103920 2024-04-25T01:57:40.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.717+0000 7fb39a000640 1 -- 172.21.15.102:0/164179142 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb38c021920 con 0x7fb39c103920 2024-04-25T01:57:40.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.717+0000 7fb39a000640 1 -- 172.21.15.102:0/164179142 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c0298e0 con 0x7fb39c103920 2024-04-25T01:57:40.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.717+0000 7fb3a4200640 1 -- 172.21.15.102:0/164179142 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39c06b1c0 con 0x7fb39c103920 2024-04-25T01:57:40.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.717+0000 7fb3a4200640 1 -- 172.21.15.102:0/164179142 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c06b6c0 con 0x7fb39c103920 2024-04-25T01:57:40.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.725+0000 7fb3a71cf740 1 Processor -- start 2024-04-25T01:57:40.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.725+0000 7fb3a71cf740 1 -- start start 2024-04-25T01:57:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.725+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc606e000 0x55ebc606e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.725+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e920 0x55ebc6078ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.725+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc6078ff0 0x55ebc607b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.725+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ebc5fd2560 con 0x55ebc6078ff0 2024-04-25T01:57:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.725+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ebc5f7a930 con 0x55ebc606e000 2024-04-25T01:57:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.725+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ebc5fce520 con 0x55ebc606e920 2024-04-25T01:57:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.726+0000 7fb3a2200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc606e000 0x55ebc606e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.726+0000 7fb3a2200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc606e000 0x55ebc606e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45208/0 (socket says 172.21.15.102:45208) 2024-04-25T01:57:40.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.726+0000 7fb3a2200640 1 -- 172.21.15.102:0/75856351 learned_addr learned my addr 172.21.15.102:0/75856351 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.727+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/75856351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc6078ff0 0x55ebc607b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.727+0000 7fb39a000640 1 -- 172.21.15.102:0/164179142 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb38c021440 con 0x7fb39c103920 2024-04-25T01:57:40.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.727+0000 7fb3a2200640 1 -- 172.21.15.102:0/75856351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e920 msgr2=0x55ebc6078ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:40.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.727+0000 7fb3a2200640 1 --2- 172.21.15.102:0/75856351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e920 0x55ebc6078ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.727+0000 7fb3a2200640 1 -- 172.21.15.102:0/75856351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc6078ff0 msgr2=0x55ebc607b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.727+0000 7fb3a2200640 1 --2- 172.21.15.102:0/75856351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc6078ff0 0x55ebc607b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.727+0000 7fb3a2200640 1 -- 172.21.15.102:0/75856351 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc5db07d0 con 0x55ebc606e000 2024-04-25T01:57:40.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.727+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/75856351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc6078ff0 0x55ebc607b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:40.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.728+0000 7fb3a2200640 1 --2- 172.21.15.102:0/75856351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc606e000 0x55ebc606e3e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb39402ae30 tx=0x7fb394005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53186efa05db5e19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.728+0000 7fb38aa00640 1 -- 172.21.15.102:0/75856351 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394031ba0 con 0x55ebc606e000 2024-04-25T01:57:40.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.728+0000 7fb38aa00640 1 -- 172.21.15.102:0/75856351 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb39403f070 con 0x55ebc606e000 2024-04-25T01:57:40.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.728+0000 7fb38aa00640 1 -- 172.21.15.102:0/75856351 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394004040 con 0x55ebc606e000 2024-04-25T01:57:40.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.729+0000 7fb39a000640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3840a2dc0 0x7fb3840a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.729+0000 7fb3a1800640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3840a2dc0 0x7fb3840a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a71cf740 1 -- 172.21.15.102:0/75856351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc606e000 msgr2=0x55ebc606e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/75856351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc606e000 0x55ebc606e3e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb39402ae30 tx=0x7fb394005ef0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb39a000640 1 -- 172.21.15.102:0/164179142 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb38c0e0c90 con 0x7fb39c103920 2024-04-25T01:57:40.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a71cf740 1 -- 172.21.15.102:0/75856351 shutdown_connections 2024-04-25T01:57:40.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/75856351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc6078ff0 0x55ebc607b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/75856351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e920 0x55ebc6078ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/75856351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc606e000 0x55ebc606e3e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a71cf740 1 -- 172.21.15.102:0/75856351 >> 172.21.15.102:0/75856351 conn(0x55ebc606a320 msgr2=0x55ebc606a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a1800640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3840a2dc0 0x7fb3840a5280 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55ebc606f5c0 tx=0x7fb38c05c130 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a71cf740 1 -- 172.21.15.102:0/75856351 shutdown_connections 2024-04-25T01:57:40.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.730+0000 7fb3a71cf740 1 -- 172.21.15.102:0/75856351 wait complete. 2024-04-25T01:57:40.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.731+0000 7fb3a71cf740 1 -- start start 2024-04-25T01:57:40.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.731+0000 7fb3a71cf740 1 Processor -- start 2024-04-25T01:57:40.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6076020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc6076560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6076aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ebc607bdd0 con 0x55ebc606e920 2024-04-25T01:57:40.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ebc5f8dd90 con 0x55ebc6078ff0 2024-04-25T01:57:40.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ebc607bf50 con 0x55ebc606e000 2024-04-25T01:57:40.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a71cf740 10 client.0 _ll_register_callbacks cb 0x55ebc5f8d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:40.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a2200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6076020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a2200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6076020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44678/0 (socket says 172.21.15.102:44678) 2024-04-25T01:57:40.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.732+0000 7fb3a2200640 1 -- 172.21.15.102:0/9153864 learned_addr learned my addr 172.21.15.102:0/9153864 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb3a2200640 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6076aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc6076560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb3a2200640 1 -- 172.21.15.102:0/9153864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 msgr2=0x55ebc6076aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb3a2200640 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6076aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb3a2200640 1 -- 172.21.15.102:0/9153864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 msgr2=0x55ebc6076560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb3a2200640 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc6076560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb3a2200640 1 -- 172.21.15.102:0/9153864 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc5ce63f0 con 0x55ebc606e000 2024-04-25T01:57:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb3a2200640 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6076020 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb394009620 tx=0x7fb394031780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fded8b57f4fc260a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb37f400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.733+0000 7fb37f400640 1 -- 172.21.15.102:0/9153864 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394003970 con 0x55ebc606e000 2024-04-25T01:57:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.734+0000 7fb37f400640 1 -- 172.21.15.102:0/9153864 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb394003b10 con 0x55ebc606e000 2024-04-25T01:57:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.734+0000 7fb3a71cf740 1 -- 172.21.15.102:0/9153864 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb39403f070 con 0x55ebc606e000 2024-04-25T01:57:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.734+0000 7fb37f400640 1 -- 172.21.15.102:0/9153864 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb394003d10 con 0x55ebc606e000 2024-04-25T01:57:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.734+0000 7fb3a71cf740 10 client.25969 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.734+0000 7fb3a71cf740 1 -- 172.21.15.102:0/9153864 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ebc5ca60f0 con 0x55ebc606e000 2024-04-25T01:57:40.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.735+0000 7fb37f400640 1 -- 172.21.15.102:0/9153864 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb39403f070 con 0x55ebc606e000 2024-04-25T01:57:40.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.735+0000 7fb37f400640 1 -- 172.21.15.102:0/9153864 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb39402f610 con 0x55ebc606e000 2024-04-25T01:57:40.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.735+0000 7fb3a71cf740 10 client.25969 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:40.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.735+0000 7fb3a71cf740 10 client.25969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:40.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.735+0000 7fb3a71cf740 20 client.25969 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:40.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.735+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ebc6089220 0x55ebc608b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.735+0000 7fb3a71cf740 4 client.25969 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:40.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.735+0000 7fb3a71cf740 1 -- 172.21.15.102:0/9153864 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebc5fe5d90 con 0x55ebc6089220 2024-04-25T01:57:40.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.736+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ebc6089220 0x55ebc608b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.737+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ebc6089220 0x55ebc608b610 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61865947f0871172 server_cookie=5c26fe459764b75c in_seq=0 out_seq=0 2024-04-25T01:57:40.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.737+0000 7fb37f400640 10 client.25969 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:40.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.738+0000 7fb37f400640 1 -- 172.21.15.102:0/9153864 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebc5fe5d90 con 0x55ebc6089220 2024-04-25T01:57:40.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.738+0000 7fb37f400640 10 client.25969 handle_command_reply: tid=0 2024-04-25T01:57:40.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb3a71cf740 1 client.25969 shutdown 2024-04-25T01:57:40.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb3a71cf740 1 -- 172.21.15.102:0/9153864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 msgr2=0x55ebc6076020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6076020 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb394009620 tx=0x7fb394031780 comp rx=0 tx=0).stop 2024-04-25T01:57:40.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb3a71cf740 1 -- 172.21.15.102:0/9153864 shutdown_connections 2024-04-25T01:57:40.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ebc6089220 0x55ebc608b610 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6076aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc6076560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/9153864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6076020 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb3a71cf740 1 -- 172.21.15.102:0/9153864 >> 172.21.15.102:0/9153864 conn(0x55ebc606a320 msgr2=0x55ebc6070120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.739+0000 7fb37f400640 0 client.25969 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:40.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.740+0000 7fb3a71cf740 1 -- 172.21.15.102:0/9153864 shutdown_connections 2024-04-25T01:57:40.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.740+0000 7fb3a71cf740 1 -- 172.21.15.102:0/9153864 wait complete. 2024-04-25T01:57:40.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.741+0000 7fb3a71cf740 20 client.25969 trim_cache size 0 max 16384 2024-04-25T01:57:40.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.779+0000 7fb3a71cf740 1 Processor -- start 2024-04-25T01:57:40.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.779+0000 7fb3a71cf740 1 -- start start 2024-04-25T01:57:40.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6084cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc6074f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6070760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1adc10 con 0x55ebc606e920 2024-04-25T01:57:40.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1ab480 con 0x55ebc6078ff0 2024-04-25T01:57:40.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c072150 con 0x55ebc606e000 2024-04-25T01:57:40.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a2200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6084cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a2200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6084cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44692/0 (socket says 172.21.15.102:44692) 2024-04-25T01:57:40.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a2200640 1 -- 172.21.15.102:0/3390925869 learned_addr learned my addr 172.21.15.102:0/3390925869 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a2200640 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a2200640 1 -- 172.21.15.102:0/3390925869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 msgr2=0x55ebc6070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a2200640 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6070760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.780+0000 7fb3a2200640 1 -- 172.21.15.102:0/3390925869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 msgr2=0x55ebc6074f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:57:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.781+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc6074f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.781+0000 7fb3a2200640 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc6074f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.781+0000 7fb3a2200640 1 -- 172.21.15.102:0/3390925869 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc607b610 con 0x55ebc606e000 2024-04-25T01:57:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.781+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc6074f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.781+0000 7fb3a2200640 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6084cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb394031f30 tx=0x7fb39402a8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31002c53c0bc81d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.782+0000 7fb38a000640 1 -- 172.21.15.102:0/3390925869 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394003970 con 0x55ebc606e000 2024-04-25T01:57:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.782+0000 7fb38a000640 1 -- 172.21.15.102:0/3390925869 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb39403f040 con 0x55ebc606e000 2024-04-25T01:57:40.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.782+0000 7fb38a000640 1 -- 172.21.15.102:0/3390925869 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39402fc70 con 0x55ebc606e000 2024-04-25T01:57:40.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.783+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3390925869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 msgr2=0x55ebc6084cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.783+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6084cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb394031f30 tx=0x7fb39402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.783+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3390925869 shutdown_connections 2024-04-25T01:57:40.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.783+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6070760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.783+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc6074f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.783+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3390925869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6084cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.783+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3390925869 >> 172.21.15.102:0/3390925869 conn(0x55ebc606a320 msgr2=0x55ebc6069480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.783+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3390925869 shutdown_connections 2024-04-25T01:57:40.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.783+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3390925869 wait complete. 2024-04-25T01:57:40.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.784+0000 7fb3a71cf740 1 -- start start 2024-04-25T01:57:40.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.784+0000 7fb3a71cf740 1 Processor -- start 2024-04-25T01:57:40.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.785+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6081430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc60728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a71cf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6072e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a2200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6081430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c0727c0 con 0x55ebc606e920 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1aadb0 con 0x55ebc6078ff0 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a2200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6081430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44702/0 (socket says 172.21.15.102:44702) 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a71cf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c072e90 con 0x55ebc606e000 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a2200640 1 -- 172.21.15.102:0/3260743623 learned_addr learned my addr 172.21.15.102:0/3260743623 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a71cf740 10 client.0 _ll_register_callbacks cb 0x7fb394004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a2200640 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6072e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a2200640 1 -- 172.21.15.102:0/3260743623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 msgr2=0x55ebc6072e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a2200640 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6072e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.786+0000 7fb3a2200640 1 -- 172.21.15.102:0/3260743623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 msgr2=0x55ebc60728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:40.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.787+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc60728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.788+0000 7fb3a2200640 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc60728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.788+0000 7fb3a2200640 1 -- 172.21.15.102:0/3260743623 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc60734d0 con 0x55ebc606e000 2024-04-25T01:57:40.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.788+0000 7fb3a2200640 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6081430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb39402f470 tx=0x7fb39402ae40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce9badd6e2f8a991 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:40.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.788+0000 7fb37f400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:40.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.788+0000 7fb37f400640 1 -- 172.21.15.102:0/3260743623 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39402f810 con 0x55ebc606e000 2024-04-25T01:57:40.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.788+0000 7fb37f400640 1 -- 172.21.15.102:0/3260743623 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb39402fc70 con 0x55ebc606e000 2024-04-25T01:57:40.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.788+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc60728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:40.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.789+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3260743623 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb39403f040 con 0x55ebc606e000 2024-04-25T01:57:40.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.789+0000 7fb37f400640 1 -- 172.21.15.102:0/3260743623 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb39402fe70 con 0x55ebc606e000 2024-04-25T01:57:40.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.789+0000 7fb3a71cf740 10 client.25975 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:40.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.789+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3260743623 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ebc6100220 con 0x55ebc606e000 2024-04-25T01:57:40.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.790+0000 7fb37f400640 1 -- 172.21.15.102:0/3260743623 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb39403f040 con 0x55ebc606e000 2024-04-25T01:57:40.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.790+0000 7fb37f400640 1 -- 172.21.15.102:0/3260743623 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb39405e040 con 0x55ebc606e000 2024-04-25T01:57:40.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.790+0000 7fb3a71cf740 10 client.25975 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:40.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.790+0000 7fb3a71cf740 10 client.25975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:40.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.790+0000 7fb3a71cf740 20 client.25975 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:40.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.790+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ebc6100670 0x55ebc6102a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:40.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.790+0000 7fb3a71cf740 4 client.25975 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:57:40.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.790+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3260743623 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ebc5fdfaa0 con 0x55ebc6100670 2024-04-25T01:57:40.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.791+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ebc6100670 0x55ebc6102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:40.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.793+0000 7fb3a0e00640 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ebc6100670 0x55ebc6102a90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c761c750ab56dd9 server_cookie=daff65e7fefc9e30 in_seq=0 out_seq=0 2024-04-25T01:57:40.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.793+0000 7fb37f400640 10 client.25975 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:40.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.793+0000 7fb37f400640 1 -- 172.21.15.102:0/3260743623 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ebc5fdfaa0 con 0x55ebc6100670 2024-04-25T01:57:40.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.793+0000 7fb37f400640 10 client.25975 handle_command_reply: tid=0 2024-04-25T01:57:40.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.793+0000 7fb3a71cf740 1 client.25975 shutdown 2024-04-25T01:57:40.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.793+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3260743623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 msgr2=0x55ebc6081430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.793+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6081430 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb39402f470 tx=0x7fb39402ae40 comp rx=0 tx=0).stop 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3260743623 shutdown_connections 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ebc6100670 0x55ebc6102a90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ebc6078ff0 0x55ebc6072e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ebc606e920 0x55ebc60728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb3a71cf740 1 --2- 172.21.15.102:0/3260743623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebc606e000 0x55ebc6081430 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3260743623 >> 172.21.15.102:0/3260743623 conn(0x55ebc606a320 msgr2=0x55ebc6075f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb37f400640 0 client.25975 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3260743623 shutdown_connections 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb3a71cf740 1 -- 172.21.15.102:0/3260743623 wait complete. 2024-04-25T01:57:40.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.794+0000 7fb3a71cf740 20 client.25975 trim_cache size 0 max 16384 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout: "8dfce396-555d-477d-adef-3963ee7e3880": { 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "8dfce396-555d-477d-adef-3963ee7e3880", 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:57:40.799 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:57:40.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 -- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3840a2dc0 msgr2=0x7fb3840a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3840a2dc0 0x7fb3840a5280 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55ebc606f5c0 tx=0x7fb38c05c130 comp rx=0 tx=0).stop 2024-04-25T01:57:40.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 -- 172.21.15.102:0/164179142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c103920 msgr2=0x7fb39c1106d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:40.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c103920 0x7fb39c1106d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb38c01f980 tx=0x7fb38c01fe40 comp rx=0 tx=0).stop 2024-04-25T01:57:40.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 -- 172.21.15.102:0/164179142 shutdown_connections 2024-04-25T01:57:40.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb3840a2dc0 0x7fb3840a5280 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb39c109930 0x7fb39c10dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb39c1042e0 0x7fb39c10d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 --2- 172.21.15.102:0/164179142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb39c103920 0x7fb39c1106d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:40.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.796+0000 7fb3a4200640 1 -- 172.21.15.102:0/164179142 >> 172.21.15.102:0/164179142 conn(0x7fb39c0ff3b0 msgr2=0x7fb39c0ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:40.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.797+0000 7fb3a4200640 1 -- 172.21.15.102:0/164179142 shutdown_connections 2024-04-25T01:57:40.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:57:40.797+0000 7fb3a4200640 1 -- 172.21.15.102:0/164179142 wait complete. 2024-04-25T01:57:40.827 INFO:tasks.cephfs.filesystem:scrub status for tag:8dfce396-555d-477d-adef-3963ee7e3880 - {'path': '/', 'tag': '8dfce396-555d-477d-adef-3963ee7e3880', 'options': 'recursive,force'} 2024-04-25T01:57:41.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:40 smithi102 ceph-mon[27796]: pgmap v75: 97 pgs: 97 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.1 MiB/s wr, 652 op/s 2024-04-25T01:57:41.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:40 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T01:57:41.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:40 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:40 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:41.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:40 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:41.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:40 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:40 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:41.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:40 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:40 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:40 smithi110 ceph-mon[32848]: pgmap v75: 97 pgs: 97 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.1 MiB/s wr, 652 op/s 2024-04-25T01:57:41.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:40 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T01:57:41.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:40 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:40 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:41.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:40 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:41.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:40 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:40 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:41.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:40 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:40 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:40 smithi193 ceph-mon[32981]: pgmap v75: 97 pgs: 97 active+clean; 1.7 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.1 MiB/s wr, 652 op/s 2024-04-25T01:57:41.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:40 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T01:57:41.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:40 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:40 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:41.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:40 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:41.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:40 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:40 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:41.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:40 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:41.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:40 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:42.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:41 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:42.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:41 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:42.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:41 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:42.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:41 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:42.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:41 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:57:42.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:41 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:42.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:41 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:42.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:41 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:42.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:41 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:42.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:41 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:42.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:41 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:57:42.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:41 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:42.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:41 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:42.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:41 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:42.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:41 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:42.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:41 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:42.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:41 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:57:42.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:41 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:57:43.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 110.0 s, 3285.2 tps, lat 30.494 ms stddev 35.586 2024-04-25T01:57:43.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:42 smithi110 ceph-mon[32848]: pgmap v76: 97 pgs: 97 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-04-25T01:57:43.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:42 smithi193 ceph-mon[32981]: pgmap v76: 97 pgs: 97 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-04-25T01:57:43.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:42 smithi102 ceph-mon[27796]: pgmap v76: 97 pgs: 97 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-04-25T01:57:45.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:44 smithi110 ceph-mon[32848]: pgmap v77: 97 pgs: 97 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 7.5 MiB/s wr, 715 op/s 2024-04-25T01:57:45.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:44 smithi193 ceph-mon[32981]: pgmap v77: 97 pgs: 97 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 7.5 MiB/s wr, 715 op/s 2024-04-25T01:57:45.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:44 smithi102 ceph-mon[27796]: pgmap v77: 97 pgs: 97 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 7.5 MiB/s wr, 715 op/s 2024-04-25T01:57:47.227 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:46 smithi102 ceph-mon[27796]: pgmap v78: 97 pgs: 97 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 7.5 MiB/s wr, 715 op/s 2024-04-25T01:57:47.227 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:46 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:57:47.227 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:46 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:57:47.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:46 smithi110 ceph-mon[32848]: pgmap v78: 97 pgs: 97 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 7.5 MiB/s wr, 715 op/s 2024-04-25T01:57:47.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:46 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:57:47.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:46 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:57:47.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:46 smithi193 ceph-mon[32981]: pgmap v78: 97 pgs: 97 active+clean; 1.7 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 7.5 MiB/s wr, 715 op/s 2024-04-25T01:57:47.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:46 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:57:47.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:46 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:57:48.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 115.0 s, 4477.3 tps, lat 22.304 ms stddev 22.144 2024-04-25T01:57:48.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:47 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:57:48.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:47 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:57:48.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:47 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:57:49.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:48 smithi110 ceph-mon[32848]: pgmap v79: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 13 MiB/s wr, 1.09k op/s 2024-04-25T01:57:49.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:48 smithi193 ceph-mon[32981]: pgmap v79: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 13 MiB/s wr, 1.09k op/s 2024-04-25T01:57:49.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:48 smithi102 ceph-mon[27796]: pgmap v79: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 13 MiB/s wr, 1.09k op/s 2024-04-25T01:57:51.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:50 smithi110 ceph-mon[32848]: pgmap v80: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 8.8 MiB/s wr, 769 op/s 2024-04-25T01:57:51.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:50 smithi193 ceph-mon[32981]: pgmap v80: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 8.8 MiB/s wr, 769 op/s 2024-04-25T01:57:51.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:50 smithi102 ceph-mon[27796]: pgmap v80: 97 pgs: 97 active+clean; 1.7 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 8.8 MiB/s wr, 769 op/s 2024-04-25T01:57:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.956+0000 7f765ec00640 1 -- 172.21.15.102:0/445666205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765806a260 msgr2=0x7f765806a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.956+0000 7f765ec00640 1 --2- 172.21.15.102:0/445666205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765806a260 0x7f765806a640 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f764800dc90 tx=0x7f764802e370 comp rx=0 tx=0).stop 2024-04-25T01:57:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.957+0000 7f765ec00640 1 -- 172.21.15.102:0/445666205 shutdown_connections 2024-04-25T01:57:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.957+0000 7f765ec00640 1 --2- 172.21.15.102:0/445666205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765810f2c0 0x7f7658116e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.957+0000 7f765ec00640 1 --2- 172.21.15.102:0/445666205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f765806ac10 0x7f765810ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.957+0000 7f765ec00640 1 --2- 172.21.15.102:0/445666205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765806a260 0x7f765806a640 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.957+0000 7f765ec00640 1 -- 172.21.15.102:0/445666205 >> 172.21.15.102:0/445666205 conn(0x7f765806b730 msgr2=0x7f765806db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.957+0000 7f765ec00640 1 -- 172.21.15.102:0/445666205 shutdown_connections 2024-04-25T01:57:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.957+0000 7f765ec00640 1 -- 172.21.15.102:0/445666205 wait complete. 2024-04-25T01:57:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.958+0000 7f765ec00640 1 Processor -- start 2024-04-25T01:57:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.959+0000 7f765ec00640 1 -- start start 2024-04-25T01:57:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.959+0000 7f765ec00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765806a260 0x7f765819e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.959+0000 7f765ec00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765810f2c0 0x7f765819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.959+0000 7f765ec00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76581a2df0 0x7f76581a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.960+0000 7f765ec00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7658118190 con 0x7f765806a260 2024-04-25T01:57:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.960+0000 7f765ec00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7658118010 con 0x7f76581a2df0 2024-04-25T01:57:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.960+0000 7f765ec00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7658118310 con 0x7f765810f2c0 2024-04-25T01:57:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.960+0000 7f7657400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765810f2c0 0x7f765819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.960+0000 7f7657400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765810f2c0 0x7f765819ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60176/0 (socket says 172.21.15.102:60176) 2024-04-25T01:57:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.960+0000 7f7657400640 1 -- 172.21.15.102:0/3260187498 learned_addr learned my addr 172.21.15.102:0/3260187498 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.960+0000 7f765cc00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76581a2df0 0x7f76581a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.960+0000 7f7657e00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765806a260 0x7f765819e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.961+0000 7f7657400640 1 -- 172.21.15.102:0/3260187498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76581a2df0 msgr2=0x7f76581a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.961+0000 7f7657400640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76581a2df0 0x7f76581a32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.961+0000 7f7657400640 1 -- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765806a260 msgr2=0x7f765819e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.961+0000 7f7657400640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765806a260 0x7f765819e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.961+0000 7f7657400640 1 -- 172.21.15.102:0/3260187498 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76581a38e0 con 0x7f765810f2c0 2024-04-25T01:57:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.961+0000 7f765cc00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76581a2df0 0x7f76581a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.961+0000 7f7657e00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765806a260 0x7f765819e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:57:51.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.962+0000 7f7657400640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765810f2c0 0x7f765819ea70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f764400d950 tx=0x7f764400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.963+0000 7f7654c00640 1 -- 172.21.15.102:0/3260187498 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644017070 con 0x7f765810f2c0 2024-04-25T01:57:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.964+0000 7f7654c00640 1 -- 172.21.15.102:0/3260187498 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7644009c30 con 0x7f765810f2c0 2024-04-25T01:57:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.964+0000 7f7654c00640 1 -- 172.21.15.102:0/3260187498 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76440047a0 con 0x7f765810f2c0 2024-04-25T01:57:51.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.964+0000 7f765ec00640 1 -- 172.21.15.102:0/3260187498 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76581a3bd0 con 0x7f765810f2c0 2024-04-25T01:57:51.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.964+0000 7f765ec00640 1 -- 172.21.15.102:0/3260187498 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76581ab370 con 0x7f765810f2c0 2024-04-25T01:57:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7654c00640 1 -- 172.21.15.102:0/3260187498 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7644004020 con 0x7f765810f2c0 2024-04-25T01:57:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7661a07740 1 Processor -- start 2024-04-25T01:57:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7661a07740 1 -- start start 2024-04-25T01:57:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7661a07740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d94621e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7661a07740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7661a07740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 0x5572d946f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7661a07740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5572d93c5ab0 con 0x5572d9462720 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7661a07740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5572d936e640 con 0x5572d9461e00 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7661a07740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5572d93c1a70 con 0x5572d946cfe0 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.973+0000 7f765cc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d94621e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.973+0000 7f765cc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d94621e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53660/0 (socket says 172.21.15.102:53660) 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.973+0000 7f765cc00640 1 -- 172.21.15.102:0/1520777454 learned_addr learned my addr 172.21.15.102:0/1520777454 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.972+0000 7f7657400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 0x5572d946f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.973+0000 7f7657e00640 1 --2- 172.21.15.102:0/1520777454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.973+0000 7f765cc00640 1 -- 172.21.15.102:0/1520777454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 msgr2=0x5572d946f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.973+0000 7f765cc00640 1 --2- 172.21.15.102:0/1520777454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 0x5572d946f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.973+0000 7f765cc00640 1 -- 172.21.15.102:0/1520777454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 msgr2=0x5572d946caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.974+0000 7f765cc00640 1 --2- 172.21.15.102:0/1520777454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.974+0000 7f765cc00640 1 -- 172.21.15.102:0/1520777454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572d90da210 con 0x5572d9461e00 2024-04-25T01:57:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.975+0000 7f7657e00640 1 --2- 172.21.15.102:0/1520777454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:57:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.975+0000 7f765cc00640 1 --2- 172.21.15.102:0/1520777454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d94621e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f765000f790 tx=0x7f765000fc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=741a6762128ac7aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.975+0000 7f7654c00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f763c0a2e70 0x7f763c0a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.975+0000 7f764d200640 1 -- 172.21.15.102:0/1520777454 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650004d20 con 0x5572d9461e00 2024-04-25T01:57:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.976+0000 7f764d200640 1 -- 172.21.15.102:0/1520777454 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f765000de00 con 0x5572d9461e00 2024-04-25T01:57:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.976+0000 7f764d200640 1 -- 172.21.15.102:0/1520777454 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650010430 con 0x5572d9461e00 2024-04-25T01:57:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.976+0000 7f7657e00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f763c0a2e70 0x7f763c0a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.977+0000 7f7661a07740 1 -- 172.21.15.102:0/1520777454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 msgr2=0x5572d94621e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.977+0000 7f7661a07740 1 --2- 172.21.15.102:0/1520777454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d94621e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f765000f790 tx=0x7f765000fc50 comp rx=0 tx=0).stop 2024-04-25T01:57:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.977+0000 7f7661a07740 1 -- 172.21.15.102:0/1520777454 shutdown_connections 2024-04-25T01:57:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.977+0000 7f7661a07740 1 --2- 172.21.15.102:0/1520777454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 0x5572d946f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.977+0000 7f7661a07740 1 --2- 172.21.15.102:0/1520777454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.977+0000 7f7661a07740 1 --2- 172.21.15.102:0/1520777454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d94621e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.977+0000 7f7661a07740 1 -- 172.21.15.102:0/1520777454 >> 172.21.15.102:0/1520777454 conn(0x5572d945e310 msgr2=0x5572d945e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.977+0000 7f7654c00640 1 -- 172.21.15.102:0/3260187498 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f76440cef60 con 0x7f765810f2c0 2024-04-25T01:57:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.979+0000 7f7657e00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f763c0a2e70 0x7f763c0a5330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f764800dad0 tx=0x7f7648030040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.980+0000 7f7661a07740 1 -- 172.21.15.102:0/1520777454 shutdown_connections 2024-04-25T01:57:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.980+0000 7f7661a07740 1 -- 172.21.15.102:0/1520777454 wait complete. 2024-04-25T01:57:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.981+0000 7f7661a07740 1 -- start start 2024-04-25T01:57:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.981+0000 7f7661a07740 1 Processor -- start 2024-04-25T01:57:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.982+0000 7f7661a07740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9469f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.982+0000 7f7661a07740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9462720 0x5572d946a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f7661a07740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d946cfe0 0x5572d946aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f7657e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9469f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f7661a07740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5572d946fc10 con 0x5572d946cfe0 2024-04-25T01:57:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f7661a07740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5572d9381990 con 0x5572d9462720 2024-04-25T01:57:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f7661a07740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5572d946fd90 con 0x5572d9461e00 2024-04-25T01:57:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f7657e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9469f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60198/0 (socket says 172.21.15.102:60198) 2024-04-25T01:57:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f7657e00640 1 -- 172.21.15.102:0/935981280 learned_addr learned my addr 172.21.15.102:0/935981280 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f7657400640 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9462720 0x5572d946a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f765cc00640 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d946cfe0 0x5572d946aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.983+0000 7f7661a07740 10 client.0 _ll_register_callbacks cb 0x5572d9383950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.984+0000 7f7657400640 1 -- 172.21.15.102:0/935981280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 msgr2=0x5572d9469f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.984+0000 7f7657400640 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9469f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.984+0000 7f7657400640 1 -- 172.21.15.102:0/935981280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d946cfe0 msgr2=0x5572d946aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.984+0000 7f7657400640 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d946cfe0 0x5572d946aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.984+0000 7f7657400640 1 -- 172.21.15.102:0/935981280 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572d91a6620 con 0x5572d9462720 2024-04-25T01:57:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.984+0000 7f7661a07740 1 -- 172.21.15.102:0/935981280 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f765000de00 con 0x5572d9462720 2024-04-25T01:57:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.984+0000 7f7641600640 10 client.26015 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:57:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.985+0000 7f7657400640 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9462720 0x5572d946a4c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f76440cc6e0 tx=0x7f7644097110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2554725200e9ab4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.985+0000 7f7641600640 1 -- 172.21.15.102:0/935981280 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764409aff0 con 0x5572d9462720 2024-04-25T01:57:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.985+0000 7f7641600640 1 -- 172.21.15.102:0/935981280 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7644097b20 con 0x5572d9462720 2024-04-25T01:57:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.985+0000 7f7641600640 1 -- 172.21.15.102:0/935981280 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f764409ae10 con 0x5572d9462720 2024-04-25T01:57:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.985+0000 7f7661a07740 10 client.26015 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.985+0000 7f7661a07740 1 -- 172.21.15.102:0/935981280 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5572d947ba00 con 0x5572d9462720 2024-04-25T01:57:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.986+0000 7f7641600640 1 -- 172.21.15.102:0/935981280 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f76440d10e0 con 0x5572d9462720 2024-04-25T01:57:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.986+0000 7f7661a07740 10 client.26015 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.986+0000 7f7661a07740 10 client.26015 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.986+0000 7f7661a07740 20 client.26015 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.986+0000 7f7661a07740 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5572d947ce70 0x5572d947f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.986+0000 7f7661a07740 4 client.26015 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:57:51.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.986+0000 7f7661a07740 1 -- 172.21.15.102:0/935981280 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572d93d2ff0 con 0x5572d947ce70 2024-04-25T01:57:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.987+0000 7f7657e00640 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5572d947ce70 0x5572d947f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:51.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.988+0000 7f7657e00640 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5572d947ce70 0x5572d947f280 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=533b31c9d9981e5d server_cookie=dcdd6f2f27eae546 in_seq=0 out_seq=0 2024-04-25T01:57:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.988+0000 7f7641600640 10 client.26015 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.989+0000 7f7641600640 1 -- 172.21.15.102:0/935981280 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5572d93d2ff0 con 0x5572d947ce70 2024-04-25T01:57:51.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.989+0000 7f7641600640 10 client.26015 handle_command_reply: tid=0 2024-04-25T01:57:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.989+0000 7f7661a07740 1 client.26015 shutdown 2024-04-25T01:57:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.990+0000 7f7661a07740 1 -- 172.21.15.102:0/935981280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9462720 msgr2=0x5572d946a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:51.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.990+0000 7f7661a07740 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9462720 0x5572d946a4c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f76440cc6e0 tx=0x7f7644097110 comp rx=0 tx=0).stop 2024-04-25T01:57:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.991+0000 7f7661a07740 1 -- 172.21.15.102:0/935981280 shutdown_connections 2024-04-25T01:57:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.991+0000 7f7661a07740 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5572d947ce70 0x5572d947f280 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.991+0000 7f7661a07740 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d946cfe0 0x5572d946aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.991+0000 7f7661a07740 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9462720 0x5572d946a4c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.991+0000 7f7661a07740 1 --2- 172.21.15.102:0/935981280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9469f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.991+0000 7f7641600640 0 client.26015 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:51.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.991+0000 7f7661a07740 1 -- 172.21.15.102:0/935981280 >> 172.21.15.102:0/935981280 conn(0x5572d945e310 msgr2=0x5572d9464090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.992+0000 7f7661a07740 1 -- 172.21.15.102:0/935981280 shutdown_connections 2024-04-25T01:57:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.992+0000 7f7661a07740 1 -- 172.21.15.102:0/935981280 wait complete. 2024-04-25T01:57:51.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:51.992+0000 7f7661a07740 20 client.26015 trim_cache size 0 max 16384 2024-04-25T01:57:52.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.042+0000 7f7661a07740 1 Processor -- start 2024-04-25T01:57:52.046 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.042+0000 7f7661a07740 1 -- start start 2024-04-25T01:57:52.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.043+0000 7f7661a07740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d945dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:52.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.043+0000 7f7661a07740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:52.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.043+0000 7f7661a07740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 0x5572d9484b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:52.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.043+0000 7f7661a07740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f76581b6300 con 0x5572d9462720 2024-04-25T01:57:52.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.043+0000 7f7661a07740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76581b3b70 con 0x5572d9461e00 2024-04-25T01:57:52.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.043+0000 7f7661a07740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76581b19c0 con 0x5572d946cfe0 2024-04-25T01:57:52.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.043+0000 7f7657e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d945dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:52.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.044+0000 7f7657e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d945dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53676/0 (socket says 172.21.15.102:53676) 2024-04-25T01:57:52.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.044+0000 7f7657e00640 1 -- 172.21.15.102:0/4052976580 learned_addr learned my addr 172.21.15.102:0/4052976580 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:52.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.044+0000 7f765cc00640 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 0x5572d9484b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:52.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.044+0000 7f7657e00640 1 -- 172.21.15.102:0/4052976580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 msgr2=0x5572d9484b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:52.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.044+0000 7f7657400640 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:52.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.045+0000 7f7657e00640 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 0x5572d9484b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.045+0000 7f7657e00640 1 -- 172.21.15.102:0/4052976580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 msgr2=0x5572d946a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:52.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.045+0000 7f7657e00640 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.046+0000 7f7657e00640 1 -- 172.21.15.102:0/4052976580 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572d947f900 con 0x5572d9461e00 2024-04-25T01:57:52.050 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.046+0000 7f765cc00640 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 0x5572d9484b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:52.050 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.046+0000 7f7657400640 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:52.050 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.047+0000 7f7657e00640 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d945dc00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7648030010 tx=0x7f764800d670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7578484081bfa4e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:52.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.048+0000 7f7643e00640 1 -- 172.21.15.102:0/4052976580 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648053030 con 0x5572d9461e00 2024-04-25T01:57:52.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.049+0000 7f7643e00640 1 -- 172.21.15.102:0/4052976580 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7648041450 con 0x5572d9461e00 2024-04-25T01:57:52.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.049+0000 7f7643e00640 1 -- 172.21.15.102:0/4052976580 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648053030 con 0x5572d9461e00 2024-04-25T01:57:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.051+0000 7f7661a07740 1 -- 172.21.15.102:0/4052976580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 msgr2=0x5572d945dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:52.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.051+0000 7f7661a07740 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d945dc00 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7648030010 tx=0x7f764800d670 comp rx=0 tx=0).stop 2024-04-25T01:57:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.052+0000 7f7661a07740 1 -- 172.21.15.102:0/4052976580 shutdown_connections 2024-04-25T01:57:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.052+0000 7f7661a07740 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d946cfe0 0x5572d9484b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.052+0000 7f7661a07740 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d946a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.056 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.052+0000 7f7661a07740 1 --2- 172.21.15.102:0/4052976580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d9461e00 0x5572d945dc00 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.056 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.052+0000 7f7661a07740 1 -- 172.21.15.102:0/4052976580 >> 172.21.15.102:0/4052976580 conn(0x5572d945e310 msgr2=0x5572d9464040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.052+0000 7f7661a07740 1 -- 172.21.15.102:0/4052976580 shutdown_connections 2024-04-25T01:57:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.053+0000 7f7661a07740 1 -- 172.21.15.102:0/4052976580 wait complete. 2024-04-25T01:57:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.053+0000 7f7661a07740 1 -- start start 2024-04-25T01:57:52.057 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.054+0000 7f7661a07740 1 Processor -- start 2024-04-25T01:57:52.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.055+0000 7f7661a07740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9474fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:52.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.056+0000 7f7661a07740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d9475520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:52.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.056+0000 7f7661a07740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d946cfe0 0x5572d9471fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:52.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.056+0000 7f7661a07740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f76581b2030 con 0x5572d9462720 2024-04-25T01:57:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.056+0000 7f7661a07740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76581b34a0 con 0x5572d946cfe0 2024-04-25T01:57:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.056+0000 7f7661a07740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76581b2700 con 0x5572d9461e00 2024-04-25T01:57:52.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.056+0000 7f7661a07740 10 client.0 _ll_register_callbacks cb 0x7f7644005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:57:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.056+0000 7f7657e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9474fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.056+0000 7f7657e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9474fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60218/0 (socket says 172.21.15.102:60218) 2024-04-25T01:57:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.056+0000 7f7657e00640 1 -- 172.21.15.102:0/2723364297 learned_addr learned my addr 172.21.15.102:0/2723364297 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:57:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.057+0000 7f765cc00640 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d946cfe0 0x5572d9471fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.057+0000 7f7657400640 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d9475520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.057+0000 7f7657e00640 1 -- 172.21.15.102:0/2723364297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d946cfe0 msgr2=0x5572d9471fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.057+0000 7f7657e00640 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d946cfe0 0x5572d9471fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.057+0000 7f7657e00640 1 -- 172.21.15.102:0/2723364297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 msgr2=0x5572d9475520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:57:52.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.058+0000 7f7657e00640 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d9475520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.058+0000 7f7657e00640 1 -- 172.21.15.102:0/2723364297 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572d945e140 con 0x5572d9461e00 2024-04-25T01:57:52.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.058+0000 7f7657400640 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d9475520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:57:52.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.059+0000 7f7657e00640 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9474fe0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f764800ddb0 tx=0x7f7648005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d90106104082814 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:57:52.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.059+0000 7f7641600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:57:52.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.059+0000 7f7641600640 1 -- 172.21.15.102:0/2723364297 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648053030 con 0x5572d9461e00 2024-04-25T01:57:52.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.059+0000 7f7661a07740 1 -- 172.21.15.102:0/2723364297 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7648041450 con 0x5572d9461e00 2024-04-25T01:57:52.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.059+0000 7f7641600640 1 -- 172.21.15.102:0/2723364297 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7648009040 con 0x5572d9461e00 2024-04-25T01:57:52.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.060+0000 7f7641600640 1 -- 172.21.15.102:0/2723364297 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7648041810 con 0x5572d9461e00 2024-04-25T01:57:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.060+0000 7f7661a07740 10 client.25990 fetch_fsmap learned FSMap version 27 2024-04-25T01:57:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.060+0000 7f7661a07740 1 -- 172.21.15.102:0/2723364297 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5572d94ec3a0 con 0x5572d9461e00 2024-04-25T01:57:52.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.060+0000 7f7641600640 1 -- 172.21.15.102:0/2723364297 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7648041450 con 0x5572d9461e00 2024-04-25T01:57:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.060+0000 7f7641600640 1 -- 172.21.15.102:0/2723364297 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7648036570 con 0x5572d9461e00 2024-04-25T01:57:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.061+0000 7f7661a07740 10 client.25990 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:57:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.061+0000 7f7661a07740 10 client.25990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:57:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.061+0000 7f7661a07740 20 client.25990 populate_metadata read hostname 'smithi102' 2024-04-25T01:57:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.061+0000 7f7661a07740 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5572d94ecad0 0x5572d94eeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:57:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.061+0000 7f7661a07740 4 client.25990 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:57:52.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.061+0000 7f7661a07740 1 -- 172.21.15.102:0/2723364297 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5572d93cd340 con 0x5572d94ecad0 2024-04-25T01:57:52.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.062+0000 7f7657400640 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5572d94ecad0 0x5572d94eeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:57:52.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.064+0000 7f7641600640 10 client.25990 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:52.068 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.064+0000 7f7657400640 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5572d94ecad0 0x5572d94eeef0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16ee933cb64f1fcf server_cookie=d3d50bb74097b352 in_seq=0 out_seq=0 2024-04-25T01:57:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.094+0000 7f7641600640 1 -- 172.21.15.102:0/2723364297 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5572d93cd340 con 0x5572d94ecad0 2024-04-25T01:57:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.094+0000 7f7641600640 10 client.25990 handle_command_reply: tid=0 2024-04-25T01:57:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.094+0000 7f7661a07740 1 client.25990 shutdown 2024-04-25T01:57:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.094+0000 7f7661a07740 1 -- 172.21.15.102:0/2723364297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 msgr2=0x5572d9474fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.094+0000 7f7661a07740 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9474fe0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f764800ddb0 tx=0x7f7648005e00 comp rx=0 tx=0).stop 2024-04-25T01:57:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.095+0000 7f7661a07740 1 -- 172.21.15.102:0/2723364297 shutdown_connections 2024-04-25T01:57:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.095+0000 7f7661a07740 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5572d94ecad0 0x5572d94eeef0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.095+0000 7f7661a07740 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5572d946cfe0 0x5572d9471fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.095+0000 7f7661a07740 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5572d9462720 0x5572d9475520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.095+0000 7f7661a07740 1 --2- 172.21.15.102:0/2723364297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5572d9461e00 0x5572d9474fe0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.095+0000 7f7661a07740 1 -- 172.21.15.102:0/2723364297 >> 172.21.15.102:0/2723364297 conn(0x5572d945e310 msgr2=0x5572d946b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.095+0000 7f7641600640 0 client.25990 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:57:52.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.096+0000 7f7661a07740 1 -- 172.21.15.102:0/2723364297 shutdown_connections 2024-04-25T01:57:52.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.097+0000 7f7661a07740 1 -- 172.21.15.102:0/2723364297 wait complete. 2024-04-25T01:57:52.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.097+0000 7f7661a07740 20 client.25990 trim_cache size 0 max 16384 2024-04-25T01:57:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:57:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:57:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:57:52.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:57:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.098+0000 7f765ec00640 1 -- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f763c0a2e70 msgr2=0x7f763c0a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.098+0000 7f765ec00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f763c0a2e70 0x7f763c0a5330 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f764800dad0 tx=0x7f7648030040 comp rx=0 tx=0).stop 2024-04-25T01:57:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.098+0000 7f765ec00640 1 -- 172.21.15.102:0/3260187498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765810f2c0 msgr2=0x7f765819ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:57:52.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.098+0000 7f765ec00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765810f2c0 0x7f765819ea70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f764400d950 tx=0x7f764400de10 comp rx=0 tx=0).stop 2024-04-25T01:57:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.099+0000 7f765ec00640 1 -- 172.21.15.102:0/3260187498 shutdown_connections 2024-04-25T01:57:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.099+0000 7f765ec00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f763c0a2e70 0x7f763c0a5330 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.099+0000 7f765ec00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76581a2df0 0x7f76581a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.099+0000 7f765ec00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765810f2c0 0x7f765819ea70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.099+0000 7f765ec00640 1 --2- 172.21.15.102:0/3260187498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765806a260 0x7f765819e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:57:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.099+0000 7f765ec00640 1 -- 172.21.15.102:0/3260187498 >> 172.21.15.102:0/3260187498 conn(0x7f765806b730 msgr2=0x7f765806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:57:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.099+0000 7f765ec00640 1 -- 172.21.15.102:0/3260187498 shutdown_connections 2024-04-25T01:57:52.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:57:52.099+0000 7f765ec00640 1 -- 172.21.15.102:0/3260187498 wait complete. 2024-04-25T01:57:53.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 120.0 s, 3639.5 tps, lat 27.453 ms stddev 30.971 2024-04-25T01:57:53.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:52 smithi110 ceph-mon[32848]: pgmap v81: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 11 MiB/s wr, 980 op/s 2024-04-25T01:57:53.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:52 smithi193 ceph-mon[32981]: pgmap v81: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 11 MiB/s wr, 980 op/s 2024-04-25T01:57:53.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:52 smithi102 ceph-mon[27796]: pgmap v81: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 11 MiB/s wr, 980 op/s 2024-04-25T01:57:55.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:54 smithi110 ceph-mon[32848]: pgmap v82: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 717 op/s 2024-04-25T01:57:55.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:54 smithi193 ceph-mon[32981]: pgmap v82: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 717 op/s 2024-04-25T01:57:55.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:54 smithi102 ceph-mon[27796]: pgmap v82: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 717 op/s 2024-04-25T01:57:57.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:56 smithi110 ceph-mon[32848]: pgmap v83: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 717 op/s 2024-04-25T01:57:57.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:56 smithi193 ceph-mon[32981]: pgmap v83: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 717 op/s 2024-04-25T01:57:57.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:56 smithi102 ceph-mon[27796]: pgmap v83: 97 pgs: 97 active+clean; 1.7 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 717 op/s 2024-04-25T01:57:58.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 125.0 s, 3061.3 tps, lat 32.526 ms stddev 37.200 2024-04-25T01:57:59.239 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:57:58 smithi110 ceph-mon[32848]: pgmap v84: 97 pgs: 97 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 14 MiB/s wr, 1.07k op/s 2024-04-25T01:57:59.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:57:58 smithi193 ceph-mon[32981]: pgmap v84: 97 pgs: 97 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 14 MiB/s wr, 1.07k op/s 2024-04-25T01:57:59.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:57:58 smithi102 ceph-mon[27796]: pgmap v84: 97 pgs: 97 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 14 MiB/s wr, 1.07k op/s 2024-04-25T01:58:01.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:00 smithi193 ceph-mon[32981]: pgmap v85: 97 pgs: 97 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.6 MiB/s wr, 692 op/s 2024-04-25T01:58:01.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:00 smithi102 ceph-mon[27796]: pgmap v85: 97 pgs: 97 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.6 MiB/s wr, 692 op/s 2024-04-25T01:58:01.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:00 smithi110 ceph-mon[32848]: pgmap v85: 97 pgs: 97 active+clean; 1.8 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.6 MiB/s wr, 692 op/s 2024-04-25T01:58:03.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 130.0 s, 1904.4 tps, lat 50.490 ms stddev 107.348 2024-04-25T01:58:03.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:02 smithi193 ceph-mon[32981]: pgmap v86: 97 pgs: 97 active+clean; 1.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 11 MiB/s wr, 868 op/s 2024-04-25T01:58:03.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:02 smithi102 ceph-mon[27796]: pgmap v86: 97 pgs: 97 active+clean; 1.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 11 MiB/s wr, 868 op/s 2024-04-25T01:58:03.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:02 smithi110 ceph-mon[32848]: pgmap v86: 97 pgs: 97 active+clean; 1.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 11 MiB/s wr, 868 op/s 2024-04-25T01:58:05.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:04 smithi193 ceph-mon[32981]: pgmap v87: 97 pgs: 97 active+clean; 1.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 10 MiB/s wr, 786 op/s 2024-04-25T01:58:05.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.279+0000 7fec92a00640 1 -- 172.21.15.102:0/3486809833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 msgr2=0x7fec8c1006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:05.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.279+0000 7fec92a00640 1 --2- 172.21.15.102:0/3486809833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 0x7fec8c1006c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009900 tx=0x7fec7c02a3f0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.280+0000 7fec92a00640 1 -- 172.21.15.102:0/3486809833 shutdown_connections 2024-04-25T01:58:05.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.280+0000 7fec92a00640 1 --2- 172.21.15.102:0/3486809833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec8c10de90 0x7fec8c110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.280+0000 7fec92a00640 1 --2- 172.21.15.102:0/3486809833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec8c100c90 0x7fec8c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.280+0000 7fec92a00640 1 --2- 172.21.15.102:0/3486809833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 0x7fec8c1006c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.280+0000 7fec92a00640 1 -- 172.21.15.102:0/3486809833 >> 172.21.15.102:0/3486809833 conn(0x7fec8c0fbf60 msgr2=0x7fec8c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:05.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.281+0000 7fec92a00640 1 -- 172.21.15.102:0/3486809833 shutdown_connections 2024-04-25T01:58:05.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.281+0000 7fec92a00640 1 -- 172.21.15.102:0/3486809833 wait complete. 2024-04-25T01:58:05.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.282+0000 7fec92a00640 1 Processor -- start 2024-04-25T01:58:05.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.283+0000 7fec92a00640 1 -- start start 2024-04-25T01:58:05.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec92a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 0x7fec8c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec92a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec8c100c90 0x7fec8c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec92a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec8c10de90 0x7fec8c19e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec92a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c113b90 con 0x7fec8c100c90 2024-04-25T01:58:05.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec92a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c113a10 con 0x7fec8c10de90 2024-04-25T01:58:05.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec92a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c113d10 con 0x7fec8c1002e0 2024-04-25T01:58:05.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec91600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 0x7fec8c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec92000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec8c10de90 0x7fec8c19e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec91600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 0x7fec8c19a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34262/0 (socket says 172.21.15.102:34262) 2024-04-25T01:58:05.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.284+0000 7fec91600640 1 -- 172.21.15.102:0/2736653762 learned_addr learned my addr 172.21.15.102:0/2736653762 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.285+0000 7fec91600640 1 -- 172.21.15.102:0/2736653762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec8c10de90 msgr2=0x7fec8c19e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.285+0000 7fec91600640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec8c10de90 0x7fec8c19e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.285+0000 7fec91600640 1 -- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec8c100c90 msgr2=0x7fec8c19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.285+0000 7fec90c00640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec8c100c90 0x7fec8c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.285+0000 7fec91600640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec8c100c90 0x7fec8c19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.285+0000 7fec91600640 1 -- 172.21.15.102:0/2736653762 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8c19f040 con 0x7fec8c1002e0 2024-04-25T01:58:05.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.285+0000 7fec90c00640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec8c100c90 0x7fec8c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.286+0000 7fec91600640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 0x7fec8c19a050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fec7c004770 tx=0x7fec7c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.286+0000 7fec8a000640 1 -- 172.21.15.102:0/2736653762 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c03f070 con 0x7fec8c1002e0 2024-04-25T01:58:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.286+0000 7fec92a00640 1 -- 172.21.15.102:0/2736653762 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8c19f2d0 con 0x7fec8c1002e0 2024-04-25T01:58:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.286+0000 7fec8a000640 1 -- 172.21.15.102:0/2736653762 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec7c031ca0 con 0x7fec8c1002e0 2024-04-25T01:58:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.286+0000 7fec92a00640 1 -- 172.21.15.102:0/2736653762 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8c1a6bc0 con 0x7fec8c1002e0 2024-04-25T01:58:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.286+0000 7fec8a000640 1 -- 172.21.15.102:0/2736653762 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c035030 con 0x7fec8c1002e0 2024-04-25T01:58:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.291+0000 7fec96e04740 1 Processor -- start 2024-04-25T01:58:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.292+0000 7fec96e04740 1 -- start start 2024-04-25T01:58:05.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.292+0000 7fec8a000640 1 -- 172.21.15.102:0/2736653762 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fec7c042070 con 0x7fec8c1002e0 2024-04-25T01:58:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.292+0000 7fec96e04740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86e97c0 0x5566e86e9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.292+0000 7fec96e04740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.292+0000 7fec96e04740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86f49a0 0x5566e86f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.292+0000 7fec96e04740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5566e8644ab0 con 0x5566e86ea0e0 2024-04-25T01:58:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.292+0000 7fec96e04740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5566e85ed640 con 0x5566e86f49a0 2024-04-25T01:58:05.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.292+0000 7fec96e04740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5566e8640a70 con 0x5566e86e97c0 2024-04-25T01:58:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.293+0000 7fec92000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86e97c0 0x5566e86e9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.293+0000 7fec92000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86e97c0 0x5566e86e9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34276/0 (socket says 172.21.15.102:34276) 2024-04-25T01:58:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.293+0000 7fec92000640 1 -- 172.21.15.102:0/699615682 learned_addr learned my addr 172.21.15.102:0/699615682 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.293+0000 7fec90c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86f49a0 0x5566e86f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.293+0000 7fec91600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec90c00640 1 -- 172.21.15.102:0/699615682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86e97c0 msgr2=0x5566e86e9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec90c00640 1 --2- 172.21.15.102:0/699615682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86e97c0 0x5566e86e9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec90c00640 1 -- 172.21.15.102:0/699615682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 msgr2=0x5566e86f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec90c00640 1 --2- 172.21.15.102:0/699615682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec90c00640 1 -- 172.21.15.102:0/699615682 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566e8359210 con 0x5566e86f49a0 2024-04-25T01:58:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec91600640 1 --2- 172.21.15.102:0/699615682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec90c00640 1 --2- 172.21.15.102:0/699615682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86f49a0 0x5566e86f6d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fec78009b60 tx=0x7fec7800e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c9151083e7a3beb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec82600640 1 -- 172.21.15.102:0/699615682 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78016070 con 0x5566e86f49a0 2024-04-25T01:58:05.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec82600640 1 -- 172.21.15.102:0/699615682 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec78009e50 con 0x5566e86f49a0 2024-04-25T01:58:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.294+0000 7fec82600640 1 -- 172.21.15.102:0/699615682 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78011470 con 0x5566e86f49a0 2024-04-25T01:58:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.295+0000 7fec8a000640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fec700a2d70 0x7fec700a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.295+0000 7fec8a000640 1 -- 172.21.15.102:0/2736653762 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fec7c0ef7d0 con 0x7fec8c1002e0 2024-04-25T01:58:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.295+0000 7fec92000640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fec700a2d70 0x7fec700a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.296+0000 7fec96e04740 1 -- 172.21.15.102:0/699615682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86f49a0 msgr2=0x5566e86f6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:05.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.296+0000 7fec96e04740 1 --2- 172.21.15.102:0/699615682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86f49a0 0x5566e86f6d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fec78009b60 tx=0x7fec7800e6c0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.296+0000 7fec96e04740 1 -- 172.21.15.102:0/699615682 shutdown_connections 2024-04-25T01:58:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.296+0000 7fec96e04740 1 --2- 172.21.15.102:0/699615682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86f49a0 0x5566e86f6d90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.296+0000 7fec96e04740 1 --2- 172.21.15.102:0/699615682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.296+0000 7fec96e04740 1 --2- 172.21.15.102:0/699615682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86e97c0 0x5566e86e9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.296+0000 7fec96e04740 1 -- 172.21.15.102:0/699615682 >> 172.21.15.102:0/699615682 conn(0x5566e86e5cd0 msgr2=0x5566e86e60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.297+0000 7fec96e04740 1 -- 172.21.15.102:0/699615682 shutdown_connections 2024-04-25T01:58:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.297+0000 7fec92000640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fec700a2d70 0x7fec700a5230 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fec840124a0 tx=0x7fec840097e0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.297+0000 7fec96e04740 1 -- 172.21.15.102:0/699615682 wait complete. 2024-04-25T01:58:05.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.297+0000 7fec96e04740 1 -- start start 2024-04-25T01:58:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.297+0000 7fec96e04740 1 Processor -- start 2024-04-25T01:58:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec96e04740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec96e04740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e86f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec96e04740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86f49a0 0x5566e86f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec96e04740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5566e86f75d0 con 0x5566e86f49a0 2024-04-25T01:58:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec96e04740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5566e8600990 con 0x5566e86e97c0 2024-04-25T01:58:05.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec96e04740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5566e86f7750 con 0x5566e86ea0e0 2024-04-25T01:58:05.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec96e04740 10 client.0 _ll_register_callbacks cb 0x5566e8602950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:05.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec90c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e86f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec90c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e86f1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34278/0 (socket says 172.21.15.102:34278) 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec90c00640 1 -- 172.21.15.102:0/4267992157 learned_addr learned my addr 172.21.15.102:0/4267992157 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.298+0000 7fec91600640 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec90c00640 1 -- 172.21.15.102:0/4267992157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 msgr2=0x5566e86f1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec90c00640 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec90c00640 1 -- 172.21.15.102:0/4267992157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86f49a0 msgr2=0x5566e86f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec90c00640 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86f49a0 0x5566e86f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec90c00640 1 -- 172.21.15.102:0/4267992157 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566e8425620 con 0x5566e86ea0e0 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec90c00640 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e86f1d90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fec78000c00 tx=0x7fec780054a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=348edb4dcb8507c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec76a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec76a00640 1 -- 172.21.15.102:0/4267992157 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78016040 con 0x5566e86ea0e0 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec76a00640 1 -- 172.21.15.102:0/4267992157 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fec78009c50 con 0x5566e86ea0e0 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.299+0000 7fec96e04740 1 -- 172.21.15.102:0/4267992157 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec78009e50 con 0x5566e86ea0e0 2024-04-25T01:58:05.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.300+0000 7fec76a00640 1 -- 172.21.15.102:0/4267992157 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec78005590 con 0x5566e86ea0e0 2024-04-25T01:58:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.300+0000 7fec96e04740 10 client.26002 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:05.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.300+0000 7fec96e04740 1 -- 172.21.15.102:0/4267992157 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5566e8703460 con 0x5566e86ea0e0 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.300+0000 7fec76a00640 1 -- 172.21.15.102:0/4267992157 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec78009e50 con 0x5566e86ea0e0 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.300+0000 7fec76a00640 1 -- 172.21.15.102:0/4267992157 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fec7800f040 con 0x5566e86ea0e0 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.301+0000 7fec96e04740 10 client.26002 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.301+0000 7fec96e04740 10 client.26002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.301+0000 7fec96e04740 20 client.26002 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.301+0000 7fec96e04740 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5566e8704870 0x5566e8706c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.301+0000 7fec96e04740 4 client.26002 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.301+0000 7fec96e04740 1 -- 172.21.15.102:0/4267992157 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566e8651ff0 con 0x5566e8704870 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.301+0000 7fec91600640 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5566e8704870 0x5566e8706c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.302+0000 7fec91600640 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5566e8704870 0x5566e8706c60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=618e4d7004c45eb0 server_cookie=a748da3622bb6e64 in_seq=0 out_seq=0 2024-04-25T01:58:05.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.302+0000 7fec76a00640 10 client.26002 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.302+0000 7fec76a00640 1 -- 172.21.15.102:0/4267992157 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566e8651ff0 con 0x5566e8704870 2024-04-25T01:58:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.302+0000 7fec76a00640 10 client.26002 handle_command_reply: tid=0 2024-04-25T01:58:05.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.302+0000 7fec96e04740 1 client.26002 shutdown 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 -- 172.21.15.102:0/4267992157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 msgr2=0x5566e86f1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e86f1d90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fec78000c00 tx=0x7fec780054a0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 -- 172.21.15.102:0/4267992157 shutdown_connections 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5566e8704870 0x5566e8706c60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86f49a0 0x5566e86f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e86f1d90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 --2- 172.21.15.102:0/4267992157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec76a00640 0 client.26002 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 -- 172.21.15.102:0/4267992157 >> 172.21.15.102:0/4267992157 conn(0x5566e86e5cd0 msgr2=0x5566e86dc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 -- 172.21.15.102:0/4267992157 shutdown_connections 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.303+0000 7fec96e04740 1 -- 172.21.15.102:0/4267992157 wait complete. 2024-04-25T01:58:05.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.304+0000 7fec96e04740 20 client.26002 trim_cache size 0 max 16384 2024-04-25T01:58:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.322+0000 7fec96e04740 1 Processor -- start 2024-04-25T01:58:05.325 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.322+0000 7fec96e04740 1 -- start start 2024-04-25T01:58:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec96e04740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e87004c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec96e04740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e870c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec96e04740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86f49a0 0x5566e870c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec96e04740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c010ff0 con 0x5566e86f49a0 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec96e04740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c00e860 con 0x5566e86e97c0 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec96e04740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c00c6b0 con 0x5566e86ea0e0 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec91600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e87004c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec90c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e870c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec91600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e87004c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45542/0 (socket says 172.21.15.102:45542) 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec91600640 1 -- 172.21.15.102:0/1345661756 learned_addr learned my addr 172.21.15.102:0/1345661756 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec90c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e870c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34282/0 (socket says 172.21.15.102:34282) 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec92000640 1 --2- 172.21.15.102:0/1345661756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86f49a0 0x5566e870c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec91600640 1 -- 172.21.15.102:0/1345661756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 msgr2=0x5566e870c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec91600640 1 --2- 172.21.15.102:0/1345661756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e870c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec91600640 1 -- 172.21.15.102:0/1345661756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86f49a0 msgr2=0x5566e870c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec91600640 1 --2- 172.21.15.102:0/1345661756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86f49a0 0x5566e870c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec91600640 1 -- 172.21.15.102:0/1345661756 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566e86e5b00 con 0x5566e86e97c0 2024-04-25T01:58:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.323+0000 7fec90c00640 1 --2- 172.21.15.102:0/1345661756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e870c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.324+0000 7fec91600640 1 --2- 172.21.15.102:0/1345661756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e87004c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fec7c006010 tx=0x7fec7c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=830c6307b843c8a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.324+0000 7fec81c00640 1 -- 172.21.15.102:0/1345661756 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c049040 con 0x5566e86e97c0 2024-04-25T01:58:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.324+0000 7fec81c00640 1 -- 172.21.15.102:0/1345661756 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec7c031e50 con 0x5566e86e97c0 2024-04-25T01:58:05.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.324+0000 7fec81c00640 1 -- 172.21.15.102:0/1345661756 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0ed360 con 0x5566e86e97c0 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.325+0000 7fec96e04740 1 -- 172.21.15.102:0/1345661756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 msgr2=0x5566e87004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.325+0000 7fec96e04740 1 --2- 172.21.15.102:0/1345661756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e87004c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fec7c006010 tx=0x7fec7c009260 comp rx=0 tx=0).stop 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.325+0000 7fec96e04740 1 -- 172.21.15.102:0/1345661756 shutdown_connections 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.325+0000 7fec96e04740 1 --2- 172.21.15.102:0/1345661756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86f49a0 0x5566e870c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.325+0000 7fec96e04740 1 --2- 172.21.15.102:0/1345661756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86ea0e0 0x5566e870c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.325+0000 7fec96e04740 1 --2- 172.21.15.102:0/1345661756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e87004c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.325+0000 7fec96e04740 1 -- 172.21.15.102:0/1345661756 >> 172.21.15.102:0/1345661756 conn(0x5566e86e5cd0 msgr2=0x5566e86dc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.325+0000 7fec96e04740 1 -- 172.21.15.102:0/1345661756 shutdown_connections 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.325+0000 7fec96e04740 1 -- 172.21.15.102:0/1345661756 wait complete. 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.326+0000 7fec96e04740 1 -- start start 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.326+0000 7fec96e04740 1 Processor -- start 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec96e04740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec96e04740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec96e04740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86f49a0 0x5566e86fb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec96e04740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c00cd20 con 0x5566e86ea0e0 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec96e04740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c00e190 con 0x5566e86e97c0 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec96e04740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c00d3f0 con 0x5566e86f49a0 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec96e04740 10 client.0 _ll_register_callbacks cb 0x7fec84009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec91600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec91600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45548/0 (socket says 172.21.15.102:45548) 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec91600640 1 -- 172.21.15.102:0/3234019900 learned_addr learned my addr 172.21.15.102:0/3234019900 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec90c00640 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec92000640 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86f49a0 0x5566e86fb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec91600640 1 -- 172.21.15.102:0/3234019900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86f49a0 msgr2=0x5566e86fb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec91600640 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86f49a0 0x5566e86fb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec91600640 1 -- 172.21.15.102:0/3234019900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 msgr2=0x5566e86f63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec91600640 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec91600640 1 -- 172.21.15.102:0/3234019900 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566e86ec630 con 0x5566e86e97c0 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.327+0000 7fec90c00640 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.328+0000 7fec91600640 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f5ea0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0b8180 tx=0x7fec7c0bbde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d99378d35b78974e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.328+0000 7fec76a00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.328+0000 7fec76a00640 1 -- 172.21.15.102:0/3234019900 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c049070 con 0x5566e86e97c0 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.328+0000 7fec76a00640 1 -- 172.21.15.102:0/3234019900 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fec7c0ed1d0 con 0x5566e86e97c0 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.328+0000 7fec96e04740 1 -- 172.21.15.102:0/3234019900 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec7c031e50 con 0x5566e86e97c0 2024-04-25T01:58:05.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.328+0000 7fec76a00640 1 -- 172.21.15.102:0/3234019900 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec7c0c00d0 con 0x5566e86e97c0 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.328+0000 7fec96e04740 10 client.26039 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.328+0000 7fec96e04740 1 -- 172.21.15.102:0/3234019900 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5566e876b190 con 0x5566e86e97c0 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.329+0000 7fec76a00640 1 -- 172.21.15.102:0/3234019900 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec7c031e50 con 0x5566e86e97c0 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.329+0000 7fec76a00640 1 -- 172.21.15.102:0/3234019900 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fec7c0f1ce0 con 0x5566e86e97c0 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.329+0000 7fec96e04740 10 client.26039 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.329+0000 7fec96e04740 10 client.26039 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.329+0000 7fec96e04740 20 client.26039 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.329+0000 7fec96e04740 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5566e876b6c0 0x5566e876dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.329+0000 7fec96e04740 4 client.26039 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.329+0000 7fec96e04740 1 -- 172.21.15.102:0/3234019900 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5566e864c340 con 0x5566e876b6c0 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.329+0000 7fec90c00640 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5566e876b6c0 0x5566e876dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.330+0000 7fec76a00640 10 client.26039 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:05.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.330+0000 7fec90c00640 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5566e876b6c0 0x5566e876dae0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89b8ac34cf82fd7c server_cookie=bd3ca2de3ae1bf1e in_seq=0 out_seq=0 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.342+0000 7fec76a00640 1 -- 172.21.15.102:0/3234019900 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5566e864c340 con 0x5566e876b6c0 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.342+0000 7fec76a00640 10 client.26039 handle_command_reply: tid=0 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 client.26039 shutdown 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 -- 172.21.15.102:0/3234019900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 msgr2=0x5566e86f5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f5ea0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0b8180 tx=0x7fec7c0bbde0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 -- 172.21.15.102:0/3234019900 shutdown_connections 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5566e876b6c0 0x5566e876dae0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5566e86f49a0 0x5566e86fb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5566e86ea0e0 0x5566e86f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 --2- 172.21.15.102:0/3234019900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5566e86e97c0 0x5566e86f5ea0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec76a00640 0 client.26039 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 -- 172.21.15.102:0/3234019900 >> 172.21.15.102:0/3234019900 conn(0x5566e86e5cd0 msgr2=0x5566e86f17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 -- 172.21.15.102:0/3234019900 shutdown_connections 2024-04-25T01:58:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 1 -- 172.21.15.102:0/3234019900 wait complete. 2024-04-25T01:58:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.343+0000 7fec96e04740 20 client.26039 trim_cache size 0 max 16384 2024-04-25T01:58:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:58:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:58:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:58:05.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:58:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 -- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fec700a2d70 msgr2=0x7fec700a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fec700a2d70 0x7fec700a5230 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fec840124a0 tx=0x7fec840097e0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 -- 172.21.15.102:0/2736653762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 msgr2=0x7fec8c19a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 0x7fec8c19a050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fec7c004770 tx=0x7fec7c003c10 comp rx=0 tx=0).stop 2024-04-25T01:58:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 -- 172.21.15.102:0/2736653762 shutdown_connections 2024-04-25T01:58:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fec700a2d70 0x7fec700a5230 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec8c10de90 0x7fec8c19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec8c100c90 0x7fec8c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 --2- 172.21.15.102:0/2736653762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec8c1002e0 0x7fec8c19a050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 -- 172.21.15.102:0/2736653762 >> 172.21.15.102:0/2736653762 conn(0x7fec8c0fbf60 msgr2=0x7fec8c0fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 -- 172.21.15.102:0/2736653762 shutdown_connections 2024-04-25T01:58:05.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:05.344+0000 7fec92a00640 1 -- 172.21.15.102:0/2736653762 wait complete. 2024-04-25T01:58:05.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:04 smithi102 ceph-mon[27796]: pgmap v87: 97 pgs: 97 active+clean; 1.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 10 MiB/s wr, 786 op/s 2024-04-25T01:58:05.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:04 smithi110 ceph-mon[32848]: pgmap v87: 97 pgs: 97 active+clean; 1.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 10 MiB/s wr, 786 op/s 2024-04-25T01:58:07.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:07 smithi102 ceph-mon[27796]: pgmap v88: 97 pgs: 97 active+clean; 1.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.6 MiB/s wr, 655 op/s 2024-04-25T01:58:07.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:07 smithi110 ceph-mon[32848]: pgmap v88: 97 pgs: 97 active+clean; 1.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.6 MiB/s wr, 655 op/s 2024-04-25T01:58:07.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:07 smithi193 ceph-mon[32981]: pgmap v88: 97 pgs: 97 active+clean; 1.8 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.6 MiB/s wr, 655 op/s 2024-04-25T01:58:08.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 135.0 s, 785.8 tps, lat 131.936 ms stddev 297.293 2024-04-25T01:58:09.291 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:09 smithi110 ceph-mon[32848]: pgmap v89: 97 pgs: 97 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 33 MiB/s wr, 1.23k op/s 2024-04-25T01:58:09.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:09 smithi102 ceph-mon[27796]: pgmap v89: 97 pgs: 97 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 33 MiB/s wr, 1.23k op/s 2024-04-25T01:58:09.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:09 smithi193 ceph-mon[32981]: pgmap v89: 97 pgs: 97 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 33 MiB/s wr, 1.23k op/s 2024-04-25T01:58:10.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:58:10.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:58:11.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.094+0000 7f76df400640 1 -- 172.21.15.102:0/2696689216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d810d040 msgr2=0x7f76d8114c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.094+0000 7f76df400640 1 --2- 172.21.15.102:0/2696689216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d810d040 0x7f76d8114c10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009be0 tx=0x7f76d002a6c0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.095+0000 7f76df400640 1 -- 172.21.15.102:0/2696689216 shutdown_connections 2024-04-25T01:58:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.095+0000 7f76df400640 1 --2- 172.21.15.102:0/2696689216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d810d040 0x7f76d8114c10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.095+0000 7f76df400640 1 --2- 172.21.15.102:0/2696689216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d8108970 0x7f76d810c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.095+0000 7f76df400640 1 --2- 172.21.15.102:0/2696689216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76d8107fc0 0x7f76d81083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.095+0000 7f76df400640 1 -- 172.21.15.102:0/2696689216 >> 172.21.15.102:0/2696689216 conn(0x7f76d8075c80 msgr2=0x7f76d80780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.095+0000 7f76df400640 1 -- 172.21.15.102:0/2696689216 shutdown_connections 2024-04-25T01:58:11.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.095+0000 7f76df400640 1 -- 172.21.15.102:0/2696689216 wait complete. 2024-04-25T01:58:11.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.097+0000 7f76df400640 1 Processor -- start 2024-04-25T01:58:11.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.097+0000 7f76df400640 1 -- start start 2024-04-25T01:58:11.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.097+0000 7f76df400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76d8107fc0 0x7f76d819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.097+0000 7f76df400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d810d040 0x7f76d819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.097+0000 7f76df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d81a2e90 0x7f76d81a3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.097+0000 7f76df400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81180d0 con 0x7f76d8107fc0 2024-04-25T01:58:11.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.098+0000 7f76df400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8117f50 con 0x7f76d810d040 2024-04-25T01:58:11.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.098+0000 7f76df400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8118250 con 0x7f76d81a2e90 2024-04-25T01:58:11.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.098+0000 7f76dd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d81a2e90 0x7f76d81a3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.098+0000 7f76dd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d81a2e90 0x7f76d81a3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34296/0 (socket says 172.21.15.102:34296) 2024-04-25T01:58:11.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.098+0000 7f76dd400640 1 -- 172.21.15.102:0/3396184671 learned_addr learned my addr 172.21.15.102:0/3396184671 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:11.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.098+0000 7f76d7e00640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d810d040 0x7f76d819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.099+0000 7f76dca00640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76d8107fc0 0x7f76d819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.099+0000 7f76dd400640 1 -- 172.21.15.102:0/3396184671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d810d040 msgr2=0x7f76d819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.099+0000 7f76dd400640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d810d040 0x7f76d819eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.099+0000 7f76dd400640 1 -- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76d8107fc0 msgr2=0x7f76d819e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.099+0000 7f76dd400640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76d8107fc0 0x7f76d819e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.099+0000 7f76dd400640 1 -- 172.21.15.102:0/3396184671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d81a3980 con 0x7f76d81a2e90 2024-04-25T01:58:11.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.100+0000 7f76dd400640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d81a2e90 0x7f76d81a3340 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009b30 tx=0x7f76d00305b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:11.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.100+0000 7f76d5600640 1 -- 172.21.15.102:0/3396184671 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0041020 con 0x7f76d81a2e90 2024-04-25T01:58:11.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.100+0000 7f76d5600640 1 -- 172.21.15.102:0/3396184671 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d0003e50 con 0x7f76d81a2e90 2024-04-25T01:58:11.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.100+0000 7f76d5600640 1 -- 172.21.15.102:0/3396184671 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0036040 con 0x7f76d81a2e90 2024-04-25T01:58:11.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.101+0000 7f76df400640 1 -- 172.21.15.102:0/3396184671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d81a3c10 con 0x7f76d81a2e90 2024-04-25T01:58:11.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.101+0000 7f76df400640 1 -- 172.21.15.102:0/3396184671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d807a1d0 con 0x7f76d81a2e90 2024-04-25T01:58:11.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.106+0000 7f76d5600640 1 -- 172.21.15.102:0/3396184671 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f76d0044020 con 0x7f76d81a2e90 2024-04-25T01:58:11.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.108+0000 7f76e221a740 1 Processor -- start 2024-04-25T01:58:11.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.109+0000 7f76d5600640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f76c00a2d80 0x7f76c00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.109+0000 7f76d5600640 1 -- 172.21.15.102:0/3396184671 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f76d00f0020 con 0x7f76d81a2e90 2024-04-25T01:58:11.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.109+0000 7f76e221a740 1 -- start start 2024-04-25T01:58:11.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.110+0000 7f76d7e00640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f76c00a2d80 0x7f76c00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.110+0000 7f76e221a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.110+0000 7f76e221a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22d5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.110+0000 7f76e221a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22d5ff0 0x5621d22d83e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.110+0000 7f76e221a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5621d222f560 con 0x5621d22d5ff0 2024-04-25T01:58:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.110+0000 7f76e221a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5621d21d7930 con 0x5621d22cb920 2024-04-25T01:58:11.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.110+0000 7f76e221a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5621d222b520 con 0x5621d22cb000 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34302/0 (socket says 172.21.15.102:34302) 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76dca00640 1 -- 172.21.15.102:0/3278314725 learned_addr learned my addr 172.21.15.102:0/3278314725 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76dd400640 1 --2- 172.21.15.102:0/3278314725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22d5ff0 0x5621d22d83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76d7e00640 1 --2- 172.21.15.102:0/3278314725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22d5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76dca00640 1 -- 172.21.15.102:0/3278314725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 msgr2=0x5621d22d5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76dca00640 1 --2- 172.21.15.102:0/3278314725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22d5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76dca00640 1 -- 172.21.15.102:0/3278314725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22d5ff0 msgr2=0x5621d22d83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76dca00640 1 --2- 172.21.15.102:0/3278314725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22d5ff0 0x5621d22d83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.111+0000 7f76dca00640 1 -- 172.21.15.102:0/3278314725 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621d200d7d0 con 0x5621d22cb000 2024-04-25T01:58:11.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.112+0000 7f76d7e00640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f76c00a2d80 0x7f76c00a5240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f76c40096c0 tx=0x7f76c40103d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:11.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.112+0000 7f76dca00640 1 --2- 172.21.15.102:0/3278314725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22cb3e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009b60 tx=0x7f76cc00e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9cbfb193677bfb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:11.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.113+0000 7f76c9c00640 1 -- 172.21.15.102:0/3278314725 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc016070 con 0x5621d22cb000 2024-04-25T01:58:11.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.113+0000 7f76c9c00640 1 -- 172.21.15.102:0/3278314725 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76cc009e50 con 0x5621d22cb000 2024-04-25T01:58:11.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.113+0000 7f76c9c00640 1 -- 172.21.15.102:0/3278314725 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc016070 con 0x5621d22cb000 2024-04-25T01:58:11.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.114+0000 7f76e221a740 1 -- 172.21.15.102:0/3278314725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 msgr2=0x5621d22cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.114+0000 7f76e221a740 1 --2- 172.21.15.102:0/3278314725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22cb3e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009b60 tx=0x7f76cc00e6c0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.114+0000 7f76e221a740 1 -- 172.21.15.102:0/3278314725 shutdown_connections 2024-04-25T01:58:11.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.114+0000 7f76e221a740 1 --2- 172.21.15.102:0/3278314725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22d5ff0 0x5621d22d83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.114+0000 7f76e221a740 1 --2- 172.21.15.102:0/3278314725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22d5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.114+0000 7f76e221a740 1 --2- 172.21.15.102:0/3278314725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22cb3e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.114+0000 7f76e221a740 1 -- 172.21.15.102:0/3278314725 >> 172.21.15.102:0/3278314725 conn(0x5621d22c7320 msgr2=0x5621d22c7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:11.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.115+0000 7f76e221a740 1 -- 172.21.15.102:0/3278314725 shutdown_connections 2024-04-25T01:58:11.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.115+0000 7f76e221a740 1 -- 172.21.15.102:0/3278314725 wait complete. 2024-04-25T01:58:11.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.116+0000 7f76e221a740 1 -- start start 2024-04-25T01:58:11.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.117+0000 7f76e221a740 1 Processor -- start 2024-04-25T01:58:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.118+0000 7f76e221a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22d2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.118+0000 7f76e221a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb920 0x5621d22d34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.118+0000 7f76e221a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22d5ff0 0x5621d22d3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.119+0000 7f76e221a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5621d22d8de0 con 0x5621d22cb920 2024-04-25T01:58:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.119+0000 7f76e221a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5621d21ead90 con 0x5621d22d5ff0 2024-04-25T01:58:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.119+0000 7f76e221a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5621d22d8f60 con 0x5621d22cb000 2024-04-25T01:58:11.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.119+0000 7f76e221a740 10 client.0 _ll_register_callbacks cb 0x5621d21ea780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:11.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.119+0000 7f76dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22d2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.119+0000 7f76dd400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22d5ff0 0x5621d22d3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.119+0000 7f76dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22d2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34314/0 (socket says 172.21.15.102:34314) 2024-04-25T01:58:11.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.119+0000 7f76dca00640 1 -- 172.21.15.102:0/1687183267 learned_addr learned my addr 172.21.15.102:0/1687183267 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:11.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.119+0000 7f76dd400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22d5ff0 0x5621d22d3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45598/0 (socket says 172.21.15.102:45598) 2024-04-25T01:58:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.120+0000 7f76dd400640 1 -- 172.21.15.102:0/1687183267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 msgr2=0x5621d22d2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.120+0000 7f76dd400640 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22d2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.120+0000 7f76dd400640 1 -- 172.21.15.102:0/1687183267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb920 msgr2=0x5621d22d34c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:58:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.120+0000 7f76dd400640 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb920 0x5621d22d34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.120+0000 7f76dd400640 1 -- 172.21.15.102:0/1687183267 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621d1f433f0 con 0x5621d22d5ff0 2024-04-25T01:58:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.121+0000 7f76dd400640 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22d5ff0 0x5621d22d3a00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f76d00ed740 tx=0x7f76d0032b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e41d3363e1c08162 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.121+0000 7f76be000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.121+0000 7f76be000640 1 -- 172.21.15.102:0/1687183267 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d00d5060 con 0x5621d22d5ff0 2024-04-25T01:58:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.121+0000 7f76be000640 1 -- 172.21.15.102:0/1687183267 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f76d00ed9c0 con 0x5621d22d5ff0 2024-04-25T01:58:11.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.121+0000 7f76e221a740 1 -- 172.21.15.102:0/1687183267 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76cc009e50 con 0x5621d22d5ff0 2024-04-25T01:58:11.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.121+0000 7f76be000640 1 -- 172.21.15.102:0/1687183267 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f76d00da050 con 0x5621d22d5ff0 2024-04-25T01:58:11.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76e221a740 10 client.26051 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:11.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76e221a740 1 -- 172.21.15.102:0/1687183267 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5621d1f030f0 con 0x5621d22d5ff0 2024-04-25T01:58:11.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76be000640 1 -- 172.21.15.102:0/1687183267 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76cc009e50 con 0x5621d22d5ff0 2024-04-25T01:58:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76be000640 1 -- 172.21.15.102:0/1687183267 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f76d004e060 con 0x5621d22d5ff0 2024-04-25T01:58:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76e221a740 10 client.26051 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76e221a740 10 client.26051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76e221a740 20 client.26051 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76e221a740 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621d22e6370 0x5621d22e8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76e221a740 4 client.26051 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.122+0000 7f76e221a740 1 -- 172.21.15.102:0/1687183267 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621d2242d90 con 0x5621d22e6370 2024-04-25T01:58:11.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.123+0000 7f76dca00640 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621d22e6370 0x5621d22e8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.123+0000 7f76dca00640 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621d22e6370 0x5621d22e8810 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=630b592a6fe9f1fa server_cookie=54bcc6621816dbd8 in_seq=0 out_seq=0 2024-04-25T01:58:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.123+0000 7f76be000640 10 client.26051 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.124+0000 7f76be000640 1 -- 172.21.15.102:0/1687183267 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621d2242d90 con 0x5621d22e6370 2024-04-25T01:58:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.124+0000 7f76be000640 10 client.26051 handle_command_reply: tid=0 2024-04-25T01:58:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.124+0000 7f76e221a740 1 client.26051 shutdown 2024-04-25T01:58:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.125+0000 7f76e221a740 1 -- 172.21.15.102:0/1687183267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22d5ff0 msgr2=0x5621d22d3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.125+0000 7f76e221a740 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22d5ff0 0x5621d22d3a00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f76d00ed740 tx=0x7f76d0032b30 comp rx=0 tx=0).stop 2024-04-25T01:58:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.125+0000 7f76e221a740 1 -- 172.21.15.102:0/1687183267 shutdown_connections 2024-04-25T01:58:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.125+0000 7f76e221a740 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621d22e6370 0x5621d22e8810 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.125+0000 7f76e221a740 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22d5ff0 0x5621d22d3a00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.125+0000 7f76e221a740 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb920 0x5621d22d34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.125+0000 7f76e221a740 1 --2- 172.21.15.102:0/1687183267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb000 0x5621d22d2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.125+0000 7f76e221a740 1 -- 172.21.15.102:0/1687183267 >> 172.21.15.102:0/1687183267 conn(0x5621d22c7320 msgr2=0x5621d22cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.125+0000 7f76be000640 0 client.26051 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.126+0000 7f76e221a740 1 -- 172.21.15.102:0/1687183267 shutdown_connections 2024-04-25T01:58:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.126+0000 7f76e221a740 1 -- 172.21.15.102:0/1687183267 wait complete. 2024-04-25T01:58:11.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.127+0000 7f76e221a740 20 client.26051 trim_cache size 0 max 16384 2024-04-25T01:58:11.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.164+0000 7f76e221a740 1 Processor -- start 2024-04-25T01:58:11.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.165+0000 7f76e221a740 1 -- start start 2024-04-25T01:58:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76e221a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22c7320 0x5621d22e1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76e221a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb000 0x5621d22d1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76e221a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb920 0x5621d22cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76e221a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f76d806db80 con 0x5621d22c7320 2024-04-25T01:58:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76e221a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76d806b3f0 con 0x5621d22cb000 2024-04-25T01:58:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76e221a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8080880 con 0x5621d22cb920 2024-04-25T01:58:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76d7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb000 0x5621d22d1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76d7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb000 0x5621d22d1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45602/0 (socket says 172.21.15.102:45602) 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76d7e00640 1 -- 172.21.15.102:0/3237440159 learned_addr learned my addr 172.21.15.102:0/3237440159 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.166+0000 7f76dca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22c7320 0x5621d22e1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.167+0000 7f76dd400640 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb920 0x5621d22cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.167+0000 7f76d7e00640 1 -- 172.21.15.102:0/3237440159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb920 msgr2=0x5621d22cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.167+0000 7f76d7e00640 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb920 0x5621d22cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.167+0000 7f76d7e00640 1 -- 172.21.15.102:0/3237440159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22c7320 msgr2=0x5621d22e1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.167+0000 7f76d7e00640 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22c7320 0x5621d22e1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.167+0000 7f76d7e00640 1 -- 172.21.15.102:0/3237440159 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621d22c6d90 con 0x5621d22cb000 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.167+0000 7f76dd400640 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb920 0x5621d22cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.167+0000 7f76dca00640 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22c7320 0x5621d22e1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:11.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.167+0000 7f76d7e00640 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb000 0x5621d22d1ec0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f76c4010fc0 tx=0x7f76c4015070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a7bad547174c1ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:11.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.168+0000 7f76c9200640 1 -- 172.21.15.102:0/3237440159 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4028470 con 0x5621d22cb000 2024-04-25T01:58:11.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.168+0000 7f76c9200640 1 -- 172.21.15.102:0/3237440159 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c4023070 con 0x5621d22cb000 2024-04-25T01:58:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.168+0000 7f76c9200640 1 -- 172.21.15.102:0/3237440159 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4011840 con 0x5621d22cb000 2024-04-25T01:58:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.169+0000 7f76e221a740 1 -- 172.21.15.102:0/3237440159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb000 msgr2=0x5621d22d1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.169+0000 7f76e221a740 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb000 0x5621d22d1ec0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f76c4010fc0 tx=0x7f76c4015070 comp rx=0 tx=0).stop 2024-04-25T01:58:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.169+0000 7f76e221a740 1 -- 172.21.15.102:0/3237440159 shutdown_connections 2024-04-25T01:58:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.169+0000 7f76e221a740 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22cb920 0x5621d22cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.169+0000 7f76e221a740 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb000 0x5621d22d1ec0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.169+0000 7f76e221a740 1 --2- 172.21.15.102:0/3237440159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22c7320 0x5621d22e1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.169+0000 7f76e221a740 1 -- 172.21.15.102:0/3237440159 >> 172.21.15.102:0/3237440159 conn(0x5621d22ce950 msgr2=0x5621d22d37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:11.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.169+0000 7f76e221a740 1 -- 172.21.15.102:0/3237440159 shutdown_connections 2024-04-25T01:58:11.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.169+0000 7f76e221a740 1 -- 172.21.15.102:0/3237440159 wait complete. 2024-04-25T01:58:11.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.170+0000 7f76e221a740 1 -- start start 2024-04-25T01:58:11.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.171+0000 7f76e221a740 1 Processor -- start 2024-04-25T01:58:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.172+0000 7f76e221a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22c7320 0x5621d22d9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.172+0000 7f76e221a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb000 0x5621d22da2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.172+0000 7f76e221a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22da830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.172+0000 7f76e221a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8080ef0 con 0x5621d22cb000 2024-04-25T01:58:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.172+0000 7f76e221a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8082360 con 0x5621d22cb920 2024-04-25T01:58:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.172+0000 7f76e221a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76d80815c0 con 0x5621d22c7320 2024-04-25T01:58:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.172+0000 7f76e221a740 10 client.0 _ll_register_callbacks cb 0x7f76d00bc050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:11.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:11 smithi102 ceph-mon[27796]: pgmap v90: 97 pgs: 97 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 29 MiB/s wr, 880 op/s 2024-04-25T01:58:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.173+0000 7f76dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22c7320 0x5621d22d9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.173+0000 7f76d7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb000 0x5621d22da2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.173+0000 7f76dd400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22da830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.173+0000 7f76dd400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22da830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45608/0 (socket says 172.21.15.102:45608) 2024-04-25T01:58:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.173+0000 7f76dd400640 1 -- 172.21.15.102:0/2446192934 learned_addr learned my addr 172.21.15.102:0/2446192934 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:11.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.174+0000 7f76dd400640 1 -- 172.21.15.102:0/2446192934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22c7320 msgr2=0x5621d22d9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:11.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.174+0000 7f76dd400640 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22c7320 0x5621d22d9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.174+0000 7f76dd400640 1 -- 172.21.15.102:0/2446192934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb000 msgr2=0x5621d22da2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:11.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.174+0000 7f76dd400640 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb000 0x5621d22da2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.174+0000 7f76dd400640 1 -- 172.21.15.102:0/2446192934 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621d234bb60 con 0x5621d22cb920 2024-04-25T01:58:11.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.174+0000 7f76d7e00640 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb000 0x5621d22da2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:11.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.175+0000 7f76dca00640 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22c7320 0x5621d22d9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:11.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.175+0000 7f76dd400640 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22da830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f76d00c5340 tx=0x7f76d00edbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7002e847ddf0f4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:11.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.175+0000 7f76be000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:11.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.176+0000 7f76be000640 1 -- 172.21.15.102:0/2446192934 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0086080 con 0x5621d22cb920 2024-04-25T01:58:11.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.176+0000 7f76be000640 1 -- 172.21.15.102:0/2446192934 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f76d00d5060 con 0x5621d22cb920 2024-04-25T01:58:11.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.176+0000 7f76e221a740 1 -- 172.21.15.102:0/2446192934 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76c4023070 con 0x5621d22cb920 2024-04-25T01:58:11.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.177+0000 7f76be000640 1 -- 172.21.15.102:0/2446192934 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f76d00bc510 con 0x5621d22cb920 2024-04-25T01:58:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.177+0000 7f76e221a740 10 client.26057 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.177+0000 7f76e221a740 1 -- 172.21.15.102:0/2446192934 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5621d234bb60 con 0x5621d22cb920 2024-04-25T01:58:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.178+0000 7f76be000640 1 -- 172.21.15.102:0/2446192934 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c4023070 con 0x5621d22cb920 2024-04-25T01:58:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.178+0000 7f76be000640 1 -- 172.21.15.102:0/2446192934 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f76d00bb960 con 0x5621d22cb920 2024-04-25T01:58:11.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.178+0000 7f76e221a740 10 client.26057 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:11.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.178+0000 7f76e221a740 10 client.26057 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:11.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.178+0000 7f76e221a740 20 client.26057 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:11.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.178+0000 7f76e221a740 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621d22d5ff0 0x5621d234f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.178+0000 7f76e221a740 4 client.26057 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:58:11.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.178+0000 7f76e221a740 1 -- 172.21.15.102:0/2446192934 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5621d223caa0 con 0x5621d22d5ff0 2024-04-25T01:58:11.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.179+0000 7f76dca00640 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621d22d5ff0 0x5621d234f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.179+0000 7f76be000640 10 client.26057 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:11.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.180+0000 7f76dca00640 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621d22d5ff0 0x5621d234f0d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=158859177f011f08 server_cookie=fad287d8bc632325 in_seq=0 out_seq=0 2024-04-25T01:58:11.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.180+0000 7f76be000640 1 -- 172.21.15.102:0/2446192934 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5621d223caa0 con 0x5621d22d5ff0 2024-04-25T01:58:11.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.180+0000 7f76be000640 10 client.26057 handle_command_reply: tid=0 2024-04-25T01:58:11.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.180+0000 7f76e221a740 1 client.26057 shutdown 2024-04-25T01:58:11.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.180+0000 7f76e221a740 1 -- 172.21.15.102:0/2446192934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 msgr2=0x5621d22da830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.180+0000 7f76e221a740 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22da830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f76d00c5340 tx=0x7f76d00edbf0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.182+0000 7f76e221a740 1 -- 172.21.15.102:0/2446192934 shutdown_connections 2024-04-25T01:58:11.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.182+0000 7f76e221a740 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621d22d5ff0 0x5621d234f0d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.182+0000 7f76e221a740 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621d22cb920 0x5621d22da830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.182+0000 7f76e221a740 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621d22cb000 0x5621d22da2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.182+0000 7f76e221a740 1 --2- 172.21.15.102:0/2446192934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621d22c7320 0x5621d22d9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.182+0000 7f76e221a740 1 -- 172.21.15.102:0/2446192934 >> 172.21.15.102:0/2446192934 conn(0x5621d22ce950 msgr2=0x5621d22c5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:11.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.182+0000 7f76be000640 0 client.26057 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:11.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.182+0000 7f76e221a740 1 -- 172.21.15.102:0/2446192934 shutdown_connections 2024-04-25T01:58:11.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.182+0000 7f76e221a740 1 -- 172.21.15.102:0/2446192934 wait complete. 2024-04-25T01:58:11.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.183+0000 7f76e221a740 20 client.26057 trim_cache size 0 max 16384 2024-04-25T01:58:11.186 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:58:11.186 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T01:58:11.186 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T01:58:11.186 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:58:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 -- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f76c00a2d80 msgr2=0x7f76c00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f76c00a2d80 0x7f76c00a5240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f76c40096c0 tx=0x7f76c40103d0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 -- 172.21.15.102:0/3396184671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d81a2e90 msgr2=0x7f76d81a3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d81a2e90 0x7f76d81a3340 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009b30 tx=0x7f76d00305b0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 -- 172.21.15.102:0/3396184671 shutdown_connections 2024-04-25T01:58:11.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f76c00a2d80 0x7f76c00a5240 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d81a2e90 0x7f76d81a3340 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76d810d040 0x7f76d819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 --2- 172.21.15.102:0/3396184671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76d8107fc0 0x7f76d819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.184+0000 7f76df400640 1 -- 172.21.15.102:0/3396184671 >> 172.21.15.102:0/3396184671 conn(0x7f76d8075c80 msgr2=0x7f76d8077ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:11.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.185+0000 7f76df400640 1 -- 172.21.15.102:0/3396184671 shutdown_connections 2024-04-25T01:58:11.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.186+0000 7f76df400640 1 -- 172.21.15.102:0/3396184671 wait complete. 2024-04-25T01:58:11.221 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:58:11.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:58:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.484+0000 7feb13e00640 1 -- 172.21.15.102:0/3615443176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c108970 msgr2=0x7feb0c10c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.484+0000 7feb13e00640 1 --2- 172.21.15.102:0/3615443176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c108970 0x7feb0c10c9b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009860 tx=0x7feaf802a340 comp rx=0 tx=0).stop 2024-04-25T01:58:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.485+0000 7feb13e00640 1 -- 172.21.15.102:0/3615443176 shutdown_connections 2024-04-25T01:58:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.485+0000 7feb13e00640 1 --2- 172.21.15.102:0/3615443176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb0c10d090 0x7feb0c114c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.485+0000 7feb13e00640 1 --2- 172.21.15.102:0/3615443176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c108970 0x7feb0c10c9b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.485+0000 7feb13e00640 1 --2- 172.21.15.102:0/3615443176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb0c107fc0 0x7feb0c1083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.485+0000 7feb13e00640 1 -- 172.21.15.102:0/3615443176 >> 172.21.15.102:0/3615443176 conn(0x7feb0c074f40 msgr2=0x7feb0c106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.485+0000 7feb13e00640 1 -- 172.21.15.102:0/3615443176 shutdown_connections 2024-04-25T01:58:11.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.485+0000 7feb13e00640 1 -- 172.21.15.102:0/3615443176 wait complete. 2024-04-25T01:58:11.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.486+0000 7feb13e00640 1 Processor -- start 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.486+0000 7feb13e00640 1 -- start start 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.486+0000 7feb13e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c107fc0 0x7feb0c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.486+0000 7feb13e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb0c10d090 0x7feb0c19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.486+0000 7feb13e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb0c1a2f50 0x7feb0c1a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.486+0000 7feb13e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feb0c118180 con 0x7feb0c10d090 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.486+0000 7feb13e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feb0c118000 con 0x7feb0c1a2f50 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.486+0000 7feb13e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feb0c118300 con 0x7feb0c107fc0 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb10a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb0c10d090 0x7feb0c19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c107fc0 0x7feb0c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c107fc0 0x7feb0c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34338/0 (socket says 172.21.15.102:34338) 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb10a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb0c10d090 0x7feb0c19ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36848/0 (socket says 172.21.15.102:36848) 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11400640 1 -- 172.21.15.102:0/1792014214 learned_addr learned my addr 172.21.15.102:0/1792014214 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11e00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb0c1a2f50 0x7feb0c1a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11400640 1 -- 172.21.15.102:0/1792014214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb0c1a2f50 msgr2=0x7feb0c1a3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11400640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb0c1a2f50 0x7feb0c1a3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11400640 1 -- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb0c10d090 msgr2=0x7feb0c19ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11400640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb0c10d090 0x7feb0c19ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11400640 1 -- 172.21.15.102:0/1792014214 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb0c1a3a40 con 0x7feb0c107fc0 2024-04-25T01:58:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb10a00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb0c10d090 0x7feb0c19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.487+0000 7feb11400640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c107fc0 0x7feb0c19e6e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feb0000e9c0 tx=0x7feb0000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.488+0000 7feb0a000640 1 -- 172.21.15.102:0/1792014214 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0000ccc0 con 0x7feb0c107fc0 2024-04-25T01:58:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.488+0000 7feb0a000640 1 -- 172.21.15.102:0/1792014214 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb00005500 con 0x7feb0c107fc0 2024-04-25T01:58:11.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.488+0000 7feb0a000640 1 -- 172.21.15.102:0/1792014214 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb00010430 con 0x7feb0c107fc0 2024-04-25T01:58:11.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:11 smithi110 ceph-mon[32848]: pgmap v90: 97 pgs: 97 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 29 MiB/s wr, 880 op/s 2024-04-25T01:58:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.489+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb0c1a3d30 con 0x7feb0c107fc0 2024-04-25T01:58:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.489+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb0c1ab4a0 con 0x7feb0c107fc0 2024-04-25T01:58:11.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.491+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb0c0689b0 con 0x7feb0c107fc0 2024-04-25T01:58:11.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.495+0000 7feb0a000640 1 -- 172.21.15.102:0/1792014214 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feb000046c0 con 0x7feb0c107fc0 2024-04-25T01:58:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.497+0000 7feb0a000640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feaf40a2d00 0x7feaf40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.497+0000 7feb0a000640 1 -- 172.21.15.102:0/1792014214 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7feb000cf7c0 con 0x7feb0c107fc0 2024-04-25T01:58:11.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.498+0000 7feb10a00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feaf40a2d00 0x7feaf40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:11.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.499+0000 7feb10a00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feaf40a2d00 0x7feaf40a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009830 tx=0x7feaf80056d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:11.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.501+0000 7feb0a000640 1 -- 172.21.15.102:0/1792014214 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb0009c410 con 0x7feb0c107fc0 2024-04-25T01:58:11.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:11 smithi193 ceph-mon[32981]: pgmap v90: 97 pgs: 97 active+clean; 2.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 29 MiB/s wr, 880 op/s 2024-04-25T01:58:11.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.927+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb0c108970 con 0x7feb0c107fc0 2024-04-25T01:58:11.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.929+0000 7feb0a000640 1 -- 172.21.15.102:0/1792014214 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11901 (secure 0 0 0) 0x7feb0009bd10 con 0x7feb0c107fc0 2024-04-25T01:58:11.932 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:58:11.932 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:56:02.805319+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24541,24587,24533]},"id":1}]} 2024-04-25T01:58:11.932 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 27 2024-04-25T01:58:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.932+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feaf40a2d00 msgr2=0x7feaf40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feaf40a2d00 0x7feaf40a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009830 tx=0x7feaf80056d0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c107fc0 msgr2=0x7feb0c19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:11.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c107fc0 0x7feb0c19e6e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feb0000e9c0 tx=0x7feb0000ee80 comp rx=0 tx=0).stop 2024-04-25T01:58:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 shutdown_connections 2024-04-25T01:58:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feaf40a2d00 0x7feaf40a51c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb0c1a2f50 0x7feb0c1a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb0c10d090 0x7feb0c19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 --2- 172.21.15.102:0/1792014214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb0c107fc0 0x7feb0c19e6e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 >> 172.21.15.102:0/1792014214 conn(0x7feb0c074f40 msgr2=0x7feb0c076b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:11.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.933+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 shutdown_connections 2024-04-25T01:58:11.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:11.934+0000 7feb13e00640 1 -- 172.21.15.102:0/1792014214 wait complete. 2024-04-25T01:58:11.963 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:58:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.239+0000 7f7000800640 1 -- 172.21.15.102:0/2872587811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ff8075c70 msgr2=0x7f6ff8076050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.239+0000 7f7000800640 1 --2- 172.21.15.102:0/2872587811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ff8075c70 0x7f6ff8076050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f6ff001ace0 tx=0x7f6ff003b950 comp rx=0 tx=0).stop 2024-04-25T01:58:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.239+0000 7f7000800640 1 -- 172.21.15.102:0/2872587811 shutdown_connections 2024-04-25T01:58:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.239+0000 7f7000800640 1 --2- 172.21.15.102:0/2872587811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff8112620 0x7f6ff8114a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.239+0000 7f7000800640 1 --2- 172.21.15.102:0/2872587811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff8076590 0x7f6ff81120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.239+0000 7f7000800640 1 --2- 172.21.15.102:0/2872587811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ff8075c70 0x7f6ff8076050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.239+0000 7f7000800640 1 -- 172.21.15.102:0/2872587811 >> 172.21.15.102:0/2872587811 conn(0x7f6ff80ff4d0 msgr2=0x7f6ff81018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.239+0000 7f7000800640 1 -- 172.21.15.102:0/2872587811 shutdown_connections 2024-04-25T01:58:12.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.239+0000 7f7000800640 1 -- 172.21.15.102:0/2872587811 wait complete. 2024-04-25T01:58:12.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.242+0000 7f7000800640 1 Processor -- start 2024-04-25T01:58:12.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.243+0000 7f7000800640 1 -- start start 2024-04-25T01:58:12.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.244+0000 7f7000800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff8075c70 0x7f6ff8111dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.244+0000 7f7000800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff8076590 0x7f6ff810de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.244+0000 7f7000800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ff8112620 0x7f6ff810e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.244+0000 7f7000800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff8106270 con 0x7f6ff8076590 2024-04-25T01:58:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.244+0000 7f7000800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81060f0 con 0x7f6ff8075c70 2024-04-25T01:58:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.244+0000 7f7000800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81063f0 con 0x7f6ff8112620 2024-04-25T01:58:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff8075c70 0x7f6ff8111dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff8075c70 0x7f6ff8111dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45624/0 (socket says 172.21.15.102:45624) 2024-04-25T01:58:12.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffde00640 1 -- 172.21.15.102:0/2523170800 learned_addr learned my addr 172.21.15.102:0/2523170800 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffd400640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff8076590 0x7f6ff810de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffde00640 1 -- 172.21.15.102:0/2523170800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ff8112620 msgr2=0x7f6ff810e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffde00640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ff8112620 0x7f6ff810e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffde00640 1 -- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff8076590 msgr2=0x7f6ff810de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffde00640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff8076590 0x7f6ff810de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffde00640 1 -- 172.21.15.102:0/2523170800 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff810eb20 con 0x7f6ff8075c70 2024-04-25T01:58:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.245+0000 7f6ffd400640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff8076590 0x7f6ff810de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:12.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.247+0000 7f6ffde00640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff8075c70 0x7f6ff8111dd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6ff001acb0 tx=0x7f6ff0040750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.247+0000 7f6feea00640 1 -- 172.21.15.102:0/2523170800 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0045070 con 0x7f6ff8075c70 2024-04-25T01:58:12.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.249+0000 7f6feea00640 1 -- 172.21.15.102:0/2523170800 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff00408f0 con 0x7f6ff8075c70 2024-04-25T01:58:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.249+0000 7f7000800640 1 -- 172.21.15.102:0/2523170800 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff81a2550 con 0x7f6ff8075c70 2024-04-25T01:58:12.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.249+0000 7f6feea00640 1 -- 172.21.15.102:0/2523170800 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0004500 con 0x7f6ff8075c70 2024-04-25T01:58:12.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.249+0000 7f7000800640 1 -- 172.21.15.102:0/2523170800 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff81a2950 con 0x7f6ff8075c70 2024-04-25T01:58:12.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.257+0000 7f6feea00640 1 -- 172.21.15.102:0/2523170800 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6ff0048020 con 0x7f6ff8075c70 2024-04-25T01:58:12.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.260+0000 7f6feea00640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6fe40a2ed0 0x7f6fe40a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.261+0000 7f6feea00640 1 -- 172.21.15.102:0/2523170800 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6ff0100e70 con 0x7f6ff8075c70 2024-04-25T01:58:12.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.261+0000 7f7003625740 1 Processor -- start 2024-04-25T01:58:12.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.262+0000 7f7003625740 1 -- start start 2024-04-25T01:58:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.262+0000 7f7003625740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d0a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.262+0000 7f7003625740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d15070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.262+0000 7f7003625740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d155b0 0x55e1d1d179a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.262+0000 7f7003625740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55e1d1c6f4e0 con 0x55e1d1d155b0 2024-04-25T01:58:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.262+0000 7f7003625740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e1d1c17a70 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.262+0000 7f7003625740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e1d1c6b4a0 con 0x55e1d1d0aee0 2024-04-25T01:58:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.263+0000 7f6ffde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d0a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.263+0000 7f6ffd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d15070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.263+0000 7f6ffde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d0a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45640/0 (socket says 172.21.15.102:45640) 2024-04-25T01:58:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.263+0000 7f6ffde00640 1 -- 172.21.15.102:0/3186436554 learned_addr learned my addr 172.21.15.102:0/3186436554 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.263+0000 7f6ffd400640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6fe40a2ed0 0x7f6fe40a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.263+0000 7f6ffe800640 1 --2- 172.21.15.102:0/3186436554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d155b0 0x55e1d1d179a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.265+0000 7f6ffe800640 1 -- 172.21.15.102:0/3186436554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 msgr2=0x55e1d1d15070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.265+0000 7f6ffe800640 1 --2- 172.21.15.102:0/3186436554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d15070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.265+0000 7f6ffe800640 1 -- 172.21.15.102:0/3186436554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 msgr2=0x55e1d1d0a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.265+0000 7f6ffe800640 1 --2- 172.21.15.102:0/3186436554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d0a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.265+0000 7f6ffe800640 1 -- 172.21.15.102:0/3186436554 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1d19833f0 con 0x55e1d1d155b0 2024-04-25T01:58:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.266+0000 7f6ffe800640 1 --2- 172.21.15.102:0/3186436554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d155b0 0x55e1d1d179a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6ff806ac20 tx=0x7f6ff400ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb58d39c9c09360 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.266+0000 7f6ffd400640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6fe40a2ed0 0x7f6fe40a5390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6ff810f620 tx=0x7f6fe8009770 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.269+0000 7f6fe3000640 1 -- 172.21.15.102:0/3186436554 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400ccc0 con 0x55e1d1d155b0 2024-04-25T01:58:12.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.269+0000 7f6fe3000640 1 -- 172.21.15.102:0/3186436554 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff40035f0 con 0x55e1d1d155b0 2024-04-25T01:58:12.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.269+0000 7f6fe3000640 1 -- 172.21.15.102:0/3186436554 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400ce60 con 0x55e1d1d155b0 2024-04-25T01:58:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.270+0000 7f7003625740 1 -- 172.21.15.102:0/3186436554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d155b0 msgr2=0x55e1d1d179a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.270+0000 7f7003625740 1 --2- 172.21.15.102:0/3186436554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d155b0 0x55e1d1d179a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6ff806ac20 tx=0x7f6ff400ac70 comp rx=0 tx=0).stop 2024-04-25T01:58:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.270+0000 7f7003625740 1 -- 172.21.15.102:0/3186436554 shutdown_connections 2024-04-25T01:58:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.270+0000 7f7003625740 1 --2- 172.21.15.102:0/3186436554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d155b0 0x55e1d1d179a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.270+0000 7f7003625740 1 --2- 172.21.15.102:0/3186436554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d15070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.270+0000 7f7003625740 1 --2- 172.21.15.102:0/3186436554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d0a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.270+0000 7f7003625740 1 -- 172.21.15.102:0/3186436554 >> 172.21.15.102:0/3186436554 conn(0x55e1d1d068e0 msgr2=0x55e1d1d06cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.271+0000 7f7003625740 1 -- 172.21.15.102:0/3186436554 shutdown_connections 2024-04-25T01:58:12.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.271+0000 7f7003625740 1 -- 172.21.15.102:0/3186436554 wait complete. 2024-04-25T01:58:12.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.272+0000 7f7003625740 1 -- start start 2024-04-25T01:58:12.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.273+0000 7f7003625740 1 Processor -- start 2024-04-25T01:58:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.273+0000 7f7003625740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d12560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.273+0000 7f7003625740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d12aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.274+0000 7f7003625740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d155b0 0x55e1d1d12fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.274+0000 7f7003625740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55e1d1d183a0 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.274+0000 7f7003625740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e1d1c2aec0 con 0x55e1d1d155b0 2024-04-25T01:58:12.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.274+0000 7f7003625740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e1d1d18520 con 0x55e1d1d0aee0 2024-04-25T01:58:12.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.274+0000 7f7003625740 10 client.0 _ll_register_callbacks cb 0x55e1d1c673f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:12.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.274+0000 7f6ffe800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d155b0 0x55e1d1d12fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.274+0000 7f6ffe800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d155b0 0x55e1d1d12fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45652/0 (socket says 172.21.15.102:45652) 2024-04-25T01:58:12.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.274+0000 7f6ffe800640 1 -- 172.21.15.102:0/2135369300 learned_addr learned my addr 172.21.15.102:0/2135369300 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.275+0000 7f6ffd400640 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d12aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.275+0000 7f6ffe800640 1 -- 172.21.15.102:0/2135369300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 msgr2=0x55e1d1d12aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.275+0000 7f6ffe800640 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d12aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.275+0000 7f6ffde00640 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d12560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.275+0000 7f6ffe800640 1 -- 172.21.15.102:0/2135369300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 msgr2=0x55e1d1d12560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.275+0000 7f6ffe800640 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d12560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.275+0000 7f6ffe800640 1 -- 172.21.15.102:0/2135369300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1d1a4d7d0 con 0x55e1d1d155b0 2024-04-25T01:58:12.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.275+0000 7f6ffde00640 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d12560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.277+0000 7f6ffe800640 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d155b0 0x55e1d1d12fe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4004700 tx=0x7f6ff4003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b82c24afd3b898 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.277+0000 7f6fd7400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.277+0000 7f6fd7400640 1 -- 172.21.15.102:0/2135369300 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4028060 con 0x55e1d1d155b0 2024-04-25T01:58:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.277+0000 7f6fd7400640 1 -- 172.21.15.102:0/2135369300 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6ff400ccd0 con 0x55e1d1d155b0 2024-04-25T01:58:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.278+0000 7f7003625740 1 -- 172.21.15.102:0/2135369300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff40035f0 con 0x55e1d1d155b0 2024-04-25T01:58:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.278+0000 7f6fd7400640 1 -- 172.21.15.102:0/2135369300 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6ff4004020 con 0x55e1d1d155b0 2024-04-25T01:58:12.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.279+0000 7f7003625740 10 client.26075 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.279+0000 7f7003625740 1 -- 172.21.15.102:0/2135369300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e1d1d24660 con 0x55e1d1d155b0 2024-04-25T01:58:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.279+0000 7f6fd7400640 1 -- 172.21.15.102:0/2135369300 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff40035f0 con 0x55e1d1d155b0 2024-04-25T01:58:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.279+0000 7f6fd7400640 1 -- 172.21.15.102:0/2135369300 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6ff400fad0 con 0x55e1d1d155b0 2024-04-25T01:58:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.279+0000 7f7003625740 10 client.26075 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.279+0000 7f7003625740 10 client.26075 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:58:12.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.279+0000 7f7003625740 20 client.26075 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.279+0000 7f7003625740 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55e1d1d25ac0 0x55e1d1d27eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.280+0000 7f7003625740 4 client.26075 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.280+0000 7f7003625740 1 -- 172.21.15.102:0/2135369300 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1d1c7ca20 con 0x55e1d1d25ac0 2024-04-25T01:58:12.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.280+0000 7f6ffde00640 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55e1d1d25ac0 0x55e1d1d27eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.281+0000 7f6ffde00640 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55e1d1d25ac0 0x55e1d1d27eb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=81dc39f63b53f3db server_cookie=2720b90b0e26c99d in_seq=0 out_seq=0 2024-04-25T01:58:12.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.281+0000 7f6fd7400640 10 client.26075 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:58:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.283+0000 7f6fd7400640 1 -- 172.21.15.102:0/2135369300 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e1d1c7ca20 con 0x55e1d1d25ac0 2024-04-25T01:58:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.283+0000 7f6fd7400640 10 client.26075 handle_command_reply: tid=0 2024-04-25T01:58:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.283+0000 7f7003625740 1 client.26075 shutdown 2024-04-25T01:58:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.283+0000 7f7003625740 1 -- 172.21.15.102:0/2135369300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d155b0 msgr2=0x55e1d1d12fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.284+0000 7f7003625740 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d155b0 0x55e1d1d12fe0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4004700 tx=0x7f6ff4003c80 comp rx=0 tx=0).stop 2024-04-25T01:58:12.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.284+0000 7f7003625740 1 -- 172.21.15.102:0/2135369300 shutdown_connections 2024-04-25T01:58:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.284+0000 7f7003625740 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55e1d1d25ac0 0x55e1d1d27eb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.284+0000 7f7003625740 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d155b0 0x55e1d1d12fe0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.284+0000 7f7003625740 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d12aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.284+0000 7f7003625740 1 --2- 172.21.15.102:0/2135369300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d12560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.284+0000 7f7003625740 1 -- 172.21.15.102:0/2135369300 >> 172.21.15.102:0/2135369300 conn(0x55e1d1d068e0 msgr2=0x55e1d1d0c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.284+0000 7f6fd7400640 0 client.26075 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:58:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.285+0000 7f7003625740 1 -- 172.21.15.102:0/2135369300 shutdown_connections 2024-04-25T01:58:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.285+0000 7f7003625740 1 -- 172.21.15.102:0/2135369300 wait complete. 2024-04-25T01:58:12.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.285+0000 7f7003625740 20 client.26075 trim_cache size 0 max 16384 2024-04-25T01:58:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.309+0000 7f7003625740 1 Processor -- start 2024-04-25T01:58:12.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.309+0000 7f7003625740 1 -- start start 2024-04-25T01:58:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f7003625740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d21260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f7003625740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f7003625740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 0x55e1d1d2d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f7003625740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81adc30 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f7003625740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81ab4a0 con 0x55e1d1d0aee0 2024-04-25T01:58:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f7003625740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81a92f0 con 0x55e1d1d155b0 2024-04-25T01:58:12.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f6ffd400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f6ffe800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 0x55e1d1d2d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f6ffd400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45654/0 (socket says 172.21.15.102:45654) 2024-04-25T01:58:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f6ffd400640 1 -- 172.21.15.102:0/3002720380 learned_addr learned my addr 172.21.15.102:0/3002720380 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.310+0000 7f6ffde00640 1 --2- 172.21.15.102:0/3002720380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d21260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.311+0000 7f6ffe800640 1 -- 172.21.15.102:0/3002720380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0aee0 msgr2=0x55e1d1d0ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.311+0000 7f6ffe800640 1 --2- 172.21.15.102:0/3002720380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.311+0000 7f6ffe800640 1 -- 172.21.15.102:0/3002720380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 msgr2=0x55e1d1d21260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.311+0000 7f6ffe800640 1 --2- 172.21.15.102:0/3002720380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d21260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.311+0000 7f6ffe800640 1 -- 172.21.15.102:0/3002720380 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1d1d0d370 con 0x55e1d1d155b0 2024-04-25T01:58:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.311+0000 7f6ffe800640 1 --2- 172.21.15.102:0/3002720380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 0x55e1d1d2d4e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40184a0 tx=0x7f6ff400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abd0a4a8ee94d148 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.311+0000 7f6fe2600640 1 -- 172.21.15.102:0/3002720380 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff401c020 con 0x55e1d1d155b0 2024-04-25T01:58:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.312+0000 7f6fe2600640 1 -- 172.21.15.102:0/3002720380 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff4005a10 con 0x55e1d1d155b0 2024-04-25T01:58:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.312+0000 7f6fe2600640 1 -- 172.21.15.102:0/3002720380 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4028040 con 0x55e1d1d155b0 2024-04-25T01:58:12.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.312+0000 7f7003625740 1 -- 172.21.15.102:0/3002720380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 msgr2=0x55e1d1d2d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.312+0000 7f7003625740 1 --2- 172.21.15.102:0/3002720380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 0x55e1d1d2d4e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40184a0 tx=0x7f6ff400c6a0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.313+0000 7f7003625740 1 -- 172.21.15.102:0/3002720380 shutdown_connections 2024-04-25T01:58:12.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.313+0000 7f7003625740 1 --2- 172.21.15.102:0/3002720380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 0x55e1d1d2d4e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.313+0000 7f7003625740 1 --2- 172.21.15.102:0/3002720380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.313+0000 7f7003625740 1 --2- 172.21.15.102:0/3002720380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d21260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.313+0000 7f7003625740 1 -- 172.21.15.102:0/3002720380 >> 172.21.15.102:0/3002720380 conn(0x55e1d1d068e0 msgr2=0x55e1d1d0d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.313+0000 7f7003625740 1 -- 172.21.15.102:0/3002720380 shutdown_connections 2024-04-25T01:58:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.313+0000 7f7003625740 1 -- 172.21.15.102:0/3002720380 wait complete. 2024-04-25T01:58:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.314+0000 7f7003625740 1 -- start start 2024-04-25T01:58:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.314+0000 7f7003625740 1 Processor -- start 2024-04-25T01:58:12.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f7003625740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d1d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f7003625740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f7003625740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 0x55e1d1d0f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f7003625740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81a9960 con 0x55e1d1d0aee0 2024-04-25T01:58:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f7003625740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81aadd0 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f7003625740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff81aa030 con 0x55e1d1d155b0 2024-04-25T01:58:12.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f7003625740 10 client.0 _ll_register_callbacks cb 0x7f6ff00c9070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d1d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d1d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45656/0 (socket says 172.21.15.102:45656) 2024-04-25T01:58:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffde00640 1 -- 172.21.15.102:0/387861600 learned_addr learned my addr 172.21.15.102:0/387861600 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffd400640 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffde00640 1 -- 172.21.15.102:0/387861600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 msgr2=0x55e1d1d0f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffde00640 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 0x55e1d1d0f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffde00640 1 -- 172.21.15.102:0/387861600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0aee0 msgr2=0x55e1d1d0ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffde00640 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffde00640 1 -- 172.21.15.102:0/387861600 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1d1d0f660 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.315+0000 7f6ffd400640 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.316+0000 7f6ffde00640 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d1d960 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00c98d0 tx=0x7f6ff00c9790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fb3e2e1f719fff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.316+0000 7f6fd7400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.316+0000 7f6fd7400640 1 -- 172.21.15.102:0/387861600 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00e6070 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.316+0000 7f6fd7400640 1 -- 172.21.15.102:0/387861600 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6ff00c9ae0 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.316+0000 7f7003625740 1 -- 172.21.15.102:0/387861600 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ff4005a10 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.316+0000 7f6fd7400640 1 -- 172.21.15.102:0/387861600 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6ff00fe810 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.316+0000 7f7003625740 10 client.26081 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.316+0000 7f7003625740 1 -- 172.21.15.102:0/387861600 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e1d1d8ba60 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.317+0000 7f6fd7400640 1 -- 172.21.15.102:0/387861600 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ff4005a10 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.318+0000 7f6fd7400640 1 -- 172.21.15.102:0/387861600 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6ff00e6210 con 0x55e1d1d0a5c0 2024-04-25T01:58:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.318+0000 7f7003625740 10 client.26081 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.318+0000 7f7003625740 10 client.26081 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:58:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.318+0000 7f7003625740 20 client.26081 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.318+0000 7f7003625740 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55e1d1d8c210 0x55e1d1d8e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.318+0000 7f7003625740 4 client.26081 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:58:12.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.318+0000 7f7003625740 1 -- 172.21.15.102:0/387861600 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e1d1c76d70 con 0x55e1d1d8c210 2024-04-25T01:58:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.318+0000 7f6ffd400640 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55e1d1d8c210 0x55e1d1d8e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.319+0000 7f6ffd400640 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55e1d1d8c210 0x55e1d1d8e650 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=555ea7a21d25b8fe server_cookie=730a21026fd39078 in_seq=0 out_seq=0 2024-04-25T01:58:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.319+0000 7f6fd7400640 10 client.26081 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:58:12.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f6fd7400640 1 -- 172.21.15.102:0/387861600 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e1d1c76d70 con 0x55e1d1d8c210 2024-04-25T01:58:12.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f6fd7400640 10 client.26081 handle_command_reply: tid=0 2024-04-25T01:58:12.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f7003625740 1 client.26081 shutdown 2024-04-25T01:58:12.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f7003625740 1 -- 172.21.15.102:0/387861600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 msgr2=0x55e1d1d1d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f7003625740 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d1d960 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00c98d0 tx=0x7f6ff00c9790 comp rx=0 tx=0).stop 2024-04-25T01:58:12.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f7003625740 1 -- 172.21.15.102:0/387861600 shutdown_connections 2024-04-25T01:58:12.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f7003625740 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e1d1d155b0 0x55e1d1d0f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f7003625740 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e1d1d0aee0 0x55e1d1d0ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f7003625740 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55e1d1d8c210 0x55e1d1d8e650 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f7003625740 1 --2- 172.21.15.102:0/387861600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e1d1d0a5c0 0x55e1d1d1d960 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f7003625740 1 -- 172.21.15.102:0/387861600 >> 172.21.15.102:0/387861600 conn(0x55e1d1d068e0 msgr2=0x55e1d1d052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.320+0000 7f6fd7400640 0 client.26081 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:58:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.321+0000 7f7003625740 1 -- 172.21.15.102:0/387861600 shutdown_connections 2024-04-25T01:58:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.321+0000 7f7003625740 1 -- 172.21.15.102:0/387861600 wait complete. 2024-04-25T01:58:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.321+0000 7f7003625740 20 client.26081 trim_cache size 0 max 16384 2024-04-25T01:58:12.333 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:58:12.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 -- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6fe40a2ed0 msgr2=0x7f6fe40a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6fe40a2ed0 0x7f6fe40a5390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6ff810f620 tx=0x7f6fe8009770 comp rx=0 tx=0).stop 2024-04-25T01:58:12.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 -- 172.21.15.102:0/2523170800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff8075c70 msgr2=0x7f6ff8111dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff8075c70 0x7f6ff8111dd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6ff001acb0 tx=0x7f6ff0040750 comp rx=0 tx=0).stop 2024-04-25T01:58:12.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 -- 172.21.15.102:0/2523170800 shutdown_connections 2024-04-25T01:58:12.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6fe40a2ed0 0x7f6fe40a5390 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ff8112620 0x7f6ff810e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ff8076590 0x7f6ff810de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 --2- 172.21.15.102:0/2523170800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff8075c70 0x7f6ff8111dd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 -- 172.21.15.102:0/2523170800 >> 172.21.15.102:0/2523170800 conn(0x7f6ff80ff4d0 msgr2=0x7f6ff8113d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 -- 172.21.15.102:0/2523170800 shutdown_connections 2024-04-25T01:58:12.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.322+0000 7f7000800640 1 -- 172.21.15.102:0/2523170800 wait complete. 2024-04-25T01:58:12.345 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:12 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1792014214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:58:12.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:58:12.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:12 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1792014214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:58:12.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:12 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1792014214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:58:12.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.615+0000 7fa820a00640 1 -- 172.21.15.102:0/2074915033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa81c10ff90 msgr2=0x7fa81c10da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.615+0000 7fa820a00640 1 --2- 172.21.15.102:0/2074915033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa81c10ff90 0x7fa81c10da80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa80c009930 tx=0x7fa80c02a420 comp rx=0 tx=0).stop 2024-04-25T01:58:12.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.615+0000 7fa820a00640 1 -- 172.21.15.102:0/2074915033 shutdown_connections 2024-04-25T01:58:12.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.615+0000 7fa820a00640 1 --2- 172.21.15.102:0/2074915033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa81c11af20 0x7fa81c11d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.615+0000 7fa820a00640 1 --2- 172.21.15.102:0/2074915033 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa81c10dfc0 0x7fa81c10e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.615+0000 7fa820a00640 1 --2- 172.21.15.102:0/2074915033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa81c10ff90 0x7fa81c10da80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.615+0000 7fa820a00640 1 -- 172.21.15.102:0/2074915033 >> 172.21.15.102:0/2074915033 conn(0x7fa81c06c850 msgr2=0x7fa81c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.616+0000 7fa820a00640 1 -- 172.21.15.102:0/2074915033 shutdown_connections 2024-04-25T01:58:12.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.616+0000 7fa820a00640 1 -- 172.21.15.102:0/2074915033 wait complete. 2024-04-25T01:58:12.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.617+0000 7fa820a00640 1 Processor -- start 2024-04-25T01:58:12.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.618+0000 7fa820a00640 1 -- start start 2024-04-25T01:58:12.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa820a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa81c10dfc0 0x7fa81c1a6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa820a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa81c10ff90 0x7fa81c1a7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa820a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa81c11af20 0x7fa81c1ab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa820a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c120900 con 0x7fa81c11af20 2024-04-25T01:58:12.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa820a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c120780 con 0x7fa81c10ff90 2024-04-25T01:58:12.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa820a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c120a80 con 0x7fa81c10dfc0 2024-04-25T01:58:12.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa819e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa81c10dfc0 0x7fa81c1a6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa819e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa81c10dfc0 0x7fa81c1a6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34420/0 (socket says 172.21.15.102:34420) 2024-04-25T01:58:12.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa819e00640 1 -- 172.21.15.102:0/1436060036 learned_addr learned my addr 172.21.15.102:0/1436060036 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa81a800640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa81c11af20 0x7fa81c1ab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.619+0000 7fa819400640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa81c10ff90 0x7fa81c1a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.620+0000 7fa81a800640 1 -- 172.21.15.102:0/1436060036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa81c10dfc0 msgr2=0x7fa81c1a6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.620+0000 7fa81a800640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa81c10dfc0 0x7fa81c1a6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.620+0000 7fa81a800640 1 -- 172.21.15.102:0/1436060036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa81c10ff90 msgr2=0x7fa81c1a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.620+0000 7fa81a800640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa81c10ff90 0x7fa81c1a7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.620+0000 7fa81a800640 1 -- 172.21.15.102:0/1436060036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c1abdd0 con 0x7fa81c11af20 2024-04-25T01:58:12.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.621+0000 7fa81a800640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa81c11af20 0x7fa81c1ab6a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa81400e940 tx=0x7fa81400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.621+0000 7fa812a00640 1 -- 172.21.15.102:0/1436060036 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814004d20 con 0x7fa81c11af20 2024-04-25T01:58:12.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.621+0000 7fa812a00640 1 -- 172.21.15.102:0/1436060036 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa814005020 con 0x7fa81c11af20 2024-04-25T01:58:12.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.621+0000 7fa812a00640 1 -- 172.21.15.102:0/1436060036 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140116b0 con 0x7fa81c11af20 2024-04-25T01:58:12.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.622+0000 7fa820a00640 1 -- 172.21.15.102:0/1436060036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c1ac0c0 con 0x7fa81c11af20 2024-04-25T01:58:12.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.622+0000 7fa820a00640 1 -- 172.21.15.102:0/1436060036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81c1b3950 con 0x7fa81c11af20 2024-04-25T01:58:12.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.627+0000 7fa8239bb740 1 Processor -- start 2024-04-25T01:58:12.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.628+0000 7fa812a00640 1 -- 172.21.15.102:0/1436060036 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa8140054d0 con 0x7fa81c11af20 2024-04-25T01:58:12.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.629+0000 7fa8239bb740 1 -- start start 2024-04-25T01:58:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.629+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c9529a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.629+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c95d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.629+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 0x55bc2c95f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.629+0000 7fa8239bb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55bc2c8b74e0 con 0x55bc2c9525c0 2024-04-25T01:58:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.629+0000 7fa8239bb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bc2c85fa70 con 0x55bc2c95d5b0 2024-04-25T01:58:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.629+0000 7fa8239bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bc2c8b34a0 con 0x55bc2c952ee0 2024-04-25T01:58:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.630+0000 7fa819e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c95d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.630+0000 7fa812a00640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa8000a2e50 0x7fa8000a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.630+0000 7fa819e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c95d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34424/0 (socket says 172.21.15.102:34424) 2024-04-25T01:58:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.630+0000 7fa819e00640 1 -- 172.21.15.102:0/2718262089 learned_addr learned my addr 172.21.15.102:0/2718262089 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.630+0000 7fa812a00640 1 -- 172.21.15.102:0/1436060036 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa8140cf590 con 0x7fa81c11af20 2024-04-25T01:58:12.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.631+0000 7fa81a800640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa8000a2e50 0x7fa8000a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.631+0000 7fa819400640 1 --2- 172.21.15.102:0/2718262089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 0x55bc2c95f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.631+0000 7fa819e00640 1 -- 172.21.15.102:0/2718262089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 msgr2=0x55bc2c95f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.631+0000 7fa819e00640 1 --2- 172.21.15.102:0/2718262089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 0x55bc2c95f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.631+0000 7fa819e00640 1 -- 172.21.15.102:0/2718262089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 msgr2=0x55bc2c9529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.631+0000 7fa819e00640 1 --2- 172.21.15.102:0/2718262089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c9529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.631+0000 7fa819e00640 1 -- 172.21.15.102:0/2718262089 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc2c5cb3f0 con 0x55bc2c952ee0 2024-04-25T01:58:12.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.631+0000 7fa819400640 1 --2- 172.21.15.102:0/2718262089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 0x55bc2c95f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:12.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.632+0000 7fa81a800640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa8000a2e50 0x7fa8000a5310 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa8140ccdc0 tx=0x7fa814005440 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.632+0000 7fa819e00640 1 --2- 172.21.15.102:0/2718262089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c95d070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa80c004120 tx=0x7fa80c004150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=210151d4b7420a14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.632+0000 7fa807000640 1 -- 172.21.15.102:0/2718262089 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c035040 con 0x55bc2c952ee0 2024-04-25T01:58:12.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.632+0000 7fa807000640 1 -- 172.21.15.102:0/2718262089 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c005520 con 0x55bc2c952ee0 2024-04-25T01:58:12.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.632+0000 7fa807000640 1 -- 172.21.15.102:0/2718262089 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c003850 con 0x55bc2c952ee0 2024-04-25T01:58:12.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.633+0000 7fa8239bb740 1 -- 172.21.15.102:0/2718262089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 msgr2=0x55bc2c95d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.633+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2718262089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c95d070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa80c004120 tx=0x7fa80c004150 comp rx=0 tx=0).stop 2024-04-25T01:58:12.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.633+0000 7fa8239bb740 1 -- 172.21.15.102:0/2718262089 shutdown_connections 2024-04-25T01:58:12.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.633+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2718262089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 0x55bc2c95f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.633+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2718262089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c95d070 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.633+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2718262089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c9529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.633+0000 7fa8239bb740 1 -- 172.21.15.102:0/2718262089 >> 172.21.15.102:0/2718262089 conn(0x55bc2c94e8e0 msgr2=0x55bc2c94ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.633+0000 7fa8239bb740 1 -- 172.21.15.102:0/2718262089 shutdown_connections 2024-04-25T01:58:12.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.634+0000 7fa8239bb740 1 -- 172.21.15.102:0/2718262089 wait complete. 2024-04-25T01:58:12.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.634+0000 7fa8239bb740 1 -- start start 2024-04-25T01:58:12.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.635+0000 7fa8239bb740 1 Processor -- start 2024-04-25T01:58:12.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.635+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c95a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.635+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c95aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.635+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 0x55bc2c95b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.635+0000 7fa8239bb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55bc2c960340 con 0x55bc2c9525c0 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.635+0000 7fa8239bb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bc2c872ec0 con 0x55bc2c952ee0 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.635+0000 7fa8239bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bc2c9604c0 con 0x55bc2c95d5b0 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.636+0000 7fa8239bb740 10 client.0 _ll_register_callbacks cb 0x55bc2c8af3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.636+0000 7fa819e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c95aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.636+0000 7fa819e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c95aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45702/0 (socket says 172.21.15.102:45702) 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.636+0000 7fa819e00640 1 -- 172.21.15.102:0/3449413830 learned_addr learned my addr 172.21.15.102:0/3449413830 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.636+0000 7fa819400640 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c95a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.636+0000 7fa819e00640 1 -- 172.21.15.102:0/3449413830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 msgr2=0x55bc2c95b030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:58:12.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.637+0000 7fa819e00640 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 0x55bc2c95b030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.637+0000 7fa819e00640 1 -- 172.21.15.102:0/3449413830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 msgr2=0x55bc2c95a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.637+0000 7fa819e00640 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c95a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.637+0000 7fa819e00640 1 -- 172.21.15.102:0/3449413830 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc2c6957d0 con 0x55bc2c952ee0 2024-04-25T01:58:12.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.637+0000 7fa819e00640 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c95aaf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa80c031550 tx=0x7fa80c009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4f2cc270ca9bf12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.637+0000 7fa7fb400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:12.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.638+0000 7fa7fb400640 1 -- 172.21.15.102:0/3449413830 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c052060 con 0x55bc2c952ee0 2024-04-25T01:58:12.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.638+0000 7fa8239bb740 1 -- 172.21.15.102:0/3449413830 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa80c005520 con 0x55bc2c952ee0 2024-04-25T01:58:12.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.638+0000 7fa7fb400640 1 -- 172.21.15.102:0/3449413830 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa80c0035f0 con 0x55bc2c952ee0 2024-04-25T01:58:12.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.638+0000 7fa7fb400640 1 -- 172.21.15.102:0/3449413830 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa80c035040 con 0x55bc2c952ee0 2024-04-25T01:58:12.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.638+0000 7fa8239bb740 10 client.26090 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:12.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.638+0000 7fa8239bb740 1 -- 172.21.15.102:0/3449413830 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bc2c96c5b0 con 0x55bc2c952ee0 2024-04-25T01:58:12.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.638+0000 7fa7fb400640 1 -- 172.21.15.102:0/3449413830 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c005520 con 0x55bc2c952ee0 2024-04-25T01:58:12.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.639+0000 7fa7fb400640 1 -- 172.21.15.102:0/3449413830 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa80c04c070 con 0x55bc2c952ee0 2024-04-25T01:58:12.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.639+0000 7fa8239bb740 10 client.26090 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:12.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.639+0000 7fa8239bb740 10 client.26090 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:58:12.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.639+0000 7fa8239bb740 20 client.26090 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:12.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.639+0000 7fa8239bb740 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55bc2c96d900 0x55bc2c96fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.639+0000 7fa8239bb740 4 client.26090 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:12.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.639+0000 7fa8239bb740 1 -- 172.21.15.102:0/3449413830 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc2c8c4a20 con 0x55bc2c96d900 2024-04-25T01:58:12.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.640+0000 7fa819400640 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55bc2c96d900 0x55bc2c96fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.640+0000 7fa819400640 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55bc2c96d900 0x55bc2c96fcf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=81a60eac7384923f server_cookie=d4124551a6c42814 in_seq=0 out_seq=0 2024-04-25T01:58:12.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.641+0000 7fa7fb400640 10 client.26090 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:58:12.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.641+0000 7fa7fb400640 1 -- 172.21.15.102:0/3449413830 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bc2c8c4a20 con 0x55bc2c96d900 2024-04-25T01:58:12.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.642+0000 7fa7fb400640 10 client.26090 handle_command_reply: tid=0 2024-04-25T01:58:12.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.642+0000 7fa8239bb740 1 client.26090 shutdown 2024-04-25T01:58:12.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.642+0000 7fa8239bb740 1 -- 172.21.15.102:0/3449413830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 msgr2=0x55bc2c95aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.642+0000 7fa8239bb740 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c95aaf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa80c031550 tx=0x7fa80c009650 comp rx=0 tx=0).stop 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa8239bb740 1 -- 172.21.15.102:0/3449413830 shutdown_connections 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa8239bb740 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55bc2c96d900 0x55bc2c96fcf0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa8239bb740 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 0x55bc2c95b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa8239bb740 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c95aaf0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa8239bb740 1 --2- 172.21.15.102:0/3449413830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c95a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa8239bb740 1 -- 172.21.15.102:0/3449413830 >> 172.21.15.102:0/3449413830 conn(0x55bc2c94e8e0 msgr2=0x55bc2c9546e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa7fb400640 0 client.26090 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa8239bb740 1 -- 172.21.15.102:0/3449413830 shutdown_connections 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa8239bb740 1 -- 172.21.15.102:0/3449413830 wait complete. 2024-04-25T01:58:12.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.643+0000 7fa8239bb740 20 client.26090 trim_cache size 0 max 16384 2024-04-25T01:58:12.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.682+0000 7fa8239bb740 1 Processor -- start 2024-04-25T01:58:12.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.682+0000 7fa8239bb740 1 -- start start 2024-04-25T01:58:12.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.683+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c9544c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.683+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c972d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.683+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 0x55bc2c973250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.683+0000 7fa8239bb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1beb40 con 0x55bc2c9525c0 2024-04-25T01:58:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.683+0000 7fa8239bb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1bc3b0 con 0x55bc2c952ee0 2024-04-25T01:58:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.683+0000 7fa8239bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1ba200 con 0x55bc2c95d5b0 2024-04-25T01:58:12.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.683+0000 7fa819e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c972d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.684+0000 7fa819e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c972d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45710/0 (socket says 172.21.15.102:45710) 2024-04-25T01:58:12.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.684+0000 7fa819e00640 1 -- 172.21.15.102:0/192189118 learned_addr learned my addr 172.21.15.102:0/192189118 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.684+0000 7fa819400640 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c9544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.684+0000 7fa819400640 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 0x55bc2c973250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.685+0000 7fa819e00640 1 -- 172.21.15.102:0/192189118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 msgr2=0x55bc2c973250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.685+0000 7fa819e00640 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 0x55bc2c973250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.686+0000 7fa819e00640 1 -- 172.21.15.102:0/192189118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 msgr2=0x55bc2c9544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.686+0000 7fa819e00640 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c9544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.686+0000 7fa819e00640 1 -- 172.21.15.102:0/192189118 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc2c95f9a0 con 0x55bc2c952ee0 2024-04-25T01:58:12.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.686+0000 7fa819400640 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 0x55bc2c973250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:12.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.687+0000 7fa819e00640 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c972d10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa80c004e50 tx=0x7fa80c005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=442cece9323902e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.688+0000 7fa806600640 1 -- 172.21.15.102:0/192189118 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c052070 con 0x55bc2c952ee0 2024-04-25T01:58:12.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.689+0000 7fa806600640 1 -- 172.21.15.102:0/192189118 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c004950 con 0x55bc2c952ee0 2024-04-25T01:58:12.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.689+0000 7fa806600640 1 -- 172.21.15.102:0/192189118 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c003850 con 0x55bc2c952ee0 2024-04-25T01:58:12.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.690+0000 7fa8239bb740 1 -- 172.21.15.102:0/192189118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 msgr2=0x55bc2c972d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.690+0000 7fa8239bb740 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c972d10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa80c004e50 tx=0x7fa80c005a20 comp rx=0 tx=0).stop 2024-04-25T01:58:12.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.691+0000 7fa8239bb740 1 -- 172.21.15.102:0/192189118 shutdown_connections 2024-04-25T01:58:12.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.691+0000 7fa8239bb740 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c95d5b0 0x55bc2c973250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.691+0000 7fa8239bb740 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c952ee0 0x55bc2c972d10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.691+0000 7fa8239bb740 1 --2- 172.21.15.102:0/192189118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c9544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.691+0000 7fa8239bb740 1 -- 172.21.15.102:0/192189118 >> 172.21.15.102:0/192189118 conn(0x55bc2c94e8e0 msgr2=0x55bc2c953db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.691+0000 7fa8239bb740 1 -- 172.21.15.102:0/192189118 shutdown_connections 2024-04-25T01:58:12.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.693+0000 7fa8239bb740 1 -- 172.21.15.102:0/192189118 wait complete. 2024-04-25T01:58:12.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.694+0000 7fa8239bb740 1 -- start start 2024-04-25T01:58:12.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.694+0000 7fa8239bb740 1 Processor -- start 2024-04-25T01:58:12.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.696+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c973160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.696+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c970540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.696+0000 7fa8239bb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 0x55bc2c970aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.696+0000 7fa8239bb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1ba870 con 0x55bc2c9525c0 2024-04-25T01:58:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.696+0000 7fa8239bb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1bbce0 con 0x55bc2c95d5b0 2024-04-25T01:58:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.696+0000 7fa8239bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1baf40 con 0x55bc2c952ee0 2024-04-25T01:58:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.697+0000 7fa819e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c970540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.697+0000 7fa819e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c970540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34456/0 (socket says 172.21.15.102:34456) 2024-04-25T01:58:12.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.697+0000 7fa819e00640 1 -- 172.21.15.102:0/2398977001 learned_addr learned my addr 172.21.15.102:0/2398977001 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:12.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.697+0000 7fa819400640 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 0x55bc2c970aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.697+0000 7fa8239bb740 10 client.0 _ll_register_callbacks cb 0x7fa80c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:12.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.697+0000 7fa819400640 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c973160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.698+0000 7fa819e00640 1 -- 172.21.15.102:0/2398977001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 msgr2=0x55bc2c970aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.698+0000 7fa819e00640 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 0x55bc2c970aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.698+0000 7fa819e00640 1 -- 172.21.15.102:0/2398977001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 msgr2=0x55bc2c973160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.698+0000 7fa819e00640 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c973160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.698+0000 7fa819e00640 1 -- 172.21.15.102:0/2398977001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc2c9712d0 con 0x55bc2c952ee0 2024-04-25T01:58:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.699+0000 7fa7fb400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.699+0000 7fa819e00640 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c970540 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0044d0 tx=0x7fa80c0058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9548b008d919e246 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.699+0000 7fa819400640 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c973160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.699+0000 7fa7fb400640 1 -- 172.21.15.102:0/2398977001 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c064060 con 0x55bc2c952ee0 2024-04-25T01:58:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.699+0000 7fa8239bb740 1 -- 172.21.15.102:0/2398977001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa80c004950 con 0x55bc2c952ee0 2024-04-25T01:58:12.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.699+0000 7fa7fb400640 1 -- 172.21.15.102:0/2398977001 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa80c005190 con 0x55bc2c952ee0 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.700+0000 7fa7fb400640 1 -- 172.21.15.102:0/2398977001 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa80c064060 con 0x55bc2c952ee0 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.700+0000 7fa8239bb740 10 client.26056 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.700+0000 7fa8239bb740 1 -- 172.21.15.102:0/2398977001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bc2c9d3b70 con 0x55bc2c952ee0 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.701+0000 7fa7fb400640 1 -- 172.21.15.102:0/2398977001 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c004950 con 0x55bc2c952ee0 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.701+0000 7fa7fb400640 1 -- 172.21.15.102:0/2398977001 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa80c04c070 con 0x55bc2c952ee0 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.701+0000 7fa8239bb740 10 client.26056 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.701+0000 7fa8239bb740 10 client.26056 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.701+0000 7fa8239bb740 20 client.26056 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.701+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55bc2c9d41a0 0x55bc2c9d65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.701+0000 7fa8239bb740 4 client.26056 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:58:12.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.701+0000 7fa8239bb740 1 -- 172.21.15.102:0/2398977001 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc2c8bed70 con 0x55bc2c9d41a0 2024-04-25T01:58:12.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.706+0000 7fa819400640 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55bc2c9d41a0 0x55bc2c9d65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:12.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.709+0000 7fa819400640 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55bc2c9d41a0 0x55bc2c9d65c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e947ccadac81b413 server_cookie=2b2668d2a0beb3e8 in_seq=0 out_seq=0 2024-04-25T01:58:12.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.709+0000 7fa7fb400640 10 client.26056 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:58:12.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa7fb400640 1 -- 172.21.15.102:0/2398977001 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc2c8bed70 con 0x55bc2c9d41a0 2024-04-25T01:58:12.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa7fb400640 10 client.26056 handle_command_reply: tid=0 2024-04-25T01:58:12.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa8239bb740 1 client.26056 shutdown 2024-04-25T01:58:12.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa8239bb740 1 -- 172.21.15.102:0/2398977001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 msgr2=0x55bc2c970540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c970540 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0044d0 tx=0x7fa80c0058d0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa8239bb740 1 -- 172.21.15.102:0/2398977001 shutdown_connections 2024-04-25T01:58:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55bc2c9d41a0 0x55bc2c9d65c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bc2c95d5b0 0x55bc2c970aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bc2c952ee0 0x55bc2c970540 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0044d0 tx=0x7fa80c0058d0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa8239bb740 1 --2- 172.21.15.102:0/2398977001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bc2c9525c0 0x55bc2c973160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa8239bb740 1 -- 172.21.15.102:0/2398977001 >> 172.21.15.102:0/2398977001 conn(0x55bc2c94e8e0 msgr2=0x55bc2c94db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.710+0000 7fa7fb400640 0 client.26056 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:58:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.711+0000 7fa8239bb740 1 -- 172.21.15.102:0/2398977001 shutdown_connections 2024-04-25T01:58:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.711+0000 7fa8239bb740 1 -- 172.21.15.102:0/2398977001 wait complete. 2024-04-25T01:58:12.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.713+0000 7fa8239bb740 20 client.26056 trim_cache size 0 max 16384 2024-04-25T01:58:12.716 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:58:12.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 -- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa8000a2e50 msgr2=0x7fa8000a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa8000a2e50 0x7fa8000a5310 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa8140ccdc0 tx=0x7fa814005440 comp rx=0 tx=0).stop 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 -- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa81c11af20 msgr2=0x7fa81c1ab6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa81c11af20 0x7fa81c1ab6a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa81400e940 tx=0x7fa81400ee00 comp rx=0 tx=0).stop 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 -- 172.21.15.102:0/1436060036 shutdown_connections 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa8000a2e50 0x7fa8000a5310 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa81c11af20 0x7fa81c1ab6a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa81c10ff90 0x7fa81c1a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 --2- 172.21.15.102:0/1436060036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa81c10dfc0 0x7fa81c1a6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 -- 172.21.15.102:0/1436060036 >> 172.21.15.102:0/1436060036 conn(0x7fa81c06c850 msgr2=0x7fa81c11c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 -- 172.21.15.102:0/1436060036 shutdown_connections 2024-04-25T01:58:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:12.716+0000 7fa820a00640 1 -- 172.21.15.102:0/1436060036 wait complete. 2024-04-25T01:58:12.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:58:13.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.054+0000 7f6f54e00640 1 -- 172.21.15.102:0/1319224334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f50103920 msgr2=0x7f6f50103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:13.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.054+0000 7f6f54e00640 1 --2- 172.21.15.102:0/1319224334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f50103920 0x7f6f50103da0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6f40009cd0 tx=0x7f6f4002a8f0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.054+0000 7f6f54e00640 1 -- 172.21.15.102:0/1319224334 shutdown_connections 2024-04-25T01:58:13.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.054+0000 7f6f54e00640 1 --2- 172.21.15.102:0/1319224334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f501042e0 0x7f6f501107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.054+0000 7f6f54e00640 1 --2- 172.21.15.102:0/1319224334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f50103920 0x7f6f50103da0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.054+0000 7f6f54e00640 1 --2- 172.21.15.102:0/1319224334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f50109930 0x7f6f50109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.054+0000 7f6f54e00640 1 -- 172.21.15.102:0/1319224334 >> 172.21.15.102:0/1319224334 conn(0x7f6f500ff3b0 msgr2=0x7f6f501017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:13.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.055+0000 7f6f54e00640 1 -- 172.21.15.102:0/1319224334 shutdown_connections 2024-04-25T01:58:13.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.055+0000 7f6f54e00640 1 -- 172.21.15.102:0/1319224334 wait complete. 2024-04-25T01:58:13.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.056+0000 7f6f54e00640 1 Processor -- start 2024-04-25T01:58:13.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.057+0000 7f6f54e00640 1 -- start start 2024-04-25T01:58:13.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.057+0000 7f6f54e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f50103920 0x7f6f50075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.057+0000 7f6f54e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f501042e0 0x7f6f50076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.057+0000 7f6f54e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f50109930 0x7f6f500766d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.057+0000 7f6f54e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50113d60 con 0x7f6f501042e0 2024-04-25T01:58:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.057+0000 7f6f54e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50113be0 con 0x7f6f50109930 2024-04-25T01:58:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.057+0000 7f6f54e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50113ee0 con 0x7f6f50103920 2024-04-25T01:58:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.058+0000 7f6f4de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f50103920 0x7f6f50075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.058+0000 7f6f4de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f50103920 0x7f6f50075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41664/0 (socket says 172.21.15.102:41664) 2024-04-25T01:58:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.058+0000 7f6f4de00640 1 -- 172.21.15.102:0/395121553 learned_addr learned my addr 172.21.15.102:0/395121553 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.058+0000 7f6f45400640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f501042e0 0x7f6f50076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.058+0000 7f6f4e800640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f50109930 0x7f6f500766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.059+0000 7f6f4e800640 1 -- 172.21.15.102:0/395121553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f50103920 msgr2=0x7f6f50075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:13.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.059+0000 7f6f4e800640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f50103920 0x7f6f50075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.059+0000 7f6f4e800640 1 -- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f501042e0 msgr2=0x7f6f50076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:13.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.059+0000 7f6f4e800640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f501042e0 0x7f6f50076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.059+0000 7f6f4e800640 1 -- 172.21.15.102:0/395121553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f501aa910 con 0x7f6f50109930 2024-04-25T01:58:13.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.059+0000 7f6f4e800640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f50109930 0x7f6f500766d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800e9a0 tx=0x7f6f4800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:13.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.060+0000 7f6f47400640 1 -- 172.21.15.102:0/395121553 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4800ccc0 con 0x7f6f50109930 2024-04-25T01:58:13.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.060+0000 7f6f47400640 1 -- 172.21.15.102:0/395121553 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f4800ce60 con 0x7f6f50109930 2024-04-25T01:58:13.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.060+0000 7f6f47400640 1 -- 172.21.15.102:0/395121553 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48004a90 con 0x7f6f50109930 2024-04-25T01:58:13.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.061+0000 7f6f54e00640 1 -- 172.21.15.102:0/395121553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f501aac00 con 0x7f6f50109930 2024-04-25T01:58:13.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.061+0000 7f6f54e00640 1 -- 172.21.15.102:0/395121553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f501ab0d0 con 0x7f6f50109930 2024-04-25T01:58:13.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.067+0000 7f6f47400640 1 -- 172.21.15.102:0/395121553 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6f48004c30 con 0x7f6f50109930 2024-04-25T01:58:13.071 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 140.0 s, 3127.3 tps, lat 32.159 ms stddev 36.431 2024-04-25T01:58:13.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.071+0000 7f6f47400640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6f340a2e50 0x7f6f340a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.072+0000 7f6f47400640 1 -- 172.21.15.102:0/395121553 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6f480cf370 con 0x7f6f50109930 2024-04-25T01:58:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.072+0000 7f6f4de00640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6f340a2e50 0x7f6f340a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.074+0000 7f6f4de00640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6f340a2e50 0x7f6f340a5310 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009c10 tx=0x7f6f38005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:13.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.074+0000 7f6f57d66740 1 Processor -- start 2024-04-25T01:58:13.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.075+0000 7f6f57d66740 1 -- start start 2024-04-25T01:58:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.076+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.076+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d0ee0 0x55b78f0db070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.076+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0dd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.076+0000 7f6f57d66740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b78f0354e0 con 0x55b78f0d05c0 2024-04-25T01:58:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.076+0000 7f6f57d66740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b78efdda70 con 0x55b78f0db5b0 2024-04-25T01:58:13.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.076+0000 7f6f57d66740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b78f0314a0 con 0x55b78f0d0ee0 2024-04-25T01:58:13.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.077+0000 7f6f4e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0dd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.077+0000 7f6f4e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0dd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58942/0 (socket says 172.21.15.102:58942) 2024-04-25T01:58:13.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.077+0000 7f6f4e800640 1 -- 172.21.15.102:0/4110333341 learned_addr learned my addr 172.21.15.102:0/4110333341 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:13.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.077+0000 7f6f4e800640 1 -- 172.21.15.102:0/4110333341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d0ee0 msgr2=0x55b78f0db070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:58:13.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.077+0000 7f6f4e800640 1 --2- 172.21.15.102:0/4110333341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d0ee0 0x55b78f0db070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.077+0000 7f6f4e800640 1 -- 172.21.15.102:0/4110333341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 msgr2=0x55b78f0d09a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:58:13.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.078+0000 7f6f4de00640 1 --2- 172.21.15.102:0/4110333341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.079+0000 7f6f4e800640 1 --2- 172.21.15.102:0/4110333341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.079+0000 7f6f4e800640 1 -- 172.21.15.102:0/4110333341 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b78ed493f0 con 0x55b78f0db5b0 2024-04-25T01:58:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.079+0000 7f6f4de00640 1 --2- 172.21.15.102:0/4110333341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:13.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.079+0000 7f6f4e800640 1 --2- 172.21.15.102:0/4110333341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0dd9a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6f48098290 tx=0x7f6f4809b220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1267f6ad85486b2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.080+0000 7f6f3f400640 1 -- 172.21.15.102:0/4110333341 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f480b3020 con 0x55b78f0db5b0 2024-04-25T01:58:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.080+0000 7f6f3f400640 1 -- 172.21.15.102:0/4110333341 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f4809fc70 con 0x55b78f0db5b0 2024-04-25T01:58:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.080+0000 7f6f3f400640 1 -- 172.21.15.102:0/4110333341 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4809ad10 con 0x55b78f0db5b0 2024-04-25T01:58:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.081+0000 7f6f57d66740 1 -- 172.21.15.102:0/4110333341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 msgr2=0x55b78f0dd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.081+0000 7f6f57d66740 1 --2- 172.21.15.102:0/4110333341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0dd9a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6f48098290 tx=0x7f6f4809b220 comp rx=0 tx=0).stop 2024-04-25T01:58:13.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.081+0000 7f6f57d66740 1 -- 172.21.15.102:0/4110333341 shutdown_connections 2024-04-25T01:58:13.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.081+0000 7f6f57d66740 1 --2- 172.21.15.102:0/4110333341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0dd9a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.081+0000 7f6f57d66740 1 --2- 172.21.15.102:0/4110333341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d0ee0 0x55b78f0db070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.081+0000 7f6f57d66740 1 --2- 172.21.15.102:0/4110333341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.081+0000 7f6f57d66740 1 -- 172.21.15.102:0/4110333341 >> 172.21.15.102:0/4110333341 conn(0x55b78f0cc8e0 msgr2=0x55b78f0cccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.082+0000 7f6f57d66740 1 -- 172.21.15.102:0/4110333341 shutdown_connections 2024-04-25T01:58:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.082+0000 7f6f57d66740 1 -- 172.21.15.102:0/4110333341 wait complete. 2024-04-25T01:58:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.082+0000 7f6f57d66740 1 -- start start 2024-04-25T01:58:13.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.083+0000 7f6f57d66740 1 Processor -- start 2024-04-25T01:58:13.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.084+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0d8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.084+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.084+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0db5b0 0x55b78f0d8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.084+0000 7f6f4de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0d8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.084+0000 7f6f4de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0d8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41676/0 (socket says 172.21.15.102:41676) 2024-04-25T01:58:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.085+0000 7f6f57d66740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b78f0de3a0 con 0x55b78f0db5b0 2024-04-25T01:58:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f57d66740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b78eff0ec0 con 0x55b78f0d0ee0 2024-04-25T01:58:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f57d66740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b78f0de520 con 0x55b78f0d05c0 2024-04-25T01:58:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f57d66740 10 client.0 _ll_register_callbacks cb 0x55b78f02d3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f45400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f45400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58958/0 (socket says 172.21.15.102:58958) 2024-04-25T01:58:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f45400640 1 -- 172.21.15.102:0/356600675 learned_addr learned my addr 172.21.15.102:0/356600675 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:13.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f4e800640 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0db5b0 0x55b78f0d8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f45400640 1 -- 172.21.15.102:0/356600675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 msgr2=0x55b78f0d8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f45400640 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0d8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f45400640 1 -- 172.21.15.102:0/356600675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0db5b0 msgr2=0x55b78f0d8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f45400640 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0db5b0 0x55b78f0d8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.086+0000 7f6f45400640 1 -- 172.21.15.102:0/356600675 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b78ee137d0 con 0x55b78f0d0ee0 2024-04-25T01:58:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.087+0000 7f6f4de00640 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0d8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-25T01:58:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.087+0000 7f6f45400640 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d8a70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6f4002aee0 tx=0x7f6f40003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=216549287bbada4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.087+0000 7f6f33e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.087+0000 7f6f33e00640 1 -- 172.21.15.102:0/356600675 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f40030b30 con 0x55b78f0d0ee0 2024-04-25T01:58:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.087+0000 7f6f33e00640 1 -- 172.21.15.102:0/356600675 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6f40030cd0 con 0x55b78f0d0ee0 2024-04-25T01:58:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.087+0000 7f6f57d66740 1 -- 172.21.15.102:0/356600675 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f4809fc70 con 0x55b78f0d0ee0 2024-04-25T01:58:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.088+0000 7f6f33e00640 1 -- 172.21.15.102:0/356600675 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f40030ed0 con 0x55b78f0d0ee0 2024-04-25T01:58:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.088+0000 7f6f57d66740 10 client.26108 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:13.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.088+0000 7f6f57d66740 1 -- 172.21.15.102:0/356600675 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b78f0ea5c0 con 0x55b78f0d0ee0 2024-04-25T01:58:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.088+0000 7f6f33e00640 1 -- 172.21.15.102:0/356600675 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f4809fc70 con 0x55b78f0d0ee0 2024-04-25T01:58:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.088+0000 7f6f33e00640 1 -- 172.21.15.102:0/356600675 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6f40043070 con 0x55b78f0d0ee0 2024-04-25T01:58:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.089+0000 7f6f57d66740 10 client.26108 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.089+0000 7f6f57d66740 10 client.26108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.089+0000 7f6f57d66740 20 client.26108 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.089+0000 7f6f57d66740 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b78f0eba40 0x55b78f0edee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.089+0000 7f6f57d66740 4 client.26108 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.089+0000 7f6f57d66740 1 -- 172.21.15.102:0/356600675 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b78f042a20 con 0x55b78f0eba40 2024-04-25T01:58:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.089+0000 7f6f4de00640 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b78f0eba40 0x55b78f0edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.090+0000 7f6f4de00640 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b78f0eba40 0x55b78f0edee0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8081780fe5def256 server_cookie=dd51fb2e6d3eda73 in_seq=0 out_seq=0 2024-04-25T01:58:13.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.090+0000 7f6f33e00640 10 client.26108 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:13.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.091+0000 7f6f33e00640 1 -- 172.21.15.102:0/356600675 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b78f042a20 con 0x55b78f0eba40 2024-04-25T01:58:13.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.091+0000 7f6f33e00640 10 client.26108 handle_command_reply: tid=0 2024-04-25T01:58:13.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.091+0000 7f6f57d66740 1 client.26108 shutdown 2024-04-25T01:58:13.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.092+0000 7f6f57d66740 1 -- 172.21.15.102:0/356600675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0d0ee0 msgr2=0x55b78f0d8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:13.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.092+0000 7f6f57d66740 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d8a70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6f4002aee0 tx=0x7f6f40003ba0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f57d66740 1 -- 172.21.15.102:0/356600675 shutdown_connections 2024-04-25T01:58:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f57d66740 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b78f0eba40 0x55b78f0edee0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f57d66740 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0db5b0 0x55b78f0d8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f33e00640 0 client.26108 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f57d66740 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d8a70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f57d66740 1 --2- 172.21.15.102:0/356600675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0d8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f57d66740 1 -- 172.21.15.102:0/356600675 >> 172.21.15.102:0/356600675 conn(0x55b78f0cc8e0 msgr2=0x55b78f0cbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f57d66740 1 -- 172.21.15.102:0/356600675 shutdown_connections 2024-04-25T01:58:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f57d66740 1 -- 172.21.15.102:0/356600675 wait complete. 2024-04-25T01:58:13.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.093+0000 7f6f57d66740 20 client.26108 trim_cache size 0 max 16384 2024-04-25T01:58:13.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.119+0000 7f6f57d66740 1 Processor -- start 2024-04-25T01:58:13.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.120+0000 7f6f57d66740 1 -- start start 2024-04-25T01:58:13.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.120+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0e7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.120+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.120+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0f3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.120+0000 7f6f57d66740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b6370 con 0x55b78f0d0ee0 2024-04-25T01:58:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.120+0000 7f6f57d66740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b3be0 con 0x55b78f0db5b0 2024-04-25T01:58:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.120+0000 7f6f57d66740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b1a30 con 0x55b78f0d05c0 2024-04-25T01:58:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.121+0000 7f6f4de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0e7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.121+0000 7f6f4de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0e7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41688/0 (socket says 172.21.15.102:41688) 2024-04-25T01:58:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.121+0000 7f6f4de00640 1 -- 172.21.15.102:0/2898816928 learned_addr learned my addr 172.21.15.102:0/2898816928 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:13.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.121+0000 7f6f4e800640 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.121+0000 7f6f45400640 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.122+0000 7f6f4de00640 1 -- 172.21.15.102:0/2898816928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 msgr2=0x55b78f0f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.122+0000 7f6f4de00640 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.122+0000 7f6f4de00640 1 -- 172.21.15.102:0/2898816928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d0ee0 msgr2=0x55b78f0d2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.122+0000 7f6f4de00640 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.122+0000 7f6f4de00640 1 -- 172.21.15.102:0/2898816928 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b78f0ee560 con 0x55b78f0d05c0 2024-04-25T01:58:13.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.123+0000 7f6f45400640 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:13.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.123+0000 7f6f4de00640 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0e7290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009fd0 tx=0x7f6f38018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5fb30ce04983cfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:13.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.124+0000 7f6f3ea00640 1 -- 172.21.15.102:0/2898816928 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f38004230 con 0x55b78f0d05c0 2024-04-25T01:58:13.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.124+0000 7f6f3ea00640 1 -- 172.21.15.102:0/2898816928 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f38029910 con 0x55b78f0d05c0 2024-04-25T01:58:13.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.124+0000 7f6f3ea00640 1 -- 172.21.15.102:0/2898816928 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f38027400 con 0x55b78f0d05c0 2024-04-25T01:58:13.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 -- 172.21.15.102:0/2898816928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 msgr2=0x55b78f0e7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:13.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0e7290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009fd0 tx=0x7f6f38018000 comp rx=0 tx=0).stop 2024-04-25T01:58:13.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 -- 172.21.15.102:0/2898816928 shutdown_connections 2024-04-25T01:58:13.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 --2- 172.21.15.102:0/2898816928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d05c0 0x55b78f0e7290 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 -- 172.21.15.102:0/2898816928 >> 172.21.15.102:0/2898816928 conn(0x55b78f0cc8e0 msgr2=0x55b78f0d8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:13.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 -- 172.21.15.102:0/2898816928 shutdown_connections 2024-04-25T01:58:13.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 -- 172.21.15.102:0/2898816928 wait complete. 2024-04-25T01:58:13.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.125+0000 7f6f57d66740 1 -- start start 2024-04-25T01:58:13.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.126+0000 7f6f57d66740 1 Processor -- start 2024-04-25T01:58:13.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f57d66740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0d5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f57d66740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b20a0 con 0x55b78f0d05c0 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f57d66740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b3510 con 0x55b78f0db5b0 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f57d66740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6f501b2770 con 0x55b78f0d0ee0 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f57d66740 10 client.0 _ll_register_callbacks cb 0x7f6f38005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f4e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0d5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f45400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.127+0000 7f6f4de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.128+0000 7f6f4e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0d5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58980/0 (socket says 172.21.15.102:58980) 2024-04-25T01:58:13.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.128+0000 7f6f4e800640 1 -- 172.21.15.102:0/3385763912 learned_addr learned my addr 172.21.15.102:0/3385763912 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:13.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.129+0000 7f6f4de00640 1 -- 172.21.15.102:0/3385763912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d0ee0 msgr2=0x55b78f0d5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:13.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.129+0000 7f6f4de00640 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.129+0000 7f6f4de00640 1 -- 172.21.15.102:0/3385763912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 msgr2=0x55b78f0d5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:13.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.129+0000 7f6f4de00640 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0d5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.129+0000 7f6f4de00640 1 -- 172.21.15.102:0/3385763912 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b78f151350 con 0x55b78f0d05c0 2024-04-25T01:58:13.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.129+0000 7f6f4e800640 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0d5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:13.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.130+0000 7f6f4de00640 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d5500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6f38005b60 tx=0x7f6f38009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5514ce34bd8a1831 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:13.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.130+0000 7f6f33e00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:58:13.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.130+0000 7f6f33e00640 1 -- 172.21.15.102:0/3385763912 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f380294f0 con 0x55b78f0d05c0 2024-04-25T01:58:13.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.130+0000 7f6f33e00640 1 -- 172.21.15.102:0/3385763912 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f6f38009040 con 0x55b78f0d05c0 2024-04-25T01:58:13.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.131+0000 7f6f57d66740 1 -- 172.21.15.102:0/3385763912 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f38029910 con 0x55b78f0d05c0 2024-04-25T01:58:13.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.131+0000 7f6f33e00640 1 -- 172.21.15.102:0/3385763912 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f38029690 con 0x55b78f0d05c0 2024-04-25T01:58:13.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.131+0000 7f6f57d66740 10 client.15828 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:13.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.131+0000 7f6f57d66740 1 -- 172.21.15.102:0/3385763912 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b78f1525a0 con 0x55b78f0d05c0 2024-04-25T01:58:13.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.132+0000 7f6f33e00640 1 -- 172.21.15.102:0/3385763912 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f38029910 con 0x55b78f0d05c0 2024-04-25T01:58:13.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.133+0000 7f6f33e00640 1 -- 172.21.15.102:0/3385763912 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6f38032070 con 0x55b78f0d05c0 2024-04-25T01:58:13.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.133+0000 7f6f57d66740 10 client.15828 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:13.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.133+0000 7f6f57d66740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:13.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.133+0000 7f6f57d66740 20 client.15828 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:13.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.133+0000 7f6f57d66740 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b78f152c10 0x55b78f155030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:13.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.134+0000 7f6f57d66740 4 client.15828 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:58:13.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.134+0000 7f6f57d66740 1 -- 172.21.15.102:0/3385763912 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b78f03cd70 con 0x55b78f152c10 2024-04-25T01:58:13.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.134+0000 7f6f45400640 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b78f152c10 0x55b78f155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:13.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.136+0000 7f6f33e00640 10 client.15828 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:13.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.136+0000 7f6f45400640 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b78f152c10 0x55b78f155030 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=355e3ff427f91292 server_cookie=76f864c5ff603583 in_seq=0 out_seq=0 2024-04-25T01:58:13.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.137+0000 7f6f33e00640 1 -- 172.21.15.102:0/3385763912 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b78f03cd70 con 0x55b78f152c10 2024-04-25T01:58:13.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.137+0000 7f6f33e00640 10 client.15828 handle_command_reply: tid=0 2024-04-25T01:58:13.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.137+0000 7f6f57d66740 1 client.15828 shutdown 2024-04-25T01:58:13.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f57d66740 1 -- 172.21.15.102:0/3385763912 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 msgr2=0x55b78f0d5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:13.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f57d66740 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d5500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6f38005b60 tx=0x7f6f38009a10 comp rx=0 tx=0).stop 2024-04-25T01:58:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f57d66740 1 -- 172.21.15.102:0/3385763912 shutdown_connections 2024-04-25T01:58:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f57d66740 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b78f152c10 0x55b78f155030 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f57d66740 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b78f0db5b0 0x55b78f0d5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f57d66740 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b78f0d0ee0 0x55b78f0d5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f57d66740 1 --2- 172.21.15.102:0/3385763912 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b78f0d05c0 0x55b78f0d5500 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f57d66740 1 -- 172.21.15.102:0/3385763912 >> 172.21.15.102:0/3385763912 conn(0x55b78f0cc8e0 msgr2=0x55b78f0d9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:13.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f33e00640 0 client.15828 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:13.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.139+0000 7f6f57d66740 1 -- 172.21.15.102:0/3385763912 shutdown_connections 2024-04-25T01:58:13.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.140+0000 7f6f57d66740 1 -- 172.21.15.102:0/3385763912 wait complete. 2024-04-25T01:58:13.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.140+0000 7f6f57d66740 20 client.15828 trim_cache size 0 max 16384 2024-04-25T01:58:13.143 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:58:13.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.144+0000 7f6f54e00640 1 -- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6f340a2e50 msgr2=0x7f6f340a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:13.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.144+0000 7f6f54e00640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6f340a2e50 0x7f6f340a5310 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6f38009c10 tx=0x7f6f38005f90 comp rx=0 tx=0).stop 2024-04-25T01:58:13.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.144+0000 7f6f54e00640 1 -- 172.21.15.102:0/395121553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f50109930 msgr2=0x7f6f500766d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:13.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.144+0000 7f6f54e00640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f50109930 0x7f6f500766d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800e9a0 tx=0x7f6f4800ee60 comp rx=0 tx=0).stop 2024-04-25T01:58:13.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.145+0000 7f6f54e00640 1 -- 172.21.15.102:0/395121553 shutdown_connections 2024-04-25T01:58:13.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.145+0000 7f6f54e00640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6f340a2e50 0x7f6f340a5310 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.145+0000 7f6f54e00640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f50109930 0x7f6f500766d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.145+0000 7f6f54e00640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f501042e0 0x7f6f50076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.145+0000 7f6f54e00640 1 --2- 172.21.15.102:0/395121553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6f50103920 0x7f6f50075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:13.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.145+0000 7f6f54e00640 1 -- 172.21.15.102:0/395121553 >> 172.21.15.102:0/395121553 conn(0x7f6f500ff3b0 msgr2=0x7f6f500ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:13.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.146+0000 7f6f54e00640 1 -- 172.21.15.102:0/395121553 shutdown_connections 2024-04-25T01:58:13.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:13.147+0000 7f6f54e00640 1 -- 172.21.15.102:0/395121553 wait complete. 2024-04-25T01:58:13.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:13 smithi102 ceph-mon[27796]: pgmap v91: 97 pgs: 97 active+clean; 2.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 MiB/s wr, 1.24k op/s 2024-04-25T01:58:13.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:13 smithi110 ceph-mon[32848]: pgmap v91: 97 pgs: 97 active+clean; 2.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 MiB/s wr, 1.24k op/s 2024-04-25T01:58:13.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:13 smithi193 ceph-mon[32981]: pgmap v91: 97 pgs: 97 active+clean; 2.1 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 MiB/s wr, 1.24k op/s 2024-04-25T01:58:14.193 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:58:14.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:58:14.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:58:14.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.485+0000 7faf32600640 1 -- 172.21.15.102:0/1392953152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf2c1015e0 msgr2=0x7faf2c101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.485+0000 7faf32600640 1 --2- 172.21.15.102:0/1392953152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf2c1015e0 0x7faf2c101a60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faf18009930 tx=0x7faf1802a420 comp rx=0 tx=0).stop 2024-04-25T01:58:14.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.485+0000 7faf32600640 1 -- 172.21.15.102:0/1392953152 shutdown_connections 2024-04-25T01:58:14.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.485+0000 7faf32600640 1 --2- 172.21.15.102:0/1392953152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c101fa0 0x7faf2c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.485+0000 7faf32600640 1 --2- 172.21.15.102:0/1392953152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf2c1015e0 0x7faf2c101a60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.485+0000 7faf32600640 1 --2- 172.21.15.102:0/1392953152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c107680 0x7faf2c107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.485+0000 7faf32600640 1 -- 172.21.15.102:0/1392953152 >> 172.21.15.102:0/1392953152 conn(0x7faf2c0fd110 msgr2=0x7faf2c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:14.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.485+0000 7faf32600640 1 -- 172.21.15.102:0/1392953152 shutdown_connections 2024-04-25T01:58:14.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.485+0000 7faf32600640 1 -- 172.21.15.102:0/1392953152 wait complete. 2024-04-25T01:58:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.487+0000 7faf32600640 1 Processor -- start 2024-04-25T01:58:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.487+0000 7faf32600640 1 -- start start 2024-04-25T01:58:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.488+0000 7faf32600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c1015e0 0x7faf2c073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.488+0000 7faf32600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf2c101fa0 0x7faf2c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.488+0000 7faf32600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c107680 0x7faf2c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.489+0000 7faf32600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c111a10 con 0x7faf2c107680 2024-04-25T01:58:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.489+0000 7faf32600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c111890 con 0x7faf2c1015e0 2024-04-25T01:58:14.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.489+0000 7faf32600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c111b90 con 0x7faf2c101fa0 2024-04-25T01:58:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.489+0000 7faf31200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c1015e0 0x7faf2c073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.489+0000 7faf31200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c1015e0 0x7faf2c073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58994/0 (socket says 172.21.15.102:58994) 2024-04-25T01:58:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.489+0000 7faf31200640 1 -- 172.21.15.102:0/2999342961 learned_addr learned my addr 172.21.15.102:0/2999342961 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.489+0000 7faf2be00640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf2c101fa0 0x7faf2c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.490+0000 7faf2be00640 1 -- 172.21.15.102:0/2999342961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c1015e0 msgr2=0x7faf2c073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.490+0000 7faf2be00640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c1015e0 0x7faf2c073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.490+0000 7faf2be00640 1 -- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c107680 msgr2=0x7faf2c0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.490+0000 7faf31c00640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c107680 0x7faf2c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.490+0000 7faf2be00640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c107680 0x7faf2c0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.490+0000 7faf2be00640 1 -- 172.21.15.102:0/2999342961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c1a6310 con 0x7faf2c101fa0 2024-04-25T01:58:14.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.490+0000 7faf31c00640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c107680 0x7faf2c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:14.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.491+0000 7faf2be00640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf2c101fa0 0x7faf2c073f80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faf18009850 tx=0x7faf18003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.492+0000 7faf29600640 1 -- 172.21.15.102:0/2999342961 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1803f070 con 0x7faf2c101fa0 2024-04-25T01:58:14.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.492+0000 7faf29600640 1 -- 172.21.15.102:0/2999342961 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf18031c90 con 0x7faf2c101fa0 2024-04-25T01:58:14.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.492+0000 7faf29600640 1 -- 172.21.15.102:0/2999342961 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf18035030 con 0x7faf2c101fa0 2024-04-25T01:58:14.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.492+0000 7faf32600640 1 -- 172.21.15.102:0/2999342961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2c1a65a0 con 0x7faf2c101fa0 2024-04-25T01:58:14.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.492+0000 7faf32600640 1 -- 172.21.15.102:0/2999342961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2c1a6a10 con 0x7faf2c101fa0 2024-04-25T01:58:14.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.498+0000 7faf29600640 1 -- 172.21.15.102:0/2999342961 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7faf18042070 con 0x7faf2c101fa0 2024-04-25T01:58:14.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.501+0000 7faf29600640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf100a2e00 0x7faf100a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.501+0000 7faf29600640 1 -- 172.21.15.102:0/2999342961 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7faf180efbb0 con 0x7faf2c101fa0 2024-04-25T01:58:14.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.502+0000 7faf31200640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf100a2e00 0x7faf100a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.503+0000 7faf36a2e740 1 Processor -- start 2024-04-25T01:58:14.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.503+0000 7faf36a2e740 1 -- start start 2024-04-25T01:58:14.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.503+0000 7faf31200640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf100a2e00 0x7faf100a52c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faf1c005fd0 tx=0x7faf1c005e50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.504+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb74c70 0x562cebb75050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.504+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb75620 0x562cebb881e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.504+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 0x562cebb8ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.504+0000 7faf36a2e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x562cebad3f40 con 0x562cebb74c70 2024-04-25T01:58:14.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.505+0000 7faf36a2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562cebad9190 con 0x562cebb75620 2024-04-25T01:58:14.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.505+0000 7faf36a2e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562ceba81a10 con 0x562cebb88720 2024-04-25T01:58:14.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.505+0000 7faf2be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb75620 0x562cebb881e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.505+0000 7faf2be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb75620 0x562cebb881e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58996/0 (socket says 172.21.15.102:58996) 2024-04-25T01:58:14.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.505+0000 7faf2be00640 1 -- 172.21.15.102:0/3671962928 learned_addr learned my addr 172.21.15.102:0/3671962928 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:14.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.505+0000 7faf31200640 1 --2- 172.21.15.102:0/3671962928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb74c70 0x562cebb75050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.506+0000 7faf2be00640 1 -- 172.21.15.102:0/3671962928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 msgr2=0x562cebb8ab10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:58:14.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.506+0000 7faf2be00640 1 --2- 172.21.15.102:0/3671962928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 0x562cebb8ab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.506+0000 7faf2be00640 1 -- 172.21.15.102:0/3671962928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb74c70 msgr2=0x562cebb75050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.506+0000 7faf2be00640 1 --2- 172.21.15.102:0/3671962928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb74c70 0x562cebb75050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.506+0000 7faf2be00640 1 -- 172.21.15.102:0/3671962928 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ceb8b57e0 con 0x562cebb75620 2024-04-25T01:58:14.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.507+0000 7faf2be00640 1 --2- 172.21.15.102:0/3671962928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb75620 0x562cebb881e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faf180b8500 tx=0x7faf1802a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce84b3c51a5d2a45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.508+0000 7faf31200640 1 --2- 172.21.15.102:0/3671962928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb74c70 0x562cebb75050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:14.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.508+0000 7faf22000640 1 -- 172.21.15.102:0/3671962928 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf18049050 con 0x562cebb75620 2024-04-25T01:58:14.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.508+0000 7faf22000640 1 -- 172.21.15.102:0/3671962928 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf180bc0a0 con 0x562cebb75620 2024-04-25T01:58:14.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.508+0000 7faf22000640 1 -- 172.21.15.102:0/3671962928 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf180c0080 con 0x562cebb75620 2024-04-25T01:58:14.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.510+0000 7faf36a2e740 1 -- 172.21.15.102:0/3671962928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb75620 msgr2=0x562cebb881e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.510+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3671962928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb75620 0x562cebb881e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7faf180b8500 tx=0x7faf1802a420 comp rx=0 tx=0).stop 2024-04-25T01:58:14.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.515+0000 7faf36a2e740 1 -- 172.21.15.102:0/3671962928 shutdown_connections 2024-04-25T01:58:14.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.515+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3671962928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 0x562cebb8ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.515+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3671962928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb75620 0x562cebb881e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.515+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3671962928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb74c70 0x562cebb75050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.515+0000 7faf36a2e740 1 -- 172.21.15.102:0/3671962928 >> 172.21.15.102:0/3671962928 conn(0x562cebb71240 msgr2=0x562cebb71620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:14.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.519+0000 7faf36a2e740 1 -- 172.21.15.102:0/3671962928 shutdown_connections 2024-04-25T01:58:14.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.519+0000 7faf36a2e740 1 -- 172.21.15.102:0/3671962928 wait complete. 2024-04-25T01:58:14.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.519+0000 7faf36a2e740 1 -- start start 2024-04-25T01:58:14.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.520+0000 7faf36a2e740 1 Processor -- start 2024-04-25T01:58:14.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.521+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb85700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.521+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb75620 0x562cebb85c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.521+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 0x562cebb86180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.521+0000 7faf36a2e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x562ceba94e70 con 0x562cebb75620 2024-04-25T01:58:14.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.521+0000 7faf36a2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562cebad5150 con 0x562cebb74c70 2024-04-25T01:58:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.521+0000 7faf36a2e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562cebb8b2a0 con 0x562cebb88720 2024-04-25T01:58:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.521+0000 7faf36a2e740 10 client.0 _ll_register_callbacks cb 0x562cebad01f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.522+0000 7faf31200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb85700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.522+0000 7faf31200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb85700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59010/0 (socket says 172.21.15.102:59010) 2024-04-25T01:58:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.522+0000 7faf31200640 1 -- 172.21.15.102:0/3515974443 learned_addr learned my addr 172.21.15.102:0/3515974443 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.523+0000 7faf31c00640 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 0x562cebb86180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.523+0000 7faf2be00640 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb75620 0x562cebb85c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.524+0000 7faf31c00640 1 -- 172.21.15.102:0/3515974443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 msgr2=0x562cebb85700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.524+0000 7faf31c00640 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb85700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.524+0000 7faf31c00640 1 -- 172.21.15.102:0/3515974443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb75620 msgr2=0x562cebb85c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.524+0000 7faf31c00640 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb75620 0x562cebb85c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.524+0000 7faf31c00640 1 -- 172.21.15.102:0/3515974443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ceb915f70 con 0x562cebb88720 2024-04-25T01:58:14.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.525+0000 7faf16a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:14.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.525+0000 7faf31c00640 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 0x562cebb86180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faf24005fd0 tx=0x7faf2400fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44453effab4240fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.525+0000 7faf16a00640 1 -- 172.21.15.102:0/3515974443 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24011910 con 0x562cebb88720 2024-04-25T01:58:14.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.526+0000 7faf16a00640 1 -- 172.21.15.102:0/3515974443 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7faf24011710 con 0x562cebb88720 2024-04-25T01:58:14.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.527+0000 7faf36a2e740 1 -- 172.21.15.102:0/3515974443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf180bc0a0 con 0x562cebb88720 2024-04-25T01:58:14.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.527+0000 7faf16a00640 1 -- 172.21.15.102:0/3515974443 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf24011910 con 0x562cebb88720 2024-04-25T01:58:14.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.527+0000 7faf36a2e740 10 client.26077 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:14.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.527+0000 7faf36a2e740 1 -- 172.21.15.102:0/3515974443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562ceb7ad170 con 0x562cebb88720 2024-04-25T01:58:14.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.527+0000 7faf16a00640 1 -- 172.21.15.102:0/3515974443 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf24025420 con 0x562cebb88720 2024-04-25T01:58:14.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.528+0000 7faf16a00640 1 -- 172.21.15.102:0/3515974443 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7faf2401b410 con 0x562cebb88720 2024-04-25T01:58:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.528+0000 7faf36a2e740 10 client.26077 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.528+0000 7faf36a2e740 10 client.26077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.528+0000 7faf36a2e740 20 client.26077 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.528+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562cebb98bd0 0x562cebb9aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.528+0000 7faf36a2e740 4 client.26077 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.528+0000 7faf36a2e740 1 -- 172.21.15.102:0/3515974443 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7faf180bc0a0 con 0x562cebb98bd0 2024-04-25T01:58:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.530+0000 7faf31200640 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562cebb98bd0 0x562cebb9aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.533+0000 7faf16a00640 10 client.26077 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:14.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.533+0000 7faf31200640 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562cebb98bd0 0x562cebb9aff0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cebfc8cfe7376aa server_cookie=917ca74e57a7e8f8 in_seq=0 out_seq=0 2024-04-25T01:58:14.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.534+0000 7faf16a00640 1 -- 172.21.15.102:0/3515974443 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7faf180bc0a0 con 0x562cebb98bd0 2024-04-25T01:58:14.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.534+0000 7faf16a00640 10 client.26077 handle_command_reply: tid=0 2024-04-25T01:58:14.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.534+0000 7faf36a2e740 1 client.26077 shutdown 2024-04-25T01:58:14.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.534+0000 7faf36a2e740 1 -- 172.21.15.102:0/3515974443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 msgr2=0x562cebb86180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.534+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 0x562cebb86180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faf24005fd0 tx=0x7faf2400fa70 comp rx=0 tx=0).stop 2024-04-25T01:58:14.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.535+0000 7faf36a2e740 1 -- 172.21.15.102:0/3515974443 shutdown_connections 2024-04-25T01:58:14.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.535+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562cebb98bd0 0x562cebb9aff0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.535+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb88720 0x562cebb86180 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.535+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb75620 0x562cebb85c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.535+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3515974443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb85700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.535+0000 7faf36a2e740 1 -- 172.21.15.102:0/3515974443 >> 172.21.15.102:0/3515974443 conn(0x562cebb71240 msgr2=0x562cebb77040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:14.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.535+0000 7faf16a00640 0 client.26077 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:14.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.535+0000 7faf36a2e740 1 -- 172.21.15.102:0/3515974443 shutdown_connections 2024-04-25T01:58:14.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.536+0000 7faf36a2e740 1 -- 172.21.15.102:0/3515974443 wait complete. 2024-04-25T01:58:14.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.536+0000 7faf36a2e740 20 client.26077 trim_cache size 0 max 16384 2024-04-25T01:58:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.576+0000 7faf36a2e740 1 Processor -- start 2024-04-25T01:58:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.576+0000 7faf36a2e740 1 -- start start 2024-04-25T01:58:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.577+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb94070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.577+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb75620 0x562cebb70230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.577+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebba0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.577+0000 7faf2be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb75620 0x562cebb70230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.577+0000 7faf2be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb75620 0x562cebb70230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41748/0 (socket says 172.21.15.102:41748) 2024-04-25T01:58:14.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.578+0000 7faf31200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb94070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.578+0000 7faf31c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebba0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.578+0000 7faf31c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebba0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33970/0 (socket says 172.21.15.102:33970) 2024-04-25T01:58:14.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.578+0000 7faf36a2e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1ada80 con 0x562cebb88720 2024-04-25T01:58:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.578+0000 7faf36a2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1ab2f0 con 0x562cebb74c70 2024-04-25T01:58:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.578+0000 7faf36a2e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1a9120 con 0x562cebb75620 2024-04-25T01:58:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.578+0000 7faf2be00640 1 -- 172.21.15.102:0/690001195 learned_addr learned my addr 172.21.15.102:0/690001195 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.578+0000 7faf31200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb94070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59020/0 (socket says 172.21.15.102:59020) 2024-04-25T01:58:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.579+0000 7faf2be00640 1 -- 172.21.15.102:0/690001195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 msgr2=0x562cebb94070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.579+0000 7faf2be00640 1 --2- 172.21.15.102:0/690001195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb94070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.579+0000 7faf2be00640 1 -- 172.21.15.102:0/690001195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 msgr2=0x562cebba0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.579+0000 7faf2be00640 1 --2- 172.21.15.102:0/690001195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebba0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.579+0000 7faf2be00640 1 -- 172.21.15.102:0/690001195 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cebb753c0 con 0x562cebb75620 2024-04-25T01:58:14.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.579+0000 7faf31c00640 1 --2- 172.21.15.102:0/690001195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebba0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:14.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.580+0000 7faf2be00640 1 --2- 172.21.15.102:0/690001195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb75620 0x562cebb70230 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faf180c47e0 tx=0x7faf18040fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8de12ebd3e0496b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.580+0000 7faf21600640 1 -- 172.21.15.102:0/690001195 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf180bc0a0 con 0x562cebb75620 2024-04-25T01:58:14.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.580+0000 7faf21600640 1 -- 172.21.15.102:0/690001195 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf180dc0c0 con 0x562cebb75620 2024-04-25T01:58:14.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.580+0000 7faf21600640 1 -- 172.21.15.102:0/690001195 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf180c0070 con 0x562cebb75620 2024-04-25T01:58:14.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.581+0000 7faf36a2e740 1 -- 172.21.15.102:0/690001195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb75620 msgr2=0x562cebb70230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.581+0000 7faf36a2e740 1 --2- 172.21.15.102:0/690001195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb75620 0x562cebb70230 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faf180c47e0 tx=0x7faf18040fc0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.582+0000 7faf36a2e740 1 -- 172.21.15.102:0/690001195 shutdown_connections 2024-04-25T01:58:14.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.582+0000 7faf36a2e740 1 --2- 172.21.15.102:0/690001195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebba0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.582+0000 7faf36a2e740 1 --2- 172.21.15.102:0/690001195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb75620 0x562cebb70230 secure :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faf180c47e0 tx=0x7faf18040fc0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.582+0000 7faf36a2e740 1 --2- 172.21.15.102:0/690001195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb94070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.582+0000 7faf36a2e740 1 -- 172.21.15.102:0/690001195 >> 172.21.15.102:0/690001195 conn(0x562cebb71240 msgr2=0x562cebb6fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:14.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.582+0000 7faf36a2e740 1 -- 172.21.15.102:0/690001195 shutdown_connections 2024-04-25T01:58:14.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.582+0000 7faf36a2e740 1 -- 172.21.15.102:0/690001195 wait complete. 2024-04-25T01:58:14.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.583+0000 7faf36a2e740 1 -- start start 2024-04-25T01:58:14.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.583+0000 7faf36a2e740 1 Processor -- start 2024-04-25T01:58:14.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.584+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb90580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.584+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebb90ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.584+0000 7faf36a2e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb8d550 0x562cebb8d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.585+0000 7faf36a2e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1a97b0 con 0x562cebb88720 2024-04-25T01:58:14.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.585+0000 7faf36a2e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1aac20 con 0x562cebb74c70 2024-04-25T01:58:14.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.585+0000 7faf36a2e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c1a9e80 con 0x562cebb8d550 2024-04-25T01:58:14.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.585+0000 7faf36a2e740 10 client.0 _ll_register_callbacks cb 0x7faf240098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:14.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.585+0000 7faf31c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb8d550 0x562cebb8d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.585+0000 7faf31c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb8d550 0x562cebb8d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41754/0 (socket says 172.21.15.102:41754) 2024-04-25T01:58:14.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.585+0000 7faf31c00640 1 -- 172.21.15.102:0/3761795017 learned_addr learned my addr 172.21.15.102:0/3761795017 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:14.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.585+0000 7faf31200640 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb90580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.586+0000 7faf2be00640 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebb90ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.586+0000 7faf31200640 1 -- 172.21.15.102:0/3761795017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb8d550 msgr2=0x562cebb8d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.586+0000 7faf31200640 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb8d550 0x562cebb8d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.586+0000 7faf31200640 1 -- 172.21.15.102:0/3761795017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 msgr2=0x562cebb90ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.586+0000 7faf31200640 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebb90ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.586+0000 7faf31200640 1 -- 172.21.15.102:0/3761795017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cebb8e090 con 0x562cebb74c70 2024-04-25T01:58:14.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.586+0000 7faf2be00640 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebb90ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:14.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.587+0000 7faf31c00640 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb8d550 0x562cebb8d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:14.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.587+0000 7faf31200640 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb90580 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x562cebb9b5f0 tx=0x7faf1c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a5ebac518f670e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.587+0000 7faf16a00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:14.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.587+0000 7faf16a00640 1 -- 172.21.15.102:0/3761795017 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1c0266d0 con 0x562cebb74c70 2024-04-25T01:58:14.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.587+0000 7faf16a00640 1 -- 172.21.15.102:0/3761795017 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7faf1c026870 con 0x562cebb74c70 2024-04-25T01:58:14.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.588+0000 7faf36a2e740 1 -- 172.21.15.102:0/3761795017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf180dc0c0 con 0x562cebb74c70 2024-04-25T01:58:14.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.588+0000 7faf16a00640 1 -- 172.21.15.102:0/3761795017 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf1c010e50 con 0x562cebb74c70 2024-04-25T01:58:14.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.589+0000 7faf36a2e740 10 client.26132 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:14.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.589+0000 7faf36a2e740 1 -- 172.21.15.102:0/3761795017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562cebc00050 con 0x562cebb74c70 2024-04-25T01:58:14.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.589+0000 7faf16a00640 1 -- 172.21.15.102:0/3761795017 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf180dc0c0 con 0x562cebb74c70 2024-04-25T01:58:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.590+0000 7faf16a00640 1 -- 172.21.15.102:0/3761795017 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7faf1c01d420 con 0x562cebb74c70 2024-04-25T01:58:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.590+0000 7faf36a2e740 10 client.26132 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.590+0000 7faf36a2e740 10 client.26132 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.590+0000 7faf36a2e740 20 client.26132 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.590+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562cebc007d0 0x562cebc02bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.590+0000 7faf36a2e740 4 client.26132 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:58:14.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.590+0000 7faf36a2e740 1 -- 172.21.15.102:0/3761795017 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562cebae66d0 con 0x562cebc007d0 2024-04-25T01:58:14.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.590+0000 7faf2be00640 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562cebc007d0 0x562cebc02bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.591+0000 7faf16a00640 10 client.26132 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:14.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.591+0000 7faf2be00640 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562cebc007d0 0x562cebc02bf0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1619f1c4aa380a45 server_cookie=1ca12eacc17a6407 in_seq=0 out_seq=0 2024-04-25T01:58:14.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.592+0000 7faf16a00640 1 -- 172.21.15.102:0/3761795017 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562cebae66d0 con 0x562cebc007d0 2024-04-25T01:58:14.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.592+0000 7faf16a00640 10 client.26132 handle_command_reply: tid=0 2024-04-25T01:58:14.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.592+0000 7faf36a2e740 1 client.26132 shutdown 2024-04-25T01:58:14.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.592+0000 7faf36a2e740 1 -- 172.21.15.102:0/3761795017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 msgr2=0x562cebb90580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.592+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb90580 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x562cebb9b5f0 tx=0x7faf1c004730 comp rx=0 tx=0).stop 2024-04-25T01:58:14.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.593+0000 7faf36a2e740 1 -- 172.21.15.102:0/3761795017 shutdown_connections 2024-04-25T01:58:14.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.593+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562cebc007d0 0x562cebc02bf0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.593+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562cebb8d550 0x562cebb8d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.593+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562cebb88720 0x562cebb90ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.593+0000 7faf36a2e740 1 --2- 172.21.15.102:0/3761795017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562cebb74c70 0x562cebb90580 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.593+0000 7faf36a2e740 1 -- 172.21.15.102:0/3761795017 >> 172.21.15.102:0/3761795017 conn(0x562cebb71240 msgr2=0x562cebb703b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:14.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.593+0000 7faf16a00640 0 client.26132 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:14.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.593+0000 7faf36a2e740 1 -- 172.21.15.102:0/3761795017 shutdown_connections 2024-04-25T01:58:14.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.593+0000 7faf36a2e740 1 -- 172.21.15.102:0/3761795017 wait complete. 2024-04-25T01:58:14.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.594+0000 7faf36a2e740 20 client.26132 trim_cache size 0 max 16384 2024-04-25T01:58:14.598 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:58:14.598 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T01:58:14.598 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "9b8b8c26-591a-4589-ad39-8e0f5ef9fa05", 2024-04-25T01:58:14.598 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T01:58:14.598 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:58:14.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 -- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf100a2e00 msgr2=0x7faf100a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf100a2e00 0x7faf100a52c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faf1c005fd0 tx=0x7faf1c005e50 comp rx=0 tx=0).stop 2024-04-25T01:58:14.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 -- 172.21.15.102:0/2999342961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf2c101fa0 msgr2=0x7faf2c073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf2c101fa0 0x7faf2c073f80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faf18009850 tx=0x7faf18003b90 comp rx=0 tx=0).stop 2024-04-25T01:58:14.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 -- 172.21.15.102:0/2999342961 shutdown_connections 2024-04-25T01:58:14.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf100a2e00 0x7faf100a52c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c107680 0x7faf2c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf2c101fa0 0x7faf2c073f80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 --2- 172.21.15.102:0/2999342961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf2c1015e0 0x7faf2c073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 -- 172.21.15.102:0/2999342961 >> 172.21.15.102:0/2999342961 conn(0x7faf2c0fd110 msgr2=0x7faf2c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:14.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 -- 172.21.15.102:0/2999342961 shutdown_connections 2024-04-25T01:58:14.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.596+0000 7faf32600640 1 -- 172.21.15.102:0/2999342961 wait complete. 2024-04-25T01:58:14.634 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:58:14.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:58:14.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.937+0000 7fd5c2600640 1 -- 172.21.15.102:0/302508935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5bc10d040 msgr2=0x7fd5bc114c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.937+0000 7fd5c2600640 1 --2- 172.21.15.102:0/302508935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5bc10d040 0x7fd5bc114c10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4009ca0 tx=0x7fd5b402a8b0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.938+0000 7fd5c2600640 1 -- 172.21.15.102:0/302508935 shutdown_connections 2024-04-25T01:58:14.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.938+0000 7fd5c2600640 1 --2- 172.21.15.102:0/302508935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5bc10d040 0x7fd5bc114c10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.938+0000 7fd5c2600640 1 --2- 172.21.15.102:0/302508935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5bc108970 0x7fd5bc10c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.938+0000 7fd5c2600640 1 --2- 172.21.15.102:0/302508935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc107fc0 0x7fd5bc1083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.938+0000 7fd5c2600640 1 -- 172.21.15.102:0/302508935 >> 172.21.15.102:0/302508935 conn(0x7fd5bc075c80 msgr2=0x7fd5bc0780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:14.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.939+0000 7fd5c2600640 1 -- 172.21.15.102:0/302508935 shutdown_connections 2024-04-25T01:58:14.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.939+0000 7fd5c2600640 1 -- 172.21.15.102:0/302508935 wait complete. 2024-04-25T01:58:14.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.940+0000 7fd5c2600640 1 Processor -- start 2024-04-25T01:58:14.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.941+0000 7fd5c2600640 1 -- start start 2024-04-25T01:58:14.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.942+0000 7fd5c2600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5bc107fc0 0x7fd5bc19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.942+0000 7fd5c2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5bc10d040 0x7fd5bc19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.942+0000 7fd5c2600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc1a2e60 0x7fd5bc1a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.943+0000 7fd5bbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc1a2e60 0x7fd5bc1a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.943+0000 7fd5bbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc1a2e60 0x7fd5bc1a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59052/0 (socket says 172.21.15.102:59052) 2024-04-25T01:58:14.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.943+0000 7fd5bbe00640 1 -- 172.21.15.102:0/1640504500 learned_addr learned my addr 172.21.15.102:0/1640504500 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:14.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.943+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc118150 con 0x7fd5bc10d040 2024-04-25T01:58:14.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.943+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc117fd0 con 0x7fd5bc1a2e60 2024-04-25T01:58:14.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.943+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1182d0 con 0x7fd5bc107fc0 2024-04-25T01:58:14.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.943+0000 7fd5bb400640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5bc107fc0 0x7fd5bc19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.944+0000 7fd5bbe00640 1 -- 172.21.15.102:0/1640504500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5bc107fc0 msgr2=0x7fd5bc19e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.944+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5bc107fc0 0x7fd5bc19e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.944+0000 7fd5bbe00640 1 -- 172.21.15.102:0/1640504500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5bc10d040 msgr2=0x7fd5bc19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.944+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5bc10d040 0x7fd5bc19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.944+0000 7fd5bbe00640 1 -- 172.21.15.102:0/1640504500 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5bc1a3950 con 0x7fd5bc1a2e60 2024-04-25T01:58:14.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.945+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc1a2e60 0x7fd5bc1a3310 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4030b60 tx=0x7fd5b4030b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.945+0000 7fd5b3e00640 1 -- 172.21.15.102:0/1640504500 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4004240 con 0x7fd5bc1a2e60 2024-04-25T01:58:14.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.945+0000 7fd5b3e00640 1 -- 172.21.15.102:0/1640504500 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5b4040070 con 0x7fd5bc1a2e60 2024-04-25T01:58:14.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.945+0000 7fd5b3e00640 1 -- 172.21.15.102:0/1640504500 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4045de0 con 0x7fd5bc1a2e60 2024-04-25T01:58:14.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.947+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5bc1a3be0 con 0x7fd5bc1a2e60 2024-04-25T01:58:14.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.947+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5bc1ab350 con 0x7fd5bc1a2e60 2024-04-25T01:58:14.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.953+0000 7fd5b3e00640 1 -- 172.21.15.102:0/1640504500 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd5b4045470 con 0x7fd5bc1a2e60 2024-04-25T01:58:14.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.960+0000 7fd5b3e00640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd5a00a2e80 0x7fd5a00a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.961+0000 7fd5bb400640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd5a00a2e80 0x7fd5a00a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.961+0000 7fd5c5409740 1 Processor -- start 2024-04-25T01:58:14.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.961+0000 7fd5b3e00640 1 -- 172.21.15.102:0/1640504500 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd5b40f0790 con 0x7fd5bc1a2e60 2024-04-25T01:58:14.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.962+0000 7fd5c5409740 1 -- start start 2024-04-25T01:58:14.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.962+0000 7fd5bb400640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd5a00a2e80 0x7fd5a00a5340 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac005e10 tx=0x7fd5ac005da0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.964+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a773e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.964+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a81ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.964+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a843e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.964+0000 7fd5c5409740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5595f89db560 con 0x5595f8a77000 2024-04-25T01:58:14.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.964+0000 7fd5c5409740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5595f8983930 con 0x5595f8a81ff0 2024-04-25T01:58:14.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.964+0000 7fd5c5409740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5595f89d7520 con 0x5595f8a77920 2024-04-25T01:58:14.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.965+0000 7fd5bbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a843e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.965+0000 7fd5bbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a843e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59060/0 (socket says 172.21.15.102:59060) 2024-04-25T01:58:14.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.965+0000 7fd5bbe00640 1 -- 172.21.15.102:0/3454393821 learned_addr learned my addr 172.21.15.102:0/3454393821 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:14.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.965+0000 7fd5baa00640 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a81ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.966+0000 7fd5bb400640 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a773e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.966+0000 7fd5bbe00640 1 -- 172.21.15.102:0/3454393821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 msgr2=0x5595f8a81ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.966+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a81ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.966+0000 7fd5bbe00640 1 -- 172.21.15.102:0/3454393821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 msgr2=0x5595f8a773e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.966+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a773e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.966+0000 7fd5bbe00640 1 -- 172.21.15.102:0/3454393821 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595f87b97d0 con 0x5595f8a81ff0 2024-04-25T01:58:14.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.966+0000 7fd5bb400640 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a773e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:14.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.966+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a843e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40c5550 tx=0x7fd5b40bca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ac12e5b5690f884 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.967+0000 7fd5b0c00640 1 -- 172.21.15.102:0/3454393821 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40bce10 con 0x5595f8a81ff0 2024-04-25T01:58:14.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.967+0000 7fd5b0c00640 1 -- 172.21.15.102:0/3454393821 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5b4036040 con 0x5595f8a81ff0 2024-04-25T01:58:14.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.967+0000 7fd5b0c00640 1 -- 172.21.15.102:0/3454393821 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40bd070 con 0x5595f8a81ff0 2024-04-25T01:58:14.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.969+0000 7fd5c5409740 1 -- 172.21.15.102:0/3454393821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 msgr2=0x5595f8a843e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.969+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a843e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40c5550 tx=0x7fd5b40bca10 comp rx=0 tx=0).stop 2024-04-25T01:58:14.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.970+0000 7fd5c5409740 1 -- 172.21.15.102:0/3454393821 shutdown_connections 2024-04-25T01:58:14.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.970+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a843e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.970+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a81ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.970+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3454393821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a773e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.970+0000 7fd5c5409740 1 -- 172.21.15.102:0/3454393821 >> 172.21.15.102:0/3454393821 conn(0x5595f8a73320 msgr2=0x5595f8a73700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:14.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.970+0000 7fd5c5409740 1 -- 172.21.15.102:0/3454393821 shutdown_connections 2024-04-25T01:58:14.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.970+0000 7fd5c5409740 1 -- 172.21.15.102:0/3454393821 wait complete. 2024-04-25T01:58:14.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.971+0000 7fd5c5409740 1 -- start start 2024-04-25T01:58:14.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.973+0000 7fd5c5409740 1 Processor -- start 2024-04-25T01:58:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.975+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.975+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a77920 0x5595f8a7f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.975+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a81ff0 0x5595f8a7fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.975+0000 7fd5c5409740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5595f8a84ba0 con 0x5595f8a81ff0 2024-04-25T01:58:14.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.975+0000 7fd5c5409740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5595f8996d90 con 0x5595f8a77920 2024-04-25T01:58:14.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.975+0000 7fd5c5409740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5595f8a84d20 con 0x5595f8a77000 2024-04-25T01:58:14.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.976+0000 7fd5c5409740 10 client.0 _ll_register_callbacks cb 0x5595f8996780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:14.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.975+0000 7fd5baa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a77920 0x5595f8a7f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.976+0000 7fd5baa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a77920 0x5595f8a7f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59074/0 (socket says 172.21.15.102:59074) 2024-04-25T01:58:14.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.976+0000 7fd5baa00640 1 -- 172.21.15.102:0/1275661756 learned_addr learned my addr 172.21.15.102:0/1275661756 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:14.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.975+0000 7fd5bb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.977+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a81ff0 0x5595f8a7fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.977+0000 7fd5baa00640 1 -- 172.21.15.102:0/1275661756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 msgr2=0x5595f8a7f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.977+0000 7fd5baa00640 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.977+0000 7fd5baa00640 1 -- 172.21.15.102:0/1275661756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a81ff0 msgr2=0x5595f8a7fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:14.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.977+0000 7fd5baa00640 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a81ff0 0x5595f8a7fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.977+0000 7fd5baa00640 1 -- 172.21.15.102:0/1275661756 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595f86ef3f0 con 0x5595f8a77920 2024-04-25T01:58:14.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.978+0000 7fd5baa00640 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a77920 0x5595f8a7f550 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800d160 tx=0x7fd5a800d620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e814b49aae6b045a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:14.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.979+0000 7fd5a5600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:14.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.979+0000 7fd5a5600640 1 -- 172.21.15.102:0/1275661756 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a800de50 con 0x5595f8a77920 2024-04-25T01:58:14.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.979+0000 7fd5a5600640 1 -- 172.21.15.102:0/1275661756 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd5a80125f0 con 0x5595f8a77920 2024-04-25T01:58:14.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.979+0000 7fd5c5409740 1 -- 172.21.15.102:0/1275661756 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5b4036040 con 0x5595f8a77920 2024-04-25T01:58:14.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.980+0000 7fd5a5600640 1 -- 172.21.15.102:0/1275661756 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd5a80127f0 con 0x5595f8a77920 2024-04-25T01:58:14.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.980+0000 7fd5c5409740 10 client.26141 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:14.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.980+0000 7fd5c5409740 1 -- 172.21.15.102:0/1275661756 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5595f86af0f0 con 0x5595f8a77920 2024-04-25T01:58:14.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.981+0000 7fd5a5600640 1 -- 172.21.15.102:0/1275661756 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5b4036040 con 0x5595f8a77920 2024-04-25T01:58:14.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.981+0000 7fd5a5600640 1 -- 172.21.15.102:0/1275661756 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd5a8013070 con 0x5595f8a77920 2024-04-25T01:58:14.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.982+0000 7fd5c5409740 10 client.26141 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:14.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.982+0000 7fd5c5409740 10 client.26141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:14.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.982+0000 7fd5c5409740 20 client.26141 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:14.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.982+0000 7fd5c5409740 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595f8a92180 0x5595f8a945a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:14.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.982+0000 7fd5c5409740 4 client.26141 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:14.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.982+0000 7fd5c5409740 1 -- 172.21.15.102:0/1275661756 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595f89eed90 con 0x5595f8a92180 2024-04-25T01:58:14.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.982+0000 7fd5bb400640 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595f8a92180 0x5595f8a945a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:14.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.984+0000 7fd5bb400640 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595f8a92180 0x5595f8a945a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af60c84eb87cd346 server_cookie=53a59b2aec4737d4 in_seq=0 out_seq=0 2024-04-25T01:58:14.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.984+0000 7fd5a5600640 10 client.26141 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:14.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.985+0000 7fd5a5600640 1 -- 172.21.15.102:0/1275661756 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595f89eed90 con 0x5595f8a92180 2024-04-25T01:58:14.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.985+0000 7fd5a5600640 10 client.26141 handle_command_reply: tid=0 2024-04-25T01:58:14.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.985+0000 7fd5c5409740 1 client.26141 shutdown 2024-04-25T01:58:14.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.986+0000 7fd5c5409740 1 -- 172.21.15.102:0/1275661756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a77920 msgr2=0x5595f8a7f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:14.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.986+0000 7fd5c5409740 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a77920 0x5595f8a7f550 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800d160 tx=0x7fd5a800d620 comp rx=0 tx=0).stop 2024-04-25T01:58:14.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.987+0000 7fd5c5409740 1 -- 172.21.15.102:0/1275661756 shutdown_connections 2024-04-25T01:58:14.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.987+0000 7fd5c5409740 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595f8a92180 0x5595f8a945a0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.987+0000 7fd5c5409740 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a81ff0 0x5595f8a7fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.987+0000 7fd5c5409740 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a77920 0x5595f8a7f550 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.987+0000 7fd5c5409740 1 --2- 172.21.15.102:0/1275661756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:14.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.987+0000 7fd5a5600640 0 client.26141 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:14.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.987+0000 7fd5c5409740 1 -- 172.21.15.102:0/1275661756 >> 172.21.15.102:0/1275661756 conn(0x5595f8a73320 msgr2=0x5595f8a79120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:14.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.987+0000 7fd5c5409740 1 -- 172.21.15.102:0/1275661756 shutdown_connections 2024-04-25T01:58:14.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.989+0000 7fd5c5409740 1 -- 172.21.15.102:0/1275661756 wait complete. 2024-04-25T01:58:14.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:14.991+0000 7fd5c5409740 20 client.26141 trim_cache size 0 max 16384 2024-04-25T01:58:15.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.047+0000 7fd5c5409740 1 Processor -- start 2024-04-25T01:58:15.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.049+0000 7fd5c5409740 1 -- start start 2024-04-25T01:58:15.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.050+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a7df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:15.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.050+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a7a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:15.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a8d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:15.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5c5409740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1b6310 con 0x5595f8a77000 2024-04-25T01:58:15.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5c5409740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1b3b80 con 0x5595f8a81ff0 2024-04-25T01:58:15.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5c5409740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1b19d0 con 0x5595f8a77920 2024-04-25T01:58:15.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5baa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a7a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:15.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5baa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a7a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41792/0 (socket says 172.21.15.102:41792) 2024-04-25T01:58:15.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5baa00640 1 -- 172.21.15.102:0/4005109850 learned_addr learned my addr 172.21.15.102:0/4005109850 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:15.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a8d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5baa00640 1 -- 172.21.15.102:0/4005109850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 msgr2=0x5595f8a8d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5baa00640 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a8d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5baa00640 1 -- 172.21.15.102:0/4005109850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 msgr2=0x5595f8a7df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.051+0000 7fd5bb400640 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a7df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.052+0000 7fd5baa00640 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a7df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.052+0000 7fd5baa00640 1 -- 172.21.15.102:0/4005109850 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595f8a77750 con 0x5595f8a77920 2024-04-25T01:58:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.052+0000 7fd5bb400640 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a7df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.052+0000 7fd5baa00640 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a7a370 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8006b10 tx=0x7fd5a800dbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af92074079eedb6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:15.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.053+0000 7fd5a7e00640 1 -- 172.21.15.102:0/4005109850 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a802c020 con 0x5595f8a77920 2024-04-25T01:58:15.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.053+0000 7fd5a7e00640 1 -- 172.21.15.102:0/4005109850 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5a8019070 con 0x5595f8a77920 2024-04-25T01:58:15.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.054+0000 7fd5a7e00640 1 -- 172.21.15.102:0/4005109850 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8014690 con 0x5595f8a77920 2024-04-25T01:58:15.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.055+0000 7fd5c5409740 1 -- 172.21.15.102:0/4005109850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 msgr2=0x5595f8a7a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:15.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.055+0000 7fd5c5409740 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a7a370 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8006b10 tx=0x7fd5a800dbd0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.055+0000 7fd5c5409740 1 -- 172.21.15.102:0/4005109850 shutdown_connections 2024-04-25T01:58:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.055+0000 7fd5c5409740 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a8d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.055+0000 7fd5c5409740 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77920 0x5595f8a7a370 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.055+0000 7fd5c5409740 1 --2- 172.21.15.102:0/4005109850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77000 0x5595f8a7df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.055+0000 7fd5c5409740 1 -- 172.21.15.102:0/4005109850 >> 172.21.15.102:0/4005109850 conn(0x5595f8a73320 msgr2=0x5595f8a72480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.055+0000 7fd5c5409740 1 -- 172.21.15.102:0/4005109850 shutdown_connections 2024-04-25T01:58:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.056+0000 7fd5c5409740 1 -- 172.21.15.102:0/4005109850 wait complete. 2024-04-25T01:58:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.056+0000 7fd5c5409740 1 -- start start 2024-04-25T01:58:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.056+0000 7fd5c5409740 1 Processor -- start 2024-04-25T01:58:15.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.058+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:15.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.058+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77920 0x5595f8a7c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:15.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.058+0000 7fd5c5409740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a7c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.058+0000 7fd5c5409740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1b2040 con 0x5595f8a77920 2024-04-25T01:58:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.058+0000 7fd5c5409740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1b34b0 con 0x5595f8a81ff0 2024-04-25T01:58:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.058+0000 7fd5c5409740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd5bc1b2710 con 0x5595f8a77000 2024-04-25T01:58:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.058+0000 7fd5c5409740 10 client.0 _ll_register_callbacks cb 0x7fd5a800bbb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41796/0 (socket says 172.21.15.102:41796) 2024-04-25T01:58:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bb400640 1 -- 172.21.15.102:0/3416450264 learned_addr learned my addr 172.21.15.102:0/3416450264 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a7c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bb400640 1 -- 172.21.15.102:0/3416450264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 msgr2=0x5595f8a7c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bb400640 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a7c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bb400640 1 -- 172.21.15.102:0/3416450264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77920 msgr2=0x5595f8a7c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bb400640 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77920 0x5595f8a7c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bb400640 1 -- 172.21.15.102:0/3416450264 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595f8af77d0 con 0x5595f8a77000 2024-04-25T01:58:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.059+0000 7fd5bbe00640 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a7c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.060+0000 7fd5a5600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.060+0000 7fd5bb400640 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7bdf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00ee80 tx=0x7fd5ac004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ecffca5b669932 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.060+0000 7fd5a5600640 1 -- 172.21.15.102:0/3416450264 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac026530 con 0x5595f8a77000 2024-04-25T01:58:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.060+0000 7fd5c5409740 1 -- 172.21.15.102:0/3416450264 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5a8019070 con 0x5595f8a77000 2024-04-25T01:58:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.061+0000 7fd5a5600640 1 -- 172.21.15.102:0/3416450264 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fd5ac026330 con 0x5595f8a77000 2024-04-25T01:58:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.061+0000 7fd5a5600640 1 -- 172.21.15.102:0/3416450264 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd5ac026530 con 0x5595f8a77000 2024-04-25T01:58:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.062+0000 7fd5c5409740 10 client.26101 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:15.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.062+0000 7fd5c5409740 1 -- 172.21.15.102:0/3416450264 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5595f8af8b10 con 0x5595f8a77000 2024-04-25T01:58:15.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.062+0000 7fd5a5600640 1 -- 172.21.15.102:0/3416450264 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5a8019070 con 0x5595f8a77000 2024-04-25T01:58:15.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.062+0000 7fd5a5600640 1 -- 172.21.15.102:0/3416450264 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd5ac02f420 con 0x5595f8a77000 2024-04-25T01:58:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.063+0000 7fd5c5409740 10 client.26101 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.063+0000 7fd5c5409740 10 client.26101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.063+0000 7fd5c5409740 20 client.26101 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.063+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595f8af92d0 0x5595f8afb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.063+0000 7fd5c5409740 4 client.26101 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:58:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.063+0000 7fd5c5409740 1 -- 172.21.15.102:0/3416450264 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5595f89e8aa0 con 0x5595f8af92d0 2024-04-25T01:58:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.063+0000 7fd5baa00640 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595f8af92d0 0x5595f8afb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.064+0000 7fd5baa00640 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595f8af92d0 0x5595f8afb730 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=564b245e65563dee server_cookie=7802013bb737d2b0 in_seq=0 out_seq=0 2024-04-25T01:58:15.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.064+0000 7fd5a5600640 10 client.26101 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:15.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.064+0000 7fd5a5600640 1 -- 172.21.15.102:0/3416450264 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5595f89e8aa0 con 0x5595f8af92d0 2024-04-25T01:58:15.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.064+0000 7fd5a5600640 10 client.26101 handle_command_reply: tid=0 2024-04-25T01:58:15.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.064+0000 7fd5c5409740 1 client.26101 shutdown 2024-04-25T01:58:15.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.065+0000 7fd5c5409740 1 -- 172.21.15.102:0/3416450264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 msgr2=0x5595f8a7bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:15.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.065+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7bdf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00ee80 tx=0x7fd5ac004730 comp rx=0 tx=0).stop 2024-04-25T01:58:15.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.065+0000 7fd5c5409740 1 -- 172.21.15.102:0/3416450264 shutdown_connections 2024-04-25T01:58:15.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.065+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595f8af92d0 0x5595f8afb730 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.065+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595f8a81ff0 0x5595f8a7c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.065+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595f8a77920 0x5595f8a7c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.065+0000 7fd5c5409740 1 --2- 172.21.15.102:0/3416450264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595f8a77000 0x5595f8a7bdf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.065+0000 7fd5c5409740 1 -- 172.21.15.102:0/3416450264 >> 172.21.15.102:0/3416450264 conn(0x5595f8a73320 msgr2=0x5595f8a787b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.065+0000 7fd5a5600640 0 client.26101 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.066+0000 7fd5c5409740 1 -- 172.21.15.102:0/3416450264 shutdown_connections 2024-04-25T01:58:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.066+0000 7fd5c5409740 1 -- 172.21.15.102:0/3416450264 wait complete. 2024-04-25T01:58:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.066+0000 7fd5c5409740 20 client.26101 trim_cache size 0 max 16384 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout: "9b8b8c26-591a-4589-ad39-8e0f5ef9fa05": { 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "9b8b8c26-591a-4589-ad39-8e0f5ef9fa05", 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:58:15.070 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:58:15.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd5a00a2e80 msgr2=0x7fd5a00a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd5a00a2e80 0x7fd5a00a5340 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac005e10 tx=0x7fd5ac005da0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc1a2e60 msgr2=0x7fd5bc1a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc1a2e60 0x7fd5bc1a3310 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4030b60 tx=0x7fd5b4030b90 comp rx=0 tx=0).stop 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 shutdown_connections 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd5a00a2e80 0x7fd5a00a5340 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc1a2e60 0x7fd5bc1a3310 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5bc10d040 0x7fd5bc19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 --2- 172.21.15.102:0/1640504500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5bc107fc0 0x7fd5bc19e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 >> 172.21.15.102:0/1640504500 conn(0x7fd5bc075c80 msgr2=0x7fd5bc076810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.068+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 shutdown_connections 2024-04-25T01:58:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:15.069+0000 7fd5c2600640 1 -- 172.21.15.102:0/1640504500 wait complete. 2024-04-25T01:58:15.097 INFO:tasks.cephfs.filesystem:scrub status for tag:9b8b8c26-591a-4589-ad39-8e0f5ef9fa05 - {'path': '/', 'tag': '9b8b8c26-591a-4589-ad39-8e0f5ef9fa05', 'options': 'recursive,force'} 2024-04-25T01:58:15.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: pgmap v92: 97 pgs: 97 active+clean; 2.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 40 MiB/s wr, 1.24k op/s 2024-04-25T01:58:15.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T01:58:15.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:15 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: pgmap v92: 97 pgs: 97 active+clean; 2.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 40 MiB/s wr, 1.24k op/s 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:15.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:15.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:15.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:15 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: pgmap v92: 97 pgs: 97 active+clean; 2.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 40 MiB/s wr, 1.24k op/s 2024-04-25T01:58:15.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T01:58:15.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:15 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:17.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:17 smithi102 ceph-mon[27796]: pgmap v93: 97 pgs: 97 active+clean; 2.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 38 MiB/s wr, 1.11k op/s 2024-04-25T01:58:17.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:17 smithi110 ceph-mon[32848]: pgmap v93: 97 pgs: 97 active+clean; 2.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 38 MiB/s wr, 1.11k op/s 2024-04-25T01:58:17.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:17 smithi193 ceph-mon[32981]: pgmap v93: 97 pgs: 97 active+clean; 2.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 38 MiB/s wr, 1.11k op/s 2024-04-25T01:58:18.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 145.0 s, 3959.3 tps, lat 25.174 ms stddev 27.614 2024-04-25T01:58:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.524+0000 7ff304600640 1 -- 172.21.15.102:0/1513011964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2fc073ae0 msgr2=0x7ff2fc073ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.524+0000 7ff304600640 1 --2- 172.21.15.102:0/1513011964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2fc073ae0 0x7ff2fc073ec0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009900 tx=0x7ff2ec02a480 comp rx=0 tx=0).stop 2024-04-25T01:58:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.524+0000 7ff304600640 1 -- 172.21.15.102:0/1513011964 shutdown_connections 2024-04-25T01:58:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.524+0000 7ff304600640 1 --2- 172.21.15.102:0/1513011964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc112730 0x7ff2fc114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.524+0000 7ff304600640 1 --2- 172.21.15.102:0/1513011964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2fc074400 0x7ff2fc1121f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.524+0000 7ff304600640 1 --2- 172.21.15.102:0/1513011964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2fc073ae0 0x7ff2fc073ec0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.524+0000 7ff304600640 1 -- 172.21.15.102:0/1513011964 >> 172.21.15.102:0/1513011964 conn(0x7ff2fc0fd1f0 msgr2=0x7ff2fc0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.524+0000 7ff304600640 1 -- 172.21.15.102:0/1513011964 shutdown_connections 2024-04-25T01:58:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.524+0000 7ff304600640 1 -- 172.21.15.102:0/1513011964 wait complete. 2024-04-25T01:58:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.525+0000 7ff304600640 1 Processor -- start 2024-04-25T01:58:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.526+0000 7ff304600640 1 -- start start 2024-04-25T01:58:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.526+0000 7ff304600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc073ae0 0x7ff2fc19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.526+0000 7ff304600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2fc074400 0x7ff2fc19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.526+0000 7ff304600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2fc112730 0x7ff2fc1a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.526+0000 7ff301200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2fc074400 0x7ff2fc19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.526+0000 7ff301200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2fc074400 0x7ff2fc19eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41816/0 (socket says 172.21.15.102:41816) 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.526+0000 7ff301c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc073ae0 0x7ff2fc19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff301c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc073ae0 0x7ff2fc19e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59114/0 (socket says 172.21.15.102:59114) 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc118150 con 0x7ff2fc112730 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc117fd0 con 0x7ff2fc073ae0 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc1182d0 con 0x7ff2fc074400 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff301200640 1 -- 172.21.15.102:0/2167945060 learned_addr learned my addr 172.21.15.102:0/2167945060 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff302600640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2fc112730 0x7ff2fc1a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff301c00640 1 -- 172.21.15.102:0/2167945060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2fc074400 msgr2=0x7ff2fc19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff301c00640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2fc074400 0x7ff2fc19eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff301c00640 1 -- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2fc112730 msgr2=0x7ff2fc1a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff301c00640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2fc112730 0x7ff2fc1a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.527+0000 7ff301c00640 1 -- 172.21.15.102:0/2167945060 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2fc1a35c0 con 0x7ff2fc073ae0 2024-04-25T01:58:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.528+0000 7ff301c00640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc073ae0 0x7ff2fc19e5d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec02a9a0 tx=0x7ff2ec003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.528+0000 7ff2fa000640 1 -- 172.21.15.102:0/2167945060 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec03f070 con 0x7ff2fc073ae0 2024-04-25T01:58:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.528+0000 7ff2fa000640 1 -- 172.21.15.102:0/2167945060 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2ec031c90 con 0x7ff2fc073ae0 2024-04-25T01:58:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.528+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2fc1a3850 con 0x7ff2fc073ae0 2024-04-25T01:58:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.528+0000 7ff2fa000640 1 -- 172.21.15.102:0/2167945060 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec035030 con 0x7ff2fc073ae0 2024-04-25T01:58:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.528+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2fc1ab140 con 0x7ff2fc073ae0 2024-04-25T01:58:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.533+0000 7ff307397740 1 Processor -- start 2024-04-25T01:58:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.533+0000 7ff307397740 1 -- start start 2024-04-25T01:58:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.533+0000 7ff307397740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x5598572421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.533+0000 7ff307397740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 0x55985724caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.533+0000 7ff307397740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.533+0000 7ff307397740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5598571a5ab0 con 0x55985724cfe0 2024-04-25T01:58:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.533+0000 7ff307397740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55985714e640 con 0x559857242720 2024-04-25T01:58:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.533+0000 7ff307397740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5598571a1a70 con 0x559857241e00 2024-04-25T01:58:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.534+0000 7ff302600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x5598572421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.534+0000 7ff301200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.534+0000 7ff301200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34062/0 (socket says 172.21.15.102:34062) 2024-04-25T01:58:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.534+0000 7ff301200640 1 -- 172.21.15.102:0/2977136492 learned_addr learned my addr 172.21.15.102:0/2977136492 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.534+0000 7ff2fa000640 1 -- 172.21.15.102:0/2167945060 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff2ec042070 con 0x7ff2fc073ae0 2024-04-25T01:58:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.535+0000 7ff301200640 1 -- 172.21.15.102:0/2977136492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 msgr2=0x5598572421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.535+0000 7ff301200640 1 --2- 172.21.15.102:0/2977136492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x5598572421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.535+0000 7ff301200640 1 -- 172.21.15.102:0/2977136492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 msgr2=0x55985724caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.535+0000 7ff301c00640 1 --2- 172.21.15.102:0/2977136492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 0x55985724caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.535+0000 7ff301200640 1 --2- 172.21.15.102:0/2977136492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 0x55985724caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.535+0000 7ff301200640 1 -- 172.21.15.102:0/2977136492 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559856eba210 con 0x55985724cfe0 2024-04-25T01:58:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.535+0000 7ff301c00640 1 --2- 172.21.15.102:0/2977136492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 0x55985724caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.535+0000 7ff301200640 1 --2- 172.21.15.102:0/2977136492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724f3d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009d60 tx=0x7ff2f0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62e2bac09b50fe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.536+0000 7ff2eaa00640 1 -- 172.21.15.102:0/2977136492 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0004230 con 0x55985724cfe0 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.536+0000 7ff2eaa00640 1 -- 172.21.15.102:0/2977136492 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2f00043d0 con 0x55985724cfe0 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.536+0000 7ff2eaa00640 1 -- 172.21.15.102:0/2977136492 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f00035f0 con 0x55985724cfe0 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.536+0000 7ff2fa000640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff2e40a2ea0 0x7ff2e40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.536+0000 7ff2fa000640 1 -- 172.21.15.102:0/2167945060 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff2ec0efa60 con 0x7ff2fc073ae0 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.537+0000 7ff302600640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff2e40a2ea0 0x7ff2e40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.537+0000 7ff307397740 1 -- 172.21.15.102:0/2977136492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 msgr2=0x55985724f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.537+0000 7ff307397740 1 --2- 172.21.15.102:0/2977136492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724f3d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009d60 tx=0x7ff2f0011810 comp rx=0 tx=0).stop 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.537+0000 7ff307397740 1 -- 172.21.15.102:0/2977136492 shutdown_connections 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.537+0000 7ff307397740 1 --2- 172.21.15.102:0/2977136492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724f3d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.537+0000 7ff307397740 1 --2- 172.21.15.102:0/2977136492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 0x55985724caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.537+0000 7ff307397740 1 --2- 172.21.15.102:0/2977136492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x5598572421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.537+0000 7ff307397740 1 -- 172.21.15.102:0/2977136492 >> 172.21.15.102:0/2977136492 conn(0x55985723e310 msgr2=0x55985723e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.538+0000 7ff302600640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff2e40a2ea0 0x7ff2e40a5360 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc06adc0 tx=0x7ff2f4009760 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.538+0000 7ff307397740 1 -- 172.21.15.102:0/2977136492 shutdown_connections 2024-04-25T01:58:18.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.538+0000 7ff307397740 1 -- 172.21.15.102:0/2977136492 wait complete. 2024-04-25T01:58:18.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.539+0000 7ff307397740 1 -- start start 2024-04-25T01:58:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.539+0000 7ff307397740 1 Processor -- start 2024-04-25T01:58:18.543 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.540+0000 7ff307397740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x559857249f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.541+0000 7ff307397740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 0x55985724a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.541+0000 7ff301c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x559857249f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.541+0000 7ff301c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x559857249f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41824/0 (socket says 172.21.15.102:41824) 2024-04-25T01:58:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.541+0000 7ff307397740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.541+0000 7ff307397740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55985724fc10 con 0x55985724cfe0 2024-04-25T01:58:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff307397740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559857161990 con 0x559857242720 2024-04-25T01:58:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff307397740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55985724fd90 con 0x559857241e00 2024-04-25T01:58:18.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff307397740 10 client.0 _ll_register_callbacks cb 0x559857163950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff301c00640 1 -- 172.21.15.102:0/2076646131 learned_addr learned my addr 172.21.15.102:0/2076646131 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff301200640 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 0x55985724a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff302600640 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff301c00640 1 -- 172.21.15.102:0/2076646131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 msgr2=0x55985724a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff301c00640 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 0x55985724a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff301c00640 1 -- 172.21.15.102:0/2076646131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 msgr2=0x55985724a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:18.546 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff301c00640 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.542+0000 7ff301c00640 1 -- 172.21.15.102:0/2076646131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559856f86620 con 0x559857241e00 2024-04-25T01:58:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.543+0000 7ff301c00640 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x559857249f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec004650 tx=0x7ff2ec031490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71417d1bbff53a83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.543+0000 7ff2df400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.543+0000 7ff2df400640 1 -- 172.21.15.102:0/2076646131 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec0bb8b0 con 0x559857241e00 2024-04-25T01:58:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.543+0000 7ff2df400640 1 -- 172.21.15.102:0/2076646131 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff2ec0b85c0 con 0x559857241e00 2024-04-25T01:58:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.544+0000 7ff307397740 1 -- 172.21.15.102:0/2076646131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2f00043d0 con 0x559857241e00 2024-04-25T01:58:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.544+0000 7ff2df400640 1 -- 172.21.15.102:0/2076646131 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff2ec0bc650 con 0x559857241e00 2024-04-25T01:58:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.544+0000 7ff307397740 10 client.26113 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.544+0000 7ff307397740 1 -- 172.21.15.102:0/2076646131 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55985724fc10 con 0x559857241e00 2024-04-25T01:58:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.544+0000 7ff2df400640 1 -- 172.21.15.102:0/2076646131 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2f00043d0 con 0x559857241e00 2024-04-25T01:58:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.546+0000 7ff2df400640 1 -- 172.21.15.102:0/2076646131 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff2ec0bc1b0 con 0x559857241e00 2024-04-25T01:58:18.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.546+0000 7ff307397740 10 client.26113 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:18.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.546+0000 7ff307397740 10 client.26113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:18.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.547+0000 7ff307397740 20 client.26113 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.547+0000 7ff307397740 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55985725cee0 0x55985725f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.547+0000 7ff307397740 4 client.26113 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.547+0000 7ff307397740 1 -- 172.21.15.102:0/2076646131 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598571b2ff0 con 0x55985725cee0 2024-04-25T01:58:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.547+0000 7ff301200640 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55985725cee0 0x55985725f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.548+0000 7ff301200640 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55985725cee0 0x55985725f310 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdaed0a1c94f4ce server_cookie=c0e441cf8fab8ad4 in_seq=0 out_seq=0 2024-04-25T01:58:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.548+0000 7ff2df400640 10 client.26113 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:18.551 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.549+0000 7ff2df400640 1 -- 172.21.15.102:0/2076646131 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5598571b2ff0 con 0x55985725cee0 2024-04-25T01:58:18.551 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.549+0000 7ff2df400640 10 client.26113 handle_command_reply: tid=0 2024-04-25T01:58:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.549+0000 7ff307397740 1 client.26113 shutdown 2024-04-25T01:58:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.550+0000 7ff307397740 1 -- 172.21.15.102:0/2076646131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 msgr2=0x559857249f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.550+0000 7ff307397740 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x559857249f20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec004650 tx=0x7ff2ec031490 comp rx=0 tx=0).stop 2024-04-25T01:58:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.550+0000 7ff307397740 1 -- 172.21.15.102:0/2076646131 shutdown_connections 2024-04-25T01:58:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.550+0000 7ff307397740 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55985725cee0 0x55985725f310 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.550+0000 7ff307397740 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.550+0000 7ff307397740 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559857242720 0x55985724a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.550+0000 7ff307397740 1 --2- 172.21.15.102:0/2076646131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857241e00 0x559857249f20 secure :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec004650 tx=0x7ff2ec031490 comp rx=0 tx=0).stop 2024-04-25T01:58:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.550+0000 7ff307397740 1 -- 172.21.15.102:0/2076646131 >> 172.21.15.102:0/2076646131 conn(0x55985723e310 msgr2=0x559857244040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:18.553 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.550+0000 7ff2df400640 0 client.26113 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.551+0000 7ff307397740 1 -- 172.21.15.102:0/2076646131 shutdown_connections 2024-04-25T01:58:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.551+0000 7ff307397740 1 -- 172.21.15.102:0/2076646131 wait complete. 2024-04-25T01:58:18.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.551+0000 7ff307397740 20 client.26113 trim_cache size 0 max 16384 2024-04-25T01:58:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.580+0000 7ff307397740 1 Processor -- start 2024-04-25T01:58:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.580+0000 7ff307397740 1 -- start start 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff307397740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x55985723dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff307397740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff307397740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724c760 0x559857243ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff307397740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc1b62e0 con 0x55985724cfe0 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff307397740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc1b3b50 con 0x55985724c760 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff307397740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc1b19a0 con 0x559857242720 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff302600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724c760 0x559857243ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff302600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724c760 0x559857243ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59144/0 (socket says 172.21.15.102:59144) 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff302600640 1 -- 172.21.15.102:0/737168768 learned_addr learned my addr 172.21.15.102:0/737168768 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff301c00640 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x55985723dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff301200640 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff302600640 1 -- 172.21.15.102:0/737168768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 msgr2=0x55985723dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff302600640 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x55985723dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.581+0000 7ff302600640 1 -- 172.21.15.102:0/737168768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 msgr2=0x55985724a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.582+0000 7ff302600640 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.582+0000 7ff302600640 1 -- 172.21.15.102:0/737168768 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55985723e140 con 0x55985724c760 2024-04-25T01:58:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.582+0000 7ff301c00640 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x55985723dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.583+0000 7ff301200640 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.584+0000 7ff302600640 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724c760 0x559857243ec0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4003cf0 tx=0x7ff2f4015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9475b9c2a8189d8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.585+0000 7ff2ea000640 1 -- 172.21.15.102:0/737168768 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f4003e60 con 0x55985724c760 2024-04-25T01:58:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.585+0000 7ff2ea000640 1 -- 172.21.15.102:0/737168768 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2f401eb00 con 0x55985724c760 2024-04-25T01:58:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.585+0000 7ff2ea000640 1 -- 172.21.15.102:0/737168768 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f401dce0 con 0x55985724c760 2024-04-25T01:58:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.585+0000 7ff307397740 1 -- 172.21.15.102:0/737168768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724c760 msgr2=0x559857243ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.585+0000 7ff307397740 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724c760 0x559857243ec0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4003cf0 tx=0x7ff2f4015000 comp rx=0 tx=0).stop 2024-04-25T01:58:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.586+0000 7ff307397740 1 -- 172.21.15.102:0/737168768 shutdown_connections 2024-04-25T01:58:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.586+0000 7ff307397740 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724c760 0x559857243ec0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.586+0000 7ff307397740 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724cfe0 0x55985724a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.586+0000 7ff307397740 1 --2- 172.21.15.102:0/737168768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x55985723dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.586+0000 7ff307397740 1 -- 172.21.15.102:0/737168768 >> 172.21.15.102:0/737168768 conn(0x55985723e310 msgr2=0x5598572437b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.586+0000 7ff307397740 1 -- 172.21.15.102:0/737168768 shutdown_connections 2024-04-25T01:58:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.586+0000 7ff307397740 1 -- 172.21.15.102:0/737168768 wait complete. 2024-04-25T01:58:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.586+0000 7ff307397740 1 -- start start 2024-04-25T01:58:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.587+0000 7ff307397740 1 Processor -- start 2024-04-25T01:58:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.587+0000 7ff307397740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x559857246f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.587+0000 7ff307397740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724c760 0x559857247450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.587+0000 7ff307397740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724cfe0 0x5598572c2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.587+0000 7ff307397740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc1b2010 con 0x55985724c760 2024-04-25T01:58:18.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.587+0000 7ff307397740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc1b3480 con 0x55985724cfe0 2024-04-25T01:58:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.587+0000 7ff307397740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff2fc1b26e0 con 0x559857242720 2024-04-25T01:58:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.587+0000 7ff307397740 10 client.0 _ll_register_callbacks cb 0x7ff2ec031e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.588+0000 7ff302600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724cfe0 0x5598572c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.588+0000 7ff302600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724cfe0 0x5598572c2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59150/0 (socket says 172.21.15.102:59150) 2024-04-25T01:58:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.588+0000 7ff302600640 1 -- 172.21.15.102:0/4178026194 learned_addr learned my addr 172.21.15.102:0/4178026194 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:18.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.588+0000 7ff301c00640 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x559857246f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.589+0000 7ff301200640 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724c760 0x559857247450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.589+0000 7ff301c00640 1 -- 172.21.15.102:0/4178026194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724cfe0 msgr2=0x5598572c2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.589+0000 7ff301c00640 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724cfe0 0x5598572c2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.589+0000 7ff301c00640 1 -- 172.21.15.102:0/4178026194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724c760 msgr2=0x559857247450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.589+0000 7ff301c00640 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724c760 0x559857247450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.589+0000 7ff301c00640 1 -- 172.21.15.102:0/4178026194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559857247d20 con 0x559857242720 2024-04-25T01:58:18.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.589+0000 7ff301200640 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724c760 0x559857247450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.590+0000 7ff301c00640 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x559857246f10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0ba200 tx=0x7ff2ec003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d391cd42c457fbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:18.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.590+0000 7ff2df400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.590+0000 7ff2df400640 1 -- 172.21.15.102:0/4178026194 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec0da080 con 0x559857242720 2024-04-25T01:58:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.590+0000 7ff2df400640 1 -- 172.21.15.102:0/4178026194 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff2ec0bb5a0 con 0x559857242720 2024-04-25T01:58:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.591+0000 7ff307397740 1 -- 172.21.15.102:0/4178026194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2f401eb00 con 0x559857242720 2024-04-25T01:58:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.591+0000 7ff2df400640 1 -- 172.21.15.102:0/4178026194 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff2ec0bb8b0 con 0x559857242720 2024-04-25T01:58:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.591+0000 7ff307397740 10 client.26119 fetch_fsmap learned FSMap version 27 2024-04-25T01:58:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.591+0000 7ff307397740 1 -- 172.21.15.102:0/4178026194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5598572d4a30 con 0x559857242720 2024-04-25T01:58:18.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.591+0000 7ff2df400640 1 -- 172.21.15.102:0/4178026194 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2f401eb00 con 0x559857242720 2024-04-25T01:58:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.592+0000 7ff2df400640 1 -- 172.21.15.102:0/4178026194 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff2ec0b7e30 con 0x559857242720 2024-04-25T01:58:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.592+0000 7ff307397740 10 client.26119 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T01:58:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.592+0000 7ff307397740 10 client.26119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.592+0000 7ff307397740 20 client.26119 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.592+0000 7ff307397740 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5598572d5050 0x5598572d7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.592+0000 7ff307397740 4 client.26119 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:58:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.592+0000 7ff307397740 1 -- 172.21.15.102:0/4178026194 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5598571ad340 con 0x5598572d5050 2024-04-25T01:58:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.592+0000 7ff301200640 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5598572d5050 0x5598572d7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.593+0000 7ff301200640 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5598572d5050 0x5598572d7470 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76526e12e2294c1b server_cookie=91690449458f3a72 in_seq=0 out_seq=0 2024-04-25T01:58:18.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.593+0000 7ff2df400640 10 client.26119 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.606+0000 7ff2df400640 1 -- 172.21.15.102:0/4178026194 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5598571ad340 con 0x5598572d5050 2024-04-25T01:58:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.606+0000 7ff2df400640 10 client.26119 handle_command_reply: tid=0 2024-04-25T01:58:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.606+0000 7ff307397740 1 client.26119 shutdown 2024-04-25T01:58:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 -- 172.21.15.102:0/4178026194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 msgr2=0x559857246f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x559857246f10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec0ba200 tx=0x7ff2ec003f90 comp rx=0 tx=0).stop 2024-04-25T01:58:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 -- 172.21.15.102:0/4178026194 shutdown_connections 2024-04-25T01:58:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5598572d5050 0x5598572d7470 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55985724cfe0 0x5598572c2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55985724c760 0x559857247450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 --2- 172.21.15.102:0/4178026194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559857242720 0x559857246f10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 -- 172.21.15.102:0/4178026194 >> 172.21.15.102:0/4178026194 conn(0x55985723e310 msgr2=0x559857244040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff2df400640 0 client.26119 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 -- 172.21.15.102:0/4178026194 shutdown_connections 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 1 -- 172.21.15.102:0/4178026194 wait complete. 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.607+0000 7ff307397740 20 client.26119 trim_cache size 0 max 16384 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:58:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:58:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.608+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff2e40a2ea0 msgr2=0x7ff2e40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.608+0000 7ff304600640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff2e40a2ea0 0x7ff2e40a5360 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc06adc0 tx=0x7ff2f4009760 comp rx=0 tx=0).stop 2024-04-25T01:58:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.608+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc073ae0 msgr2=0x7ff2fc19e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.608+0000 7ff304600640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc073ae0 0x7ff2fc19e5d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec02a9a0 tx=0x7ff2ec003b90 comp rx=0 tx=0).stop 2024-04-25T01:58:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.609+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 shutdown_connections 2024-04-25T01:58:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.609+0000 7ff304600640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff2e40a2ea0 0x7ff2e40a5360 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.609+0000 7ff304600640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2fc112730 0x7ff2fc1a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.609+0000 7ff304600640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff2fc074400 0x7ff2fc19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.609+0000 7ff304600640 1 --2- 172.21.15.102:0/2167945060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc073ae0 0x7ff2fc19e5d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.609+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 >> 172.21.15.102:0/2167945060 conn(0x7ff2fc0fd1f0 msgr2=0x7ff2fc0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.609+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 shutdown_connections 2024-04-25T01:58:18.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:18.609+0000 7ff304600640 1 -- 172.21.15.102:0/2167945060 wait complete. 2024-04-25T01:58:19.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:19 smithi102 ceph-mon[27796]: pgmap v94: 97 pgs: 97 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 43 MiB/s wr, 1.62k op/s 2024-04-25T01:58:19.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:19 smithi110 ceph-mon[32848]: pgmap v94: 97 pgs: 97 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 43 MiB/s wr, 1.62k op/s 2024-04-25T01:58:19.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:19 smithi193 ceph-mon[32981]: pgmap v94: 97 pgs: 97 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 43 MiB/s wr, 1.62k op/s 2024-04-25T01:58:21.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:21 smithi102 ceph-mon[27796]: pgmap v95: 97 pgs: 97 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 17 MiB/s wr, 1.04k op/s 2024-04-25T01:58:21.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:21 smithi110 ceph-mon[32848]: pgmap v95: 97 pgs: 97 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 17 MiB/s wr, 1.04k op/s 2024-04-25T01:58:21.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:21 smithi193 ceph-mon[32981]: pgmap v95: 97 pgs: 97 active+clean; 2.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 17 MiB/s wr, 1.04k op/s 2024-04-25T01:58:22.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:22 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:58:22.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:22 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:58:22.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:22 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:58:22.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:22 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:58:22.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:22 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:58:22.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:22 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:58:22.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:22 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:58:22.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:22 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:58:22.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:22 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:58:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 150.0 s, 2833.2 tps, lat 35.224 ms stddev 38.383 2024-04-25T01:58:23.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:23 smithi102 ceph-mon[27796]: pgmap v96: 97 pgs: 97 active+clean; 2.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 20 MiB/s wr, 1.26k op/s 2024-04-25T01:58:23.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:23 smithi110 ceph-mon[32848]: pgmap v96: 97 pgs: 97 active+clean; 2.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 20 MiB/s wr, 1.26k op/s 2024-04-25T01:58:23.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:23 smithi193 ceph-mon[32981]: pgmap v96: 97 pgs: 97 active+clean; 2.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 20 MiB/s wr, 1.26k op/s 2024-04-25T01:58:25.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:25 smithi102 ceph-mon[27796]: pgmap v97: 97 pgs: 97 active+clean; 2.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 13 MiB/s wr, 986 op/s 2024-04-25T01:58:25.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:25 smithi110 ceph-mon[32848]: pgmap v97: 97 pgs: 97 active+clean; 2.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 13 MiB/s wr, 986 op/s 2024-04-25T01:58:25.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:25 smithi193 ceph-mon[32981]: pgmap v97: 97 pgs: 97 active+clean; 2.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 13 MiB/s wr, 986 op/s 2024-04-25T01:58:27.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:27 smithi102 ceph-mon[27796]: pgmap v98: 97 pgs: 97 active+clean; 2.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.9 MiB/s wr, 808 op/s 2024-04-25T01:58:27.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:27 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:58:27.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:27 smithi110 ceph-mon[32848]: pgmap v98: 97 pgs: 97 active+clean; 2.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.9 MiB/s wr, 808 op/s 2024-04-25T01:58:27.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:27 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:58:27.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:27 smithi193 ceph-mon[32981]: pgmap v98: 97 pgs: 97 active+clean; 2.1 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 8.9 MiB/s wr, 808 op/s 2024-04-25T01:58:27.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:27 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:58:28.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 155.0 s, 3403.1 tps, lat 29.481 ms stddev 32.878 2024-04-25T01:58:29.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:29 smithi102 ceph-mon[27796]: pgmap v99: 97 pgs: 97 active+clean; 2.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-04-25T01:58:29.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:29 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:58:29.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:29 smithi110 ceph-mon[32848]: pgmap v99: 97 pgs: 97 active+clean; 2.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-04-25T01:58:29.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:29 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:58:29.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:29 smithi193 ceph-mon[32981]: pgmap v99: 97 pgs: 97 active+clean; 2.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 13 MiB/s wr, 1.18k op/s 2024-04-25T01:58:29.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:29 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:58:31.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:31 smithi102 ceph-mon[27796]: pgmap v100: 97 pgs: 97 active+clean; 2.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.9 MiB/s wr, 668 op/s 2024-04-25T01:58:31.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:31 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:58:31.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:31 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:58:31.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:31 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:58:31.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:31 smithi110 ceph-mon[32848]: pgmap v100: 97 pgs: 97 active+clean; 2.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.9 MiB/s wr, 668 op/s 2024-04-25T01:58:31.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:31 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:58:31.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:31 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:58:31.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:31 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:58:31.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:31 smithi193 ceph-mon[32981]: pgmap v100: 97 pgs: 97 active+clean; 2.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.9 MiB/s wr, 668 op/s 2024-04-25T01:58:31.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:31 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:58:31.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:31 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:58:31.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:31 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:58:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.787+0000 7fb535200640 1 -- 172.21.15.102:0/3349306108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb530101f30 msgr2=0x7fb53010e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.787+0000 7fb535200640 1 --2- 172.21.15.102:0/3349306108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb530101f30 0x7fb53010e3f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009930 tx=0x7fb51c02a440 comp rx=0 tx=0).stop 2024-04-25T01:58:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.787+0000 7fb535200640 1 -- 172.21.15.102:0/3349306108 shutdown_connections 2024-04-25T01:58:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.787+0000 7fb535200640 1 --2- 172.21.15.102:0/3349306108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb530101f30 0x7fb53010e3f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.787+0000 7fb535200640 1 --2- 172.21.15.102:0/3349306108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb530101570 0x7fb5301019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.787+0000 7fb535200640 1 --2- 172.21.15.102:0/3349306108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb530107630 0x7fb530107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.787+0000 7fb535200640 1 -- 172.21.15.102:0/3349306108 >> 172.21.15.102:0/3349306108 conn(0x7fb5300fd080 msgr2=0x7fb5300ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.787+0000 7fb535200640 1 -- 172.21.15.102:0/3349306108 shutdown_connections 2024-04-25T01:58:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.788+0000 7fb535200640 1 -- 172.21.15.102:0/3349306108 wait complete. 2024-04-25T01:58:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.789+0000 7fb535200640 1 Processor -- start 2024-04-25T01:58:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.789+0000 7fb535200640 1 -- start start 2024-04-25T01:58:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.790+0000 7fb535200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb530101570 0x7fb530197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.790+0000 7fb535200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb530101f30 0x7fb530198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.790+0000 7fb535200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb530107630 0x7fb53019c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.791+0000 7fb535200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb530111940 con 0x7fb530101f30 2024-04-25T01:58:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.791+0000 7fb52f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb530101570 0x7fb530197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.791+0000 7fb52f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb530101570 0x7fb530197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44916/0 (socket says 172.21.15.102:44916) 2024-04-25T01:58:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.791+0000 7fb52f400640 1 -- 172.21.15.102:0/3336060478 learned_addr learned my addr 172.21.15.102:0/3336060478 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.791+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb530107630 0x7fb53019c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.791+0000 7fb52ea00640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb530101f30 0x7fb530198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.791+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301117c0 con 0x7fb530107630 2024-04-25T01:58:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.791+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb530111ac0 con 0x7fb530101570 2024-04-25T01:58:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.792+0000 7fb52fe00640 1 -- 172.21.15.102:0/3336060478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb530101570 msgr2=0x7fb530197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.792+0000 7fb52fe00640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb530101570 0x7fb530197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.792+0000 7fb52fe00640 1 -- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb530101f30 msgr2=0x7fb530198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.792+0000 7fb52fe00640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb530101f30 0x7fb530198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.792+0000 7fb52fe00640 1 -- 172.21.15.102:0/3336060478 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53019ce10 con 0x7fb530107630 2024-04-25T01:58:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.792+0000 7fb52f400640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb530101570 0x7fb530197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.793+0000 7fb52fe00640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb530107630 0x7fb53019c6e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb51c02a960 tx=0x7fb51c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.794+0000 7fb527e00640 1 -- 172.21.15.102:0/3336060478 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c03f070 con 0x7fb530107630 2024-04-25T01:58:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.794+0000 7fb527e00640 1 -- 172.21.15.102:0/3336060478 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb51c031c90 con 0x7fb530107630 2024-04-25T01:58:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.794+0000 7fb527e00640 1 -- 172.21.15.102:0/3336060478 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c035030 con 0x7fb530107630 2024-04-25T01:58:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.794+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53019d0a0 con 0x7fb530107630 2024-04-25T01:58:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.794+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53019d510 con 0x7fb530107630 2024-04-25T01:58:31.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.797+0000 7fb527e00640 1 -- 172.21.15.102:0/3336060478 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb51c042070 con 0x7fb530107630 2024-04-25T01:58:31.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.799+0000 7fb527e00640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5140a2d70 0x7fb5140a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.800+0000 7fb527e00640 1 -- 172.21.15.102:0/3336060478 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb51c0efa20 con 0x7fb530107630 2024-04-25T01:58:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.800+0000 7fb52f400640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5140a2d70 0x7fb5140a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.803+0000 7fb52f400640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5140a2d70 0x7fb5140a5230 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb528009fd0 tx=0x7fb528009720 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.803+0000 7fb5395ff740 1 Processor -- start 2024-04-25T01:58:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.807+0000 7fb5395ff740 1 -- start start 2024-04-25T01:58:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.807+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f1692050 0x55a2f1692430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.807+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168dac0 0x55a2f168df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.807+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 0x55a2f1699fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.809+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 0x55a2f1699fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.809+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 0x55a2f1699fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44920/0 (socket says 172.21.15.102:44920) 2024-04-25T01:58:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.809+0000 7fb52fe00640 1 -- 172.21.15.102:0/2707994753 learned_addr learned my addr 172.21.15.102:0/2707994753 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.810+0000 7fb52f400640 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f1692050 0x55a2f1692430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.810+0000 7fb52ea00640 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168dac0 0x55a2f168df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.810+0000 7fb5395ff740 1 -- 172.21.15.102:0/2707994753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a2f15e7ab0 con 0x55a2f1692050 2024-04-25T01:58:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.810+0000 7fb5395ff740 1 -- 172.21.15.102:0/2707994753 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a2f1590640 con 0x55a2f168dac0 2024-04-25T01:58:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.810+0000 7fb5395ff740 1 -- 172.21.15.102:0/2707994753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a2f15e3a70 con 0x55a2f168e480 2024-04-25T01:58:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.811+0000 7fb52ea00640 1 -- 172.21.15.102:0/2707994753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 msgr2=0x55a2f1699fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.811+0000 7fb52ea00640 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 0x55a2f1699fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.811+0000 7fb52ea00640 1 -- 172.21.15.102:0/2707994753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f1692050 msgr2=0x55a2f1692430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.811+0000 7fb52ea00640 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f1692050 0x55a2f1692430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.811+0000 7fb52ea00640 1 -- 172.21.15.102:0/2707994753 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2f12fc210 con 0x55a2f168dac0 2024-04-25T01:58:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.811+0000 7fb52f400640 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f1692050 0x55a2f1692430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.813+0000 7fb52ea00640 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168dac0 0x55a2f168df40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb520005d00 tx=0x7fb5200107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a2f279bf2e80cab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.813+0000 7fb524c00640 1 -- 172.21.15.102:0/2707994753 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52001c480 con 0x55a2f168dac0 2024-04-25T01:58:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.814+0000 7fb524c00640 1 -- 172.21.15.102:0/2707994753 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb520011920 con 0x55a2f168dac0 2024-04-25T01:58:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.815+0000 7fb524c00640 1 -- 172.21.15.102:0/2707994753 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52001c480 con 0x55a2f168dac0 2024-04-25T01:58:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.815+0000 7fb5395ff740 1 -- 172.21.15.102:0/2707994753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168dac0 msgr2=0x55a2f168df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.815+0000 7fb5395ff740 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168dac0 0x55a2f168df40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb520005d00 tx=0x7fb5200107a0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.816+0000 7fb5395ff740 1 -- 172.21.15.102:0/2707994753 shutdown_connections 2024-04-25T01:58:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.816+0000 7fb5395ff740 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 0x55a2f1699fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.817+0000 7fb5395ff740 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168dac0 0x55a2f168df40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.817+0000 7fb5395ff740 1 --2- 172.21.15.102:0/2707994753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f1692050 0x55a2f1692430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.817+0000 7fb5395ff740 1 -- 172.21.15.102:0/2707994753 >> 172.21.15.102:0/2707994753 conn(0x55a2f167ff70 msgr2=0x55a2f1680350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.818+0000 7fb5395ff740 1 -- 172.21.15.102:0/2707994753 shutdown_connections 2024-04-25T01:58:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.818+0000 7fb5395ff740 1 -- 172.21.15.102:0/2707994753 wait complete. 2024-04-25T01:58:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.820+0000 7fb5395ff740 1 -- start start 2024-04-25T01:58:31.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.820+0000 7fb5395ff740 1 Processor -- start 2024-04-25T01:58:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.822+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f1697330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.826 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.822+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 0x55a2f1697870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.822+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f1692050 0x55a2f1697db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f1692050 0x55a2f1697db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f1692050 0x55a2f1697db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45822/0 (socket says 172.21.15.102:45822) 2024-04-25T01:58:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52fe00640 1 -- 172.21.15.102:0/3384881884 learned_addr learned my addr 172.21.15.102:0/3384881884 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb5395ff740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a2f169a600 con 0x55a2f168dac0 2024-04-25T01:58:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52f400640 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f1697330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a2f15a3990 con 0x55a2f1692050 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a2f169a780 con 0x55a2f168e480 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52ea00640 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 0x55a2f1697870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52fe00640 1 -- 172.21.15.102:0/3384881884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 msgr2=0x55a2f1697870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52fe00640 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 0x55a2f1697870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52fe00640 1 -- 172.21.15.102:0/3384881884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 msgr2=0x55a2f1697330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52fe00640 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f1697330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb52fe00640 1 -- 172.21.15.102:0/3384881884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2f13c8620 con 0x55a2f1692050 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.823+0000 7fb5395ff740 10 client.0 _ll_register_callbacks cb 0x55a2f15a5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.824+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb520011920 con 0x55a2f1692050 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.824+0000 7fb52fe00640 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f1692050 0x55a2f1697db0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0b8420 tx=0x7fb51c0b8300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aefa01528310d10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.824+0000 7fb519600640 10 client.26174 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.824+0000 7fb519600640 1 -- 172.21.15.102:0/3384881884 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c049040 con 0x55a2f1692050 2024-04-25T01:58:31.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.825+0000 7fb519600640 1 -- 172.21.15.102:0/3384881884 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb51c03b460 con 0x55a2f1692050 2024-04-25T01:58:31.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.825+0000 7fb519600640 1 -- 172.21.15.102:0/3384881884 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb51c0bbfc0 con 0x55a2f1692050 2024-04-25T01:58:31.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.825+0000 7fb5395ff740 10 client.26174 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.825+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55a2f16a6140 con 0x55a2f1692050 2024-04-25T01:58:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.825+0000 7fb519600640 1 -- 172.21.15.102:0/3384881884 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fb51c0c46f0 con 0x55a2f1692050 2024-04-25T01:58:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.826+0000 7fb5395ff740 10 client.26174 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.826+0000 7fb5395ff740 10 client.26174 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.826+0000 7fb5395ff740 20 client.26174 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.826+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a2f16a7740 0x55a2f16a9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.826+0000 7fb5395ff740 4 client.26174 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.826+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2f15f4ff0 con 0x55a2f16a7740 2024-04-25T01:58:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.826+0000 7fb52f400640 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a2f16a7740 0x55a2f16a9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.827+0000 7fb519600640 10 client.26174 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.827+0000 7fb52f400640 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a2f16a7740 0x55a2f16a9c00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc3d8fc4ea6132d1 server_cookie=f1d2ccf7c5d5f1e0 in_seq=0 out_seq=0 2024-04-25T01:58:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.828+0000 7fb519600640 1 -- 172.21.15.102:0/3384881884 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2f15f4ff0 con 0x55a2f16a7740 2024-04-25T01:58:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.828+0000 7fb519600640 10 client.26174 handle_command_reply: tid=0 2024-04-25T01:58:31.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.828+0000 7fb5395ff740 1 client.26174 shutdown 2024-04-25T01:58:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.829+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f1692050 msgr2=0x55a2f1697db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.829+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f1692050 0x55a2f1697db0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0b8420 tx=0x7fb51c0b8300 comp rx=0 tx=0).stop 2024-04-25T01:58:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.829+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 shutdown_connections 2024-04-25T01:58:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.829+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a2f16a7740 0x55a2f16a9c00 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.829+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f1692050 0x55a2f1697db0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.829+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f168e480 0x55a2f1697870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.829+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3384881884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f1697330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.829+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 >> 172.21.15.102:0/3384881884 conn(0x55a2f167ff70 msgr2=0x55a2f168d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.829+0000 7fb519600640 0 client.26174 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.830+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 shutdown_connections 2024-04-25T01:58:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.830+0000 7fb5395ff740 1 -- 172.21.15.102:0/3384881884 wait complete. 2024-04-25T01:58:31.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.830+0000 7fb5395ff740 20 client.26174 trim_cache size 0 max 16384 2024-04-25T01:58:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.861+0000 7fb5395ff740 1 Processor -- start 2024-04-25T01:58:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.861+0000 7fb5395ff740 1 -- start start 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.861+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f1696270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.861+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 0x55a2f1691750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.861+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f1697bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.861+0000 7fb5395ff740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301ad8b0 con 0x55a2f168dac0 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.861+0000 7fb5395ff740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301ab150 con 0x55a2f168e480 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.861+0000 7fb5395ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301a8fa0 con 0x55a2f1692050 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.862+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f1697bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.862+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f1697bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44926/0 (socket says 172.21.15.102:44926) 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.862+0000 7fb52fe00640 1 -- 172.21.15.102:0/87489440 learned_addr learned my addr 172.21.15.102:0/87489440 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.862+0000 7fb52ea00640 1 --2- 172.21.15.102:0/87489440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 0x55a2f1691750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.862+0000 7fb52f400640 1 --2- 172.21.15.102:0/87489440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f1696270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.863+0000 7fb52fe00640 1 -- 172.21.15.102:0/87489440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 msgr2=0x55a2f1691750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.863+0000 7fb52fe00640 1 --2- 172.21.15.102:0/87489440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 0x55a2f1691750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.863+0000 7fb52fe00640 1 -- 172.21.15.102:0/87489440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 msgr2=0x55a2f1696270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.863+0000 7fb52fe00640 1 --2- 172.21.15.102:0/87489440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f1696270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.863+0000 7fb52fe00640 1 -- 172.21.15.102:0/87489440 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2f1683e60 con 0x55a2f1692050 2024-04-25T01:58:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.863+0000 7fb52fe00640 1 --2- 172.21.15.102:0/87489440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f1697bd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0ed1a0 tx=0x7fb51c0ef000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf4ea9a80d9115a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.863+0000 7fb51be00640 1 -- 172.21.15.102:0/87489440 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c049040 con 0x55a2f1692050 2024-04-25T01:58:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.864+0000 7fb51be00640 1 -- 172.21.15.102:0/87489440 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb520011920 con 0x55a2f1692050 2024-04-25T01:58:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.864+0000 7fb51be00640 1 -- 172.21.15.102:0/87489440 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c0dab50 con 0x55a2f1692050 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.865+0000 7fb5395ff740 1 -- 172.21.15.102:0/87489440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 msgr2=0x55a2f1697bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.865+0000 7fb5395ff740 1 --2- 172.21.15.102:0/87489440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f1697bd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0ed1a0 tx=0x7fb51c0ef000 comp rx=0 tx=0).stop 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.866+0000 7fb5395ff740 1 -- 172.21.15.102:0/87489440 shutdown_connections 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.866+0000 7fb5395ff740 1 --2- 172.21.15.102:0/87489440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f1697bd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.866+0000 7fb5395ff740 1 --2- 172.21.15.102:0/87489440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 0x55a2f1691750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.866+0000 7fb5395ff740 1 --2- 172.21.15.102:0/87489440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f1696270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.866+0000 7fb5395ff740 1 -- 172.21.15.102:0/87489440 >> 172.21.15.102:0/87489440 conn(0x55a2f167ff70 msgr2=0x55a2f168d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.866+0000 7fb5395ff740 1 -- 172.21.15.102:0/87489440 shutdown_connections 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.866+0000 7fb5395ff740 1 -- 172.21.15.102:0/87489440 wait complete. 2024-04-25T01:58:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.866+0000 7fb5395ff740 1 -- start start 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb5395ff740 1 Processor -- start 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f168eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 0x55a2f168f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb5395ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f168f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb5395ff740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301a9610 con 0x55a2f168dac0 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb5395ff740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301aaa80 con 0x55a2f168e480 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb5395ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301a9ce0 con 0x55a2f1692050 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb5395ff740 10 client.0 _ll_register_callbacks cb 0x7fb51c0b7b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb52ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 0x55a2f168f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb52fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f168f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb52f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f168eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb52ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 0x55a2f168f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45830/0 (socket says 172.21.15.102:45830) 2024-04-25T01:58:31.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.867+0000 7fb52ea00640 1 -- 172.21.15.102:0/3649466200 learned_addr learned my addr 172.21.15.102:0/3649466200 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.868+0000 7fb52fe00640 1 -- 172.21.15.102:0/3649466200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 msgr2=0x55a2f168f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.868+0000 7fb52fe00640 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 0x55a2f168f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.868+0000 7fb52fe00640 1 -- 172.21.15.102:0/3649466200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 msgr2=0x55a2f168eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.870+0000 7fb52fe00640 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f168eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.870+0000 7fb52fe00640 1 -- 172.21.15.102:0/3649466200 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2f169ec80 con 0x55a2f1692050 2024-04-25T01:58:31.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.870+0000 7fb52f400640 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f168eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.870+0000 7fb52fe00640 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f168f720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009880 tx=0x7fb51c031490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73dfe78b92370fed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.870+0000 7fb519600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.870+0000 7fb519600640 1 -- 172.21.15.102:0/3649466200 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c049070 con 0x55a2f1692050 2024-04-25T01:58:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb519600640 1 -- 172.21.15.102:0/3649466200 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb51c0dad50 con 0x55a2f1692050 2024-04-25T01:58:31.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb5395ff740 1 -- 172.21.15.102:0/3649466200 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb520011920 con 0x55a2f1692050 2024-04-25T01:58:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb519600640 1 -- 172.21.15.102:0/3649466200 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb51c0b8260 con 0x55a2f1692050 2024-04-25T01:58:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb5395ff740 10 client.26134 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb5395ff740 1 -- 172.21.15.102:0/3649466200 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a2f169ec80 con 0x55a2f1692050 2024-04-25T01:58:31.875 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb519600640 1 -- 172.21.15.102:0/3649466200 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb520011920 con 0x55a2f1692050 2024-04-25T01:58:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb519600640 1 -- 172.21.15.102:0/3649466200 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fb51c0bb530 con 0x55a2f1692050 2024-04-25T01:58:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb5395ff740 10 client.26134 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb5395ff740 10 client.26134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb5395ff740 20 client.26134 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a2f170dd40 0x55a2f1710130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb5395ff740 4 client.26134 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:58:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.871+0000 7fb5395ff740 1 -- 172.21.15.102:0/3649466200 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a2f15ef340 con 0x55a2f170dd40 2024-04-25T01:58:31.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.872+0000 7fb52f400640 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a2f170dd40 0x55a2f1710130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.874+0000 7fb52f400640 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a2f170dd40 0x55a2f1710130 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=112e61e7b38216c server_cookie=90f0b98194854426 in_seq=0 out_seq=0 2024-04-25T01:58:31.877 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.874+0000 7fb519600640 10 client.26134 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.886+0000 7fb519600640 1 -- 172.21.15.102:0/3649466200 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a2f15ef340 con 0x55a2f170dd40 2024-04-25T01:58:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.886+0000 7fb519600640 10 client.26134 handle_command_reply: tid=0 2024-04-25T01:58:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.886+0000 7fb5395ff740 1 client.26134 shutdown 2024-04-25T01:58:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 -- 172.21.15.102:0/3649466200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 msgr2=0x55a2f168f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f168f720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009880 tx=0x7fb51c031490 comp rx=0 tx=0).stop 2024-04-25T01:58:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 -- 172.21.15.102:0/3649466200 shutdown_connections 2024-04-25T01:58:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a2f170dd40 0x55a2f1710130 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2f1692050 0x55a2f168f720 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a2f168e480 0x55a2f168f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 --2- 172.21.15.102:0/3649466200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a2f168dac0 0x55a2f168eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 -- 172.21.15.102:0/3649466200 >> 172.21.15.102:0/3649466200 conn(0x55a2f167ff70 msgr2=0x55a2f167ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb519600640 0 client.26134 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 -- 172.21.15.102:0/3649466200 shutdown_connections 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 1 -- 172.21.15.102:0/3649466200 wait complete. 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.887+0000 7fb5395ff740 20 client.26134 trim_cache size 0 max 16384 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:58:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:58:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:58:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.888+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5140a2d70 msgr2=0x7fb5140a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.888+0000 7fb535200640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5140a2d70 0x7fb5140a5230 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb528009fd0 tx=0x7fb528009720 comp rx=0 tx=0).stop 2024-04-25T01:58:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.888+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb530107630 msgr2=0x7fb53019c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.888+0000 7fb535200640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb530107630 0x7fb53019c6e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb51c02a960 tx=0x7fb51c003b90 comp rx=0 tx=0).stop 2024-04-25T01:58:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.889+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 shutdown_connections 2024-04-25T01:58:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.889+0000 7fb535200640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5140a2d70 0x7fb5140a5230 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.889+0000 7fb535200640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb530107630 0x7fb53019c6e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.889+0000 7fb535200640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb530101f30 0x7fb530198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.889+0000 7fb535200640 1 --2- 172.21.15.102:0/3336060478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb530101570 0x7fb530197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.889+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 >> 172.21.15.102:0/3336060478 conn(0x7fb5300fd080 msgr2=0x7fb5300fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.890+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 shutdown_connections 2024-04-25T01:58:31.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:31.890+0000 7fb535200640 1 -- 172.21.15.102:0/3336060478 wait complete. 2024-04-25T01:58:33.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 160.0 s, 3025.9 tps, lat 32.941 ms stddev 35.528 2024-04-25T01:58:33.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:33 smithi102 ceph-mon[27796]: pgmap v101: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 11 MiB/s wr, 877 op/s 2024-04-25T01:58:33.489 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:33 smithi110 ceph-mon[32848]: pgmap v101: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 11 MiB/s wr, 877 op/s 2024-04-25T01:58:33.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:33 smithi193 ceph-mon[32981]: pgmap v101: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 11 MiB/s wr, 877 op/s 2024-04-25T01:58:35.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:35 smithi102 ceph-mon[27796]: pgmap v102: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.7 MiB/s wr, 762 op/s 2024-04-25T01:58:35.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:35 smithi110 ceph-mon[32848]: pgmap v102: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.7 MiB/s wr, 762 op/s 2024-04-25T01:58:35.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:35 smithi193 ceph-mon[32981]: pgmap v102: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.7 MiB/s wr, 762 op/s 2024-04-25T01:58:37.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:37 smithi102 ceph-mon[27796]: pgmap v103: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.3 MiB/s wr, 679 op/s 2024-04-25T01:58:37.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:37 smithi110 ceph-mon[32848]: pgmap v103: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.3 MiB/s wr, 679 op/s 2024-04-25T01:58:37.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:37 smithi193 ceph-mon[32981]: pgmap v103: 97 pgs: 97 active+clean; 2.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.3 MiB/s wr, 679 op/s 2024-04-25T01:58:38.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 165.0 s, 2846.4 tps, lat 35.127 ms stddev 41.410 2024-04-25T01:58:39.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:39 smithi102 ceph-mon[27796]: pgmap v104: 97 pgs: 97 active+clean; 2.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 982 op/s 2024-04-25T01:58:39.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:39 smithi110 ceph-mon[32848]: pgmap v104: 97 pgs: 97 active+clean; 2.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 982 op/s 2024-04-25T01:58:39.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:39 smithi193 ceph-mon[32981]: pgmap v104: 97 pgs: 97 active+clean; 2.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 982 op/s 2024-04-25T01:58:41.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:41 smithi102 ceph-mon[27796]: pgmap v105: 97 pgs: 97 active+clean; 2.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.5 MiB/s wr, 611 op/s 2024-04-25T01:58:41.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:41 smithi110 ceph-mon[32848]: pgmap v105: 97 pgs: 97 active+clean; 2.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.5 MiB/s wr, 611 op/s 2024-04-25T01:58:41.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:41 smithi193 ceph-mon[32981]: pgmap v105: 97 pgs: 97 active+clean; 2.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.5 MiB/s wr, 611 op/s 2024-04-25T01:58:43.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 170.0 s, 2815.0 tps, lat 35.599 ms stddev 45.538 2024-04-25T01:58:43.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:43 smithi102 ceph-mon[27796]: pgmap v106: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.6 MiB/s wr, 842 op/s 2024-04-25T01:58:43.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:43 smithi110 ceph-mon[32848]: pgmap v106: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.6 MiB/s wr, 842 op/s 2024-04-25T01:58:43.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:43 smithi193 ceph-mon[32981]: pgmap v106: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.6 MiB/s wr, 842 op/s 2024-04-25T01:58:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.083+0000 7fc51ea00640 1 -- 172.21.15.102:0/2308654246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc510095f20 msgr2=0x7fc5100963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.083+0000 7fc51ea00640 1 --2- 172.21.15.102:0/2308654246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc510095f20 0x7fc5100963a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc504009960 tx=0x7fc50402a440 comp rx=0 tx=0).stop 2024-04-25T01:58:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.084+0000 7fc51ea00640 1 -- 172.21.15.102:0/2308654246 shutdown_connections 2024-04-25T01:58:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.084+0000 7fc51ea00640 1 --2- 172.21.15.102:0/2308654246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5100968e0 0x7fc5100a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.084+0000 7fc51ea00640 1 --2- 172.21.15.102:0/2308654246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc510095f20 0x7fc5100963a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.084+0000 7fc51ea00640 1 --2- 172.21.15.102:0/2308654246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc51009ace0 0x7fc51009b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.084+0000 7fc51ea00640 1 -- 172.21.15.102:0/2308654246 >> 172.21.15.102:0/2308654246 conn(0x7fc51000cb30 msgr2=0x7fc51000cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.084+0000 7fc51ea00640 1 -- 172.21.15.102:0/2308654246 shutdown_connections 2024-04-25T01:58:45.088 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.084+0000 7fc51ea00640 1 -- 172.21.15.102:0/2308654246 wait complete. 2024-04-25T01:58:45.088 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.086+0000 7fc51ea00640 1 Processor -- start 2024-04-25T01:58:45.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.086+0000 7fc51ea00640 1 -- start start 2024-04-25T01:58:45.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.087+0000 7fc51ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5100968e0 0x7fc5101334d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.087+0000 7fc51ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc51009ace0 0x7fc5100a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.087+0000 7fc51ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5100a12f0 0x7fc5100a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.087+0000 7fc51ea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc5100a6190 con 0x7fc51009ace0 2024-04-25T01:58:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.087+0000 7fc51ea00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc5100a6010 con 0x7fc5100a12f0 2024-04-25T01:58:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.087+0000 7fc517e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5100968e0 0x7fc5101334d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc517e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5100968e0 0x7fc5101334d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58462/0 (socket says 172.21.15.102:58462) 2024-04-25T01:58:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc51ea00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc5100a6310 con 0x7fc5100968e0 2024-04-25T01:58:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc517e00640 1 -- 172.21.15.102:0/207284251 learned_addr learned my addr 172.21.15.102:0/207284251 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc51ca00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5100a12f0 0x7fc5100a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc517400640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc51009ace0 0x7fc5100a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc517e00640 1 -- 172.21.15.102:0/207284251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5100a12f0 msgr2=0x7fc5100a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc517e00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5100a12f0 0x7fc5100a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc517e00640 1 -- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc51009ace0 msgr2=0x7fc5100a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc517e00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc51009ace0 0x7fc5100a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.088+0000 7fc517e00640 1 -- 172.21.15.102:0/207284251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51012ef10 con 0x7fc5100968e0 2024-04-25T01:58:45.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.089+0000 7fc517400640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc51009ace0 0x7fc5100a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:45.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.089+0000 7fc517e00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5100968e0 0x7fc5101334d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00e9f0 tx=0x7fc50c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.089+0000 7fc514c00640 1 -- 172.21.15.102:0/207284251 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c00cd80 con 0x7fc5100968e0 2024-04-25T01:58:45.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.089+0000 7fc514c00640 1 -- 172.21.15.102:0/207284251 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc50c005500 con 0x7fc5100968e0 2024-04-25T01:58:45.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.089+0000 7fc514c00640 1 -- 172.21.15.102:0/207284251 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c010690 con 0x7fc5100968e0 2024-04-25T01:58:45.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.091+0000 7fc51ea00640 1 -- 172.21.15.102:0/207284251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc51012f200 con 0x7fc5100968e0 2024-04-25T01:58:45.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.091+0000 7fc51ea00640 1 -- 172.21.15.102:0/207284251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc51012f6d0 con 0x7fc5100968e0 2024-04-25T01:58:45.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:58:45.098 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:58:45.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.095+0000 7fc5217ed740 1 Processor -- start 2024-04-25T01:58:45.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.095+0000 7fc5217ed740 1 -- start start 2024-04-25T01:58:45.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da2463f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2463f8d0 0x55da24649c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 0x55da2464c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc5217ed740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55da245a2ab0 con 0x55da2464a190 2024-04-25T01:58:45.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc5217ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55da2454b640 con 0x55da2463f8d0 2024-04-25T01:58:45.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc5217ed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55da2459ea70 con 0x55da2463efb0 2024-04-25T01:58:45.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc514c00640 1 -- 172.21.15.102:0/207284251 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc50c0046c0 con 0x7fc5100968e0 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc517e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2463f8d0 0x55da24649c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc51ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da2463f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc51ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da2463f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58472/0 (socket says 172.21.15.102:58472) 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.096+0000 7fc51ca00640 1 -- 172.21.15.102:0/2205780615 learned_addr learned my addr 172.21.15.102:0/2205780615 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.097+0000 7fc517400640 1 --2- 172.21.15.102:0/2205780615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 0x55da2464c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.097+0000 7fc517e00640 1 -- 172.21.15.102:0/2205780615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 msgr2=0x55da2463f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.097+0000 7fc517e00640 1 --2- 172.21.15.102:0/2205780615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da2463f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.097+0000 7fc517e00640 1 -- 172.21.15.102:0/2205780615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 msgr2=0x55da2464c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.097+0000 7fc517e00640 1 --2- 172.21.15.102:0/2205780615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 0x55da2464c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.097+0000 7fc517e00640 1 -- 172.21.15.102:0/2205780615 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da242b7210 con 0x55da2463f8d0 2024-04-25T01:58:45.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.098+0000 7fc517e00640 1 --2- 172.21.15.102:0/2205780615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2463f8d0 0x55da24649c50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0ccf80 tx=0x7fc50c0d5f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1876a042c64cf71d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.098+0000 7fc514c00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc5000a2d80 0x7fc5000a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.098+0000 7fc509200640 1 -- 172.21.15.102:0/2205780615 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c0de070 con 0x55da2463f8d0 2024-04-25T01:58:45.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.098+0000 7fc509200640 1 -- 172.21.15.102:0/2205780615 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc50c0d7de0 con 0x55da2463f8d0 2024-04-25T01:58:45.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.098+0000 7fc509200640 1 -- 172.21.15.102:0/2205780615 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c0d2e60 con 0x55da2463f8d0 2024-04-25T01:58:45.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.099+0000 7fc514c00640 1 -- 172.21.15.102:0/207284251 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc50c0cf800 con 0x7fc5100968e0 2024-04-25T01:58:45.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.099+0000 7fc51ca00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc5000a2d80 0x7fc5000a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.099+0000 7fc5217ed740 1 -- 172.21.15.102:0/2205780615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2463f8d0 msgr2=0x55da24649c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.099+0000 7fc5217ed740 1 --2- 172.21.15.102:0/2205780615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2463f8d0 0x55da24649c50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0ccf80 tx=0x7fc50c0d5f10 comp rx=0 tx=0).stop 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.099+0000 7fc5217ed740 1 -- 172.21.15.102:0/2205780615 shutdown_connections 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.099+0000 7fc5217ed740 1 --2- 172.21.15.102:0/2205780615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 0x55da2464c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.099+0000 7fc5217ed740 1 --2- 172.21.15.102:0/2205780615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2463f8d0 0x55da24649c50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.099+0000 7fc5217ed740 1 --2- 172.21.15.102:0/2205780615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da2463f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.099+0000 7fc5217ed740 1 -- 172.21.15.102:0/2205780615 >> 172.21.15.102:0/2205780615 conn(0x55da2463b4c0 msgr2=0x55da2463b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.100+0000 7fc5217ed740 1 -- 172.21.15.102:0/2205780615 shutdown_connections 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.100+0000 7fc5217ed740 1 -- 172.21.15.102:0/2205780615 wait complete. 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.100+0000 7fc5217ed740 1 -- start start 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.101+0000 7fc51ca00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc5000a2d80 0x7fc5000a5240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc5180648b0 tx=0x7fc518076040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.101+0000 7fc5217ed740 1 Processor -- start 2024-04-25T01:58:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.101+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 0x55da24647200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.101+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2464a190 0x55da24647740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.101+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da246561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.102+0000 7fc5217ed740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55da2464cdc0 con 0x55da2463efb0 2024-04-25T01:58:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.102+0000 7fc5217ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55da2455e990 con 0x55da24647c80 2024-04-25T01:58:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.102+0000 7fc5217ed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55da2464cf40 con 0x55da2464a190 2024-04-25T01:58:45.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.102+0000 7fc5217ed740 10 client.0 _ll_register_callbacks cb 0x55da24560950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.102+0000 7fc517e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 0x55da24647200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.102+0000 7fc517400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2464a190 0x55da24647740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.102+0000 7fc517400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2464a190 0x55da24647740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58480/0 (socket says 172.21.15.102:58480) 2024-04-25T01:58:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.102+0000 7fc517400640 1 -- 172.21.15.102:0/1287109522 learned_addr learned my addr 172.21.15.102:0/1287109522 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.102+0000 7fc51ca00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da246561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.103+0000 7fc517400640 1 -- 172.21.15.102:0/1287109522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 msgr2=0x55da246561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.103+0000 7fc517400640 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da246561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.103+0000 7fc517400640 1 -- 172.21.15.102:0/1287109522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 msgr2=0x55da24647200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.103+0000 7fc517400640 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 0x55da24647200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.103+0000 7fc517400640 1 -- 172.21.15.102:0/1287109522 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da24383620 con 0x55da2464a190 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.104+0000 7fc517400640 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2464a190 0x55da24647740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc51000df30 tx=0x7fc504005490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c03e3adeea20357f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.104+0000 7fc4fd600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.104+0000 7fc4fd600640 1 -- 172.21.15.102:0/1287109522 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504040070 con 0x55da2464a190 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.104+0000 7fc4fd600640 1 -- 172.21.15.102:0/1287109522 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc504003a00 con 0x55da2464a190 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.105+0000 7fc5217ed740 1 -- 172.21.15.102:0/1287109522 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc50c0d7de0 con 0x55da2464a190 2024-04-25T01:58:45.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.105+0000 7fc4fd600640 1 -- 172.21.15.102:0/1287109522 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc50404a050 con 0x55da2464a190 2024-04-25T01:58:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.106+0000 7fc5217ed740 10 client.26146 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.106+0000 7fc5217ed740 1 -- 172.21.15.102:0/1287109522 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da24659180 con 0x55da2464a190 2024-04-25T01:58:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.106+0000 7fc4fd600640 1 -- 172.21.15.102:0/1287109522 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc50c0d7de0 con 0x55da2464a190 2024-04-25T01:58:45.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.107+0000 7fc4fd600640 1 -- 172.21.15.102:0/1287109522 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fc504045440 con 0x55da2464a190 2024-04-25T01:58:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.107+0000 7fc5217ed740 10 client.26146 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.107+0000 7fc5217ed740 10 client.26146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.107+0000 7fc5217ed740 20 client.26146 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.107+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55da2465a750 0x55da2465cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.108+0000 7fc5217ed740 4 client.26146 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.108+0000 7fc5217ed740 1 -- 172.21.15.102:0/1287109522 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da245afff0 con 0x55da2465a750 2024-04-25T01:58:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.108+0000 7fc517e00640 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55da2465a750 0x55da2465cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.108+0000 7fc517e00640 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55da2465a750 0x55da2465cbf0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a16f226288dca2c server_cookie=9289770733a75ee0 in_seq=0 out_seq=0 2024-04-25T01:58:45.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.108+0000 7fc4fd600640 10 client.26146 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.109+0000 7fc4fd600640 1 -- 172.21.15.102:0/1287109522 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da245afff0 con 0x55da2465a750 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.109+0000 7fc4fd600640 10 client.26146 handle_command_reply: tid=0 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.109+0000 7fc5217ed740 1 client.26146 shutdown 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.109+0000 7fc5217ed740 1 -- 172.21.15.102:0/1287109522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2464a190 msgr2=0x55da24647740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.109+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2464a190 0x55da24647740 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc51000df30 tx=0x7fc504005490 comp rx=0 tx=0).stop 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc5217ed740 1 -- 172.21.15.102:0/1287109522 shutdown_connections 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55da2465a750 0x55da2465cbf0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da246561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2464a190 0x55da24647740 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1287109522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 0x55da24647200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc5217ed740 1 -- 172.21.15.102:0/1287109522 >> 172.21.15.102:0/1287109522 conn(0x55da2463b4c0 msgr2=0x55da24641240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc4fd600640 0 client.26146 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc5217ed740 1 -- 172.21.15.102:0/1287109522 shutdown_connections 2024-04-25T01:58:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc5217ed740 1 -- 172.21.15.102:0/1287109522 wait complete. 2024-04-25T01:58:45.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.110+0000 7fc5217ed740 20 client.26146 trim_cache size 0 max 16384 2024-04-25T01:58:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.145+0000 7fc5217ed740 1 Processor -- start 2024-04-25T01:58:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.146+0000 7fc5217ed740 1 -- start start 2024-04-25T01:58:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.146+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 0x55da24649140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.146+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da24647c80 0x55da24662500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.146+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2464a190 0x55da24662a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.146+0000 7fc5217ed740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc510153880 con 0x55da2463efb0 2024-04-25T01:58:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.146+0000 7fc5217ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc510151120 con 0x55da2464a190 2024-04-25T01:58:45.149 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.146+0000 7fc5217ed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc51014ef70 con 0x55da24647c80 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da24647c80 0x55da24662500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 0x55da24649140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da24647c80 0x55da24662500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58484/0 (socket says 172.21.15.102:58484) 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517400640 1 -- 172.21.15.102:0/1057916379 learned_addr learned my addr 172.21.15.102:0/1057916379 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc51ca00640 1 --2- 172.21.15.102:0/1057916379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2464a190 0x55da24662a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517400640 1 -- 172.21.15.102:0/1057916379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2464a190 msgr2=0x55da24662a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517400640 1 --2- 172.21.15.102:0/1057916379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2464a190 0x55da24662a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517400640 1 -- 172.21.15.102:0/1057916379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 msgr2=0x55da24649140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517400640 1 --2- 172.21.15.102:0/1057916379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 0x55da24649140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517400640 1 -- 172.21.15.102:0/1057916379 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da24649fb0 con 0x55da24647c80 2024-04-25T01:58:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517e00640 1 --2- 172.21.15.102:0/1057916379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 0x55da24649140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.147+0000 7fc517400640 1 --2- 172.21.15.102:0/1057916379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da24647c80 0x55da24662500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc504000c00 tx=0x7fc50402a440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7991755c131bc589 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.148+0000 7fc4ffe00640 1 -- 172.21.15.102:0/1057916379 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504040040 con 0x55da24647c80 2024-04-25T01:58:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.148+0000 7fc4ffe00640 1 -- 172.21.15.102:0/1057916379 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc504053030 con 0x55da24647c80 2024-04-25T01:58:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.148+0000 7fc4ffe00640 1 -- 172.21.15.102:0/1057916379 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc504035040 con 0x55da24647c80 2024-04-25T01:58:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.150+0000 7fc5217ed740 1 -- 172.21.15.102:0/1057916379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da24647c80 msgr2=0x55da24662500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.150+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1057916379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da24647c80 0x55da24662500 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc504000c00 tx=0x7fc50402a440 comp rx=0 tx=0).stop 2024-04-25T01:58:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.150+0000 7fc5217ed740 1 -- 172.21.15.102:0/1057916379 shutdown_connections 2024-04-25T01:58:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.150+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1057916379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da2464a190 0x55da24662a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.150+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1057916379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da24647c80 0x55da24662500 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.150+0000 7fc5217ed740 1 --2- 172.21.15.102:0/1057916379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2463efb0 0x55da24649140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.150+0000 7fc5217ed740 1 -- 172.21.15.102:0/1057916379 >> 172.21.15.102:0/1057916379 conn(0x55da2463b4c0 msgr2=0x55da24648a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.152+0000 7fc5217ed740 1 -- 172.21.15.102:0/1057916379 shutdown_connections 2024-04-25T01:58:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.152+0000 7fc5217ed740 1 -- 172.21.15.102:0/1057916379 wait complete. 2024-04-25T01:58:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.153+0000 7fc5217ed740 1 -- start start 2024-04-25T01:58:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.153+0000 7fc5217ed740 1 Processor -- start 2024-04-25T01:58:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.154+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da24652360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.154+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da2464e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.154+0000 7fc5217ed740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 0x55da2464ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.157 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.154+0000 7fc5217ed740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc51014f5e0 con 0x55da2464a190 2024-04-25T01:58:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.155+0000 7fc5217ed740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc510150a50 con 0x55da24647c80 2024-04-25T01:58:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.155+0000 7fc5217ed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc51014fcb0 con 0x55da2463efb0 2024-04-25T01:58:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.155+0000 7fc5217ed740 10 client.0 _ll_register_callbacks cb 0x7fc504009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.155+0000 7fc517400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da2464e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.155+0000 7fc517400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da2464e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42480/0 (socket says 172.21.15.102:42480) 2024-04-25T01:58:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.155+0000 7fc517400640 1 -- 172.21.15.102:0/3061224715 learned_addr learned my addr 172.21.15.102:0/3061224715 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.155+0000 7fc51ca00640 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 0x55da2464ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.155+0000 7fc517e00640 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da24652360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.156+0000 7fc517e00640 1 -- 172.21.15.102:0/3061224715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 msgr2=0x55da2464e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.156+0000 7fc517e00640 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da2464e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.156+0000 7fc517e00640 1 -- 172.21.15.102:0/3061224715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 msgr2=0x55da2464ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.157+0000 7fc517e00640 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 0x55da2464ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.157+0000 7fc517e00640 1 -- 172.21.15.102:0/3061224715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da2464f210 con 0x55da2463efb0 2024-04-25T01:58:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.157+0000 7fc517400640 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da2464e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:45.160 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.157+0000 7fc51ca00640 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 0x55da2464ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.158+0000 7fc517e00640 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da24652360 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0a4930 tx=0x7fc50c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c159f1796a940ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.158+0000 7fc4fd600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.159+0000 7fc4fd600640 1 -- 172.21.15.102:0/3061224715 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50c0de040 con 0x55da2463efb0 2024-04-25T01:58:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.159+0000 7fc4fd600640 1 -- 172.21.15.102:0/3061224715 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fc50c09a050 con 0x55da2463efb0 2024-04-25T01:58:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.159+0000 7fc5217ed740 1 -- 172.21.15.102:0/3061224715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc504053030 con 0x55da2463efb0 2024-04-25T01:58:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.160+0000 7fc4fd600640 1 -- 172.21.15.102:0/3061224715 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc50c0d2e60 con 0x55da2463efb0 2024-04-25T01:58:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.160+0000 7fc5217ed740 10 client.26152 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.160+0000 7fc5217ed740 1 -- 172.21.15.102:0/3061224715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da246c1bf0 con 0x55da2463efb0 2024-04-25T01:58:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.160+0000 7fc4fd600640 1 -- 172.21.15.102:0/3061224715 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc504053030 con 0x55da2463efb0 2024-04-25T01:58:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.160+0000 7fc4fd600640 1 -- 172.21.15.102:0/3061224715 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fc50c0cd200 con 0x55da2463efb0 2024-04-25T01:58:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.161+0000 7fc5217ed740 10 client.26152 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.161+0000 7fc5217ed740 10 client.26152 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.161+0000 7fc5217ed740 20 client.26152 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.161+0000 7fc5217ed740 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55da246c1f30 0x55da246c4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.161+0000 7fc5217ed740 4 client.26152 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:58:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.161+0000 7fc5217ed740 1 -- 172.21.15.102:0/3061224715 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55da245aa340 con 0x55da246c1f30 2024-04-25T01:58:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.161+0000 7fc517400640 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55da246c1f30 0x55da246c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.162+0000 7fc517400640 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55da246c1f30 0x55da246c4320 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8288d855c262de server_cookie=49bbd595ee98d197 in_seq=0 out_seq=0 2024-04-25T01:58:45.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.162+0000 7fc4fd600640 10 client.26152 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.176+0000 7fc4fd600640 1 -- 172.21.15.102:0/3061224715 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55da245aa340 con 0x55da246c1f30 2024-04-25T01:58:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.176+0000 7fc4fd600640 10 client.26152 handle_command_reply: tid=0 2024-04-25T01:58:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.176+0000 7fc5217ed740 1 client.26152 shutdown 2024-04-25T01:58:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.177+0000 7fc5217ed740 1 -- 172.21.15.102:0/3061224715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 msgr2=0x55da24652360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.177+0000 7fc5217ed740 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da24652360 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0a4930 tx=0x7fc50c009310 comp rx=0 tx=0).stop 2024-04-25T01:58:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.177+0000 7fc5217ed740 1 -- 172.21.15.102:0/3061224715 shutdown_connections 2024-04-25T01:58:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.177+0000 7fc5217ed740 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55da246c1f30 0x55da246c4320 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.177+0000 7fc5217ed740 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55da2464a190 0x55da2464ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.177+0000 7fc5217ed740 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55da24647c80 0x55da2464e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.177+0000 7fc5217ed740 1 --2- 172.21.15.102:0/3061224715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da2463efb0 0x55da24652360 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.177+0000 7fc5217ed740 1 -- 172.21.15.102:0/3061224715 >> 172.21.15.102:0/3061224715 conn(0x55da2463b4c0 msgr2=0x55da246492c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.177+0000 7fc4fd600640 0 client.26152 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:45.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.178+0000 7fc5217ed740 1 -- 172.21.15.102:0/3061224715 shutdown_connections 2024-04-25T01:58:45.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.178+0000 7fc5217ed740 1 -- 172.21.15.102:0/3061224715 wait complete. 2024-04-25T01:58:45.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.178+0000 7fc5217ed740 20 client.26152 trim_cache size 0 max 16384 2024-04-25T01:58:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:58:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:58:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:58:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:58:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.179+0000 7fc51ea00640 1 -- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc5000a2d80 msgr2=0x7fc5000a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.179+0000 7fc51ea00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc5000a2d80 0x7fc5000a5240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc5180648b0 tx=0x7fc518076040 comp rx=0 tx=0).stop 2024-04-25T01:58:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 -- 172.21.15.102:0/207284251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5100968e0 msgr2=0x7fc5101334d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5100968e0 0x7fc5101334d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00e9f0 tx=0x7fc50c00edf0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 -- 172.21.15.102:0/207284251 shutdown_connections 2024-04-25T01:58:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc5000a2d80 0x7fc5000a5240 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5100a12f0 0x7fc5100a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc51009ace0 0x7fc5100a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 --2- 172.21.15.102:0/207284251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5100968e0 0x7fc5101334d0 secure :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00e9f0 tx=0x7fc50c00edf0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 -- 172.21.15.102:0/207284251 >> 172.21.15.102:0/207284251 conn(0x7fc51000cb30 msgr2=0x7fc5100071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 -- 172.21.15.102:0/207284251 shutdown_connections 2024-04-25T01:58:45.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:45.180+0000 7fc51ea00640 1 -- 172.21.15.102:0/207284251 wait complete. 2024-04-25T01:58:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.358+0000 7fdc99800640 1 -- 172.21.15.102:0/2166008055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc941041b0 msgr2=0x7fdc941106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.358+0000 7fdc99800640 1 --2- 172.21.15.102:0/2166008055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc941041b0 0x7fdc941106c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009860 tx=0x7fdc8c02a340 comp rx=0 tx=0).stop 2024-04-25T01:58:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.359+0000 7fdc99800640 1 -- 172.21.15.102:0/2166008055 shutdown_connections 2024-04-25T01:58:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.359+0000 7fdc99800640 1 --2- 172.21.15.102:0/2166008055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc941041b0 0x7fdc941106c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.359+0000 7fdc99800640 1 --2- 172.21.15.102:0/2166008055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc941037f0 0x7fdc94103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.359+0000 7fdc99800640 1 --2- 172.21.15.102:0/2166008055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc94109800 0x7fdc94109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.359+0000 7fdc99800640 1 -- 172.21.15.102:0/2166008055 >> 172.21.15.102:0/2166008055 conn(0x7fdc940ff280 msgr2=0x7fdc941016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.359+0000 7fdc99800640 1 -- 172.21.15.102:0/2166008055 shutdown_connections 2024-04-25T01:58:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.359+0000 7fdc99800640 1 -- 172.21.15.102:0/2166008055 wait complete. 2024-04-25T01:58:45.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.360+0000 7fdc99800640 1 Processor -- start 2024-04-25T01:58:45.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.361+0000 7fdc99800640 1 -- start start 2024-04-25T01:58:45.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.362+0000 7fdc99800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc941037f0 0x7fdc9410f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.362+0000 7fdc99800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc941041b0 0x7fdc9410f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.362+0000 7fdc99800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc94109800 0x7fdc941a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.363+0000 7fdc98e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc94109800 0x7fdc941a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.363+0000 7fdc98e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc94109800 0x7fdc941a2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58520/0 (socket says 172.21.15.102:58520) 2024-04-25T01:58:45.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.363+0000 7fdc93400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc941041b0 0x7fdc9410f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.363+0000 7fdc93400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc941041b0 0x7fdc9410f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57908/0 (socket says 172.21.15.102:57908) 2024-04-25T01:58:45.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.363+0000 7fdc93e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc941037f0 0x7fdc9410f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.363+0000 7fdc93e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc941037f0 0x7fdc9410f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42500/0 (socket says 172.21.15.102:42500) 2024-04-25T01:58:45.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.364+0000 7fdc98e00640 1 -- 172.21.15.102:0/2069430584 learned_addr learned my addr 172.21.15.102:0/2069430584 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.364+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdc940770f0 con 0x7fdc941041b0 2024-04-25T01:58:45.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.364+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdc94076f70 con 0x7fdc941037f0 2024-04-25T01:58:45.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.364+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdc94077270 con 0x7fdc94109800 2024-04-25T01:58:45.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.364+0000 7fdc98e00640 1 -- 172.21.15.102:0/2069430584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc941037f0 msgr2=0x7fdc9410f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.365+0000 7fdc98e00640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc941037f0 0x7fdc9410f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.365+0000 7fdc98e00640 1 -- 172.21.15.102:0/2069430584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc941041b0 msgr2=0x7fdc9410f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.365+0000 7fdc98e00640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc941041b0 0x7fdc9410f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.365+0000 7fdc98e00640 1 -- 172.21.15.102:0/2069430584 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc941a3430 con 0x7fdc94109800 2024-04-25T01:58:45.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.365+0000 7fdc98e00640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc94109800 0x7fdc941a2d00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009830 tx=0x7fdc8c005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.365+0000 7fdc90c00640 1 -- 172.21.15.102:0/2069430584 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c041020 con 0x7fdc94109800 2024-04-25T01:58:45.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.365+0000 7fdc90c00640 1 -- 172.21.15.102:0/2069430584 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc8c030d60 con 0x7fdc94109800 2024-04-25T01:58:45.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.365+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc941a36c0 con 0x7fdc94109800 2024-04-25T01:58:45.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.365+0000 7fdc90c00640 1 -- 172.21.15.102:0/2069430584 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c037070 con 0x7fdc94109800 2024-04-25T01:58:45.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.366+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc941aafb0 con 0x7fdc94109800 2024-04-25T01:58:45.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.372+0000 7fdc9dbbb740 1 Processor -- start 2024-04-25T01:58:45.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.373+0000 7fdc90c00640 1 -- 172.21.15.102:0/2069430584 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdc8c044020 con 0x7fdc94109800 2024-04-25T01:58:45.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.373+0000 7fdc9dbbb740 1 -- start start 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.373+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eead35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.373+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeaddc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.373+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5571eea37560 con 0x5571eeade1d0 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5571ee9df930 con 0x5571eead3b00 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5571eea33520 con 0x5571eead31e0 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eead35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eead35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58526/0 (socket says 172.21.15.102:58526) 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93400640 1 -- 172.21.15.102:0/578667863 learned_addr learned my addr 172.21.15.102:0/578667863 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93e00640 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeaddc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93400640 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93e00640 1 -- 172.21.15.102:0/578667863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 msgr2=0x5571eead35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93e00640 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eead35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc90c00640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdc780a2d80 0x7fdc780a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93e00640 1 -- 172.21.15.102:0/578667863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 msgr2=0x5571eeae05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93e00640 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93e00640 1 -- 172.21.15.102:0/578667863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571ee8157d0 con 0x5571eead3b00 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.374+0000 7fdc93400640 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.375+0000 7fdc90c00640 1 -- 172.21.15.102:0/2069430584 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fdc8c0ef3d0 con 0x7fdc94109800 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.375+0000 7fdc93e00640 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeaddc90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009fa0 tx=0x7fdc840118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b91666fdf7308a77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.375+0000 7fdc98e00640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdc780a2d80 0x7fdc780a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.375+0000 7fdc89600640 1 -- 172.21.15.102:0/578667863 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84003c30 con 0x5571eead3b00 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.375+0000 7fdc89600640 1 -- 172.21.15.102:0/578667863 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc84003dd0 con 0x5571eead3b00 2024-04-25T01:58:45.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.375+0000 7fdc89600640 1 -- 172.21.15.102:0/578667863 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8400fb20 con 0x5571eead3b00 2024-04-25T01:58:45.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/578667863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 msgr2=0x5571eeaddc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc98e00640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdc780a2d80 0x7fdc780a5240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0b7e90 tx=0x7fdc8c0303c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeaddc90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009fa0 tx=0x7fdc840118d0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/578667863 shutdown_connections 2024-04-25T01:58:45.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeaddc90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/578667863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eead35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/578667863 >> 172.21.15.102:0/578667863 conn(0x5571eeacf500 msgr2=0x5571eeacf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/578667863 shutdown_connections 2024-04-25T01:58:45.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.378+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/578667863 wait complete. 2024-04-25T01:58:45.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.379+0000 7fdc9dbbb740 1 -- start start 2024-04-25T01:58:45.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.379+0000 7fdc9dbbb740 1 Processor -- start 2024-04-25T01:58:45.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeadb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eead3b00 0x5571eeadb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eeade1d0 0x5571eeadbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5571eeae0fc0 con 0x5571eead3b00 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5571ee9f2d90 con 0x5571eeade1d0 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5571eeae1140 con 0x5571eead31e0 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc9dbbb740 10 client.0 _ll_register_callbacks cb 0x5571ee9f2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeadb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeadb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58528/0 (socket says 172.21.15.102:58528) 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93400640 1 -- 172.21.15.102:0/3603906678 learned_addr learned my addr 172.21.15.102:0/3603906678 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93400640 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eeade1d0 0x5571eeadbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93e00640 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eead3b00 0x5571eeadb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93400640 1 -- 172.21.15.102:0/3603906678 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eeade1d0 msgr2=0x5571eeadbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93400640 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eeade1d0 0x5571eeadbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93400640 1 -- 172.21.15.102:0/3603906678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eead3b00 msgr2=0x5571eeadb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93400640 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eead3b00 0x5571eeadb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.380+0000 7fdc93400640 1 -- 172.21.15.102:0/3603906678 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571ee74b3f0 con 0x5571eead31e0 2024-04-25T01:58:45.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.381+0000 7fdc93400640 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeadb200 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c011e20 tx=0x7fdc7c013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44b0146ff55c856e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.381+0000 7fdc82000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:45.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.381+0000 7fdc82000640 1 -- 172.21.15.102:0/3603906678 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c011220 con 0x5571eead31e0 2024-04-25T01:58:45.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.381+0000 7fdc82000640 1 -- 172.21.15.102:0/3603906678 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fdc7c005020 con 0x5571eead31e0 2024-04-25T01:58:45.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.381+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/3603906678 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc84003dd0 con 0x5571eead31e0 2024-04-25T01:58:45.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.382+0000 7fdc82000640 1 -- 172.21.15.102:0/3603906678 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdc7c005220 con 0x5571eead31e0 2024-04-25T01:58:45.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.382+0000 7fdc9dbbb740 10 client.26164 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:45.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.382+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/3603906678 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5571ee70b0f0 con 0x5571eead31e0 2024-04-25T01:58:45.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.383+0000 7fdc82000640 1 -- 172.21.15.102:0/3603906678 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc84003dd0 con 0x5571eead31e0 2024-04-25T01:58:45.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.383+0000 7fdc82000640 1 -- 172.21.15.102:0/3603906678 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fdc7c00e5d0 con 0x5571eead31e0 2024-04-25T01:58:45.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.383+0000 7fdc9dbbb740 10 client.26164 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:45.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.383+0000 7fdc9dbbb740 10 client.26164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:45.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.383+0000 7fdc9dbbb740 20 client.26164 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:45.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.383+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5571eeaee550 0x5571eeaf09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.383+0000 7fdc9dbbb740 4 client.26164 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:45.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.383+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/3603906678 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571eea4ad90 con 0x5571eeaee550 2024-04-25T01:58:45.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.384+0000 7fdc93e00640 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5571eeaee550 0x5571eeaf09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.386+0000 7fdc93e00640 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5571eeaee550 0x5571eeaf09b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f7b0de691b1a8e8 server_cookie=5b18dc248dd7b417 in_seq=0 out_seq=0 2024-04-25T01:58:45.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.386+0000 7fdc82000640 10 client.26164 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:45.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.387+0000 7fdc82000640 1 -- 172.21.15.102:0/3603906678 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5571eea4ad90 con 0x5571eeaee550 2024-04-25T01:58:45.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.387+0000 7fdc82000640 10 client.26164 handle_command_reply: tid=0 2024-04-25T01:58:45.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.387+0000 7fdc9dbbb740 1 client.26164 shutdown 2024-04-25T01:58:45.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.388+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/3603906678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 msgr2=0x5571eeadb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.388+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeadb200 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c011e20 tx=0x7fdc7c013040 comp rx=0 tx=0).stop 2024-04-25T01:58:45.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.388+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/3603906678 shutdown_connections 2024-04-25T01:58:45.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.388+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5571eeaee550 0x5571eeaf09b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.388+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eeade1d0 0x5571eeadbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.388+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eead3b00 0x5571eeadb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.388+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/3603906678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeadb200 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.388+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/3603906678 >> 172.21.15.102:0/3603906678 conn(0x5571eeacf500 msgr2=0x5571eeace800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.388+0000 7fdc82000640 0 client.26164 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:45.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.389+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/3603906678 shutdown_connections 2024-04-25T01:58:45.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.389+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/3603906678 wait complete. 2024-04-25T01:58:45.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.389+0000 7fdc9dbbb740 20 client.26164 trim_cache size 0 max 16384 2024-04-25T01:58:45.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:45 smithi102 ceph-mon[27796]: pgmap v107: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.2 MiB/s wr, 685 op/s 2024-04-25T01:58:45.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.425+0000 7fdc9dbbb740 1 Processor -- start 2024-04-25T01:58:45.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.426+0000 7fdc9dbbb740 1 -- start start 2024-04-25T01:58:45.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.426+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead31e0 0x5571eeae9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.426+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead3b00 0x5571eeada140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.426+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eead5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.426+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941b61b0 con 0x5571eeade1d0 2024-04-25T01:58:45.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.426+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941b3a50 con 0x5571eead31e0 2024-04-25T01:58:45.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.426+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941b18a0 con 0x5571eead3b00 2024-04-25T01:58:45.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.427+0000 7fdc93e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead3b00 0x5571eeada140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.427+0000 7fdc93e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead3b00 0x5571eeada140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58542/0 (socket says 172.21.15.102:58542) 2024-04-25T01:58:45.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.427+0000 7fdc93e00640 1 -- 172.21.15.102:0/4013407455 learned_addr learned my addr 172.21.15.102:0/4013407455 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.428+0000 7fdc93400640 1 --2- 172.21.15.102:0/4013407455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eead5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.428+0000 7fdc93e00640 1 -- 172.21.15.102:0/4013407455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead31e0 msgr2=0x5571eeae9e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:58:45.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.428+0000 7fdc93e00640 1 --2- 172.21.15.102:0/4013407455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead31e0 0x5571eeae9e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.428+0000 7fdc93e00640 1 -- 172.21.15.102:0/4013407455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 msgr2=0x5571eead5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.428+0000 7fdc93e00640 1 --2- 172.21.15.102:0/4013407455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eead5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.428+0000 7fdc93e00640 1 -- 172.21.15.102:0/4013407455 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571eead3930 con 0x5571eead3b00 2024-04-25T01:58:45.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.429+0000 7fdc93400640 1 --2- 172.21.15.102:0/4013407455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eead5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:45.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.429+0000 7fdc93e00640 1 --2- 172.21.15.102:0/4013407455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead3b00 0x5571eeada140 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x5571eead47a0 tx=0x7fdc84000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b59d991a24e956ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.430+0000 7fdc88c00640 1 -- 172.21.15.102:0/4013407455 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84031070 con 0x5571eead3b00 2024-04-25T01:58:45.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.430+0000 7fdc88c00640 1 -- 172.21.15.102:0/4013407455 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc84003a30 con 0x5571eead3b00 2024-04-25T01:58:45.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.430+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/4013407455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead3b00 msgr2=0x5571eeada140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.430+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/4013407455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead3b00 0x5571eeada140 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x5571eead47a0 tx=0x7fdc84000f40 comp rx=0 tx=0).stop 2024-04-25T01:58:45.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.430+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/4013407455 shutdown_connections 2024-04-25T01:58:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.430+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/4013407455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eead5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.430+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/4013407455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead3b00 0x5571eeada140 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.430+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/4013407455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead31e0 0x5571eeae9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.430+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/4013407455 >> 172.21.15.102:0/4013407455 conn(0x5571eeacf500 msgr2=0x5571eeadba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.431+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/4013407455 shutdown_connections 2024-04-25T01:58:45.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.431+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/4013407455 wait complete. 2024-04-25T01:58:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.431+0000 7fdc9dbbb740 1 -- start start 2024-04-25T01:58:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.431+0000 7fdc9dbbb740 1 Processor -- start 2024-04-25T01:58:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeae6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeae68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc9dbbb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941b1f10 con 0x5571eeade1d0 2024-04-25T01:58:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941b3380 con 0x5571eead3b00 2024-04-25T01:58:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc9dbbb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdc941b25e0 con 0x5571eead31e0 2024-04-25T01:58:45.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc9dbbb740 10 client.0 _ll_register_callbacks cb 0x7fdc7c011ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:45.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc93e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeae68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc93e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeae68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42538/0 (socket says 172.21.15.102:42538) 2024-04-25T01:58:45.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc93e00640 1 -- 172.21.15.102:0/242240523 learned_addr learned my addr 172.21.15.102:0/242240523 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.432+0000 7fdc93400640 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeae6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.433+0000 7fdc93400640 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.433+0000 7fdc93e00640 1 -- 172.21.15.102:0/242240523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 msgr2=0x5571eeae6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.433+0000 7fdc93e00640 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeae6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.433+0000 7fdc93e00640 1 -- 172.21.15.102:0/242240523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 msgr2=0x5571eeae1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:45.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.433+0000 7fdc93e00640 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.433+0000 7fdc93e00640 1 -- 172.21.15.102:0/242240523 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571eeae2330 con 0x5571eead3b00 2024-04-25T01:58:45.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.434+0000 7fdc93400640 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:45.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.434+0000 7fdc93e00640 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeae68a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdc8400f700 tx=0x7fdc84009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dfb5ebd4c24bbaf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.434+0000 7fdc82000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:45.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.435+0000 7fdc82000640 1 -- 172.21.15.102:0/242240523 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8403a060 con 0x5571eead3b00 2024-04-25T01:58:45.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.435+0000 7fdc82000640 1 -- 172.21.15.102:0/242240523 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fdc8402a040 con 0x5571eead3b00 2024-04-25T01:58:45.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.435+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/242240523 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5571eea44aa0 con 0x5571eead3b00 2024-04-25T01:58:45.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.435+0000 7fdc82000640 1 -- 172.21.15.102:0/242240523 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdc84031030 con 0x5571eead3b00 2024-04-25T01:58:45.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.436+0000 7fdc9dbbb740 10 client.26213 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:45.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.436+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/242240523 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5571eeb546d0 con 0x5571eead3b00 2024-04-25T01:58:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.436+0000 7fdc82000640 1 -- 172.21.15.102:0/242240523 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5571eea44aa0 con 0x5571eead3b00 2024-04-25T01:58:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.436+0000 7fdc82000640 1 -- 172.21.15.102:0/242240523 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fdc84005340 con 0x5571eead3b00 2024-04-25T01:58:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.436+0000 7fdc9dbbb740 10 client.26213 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.436+0000 7fdc9dbbb740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.436+0000 7fdc9dbbb740 20 client.26213 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.437+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5571eeb54ab0 0x5571eeb56ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.437+0000 7fdc9dbbb740 4 client.26213 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:58:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.437+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/242240523 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5571eea3edf0 con 0x5571eeb54ab0 2024-04-25T01:58:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.437+0000 7fdc93400640 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5571eeb54ab0 0x5571eeb56ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.437+0000 7fdc93400640 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5571eeb54ab0 0x5571eeb56ed0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58628c737a5d1525 server_cookie=8057584a4b6fb294 in_seq=0 out_seq=0 2024-04-25T01:58:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.437+0000 7fdc82000640 10 client.26213 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.438+0000 7fdc82000640 1 -- 172.21.15.102:0/242240523 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5571eea3edf0 con 0x5571eeb54ab0 2024-04-25T01:58:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.438+0000 7fdc82000640 10 client.26213 handle_command_reply: tid=0 2024-04-25T01:58:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.438+0000 7fdc9dbbb740 1 client.26213 shutdown 2024-04-25T01:58:45.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.439+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/242240523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 msgr2=0x5571eeae68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.439+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeae68a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdc8400f700 tx=0x7fdc84009500 comp rx=0 tx=0).stop 2024-04-25T01:58:45.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.439+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/242240523 shutdown_connections 2024-04-25T01:58:45.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.439+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5571eeb54ab0 0x5571eeb56ed0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.439+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571eeade1d0 0x5571eeae1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.439+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5571eead3b00 0x5571eeae68a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.439+0000 7fdc9dbbb740 1 --2- 172.21.15.102:0/242240523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571eead31e0 0x5571eeae6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.439+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/242240523 >> 172.21.15.102:0/242240523 conn(0x5571eeacf500 msgr2=0x5571eeadca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.439+0000 7fdc82000640 0 client.26213 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:45.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.440+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/242240523 shutdown_connections 2024-04-25T01:58:45.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.440+0000 7fdc9dbbb740 1 -- 172.21.15.102:0/242240523 wait complete. 2024-04-25T01:58:45.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.441+0000 7fdc9dbbb740 20 client.26213 trim_cache size 0 max 16384 2024-04-25T01:58:45.444 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:58:45.444 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T01:58:45.444 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T01:58:45.444 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:58:45.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdc780a2d80 msgr2=0x7fdc780a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdc780a2d80 0x7fdc780a5240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0b7e90 tx=0x7fdc8c0303c0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc94109800 msgr2=0x7fdc941a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc94109800 0x7fdc941a2d00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009830 tx=0x7fdc8c005c90 comp rx=0 tx=0).stop 2024-04-25T01:58:45.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 shutdown_connections 2024-04-25T01:58:45.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fdc780a2d80 0x7fdc780a5240 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc94109800 0x7fdc941a2d00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc941041b0 0x7fdc9410f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 --2- 172.21.15.102:0/2069430584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc941037f0 0x7fdc9410f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 >> 172.21.15.102:0/2069430584 conn(0x7fdc940ff280 msgr2=0x7fdc94077850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 shutdown_connections 2024-04-25T01:58:45.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.442+0000 7fdc99800640 1 -- 172.21.15.102:0/2069430584 wait complete. 2024-04-25T01:58:45.479 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:58:45.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:58:45.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:45 smithi110 ceph-mon[32848]: pgmap v107: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.2 MiB/s wr, 685 op/s 2024-04-25T01:58:45.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:45 smithi193 ceph-mon[32981]: pgmap v107: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.2 MiB/s wr, 685 op/s 2024-04-25T01:58:45.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.759+0000 7f3852800640 1 -- 172.21.15.102:0/186269276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 msgr2=0x7f384c103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.759+0000 7f3852800640 1 --2- 172.21.15.102:0/186269276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 0x7f384c103db0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3838009900 tx=0x7f383802a3f0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.760+0000 7f3852800640 1 -- 172.21.15.102:0/186269276 shutdown_connections 2024-04-25T01:58:45.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.760+0000 7f3852800640 1 --2- 172.21.15.102:0/186269276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f384c1042f0 0x7f384c110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.760+0000 7f3852800640 1 --2- 172.21.15.102:0/186269276 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 0x7f384c103db0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.760+0000 7f3852800640 1 --2- 172.21.15.102:0/186269276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384c109940 0x7f384c109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.760+0000 7f3852800640 1 -- 172.21.15.102:0/186269276 >> 172.21.15.102:0/186269276 conn(0x7f384c0ff400 msgr2=0x7f384c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:45.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.761+0000 7f3852800640 1 -- 172.21.15.102:0/186269276 shutdown_connections 2024-04-25T01:58:45.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.761+0000 7f3852800640 1 -- 172.21.15.102:0/186269276 wait complete. 2024-04-25T01:58:45.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.763+0000 7f3852800640 1 Processor -- start 2024-04-25T01:58:45.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.764+0000 7f3852800640 1 -- start start 2024-04-25T01:58:45.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.764+0000 7f3852800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 0x7f384c110710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.764+0000 7f3852800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f384c1042f0 0x7f384c10d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.764+0000 7f3852800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384c109940 0x7f384c10de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.764+0000 7f3852800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f384c113da0 con 0x7f384c109940 2024-04-25T01:58:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.764+0000 7f3852800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f384c113c20 con 0x7f384c103930 2024-04-25T01:58:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.764+0000 7f3852800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f384c113f20 con 0x7f384c1042f0 2024-04-25T01:58:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 0x7f384c110710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 0x7f384c110710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42556/0 (socket says 172.21.15.102:42556) 2024-04-25T01:58:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384b400640 1 -- 172.21.15.102:0/1660828719 learned_addr learned my addr 172.21.15.102:0/1660828719 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384be00640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384c109940 0x7f384c10de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384aa00640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f384c1042f0 0x7f384c10d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384b400640 1 -- 172.21.15.102:0/1660828719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f384c1042f0 msgr2=0x7f384c10d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384b400640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f384c1042f0 0x7f384c10d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384b400640 1 -- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384c109940 msgr2=0x7f384c10de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384b400640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384c109940 0x7f384c10de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384b400640 1 -- 172.21.15.102:0/1660828719 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384c1aaa80 con 0x7f384c103930 2024-04-25T01:58:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384aa00640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f384c1042f0 0x7f384c10d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.765+0000 7f384be00640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384c109940 0x7f384c10de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:45.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.766+0000 7f384b400640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 0x7f384c110710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f383c00ea60 tx=0x7f383c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.766+0000 7f3843e00640 1 -- 172.21.15.102:0/1660828719 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c00cde0 con 0x7f384c103930 2024-04-25T01:58:45.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.767+0000 7f3843e00640 1 -- 172.21.15.102:0/1660828719 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f383c0035f0 con 0x7f384c103930 2024-04-25T01:58:45.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.767+0000 7f3843e00640 1 -- 172.21.15.102:0/1660828719 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383c010670 con 0x7f384c103930 2024-04-25T01:58:45.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.767+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f384c1aac50 con 0x7f384c103930 2024-04-25T01:58:45.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.767+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f384c1ab120 con 0x7f384c103930 2024-04-25T01:58:45.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.769+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f384c068560 con 0x7f384c103930 2024-04-25T01:58:45.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.770+0000 7f3843e00640 1 -- 172.21.15.102:0/1660828719 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f383c004020 con 0x7f384c103930 2024-04-25T01:58:45.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.773+0000 7f3843e00640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38300a2d80 0x7f38300a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:45.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.773+0000 7f3843e00640 1 -- 172.21.15.102:0/1660828719 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f383c0cf710 con 0x7f384c103930 2024-04-25T01:58:45.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.773+0000 7f384aa00640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38300a2d80 0x7f38300a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:45.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.775+0000 7f384aa00640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38300a2d80 0x7f38300a5240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3838000c00 tx=0x7f3838005a40 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:45.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:45.776+0000 7f3843e00640 1 -- 172.21.15.102:0/1660828719 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f383c09c300 con 0x7f384c103930 2024-04-25T01:58:46.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.159+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f384c0772c0 con 0x7f384c103930 2024-04-25T01:58:46.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.160+0000 7f3843e00640 1 -- 172.21.15.102:0/1660828719 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11900 (secure 0 0 0) 0x7f383c09bc00 con 0x7f384c103930 2024-04-25T01:58:46.163 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:58:46.163 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:58:25.930977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T01:58:46.163 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 28 2024-04-25T01:58:46.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38300a2d80 msgr2=0x7f38300a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38300a2d80 0x7f38300a5240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3838000c00 tx=0x7f3838005a40 comp rx=0 tx=0).stop 2024-04-25T01:58:46.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 msgr2=0x7f384c110710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 0x7f384c110710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f383c00ea60 tx=0x7f383c00ef20 comp rx=0 tx=0).stop 2024-04-25T01:58:46.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 shutdown_connections 2024-04-25T01:58:46.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38300a2d80 0x7f38300a5240 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f384c109940 0x7f384c10de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f384c1042f0 0x7f384c10d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 --2- 172.21.15.102:0/1660828719 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f384c103930 0x7f384c110710 secure :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f383c00ea60 tx=0x7f383c00ef20 comp rx=0 tx=0).stop 2024-04-25T01:58:46.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.163+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 >> 172.21.15.102:0/1660828719 conn(0x7f384c0ff400 msgr2=0x7f384c0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.164+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 shutdown_connections 2024-04-25T01:58:46.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.164+0000 7f3852800640 1 -- 172.21.15.102:0/1660828719 wait complete. 2024-04-25T01:58:46.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:58:46.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.443+0000 7fa472a00640 1 -- 172.21.15.102:0/1574918786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c1007b0 msgr2=0x7fa46c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.443+0000 7fa472a00640 1 --2- 172.21.15.102:0/1574918786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c1007b0 0x7fa46c100b90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009ca0 tx=0x7fa45c02a9e0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.444+0000 7fa472a00640 1 -- 172.21.15.102:0/1574918786 shutdown_connections 2024-04-25T01:58:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.444+0000 7fa472a00640 1 --2- 172.21.15.102:0/1574918786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa46c105ad0 0x7fa46c105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.444+0000 7fa472a00640 1 --2- 172.21.15.102:0/1574918786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa46c1010d0 0x7fa46c105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.444+0000 7fa472a00640 1 --2- 172.21.15.102:0/1574918786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c1007b0 0x7fa46c100b90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.444+0000 7fa472a00640 1 -- 172.21.15.102:0/1574918786 >> 172.21.15.102:0/1574918786 conn(0x7fa46c077740 msgr2=0x7fa46c077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.444+0000 7fa472a00640 1 -- 172.21.15.102:0/1574918786 shutdown_connections 2024-04-25T01:58:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.444+0000 7fa472a00640 1 -- 172.21.15.102:0/1574918786 wait complete. 2024-04-25T01:58:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.445+0000 7fa472a00640 1 Processor -- start 2024-04-25T01:58:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.445+0000 7fa472a00640 1 -- start start 2024-04-25T01:58:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.446+0000 7fa472a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa46c1007b0 0x7fa46c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.446+0000 7fa472a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa46c1010d0 0x7fa46c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.446+0000 7fa472a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c105ad0 0x7fa46c1a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.446+0000 7fa472a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c118140 con 0x7fa46c1010d0 2024-04-25T01:58:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.446+0000 7fa472a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c117fc0 con 0x7fa46c105ad0 2024-04-25T01:58:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.446+0000 7fa472a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1182c0 con 0x7fa46c1007b0 2024-04-25T01:58:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa470a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c105ad0 0x7fa46c1a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa470a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c105ad0 0x7fa46c1a2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42562/0 (socket says 172.21.15.102:42562) 2024-04-25T01:58:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa470a00640 1 -- 172.21.15.102:0/890952413 learned_addr learned my addr 172.21.15.102:0/890952413 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa46be00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa46c1007b0 0x7fa46c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa46b400640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa46c1010d0 0x7fa46c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa470a00640 1 -- 172.21.15.102:0/890952413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa46c1007b0 msgr2=0x7fa46c19e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa470a00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa46c1007b0 0x7fa46c19e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa470a00640 1 -- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa46c1010d0 msgr2=0x7fa46c19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa470a00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa46c1010d0 0x7fa46c19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa470a00640 1 -- 172.21.15.102:0/890952413 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46c1a35d0 con 0x7fa46c105ad0 2024-04-25T01:58:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.447+0000 7fa46b400640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa46c1010d0 0x7fa46c19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.448+0000 7fa46be00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa46c1007b0 0x7fa46c19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.448+0000 7fa470a00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c105ad0 0x7fa46c1a2ea0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa46400e970 tx=0x7fa46400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.449+0000 7fa468c00640 1 -- 172.21.15.102:0/890952413 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46400ccc0 con 0x7fa46c105ad0 2024-04-25T01:58:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.449+0000 7fa468c00640 1 -- 172.21.15.102:0/890952413 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa46400ce60 con 0x7fa46c105ad0 2024-04-25T01:58:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.449+0000 7fa468c00640 1 -- 172.21.15.102:0/890952413 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464010430 con 0x7fa46c105ad0 2024-04-25T01:58:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.449+0000 7fa472a00640 1 -- 172.21.15.102:0/890952413 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46c1a38c0 con 0x7fa46c105ad0 2024-04-25T01:58:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.449+0000 7fa472a00640 1 -- 172.21.15.102:0/890952413 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46c1ab150 con 0x7fa46c105ad0 2024-04-25T01:58:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.454+0000 7fa475859740 1 Processor -- start 2024-04-25T01:58:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.454+0000 7fa468c00640 1 -- 172.21.15.102:0/890952413 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa4640046c0 con 0x7fa46c105ad0 2024-04-25T01:58:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.455+0000 7fa475859740 1 -- start start 2024-04-25T01:58:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.455+0000 7fa475859740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a355c0 0x564e14a359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.455+0000 7fa475859740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a35ee0 0x564e14a40070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.455+0000 7fa475859740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a429a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.455+0000 7fa475859740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x564e1499a4e0 con 0x564e14a355c0 2024-04-25T01:58:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.455+0000 7fa475859740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564e14942a70 con 0x564e14a405b0 2024-04-25T01:58:46.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.455+0000 7fa475859740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564e149964a0 con 0x564e14a35ee0 2024-04-25T01:58:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a35ee0 0x564e14a40070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a35ee0 0x564e14a40070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58590/0 (socket says 172.21.15.102:58590) 2024-04-25T01:58:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46be00640 1 -- 172.21.15.102:0/4213521197 learned_addr learned my addr 172.21.15.102:0/4213521197 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46b400640 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a355c0 0x564e14a359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46b400640 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46be00640 1 -- 172.21.15.102:0/4213521197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 msgr2=0x564e14a429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46be00640 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a429a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46be00640 1 -- 172.21.15.102:0/4213521197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a355c0 msgr2=0x564e14a359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46be00640 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a355c0 0x564e14a359a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.456+0000 7fa46be00640 1 -- 172.21.15.102:0/4213521197 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e146ae3f0 con 0x564e14a35ee0 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.457+0000 7fa46b400640 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.457+0000 7fa46be00640 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a35ee0 0x564e14a40070 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009ca0 tx=0x7fa45c009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f110459b026b821 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.457+0000 7fa468c00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4500a2dc0 0x7fa4500a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.457+0000 7fa461200640 1 -- 172.21.15.102:0/4213521197 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c033050 con 0x564e14a35ee0 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.457+0000 7fa468c00640 1 -- 172.21.15.102:0/890952413 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa4640cf6d0 con 0x7fa46c105ad0 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.457+0000 7fa46b400640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4500a2dc0 0x7fa4500a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.458+0000 7fa461200640 1 -- 172.21.15.102:0/4213521197 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa45c047070 con 0x564e14a35ee0 2024-04-25T01:58:46.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.458+0000 7fa461200640 1 -- 172.21.15.102:0/4213521197 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c02fb00 con 0x564e14a35ee0 2024-04-25T01:58:46.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.459+0000 7fa475859740 1 -- 172.21.15.102:0/4213521197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a35ee0 msgr2=0x564e14a40070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.459+0000 7fa475859740 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a35ee0 0x564e14a40070 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009ca0 tx=0x7fa45c009800 comp rx=0 tx=0).stop 2024-04-25T01:58:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.459+0000 7fa475859740 1 -- 172.21.15.102:0/4213521197 shutdown_connections 2024-04-25T01:58:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.459+0000 7fa475859740 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.459+0000 7fa475859740 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a35ee0 0x564e14a40070 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.459+0000 7fa475859740 1 --2- 172.21.15.102:0/4213521197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a355c0 0x564e14a359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.459+0000 7fa475859740 1 -- 172.21.15.102:0/4213521197 >> 172.21.15.102:0/4213521197 conn(0x564e14a318e0 msgr2=0x564e14a31cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.460+0000 7fa475859740 1 -- 172.21.15.102:0/4213521197 shutdown_connections 2024-04-25T01:58:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.460+0000 7fa46b400640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4500a2dc0 0x7fa4500a5280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x564e14a32a70 tx=0x7fa458009580 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.460+0000 7fa475859740 1 -- 172.21.15.102:0/4213521197 wait complete. 2024-04-25T01:58:46.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.460+0000 7fa475859740 1 -- start start 2024-04-25T01:58:46.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.461+0000 7fa475859740 1 Processor -- start 2024-04-25T01:58:46.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.461+0000 7fa475859740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a3d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.461+0000 7fa475859740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a35ee0 0x564e14a3da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.461+0000 7fa475859740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a3df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.461+0000 7fa475859740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x564e14a43340 con 0x564e14a35ee0 2024-04-25T01:58:46.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.461+0000 7fa475859740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564e14955ec0 con 0x564e14a405b0 2024-04-25T01:58:46.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.461+0000 7fa475859740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564e14a434c0 con 0x564e14a355c0 2024-04-25T01:58:46.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.462+0000 7fa475859740 10 client.0 _ll_register_callbacks cb 0x564e149923f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.462+0000 7fa470a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a3df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.462+0000 7fa470a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a3df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42572/0 (socket says 172.21.15.102:42572) 2024-04-25T01:58:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.462+0000 7fa470a00640 1 -- 172.21.15.102:0/1170597460 learned_addr learned my addr 172.21.15.102:0/1170597460 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.462+0000 7fa46b400640 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a35ee0 0x564e14a3da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.462+0000 7fa46be00640 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a3d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.463+0000 7fa470a00640 1 -- 172.21.15.102:0/1170597460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 msgr2=0x564e14a3d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.463+0000 7fa470a00640 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a3d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.463+0000 7fa470a00640 1 -- 172.21.15.102:0/1170597460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a35ee0 msgr2=0x564e14a3da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.463+0000 7fa470a00640 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a35ee0 0x564e14a3da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.463+0000 7fa470a00640 1 -- 172.21.15.102:0/1170597460 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e147787d0 con 0x564e14a405b0 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.463+0000 7fa470a00640 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a3df80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa4640a4360 tx=0x7fa46409c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ebc736fd18c0f8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.463+0000 7fa455600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.463+0000 7fa455600640 1 -- 172.21.15.102:0/1170597460 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4640a4600 con 0x564e14a405b0 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.464+0000 7fa475859740 1 -- 172.21.15.102:0/1170597460 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa45c047070 con 0x564e14a405b0 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.464+0000 7fa455600640 1 -- 172.21.15.102:0/1170597460 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa46409b070 con 0x564e14a405b0 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.464+0000 7fa455600640 1 -- 172.21.15.102:0/1170597460 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa4640a4890 con 0x564e14a405b0 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.464+0000 7fa475859740 10 client.26228 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.464+0000 7fa475859740 1 -- 172.21.15.102:0/1170597460 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564e14a4f5b0 con 0x564e14a405b0 2024-04-25T01:58:46.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.464+0000 7fa455600640 1 -- 172.21.15.102:0/1170597460 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa45c047070 con 0x564e14a405b0 2024-04-25T01:58:46.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.465+0000 7fa455600640 1 -- 172.21.15.102:0/1170597460 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fa46409aae0 con 0x564e14a405b0 2024-04-25T01:58:46.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.465+0000 7fa475859740 10 client.26228 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:46.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.465+0000 7fa475859740 10 client.26228 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:58:46.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.465+0000 7fa475859740 20 client.26228 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:46.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.465+0000 7fa475859740 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564e14a509c0 0x564e14a52df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.465+0000 7fa475859740 4 client.26228 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:46.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.465+0000 7fa475859740 1 -- 172.21.15.102:0/1170597460 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e149a7a20 con 0x564e14a509c0 2024-04-25T01:58:46.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.466+0000 7fa46be00640 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564e14a509c0 0x564e14a52df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.466+0000 7fa46be00640 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564e14a509c0 0x564e14a52df0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=29f3aff4d37ba881 server_cookie=634505d203efd2c6 in_seq=0 out_seq=0 2024-04-25T01:58:46.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.466+0000 7fa455600640 10 client.26228 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.467+0000 7fa455600640 1 -- 172.21.15.102:0/1170597460 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e149a7a20 con 0x564e14a509c0 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.467+0000 7fa455600640 10 client.26228 handle_command_reply: tid=0 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.467+0000 7fa475859740 1 client.26228 shutdown 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 -- 172.21.15.102:0/1170597460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 msgr2=0x564e14a3df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a3df80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa4640a4360 tx=0x7fa46409c3e0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 -- 172.21.15.102:0/1170597460 shutdown_connections 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564e14a509c0 0x564e14a52df0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a3df80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a35ee0 0x564e14a3da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa455600640 0 client.26228 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 --2- 172.21.15.102:0/1170597460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a3d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 -- 172.21.15.102:0/1170597460 >> 172.21.15.102:0/1170597460 conn(0x564e14a318e0 msgr2=0x564e14a30b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 -- 172.21.15.102:0/1170597460 shutdown_connections 2024-04-25T01:58:46.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.468+0000 7fa475859740 1 -- 172.21.15.102:0/1170597460 wait complete. 2024-04-25T01:58:46.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.469+0000 7fa475859740 20 client.26228 trim_cache size 0 max 16384 2024-04-25T01:58:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.488+0000 7fa475859740 1 Processor -- start 2024-04-25T01:58:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.488+0000 7fa475859740 1 -- start start 2024-04-25T01:58:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.488+0000 7fa475859740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a4c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.488+0000 7fa475859740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a35ee0 0x564e14a3dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.488+0000 7fa475859740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a586c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.488+0000 7fa475859740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1b6220 con 0x564e14a35ee0 2024-04-25T01:58:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.488+0000 7fa475859740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1b3ac0 con 0x564e14a405b0 2024-04-25T01:58:46.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.488+0000 7fa475859740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1b1910 con 0x564e14a355c0 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa470a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa470a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42580/0 (socket says 172.21.15.102:42580) 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa470a00640 1 -- 172.21.15.102:0/3346273211 learned_addr learned my addr 172.21.15.102:0/3346273211 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa46be00640 1 --2- 172.21.15.102:0/3346273211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a4c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa470a00640 1 -- 172.21.15.102:0/3346273211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 msgr2=0x564e14a4c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa470a00640 1 --2- 172.21.15.102:0/3346273211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a4c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa470a00640 1 -- 172.21.15.102:0/3346273211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a35ee0 msgr2=0x564e14a3dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa470a00640 1 --2- 172.21.15.102:0/3346273211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a35ee0 0x564e14a3dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa470a00640 1 -- 172.21.15.102:0/3346273211 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e14a2bf50 con 0x564e14a405b0 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa46be00640 1 --2- 172.21.15.102:0/3346273211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a4c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.489+0000 7fa470a00640 1 --2- 172.21.15.102:0/3346273211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a586c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa4640ccf00 tx=0x7fa4640a0150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e5db7e9dc583fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.490+0000 7fa457e00640 1 -- 172.21.15.102:0/3346273211 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464097ec0 con 0x564e14a405b0 2024-04-25T01:58:46.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.490+0000 7fa457e00640 1 -- 172.21.15.102:0/3346273211 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa464097970 con 0x564e14a405b0 2024-04-25T01:58:46.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.490+0000 7fa457e00640 1 -- 172.21.15.102:0/3346273211 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46409b070 con 0x564e14a405b0 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.491+0000 7fa475859740 1 -- 172.21.15.102:0/3346273211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 msgr2=0x564e14a586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.491+0000 7fa475859740 1 --2- 172.21.15.102:0/3346273211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a586c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa4640ccf00 tx=0x7fa4640a0150 comp rx=0 tx=0).stop 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.491+0000 7fa475859740 1 -- 172.21.15.102:0/3346273211 shutdown_connections 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.491+0000 7fa475859740 1 --2- 172.21.15.102:0/3346273211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a405b0 0x564e14a586c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.491+0000 7fa475859740 1 --2- 172.21.15.102:0/3346273211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a35ee0 0x564e14a3dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.491+0000 7fa475859740 1 --2- 172.21.15.102:0/3346273211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a4c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.491+0000 7fa475859740 1 -- 172.21.15.102:0/3346273211 >> 172.21.15.102:0/3346273211 conn(0x564e14a318e0 msgr2=0x564e14a3d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.491+0000 7fa475859740 1 -- 172.21.15.102:0/3346273211 shutdown_connections 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.491+0000 7fa475859740 1 -- 172.21.15.102:0/3346273211 wait complete. 2024-04-25T01:58:46.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.492+0000 7fa475859740 1 -- start start 2024-04-25T01:58:46.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.492+0000 7fa475859740 1 Processor -- start 2024-04-25T01:58:46.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.493+0000 7fa475859740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a48af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.493+0000 7fa475859740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a35ee0 0x564e14a43f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.493+0000 7fa475859740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a405b0 0x564e14a444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.493+0000 7fa475859740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1b1f80 con 0x564e14a405b0 2024-04-25T01:58:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.493+0000 7fa475859740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1b33f0 con 0x564e14a35ee0 2024-04-25T01:58:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.493+0000 7fa475859740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1b2650 con 0x564e14a355c0 2024-04-25T01:58:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.493+0000 7fa475859740 10 client.0 _ll_register_callbacks cb 0x7fa464009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.493+0000 7fa46be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a48af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.493+0000 7fa46be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a48af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58618/0 (socket says 172.21.15.102:58618) 2024-04-25T01:58:46.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa46be00640 1 -- 172.21.15.102:0/3137529780 learned_addr learned my addr 172.21.15.102:0/3137529780 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa46b400640 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a35ee0 0x564e14a43f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa46be00640 1 -- 172.21.15.102:0/3137529780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a35ee0 msgr2=0x564e14a43f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa46be00640 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a35ee0 0x564e14a43f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa470a00640 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a405b0 0x564e14a444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa46be00640 1 -- 172.21.15.102:0/3137529780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a405b0 msgr2=0x564e14a444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa46be00640 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a405b0 0x564e14a444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa46be00640 1 -- 172.21.15.102:0/3137529780 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e14a44cd0 con 0x564e14a355c0 2024-04-25T01:58:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa46b400640 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a35ee0 0x564e14a43f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:46.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa470a00640 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a405b0 0x564e14a444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:46.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa46be00640 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a48af0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa45c005fe0 tx=0x7fa45c02f780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5d9eeff364eee25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.494+0000 7fa455600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:46.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.495+0000 7fa455600640 1 -- 172.21.15.102:0/3137529780 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c05a070 con 0x564e14a355c0 2024-04-25T01:58:46.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.495+0000 7fa475859740 1 -- 172.21.15.102:0/3137529780 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa464097970 con 0x564e14a355c0 2024-04-25T01:58:46.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.495+0000 7fa455600640 1 -- 172.21.15.102:0/3137529780 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa45c056dd0 con 0x564e14a355c0 2024-04-25T01:58:46.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.495+0000 7fa455600640 1 -- 172.21.15.102:0/3137529780 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa45c066050 con 0x564e14a355c0 2024-04-25T01:58:46.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.495+0000 7fa475859740 10 client.26194 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:46.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.495+0000 7fa475859740 1 -- 172.21.15.102:0/3137529780 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564e14ab6ed0 con 0x564e14a355c0 2024-04-25T01:58:46.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.496+0000 7fa455600640 1 -- 172.21.15.102:0/3137529780 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa464097970 con 0x564e14a355c0 2024-04-25T01:58:46.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.496+0000 7fa455600640 1 -- 172.21.15.102:0/3137529780 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fa45c02f7f0 con 0x564e14a355c0 2024-04-25T01:58:46.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.496+0000 7fa475859740 10 client.26194 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:46.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.496+0000 7fa475859740 10 client.26194 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.496+0000 7fa475859740 20 client.26194 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.496+0000 7fa475859740 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564e14ab71b0 0x564e14ab95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.496+0000 7fa475859740 4 client.26194 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.496+0000 7fa475859740 1 -- 172.21.15.102:0/3137529780 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e149a1d70 con 0x564e14ab71b0 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.496+0000 7fa46b400640 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564e14ab71b0 0x564e14ab95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.497+0000 7fa46b400640 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564e14ab71b0 0x564e14ab95a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bff8ede45fb383a8 server_cookie=2b566e3ac1647c29 in_seq=0 out_seq=0 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.497+0000 7fa455600640 10 client.26194 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.497+0000 7fa455600640 1 -- 172.21.15.102:0/3137529780 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e149a1d70 con 0x564e14ab71b0 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.497+0000 7fa455600640 10 client.26194 handle_command_reply: tid=0 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.497+0000 7fa475859740 1 client.26194 shutdown 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.498+0000 7fa475859740 1 -- 172.21.15.102:0/3137529780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 msgr2=0x564e14a48af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.498+0000 7fa475859740 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a48af0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa45c005fe0 tx=0x7fa45c02f780 comp rx=0 tx=0).stop 2024-04-25T01:58:46.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.498+0000 7fa475859740 1 -- 172.21.15.102:0/3137529780 shutdown_connections 2024-04-25T01:58:46.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.498+0000 7fa475859740 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564e14a405b0 0x564e14a444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.498+0000 7fa475859740 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564e14ab71b0 0x564e14ab95a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.498+0000 7fa475859740 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564e14a35ee0 0x564e14a43f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.498+0000 7fa475859740 1 --2- 172.21.15.102:0/3137529780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e14a355c0 0x564e14a48af0 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa45c005fe0 tx=0x7fa45c02f780 comp rx=0 tx=0).stop 2024-04-25T01:58:46.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.498+0000 7fa455600640 0 client.26194 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:58:46.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.498+0000 7fa475859740 1 -- 172.21.15.102:0/3137529780 >> 172.21.15.102:0/3137529780 conn(0x564e14a318e0 msgr2=0x564e14a3dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.500+0000 7fa475859740 1 -- 172.21.15.102:0/3137529780 shutdown_connections 2024-04-25T01:58:46.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.500+0000 7fa475859740 1 -- 172.21.15.102:0/3137529780 wait complete. 2024-04-25T01:58:46.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.500+0000 7fa475859740 20 client.26194 trim_cache size 0 max 16384 2024-04-25T01:58:46.504 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:58:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 -- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4500a2dc0 msgr2=0x7fa4500a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4500a2dc0 0x7fa4500a5280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x564e14a32a70 tx=0x7fa458009580 comp rx=0 tx=0).stop 2024-04-25T01:58:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 -- 172.21.15.102:0/890952413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c105ad0 msgr2=0x7fa46c1a2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c105ad0 0x7fa46c1a2ea0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa46400e970 tx=0x7fa46400ed70 comp rx=0 tx=0).stop 2024-04-25T01:58:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 -- 172.21.15.102:0/890952413 shutdown_connections 2024-04-25T01:58:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4500a2dc0 0x7fa4500a5280 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa46c105ad0 0x7fa46c1a2ea0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa46c1010d0 0x7fa46c19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 --2- 172.21.15.102:0/890952413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa46c1007b0 0x7fa46c19e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 -- 172.21.15.102:0/890952413 >> 172.21.15.102:0/890952413 conn(0x7fa46c077740 msgr2=0x7fa46c103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 -- 172.21.15.102:0/890952413 shutdown_connections 2024-04-25T01:58:46.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.501+0000 7fa472a00640 1 -- 172.21.15.102:0/890952413 wait complete. 2024-04-25T01:58:46.535 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:58:46.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.790+0000 7f2611e00640 1 -- 172.21.15.102:0/405684943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f260c10be70 msgr2=0x7f260c108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.790+0000 7f2611e00640 1 --2- 172.21.15.102:0/405684943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f260c10be70 0x7f260c108740 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009900 tx=0x7f25fc02a480 comp rx=0 tx=0).stop 2024-04-25T01:58:46.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.790+0000 7f2611e00640 1 -- 172.21.15.102:0/405684943 shutdown_connections 2024-04-25T01:58:46.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.790+0000 7f2611e00640 1 --2- 172.21.15.102:0/405684943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f260c1167d0 0x7f260c118bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.790+0000 7f2611e00640 1 --2- 172.21.15.102:0/405684943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260c108c80 0x7f260c109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.790+0000 7f2611e00640 1 --2- 172.21.15.102:0/405684943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f260c10be70 0x7f260c108740 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.790+0000 7f2611e00640 1 -- 172.21.15.102:0/405684943 >> 172.21.15.102:0/405684943 conn(0x7f260c06ccb0 msgr2=0x7f260c06d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.791+0000 7f2611e00640 1 -- 172.21.15.102:0/405684943 shutdown_connections 2024-04-25T01:58:46.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.791+0000 7f2611e00640 1 -- 172.21.15.102:0/405684943 wait complete. 2024-04-25T01:58:46.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.792+0000 7f2611e00640 1 Processor -- start 2024-04-25T01:58:46.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.792+0000 7f2611e00640 1 -- start start 2024-04-25T01:58:46.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.793+0000 7f2611e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260c108c80 0x7f260c1a8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.793+0000 7f2611e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f260c10be70 0x7f260c1a8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.793+0000 7f2611e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f260c1167d0 0x7f260c1a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.793+0000 7f2611e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f260c11c520 con 0x7f260c10be70 2024-04-25T01:58:46.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.793+0000 7f2611e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f260c11c3a0 con 0x7f260c108c80 2024-04-25T01:58:46.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.793+0000 7f2611e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f260c11c6a0 con 0x7f260c1167d0 2024-04-25T01:58:46.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.793+0000 7f260aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260c108c80 0x7f260c1a8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.794+0000 7f260b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f260c1167d0 0x7f260c1a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.795+0000 7f260aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260c108c80 0x7f260c1a8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42600/0 (socket says 172.21.15.102:42600) 2024-04-25T01:58:46.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.795+0000 7f260aa00640 1 -- 172.21.15.102:0/2764159531 learned_addr learned my addr 172.21.15.102:0/2764159531 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.794+0000 7f260a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f260c10be70 0x7f260c1a8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.795+0000 7f260aa00640 1 -- 172.21.15.102:0/2764159531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f260c1167d0 msgr2=0x7f260c1a2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.795+0000 7f260aa00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f260c1167d0 0x7f260c1a2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.795+0000 7f260aa00640 1 -- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f260c10be70 msgr2=0x7f260c1a8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.795+0000 7f260aa00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f260c10be70 0x7f260c1a8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.795+0000 7f260aa00640 1 -- 172.21.15.102:0/2764159531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f260c1a30c0 con 0x7f260c108c80 2024-04-25T01:58:46.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.795+0000 7f260b400640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f260c1167d0 0x7f260c1a2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:46.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.796+0000 7f260a000640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f260c10be70 0x7f260c1a8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:46.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.796+0000 7f260aa00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260c108c80 0x7f260c1a8790 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f25fc004770 tx=0x7f25fc003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.796+0000 7f2610a00640 1 -- 172.21.15.102:0/2764159531 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc03f070 con 0x7f260c108c80 2024-04-25T01:58:46.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.797+0000 7f2611e00640 1 -- 172.21.15.102:0/2764159531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f260c1a3350 con 0x7f260c108c80 2024-04-25T01:58:46.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.797+0000 7f2610a00640 1 -- 172.21.15.102:0/2764159531 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25fc031ca0 con 0x7f260c108c80 2024-04-25T01:58:46.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.797+0000 7f2610a00640 1 -- 172.21.15.102:0/2764159531 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc035030 con 0x7f260c108c80 2024-04-25T01:58:46.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.797+0000 7f2611e00640 1 -- 172.21.15.102:0/2764159531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f260c1af650 con 0x7f260c108c80 2024-04-25T01:58:46.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.801+0000 7f2614bea740 1 Processor -- start 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.801+0000 7f2614bea740 1 -- start start 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.801+0000 7f2614bea740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 0x55affc4549a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.801+0000 7f2614bea740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.802+0000 7f2614bea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 0x55affc4619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.802+0000 7f2614bea740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55affc3b94e0 con 0x55affc4545c0 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.802+0000 7f2614bea740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55affc361a70 con 0x55affc454ee0 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.802+0000 7f2614bea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55affc3b54a0 con 0x55affc45f5b0 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.802+0000 7f260b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 0x55affc4549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.802+0000 7f260b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 0x55affc4549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58030/0 (socket says 172.21.15.102:58030) 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.802+0000 7f260b400640 1 -- 172.21.15.102:0/987073658 learned_addr learned my addr 172.21.15.102:0/987073658 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.802+0000 7f260a000640 1 --2- 172.21.15.102:0/987073658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 0x55affc4619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.803+0000 7f260a000640 1 -- 172.21.15.102:0/987073658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 msgr2=0x55affc45f070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.803+0000 7f260a000640 1 --2- 172.21.15.102:0/987073658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.803+0000 7f260a000640 1 -- 172.21.15.102:0/987073658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 msgr2=0x55affc4549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.803+0000 7f260a000640 1 --2- 172.21.15.102:0/987073658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 0x55affc4549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.803+0000 7f260a000640 1 -- 172.21.15.102:0/987073658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affc0cd3f0 con 0x55affc45f5b0 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.803+0000 7f260a000640 1 --2- 172.21.15.102:0/987073658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 0x55affc4619a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f260c1a4130 tx=0x7f25f8010c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e865e55ad1d55c2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.803+0000 7f25f7e00640 1 -- 172.21.15.102:0/987073658 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8004d20 con 0x55affc45f5b0 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.804+0000 7f25f7e00640 1 -- 172.21.15.102:0/987073658 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f800ede0 con 0x55affc45f5b0 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.804+0000 7f25f7e00640 1 -- 172.21.15.102:0/987073658 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8005920 con 0x55affc45f5b0 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.804+0000 7f2614bea740 1 -- 172.21.15.102:0/987073658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 msgr2=0x55affc4619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.804+0000 7f2614bea740 1 --2- 172.21.15.102:0/987073658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 0x55affc4619a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f260c1a4130 tx=0x7f25f8010c00 comp rx=0 tx=0).stop 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.804+0000 7f2614bea740 1 -- 172.21.15.102:0/987073658 shutdown_connections 2024-04-25T01:58:46.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.804+0000 7f2614bea740 1 --2- 172.21.15.102:0/987073658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 0x55affc4619a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.804+0000 7f2614bea740 1 --2- 172.21.15.102:0/987073658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.804+0000 7f2614bea740 1 --2- 172.21.15.102:0/987073658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 0x55affc4549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.804+0000 7f2614bea740 1 -- 172.21.15.102:0/987073658 >> 172.21.15.102:0/987073658 conn(0x55affc4508e0 msgr2=0x55affc450cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.805+0000 7f2614bea740 1 -- 172.21.15.102:0/987073658 shutdown_connections 2024-04-25T01:58:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.805+0000 7f2614bea740 1 -- 172.21.15.102:0/987073658 wait complete. 2024-04-25T01:58:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.805+0000 7f2610a00640 1 -- 172.21.15.102:0/2764159531 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f25fc042070 con 0x7f260c108c80 2024-04-25T01:58:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.805+0000 7f2614bea740 1 -- start start 2024-04-25T01:58:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.805+0000 7f2614bea740 1 Processor -- start 2024-04-25T01:58:46.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f2614bea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 0x55affc45c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f2614bea740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f2614bea740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc45f5b0 0x55affc45d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f2614bea740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55affc4623a0 con 0x55affc45f5b0 2024-04-25T01:58:46.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f2614bea740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55affc374ec0 con 0x55affc454ee0 2024-04-25T01:58:46.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f2614bea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55affc462520 con 0x55affc4545c0 2024-04-25T01:58:46.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f2614bea740 10 client.0 _ll_register_callbacks cb 0x55affc3b13f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:46.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f260b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 0x55affc45c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f260b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 0x55affc45c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58652/0 (socket says 172.21.15.102:58652) 2024-04-25T01:58:46.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f260b400640 1 -- 172.21.15.102:0/2253226194 learned_addr learned my addr 172.21.15.102:0/2253226194 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.806+0000 7f260a000640 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc45f5b0 0x55affc45d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.807+0000 7f260b400640 1 -- 172.21.15.102:0/2253226194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 msgr2=0x55affc45caf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:58:46.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.807+0000 7f260b400640 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45caf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.807+0000 7f260b400640 1 -- 172.21.15.102:0/2253226194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc45f5b0 msgr2=0x55affc45d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.807+0000 7f260b400640 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc45f5b0 0x55affc45d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.807+0000 7f260b400640 1 -- 172.21.15.102:0/2253226194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55affc1977d0 con 0x55affc4545c0 2024-04-25T01:58:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.807+0000 7f260b400640 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 0x55affc45c5b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2604032f20 tx=0x7f2604031800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94f0b5273a49f83d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.807+0000 7f25f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.807+0000 7f2610a00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25f00a2d20 0x7f25f00a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.807+0000 7f25f4c00640 1 -- 172.21.15.102:0/2253226194 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604040060 con 0x55affc4545c0 2024-04-25T01:58:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.808+0000 7f2614bea740 1 -- 172.21.15.102:0/2253226194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25f800ede0 con 0x55affc4545c0 2024-04-25T01:58:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.808+0000 7f25f4c00640 1 -- 172.21.15.102:0/2253226194 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f260403b410 con 0x55affc4545c0 2024-04-25T01:58:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.808+0000 7f2610a00640 1 -- 172.21.15.102:0/2764159531 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f25fc0efaf0 con 0x7f260c108c80 2024-04-25T01:58:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.808+0000 7f25f4c00640 1 -- 172.21.15.102:0/2253226194 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f260403b610 con 0x55affc4545c0 2024-04-25T01:58:46.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.808+0000 7f2614bea740 10 client.26206 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:46.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.808+0000 7f2614bea740 1 -- 172.21.15.102:0/2253226194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55affc476f90 con 0x55affc4545c0 2024-04-25T01:58:46.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.808+0000 7f25f4c00640 1 -- 172.21.15.102:0/2253226194 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f800ede0 con 0x55affc4545c0 2024-04-25T01:58:46.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.809+0000 7f260aa00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25f00a2d20 0x7f25f00a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.809+0000 7f25f4c00640 1 -- 172.21.15.102:0/2253226194 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f2604045cb0 con 0x55affc4545c0 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.809+0000 7f2614bea740 10 client.26206 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.809+0000 7f2614bea740 10 client.26206 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.809+0000 7f2614bea740 20 client.26206 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.809+0000 7f2614bea740 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55affc4783a0 0x55affc47a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.809+0000 7f2614bea740 4 client.26206 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.809+0000 7f2614bea740 1 -- 172.21.15.102:0/2253226194 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55affc3c6a20 con 0x55affc4783a0 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.809+0000 7f260a000640 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55affc4783a0 0x55affc47a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.810+0000 7f260aa00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25f00a2d20 0x7f25f00a51e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55affc455b80 tx=0x7f25fc02a480 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.810+0000 7f260a000640 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55affc4783a0 0x55affc47a7d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e103a70de98deb58 server_cookie=c1c82f0ef3f55cf3 in_seq=0 out_seq=0 2024-04-25T01:58:46.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.810+0000 7f25f4c00640 10 client.26206 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:58:46.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.810+0000 7f25f4c00640 1 -- 172.21.15.102:0/2253226194 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55affc3c6a20 con 0x55affc4783a0 2024-04-25T01:58:46.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.810+0000 7f25f4c00640 10 client.26206 handle_command_reply: tid=0 2024-04-25T01:58:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 client.26206 shutdown 2024-04-25T01:58:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 -- 172.21.15.102:0/2253226194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 msgr2=0x55affc45c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 0x55affc45c5b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2604032f20 tx=0x7f2604031800 comp rx=0 tx=0).stop 2024-04-25T01:58:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 -- 172.21.15.102:0/2253226194 shutdown_connections 2024-04-25T01:58:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55affc4783a0 0x55affc47a7d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc45f5b0 0x55affc45d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 --2- 172.21.15.102:0/2253226194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 0x55affc45c5b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f25f4c00640 0 client.26206 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:58:46.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 -- 172.21.15.102:0/2253226194 >> 172.21.15.102:0/2253226194 conn(0x55affc4508e0 msgr2=0x55affc44fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 -- 172.21.15.102:0/2253226194 shutdown_connections 2024-04-25T01:58:46.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.811+0000 7f2614bea740 1 -- 172.21.15.102:0/2253226194 wait complete. 2024-04-25T01:58:46.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.812+0000 7f2614bea740 20 client.26206 trim_cache size 0 max 16384 2024-04-25T01:58:46.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.828+0000 7f2614bea740 1 Processor -- start 2024-04-25T01:58:46.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.828+0000 7f2614bea740 1 -- start start 2024-04-25T01:58:46.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.829+0000 7f2614bea740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 0x55affc473b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.829+0000 7f2614bea740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.829+0000 7f2614bea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 0x55affc45cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.829+0000 7f2614bea740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f260c07a420 con 0x55affc4545c0 2024-04-25T01:58:46.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.829+0000 7f2614bea740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f260c077cc0 con 0x55affc454ee0 2024-04-25T01:58:46.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.829+0000 7f2614bea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f260c075b30 con 0x55affc45f5b0 2024-04-25T01:58:46.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.829+0000 7f260a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.829+0000 7f260a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42616/0 (socket says 172.21.15.102:42616) 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.829+0000 7f260a000640 1 -- 172.21.15.102:0/1540140775 learned_addr learned my addr 172.21.15.102:0/1540140775 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.830+0000 7f260b400640 1 --2- 172.21.15.102:0/1540140775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 0x55affc45cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.830+0000 7f260a000640 1 -- 172.21.15.102:0/1540140775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 msgr2=0x55affc45cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.830+0000 7f260a000640 1 --2- 172.21.15.102:0/1540140775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 0x55affc45cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.830+0000 7f260a000640 1 -- 172.21.15.102:0/1540140775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 msgr2=0x55affc473b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.830+0000 7f260b400640 1 --2- 172.21.15.102:0/1540140775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 0x55affc473b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.830+0000 7f260a000640 1 --2- 172.21.15.102:0/1540140775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 0x55affc473b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.830+0000 7f260a000640 1 -- 172.21.15.102:0/1540140775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affc4543c0 con 0x55affc454ee0 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.831+0000 7f260a000640 1 --2- 172.21.15.102:0/1540140775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45b750 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f25f8011010 tx=0x7f25f8005420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96d78e25680ead9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.831+0000 7f25f7400640 1 -- 172.21.15.102:0/1540140775 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8030070 con 0x55affc454ee0 2024-04-25T01:58:46.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.831+0000 7f25f7400640 1 -- 172.21.15.102:0/1540140775 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f80059c0 con 0x55affc454ee0 2024-04-25T01:58:46.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.831+0000 7f25f7400640 1 -- 172.21.15.102:0/1540140775 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f802a4c0 con 0x55affc454ee0 2024-04-25T01:58:46.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.832+0000 7f2614bea740 1 -- 172.21.15.102:0/1540140775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 msgr2=0x55affc45b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.832+0000 7f2614bea740 1 --2- 172.21.15.102:0/1540140775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45b750 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f25f8011010 tx=0x7f25f8005420 comp rx=0 tx=0).stop 2024-04-25T01:58:46.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.832+0000 7f2614bea740 1 -- 172.21.15.102:0/1540140775 shutdown_connections 2024-04-25T01:58:46.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.832+0000 7f2614bea740 1 --2- 172.21.15.102:0/1540140775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc45f5b0 0x55affc45cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.832+0000 7f2614bea740 1 --2- 172.21.15.102:0/1540140775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc45b750 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.832+0000 7f2614bea740 1 --2- 172.21.15.102:0/1540140775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc4545c0 0x55affc473b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.832+0000 7f2614bea740 1 -- 172.21.15.102:0/1540140775 >> 172.21.15.102:0/1540140775 conn(0x55affc4508e0 msgr2=0x55affc45c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.833+0000 7f2614bea740 1 -- 172.21.15.102:0/1540140775 shutdown_connections 2024-04-25T01:58:46.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.833+0000 7f2614bea740 1 -- 172.21.15.102:0/1540140775 wait complete. 2024-04-25T01:58:46.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.833+0000 7f2614bea740 1 -- start start 2024-04-25T01:58:46.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.834+0000 7f2614bea740 1 Processor -- start 2024-04-25T01:58:46.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f2614bea740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 0x55affc470220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f2614bea740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc458e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f2614bea740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc45f5b0 0x55affc4593a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f2614bea740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f260c076180 con 0x55affc45f5b0 2024-04-25T01:58:46.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f2614bea740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f260c0775f0 con 0x55affc454ee0 2024-04-25T01:58:46.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f2614bea740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f260c076850 con 0x55affc4545c0 2024-04-25T01:58:46.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f2614bea740 10 client.0 _ll_register_callbacks cb 0x7f260400ab10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:46.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f260a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc458e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f260a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc458e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42630/0 (socket says 172.21.15.102:42630) 2024-04-25T01:58:46.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.835+0000 7f260a000640 1 -- 172.21.15.102:0/2117314173 learned_addr learned my addr 172.21.15.102:0/2117314173 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.836+0000 7f260a000640 1 -- 172.21.15.102:0/2117314173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 msgr2=0x55affc470220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.836+0000 7f260a000640 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 0x55affc470220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.836+0000 7f260a000640 1 -- 172.21.15.102:0/2117314173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc45f5b0 msgr2=0x55affc4593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.836+0000 7f260a000640 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc45f5b0 0x55affc4593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.836+0000 7f260a000640 1 -- 172.21.15.102:0/2117314173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55affc4598e0 con 0x55affc454ee0 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.836+0000 7f260a000640 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc458e60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009a00 tx=0x7f25f800ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99733509cf73dbd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.836+0000 7f25f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.837+0000 7f25f4c00640 1 -- 172.21.15.102:0/2117314173 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8039060 con 0x55affc454ee0 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.837+0000 7f2614bea740 1 -- 172.21.15.102:0/2117314173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25f80059c0 con 0x55affc454ee0 2024-04-25T01:58:46.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.837+0000 7f25f4c00640 1 -- 172.21.15.102:0/2117314173 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f25f8005070 con 0x55affc454ee0 2024-04-25T01:58:46.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.837+0000 7f25f4c00640 1 -- 172.21.15.102:0/2117314173 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f25f8030040 con 0x55affc454ee0 2024-04-25T01:58:46.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.837+0000 7f2614bea740 10 client.26243 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:46.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.837+0000 7f2614bea740 1 -- 172.21.15.102:0/2117314173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55affc4de620 con 0x55affc454ee0 2024-04-25T01:58:46.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.838+0000 7f25f4c00640 1 -- 172.21.15.102:0/2117314173 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f80059c0 con 0x55affc454ee0 2024-04-25T01:58:46.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.838+0000 7f25f4c00640 1 -- 172.21.15.102:0/2117314173 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f25f8033030 con 0x55affc454ee0 2024-04-25T01:58:46.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.838+0000 7f2614bea740 10 client.26243 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:46.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.838+0000 7f2614bea740 10 client.26243 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:58:46.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.838+0000 7f2614bea740 20 client.26243 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:46.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.838+0000 7f2614bea740 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55affc4deab0 0x55affc4e0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:46.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.838+0000 7f2614bea740 4 client.26243 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:58:46.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.838+0000 7f2614bea740 1 -- 172.21.15.102:0/2117314173 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55affc3c0d70 con 0x55affc4deab0 2024-04-25T01:58:46.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.839+0000 7f260b400640 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55affc4deab0 0x55affc4e0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:46.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.839+0000 7f260b400640 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55affc4deab0 0x55affc4e0f10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=23c57fa7676a0343 server_cookie=3483d5baa47af093 in_seq=0 out_seq=0 2024-04-25T01:58:46.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.839+0000 7f25f4c00640 10 client.26243 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.839+0000 7f25f4c00640 1 -- 172.21.15.102:0/2117314173 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55affc3c0d70 con 0x55affc4deab0 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.839+0000 7f25f4c00640 10 client.26243 handle_command_reply: tid=0 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.839+0000 7f2614bea740 1 client.26243 shutdown 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 -- 172.21.15.102:0/2117314173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 msgr2=0x55affc458e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc458e60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009a00 tx=0x7f25f800ef90 comp rx=0 tx=0).stop 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 -- 172.21.15.102:0/2117314173 shutdown_connections 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x55affc4deab0 0x55affc4e0f10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55affc45f5b0 0x55affc4593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55affc454ee0 0x55affc458e60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 --2- 172.21.15.102:0/2117314173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55affc4545c0 0x55affc470220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 -- 172.21.15.102:0/2117314173 >> 172.21.15.102:0/2117314173 conn(0x55affc4508e0 msgr2=0x55affc45b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f25f4c00640 0 client.26243 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 -- 172.21.15.102:0/2117314173 shutdown_connections 2024-04-25T01:58:46.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 1 -- 172.21.15.102:0/2117314173 wait complete. 2024-04-25T01:58:46.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.840+0000 7f2614bea740 20 client.26243 trim_cache size 0 max 16384 2024-04-25T01:58:46.845 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.841+0000 7f2611e00640 1 -- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25f00a2d20 msgr2=0x7f25f00a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.841+0000 7f2611e00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25f00a2d20 0x7f25f00a51e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x55affc455b80 tx=0x7f25fc02a480 comp rx=0 tx=0).stop 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.841+0000 7f2611e00640 1 -- 172.21.15.102:0/2764159531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260c108c80 msgr2=0x7f260c1a8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.841+0000 7f2611e00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260c108c80 0x7f260c1a8790 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f25fc004770 tx=0x7f25fc003c10 comp rx=0 tx=0).stop 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.842+0000 7f2611e00640 1 -- 172.21.15.102:0/2764159531 shutdown_connections 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.842+0000 7f2611e00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25f00a2d20 0x7f25f00a51e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.842+0000 7f2611e00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f260c1167d0 0x7f260c1a2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.842+0000 7f2611e00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f260c10be70 0x7f260c1a8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.842+0000 7f2611e00640 1 --2- 172.21.15.102:0/2764159531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260c108c80 0x7f260c1a8790 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.842+0000 7f2611e00640 1 -- 172.21.15.102:0/2764159531 >> 172.21.15.102:0/2764159531 conn(0x7f260c06ccb0 msgr2=0x7f260c116bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.842+0000 7f2611e00640 1 -- 172.21.15.102:0/2764159531 shutdown_connections 2024-04-25T01:58:46.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:46.842+0000 7f2611e00640 1 -- 172.21.15.102:0/2764159531 wait complete. 2024-04-25T01:58:46.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:58:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.118+0000 7f3541c00640 1 -- 172.21.15.102:0/3809302222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c10d040 msgr2=0x7f353c114c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.118+0000 7f3541c00640 1 --2- 172.21.15.102:0/3809302222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c10d040 0x7f353c114c10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3534009ca0 tx=0x7f353402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.118+0000 7f3541c00640 1 -- 172.21.15.102:0/3809302222 shutdown_connections 2024-04-25T01:58:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.118+0000 7f3541c00640 1 --2- 172.21.15.102:0/3809302222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c10d040 0x7f353c114c10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.118+0000 7f3541c00640 1 --2- 172.21.15.102:0/3809302222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f353c108970 0x7f353c10c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.118+0000 7f3541c00640 1 --2- 172.21.15.102:0/3809302222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353c107fc0 0x7f353c1083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.118+0000 7f3541c00640 1 -- 172.21.15.102:0/3809302222 >> 172.21.15.102:0/3809302222 conn(0x7f353c075c80 msgr2=0x7f353c0780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.119+0000 7f3541c00640 1 -- 172.21.15.102:0/3809302222 shutdown_connections 2024-04-25T01:58:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.119+0000 7f3541c00640 1 -- 172.21.15.102:0/3809302222 wait complete. 2024-04-25T01:58:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.120+0000 7f3541c00640 1 Processor -- start 2024-04-25T01:58:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.120+0000 7f3541c00640 1 -- start start 2024-04-25T01:58:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f3541c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c107fc0 0x7f353c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f3541c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353c10d040 0x7f353c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f3541c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f353c1a2ec0 0x7f353c1a3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f3541c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f353c118150 con 0x7f353c10d040 2024-04-25T01:58:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f3541c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f353c117fd0 con 0x7f353c1a2ec0 2024-04-25T01:58:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f3541c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1182d0 con 0x7f353c107fc0 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f353a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353c10d040 0x7f353c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f353aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c107fc0 0x7f353c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f353b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f353c1a2ec0 0x7f353c1a3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f353aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c107fc0 0x7f353c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58686/0 (socket says 172.21.15.102:58686) 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f353aa00640 1 -- 172.21.15.102:0/3228373880 learned_addr learned my addr 172.21.15.102:0/3228373880 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.121+0000 7f353b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f353c1a2ec0 0x7f353c1a3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42648/0 (socket says 172.21.15.102:42648) 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.122+0000 7f353aa00640 1 -- 172.21.15.102:0/3228373880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f353c1a2ec0 msgr2=0x7f353c1a3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.122+0000 7f353aa00640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f353c1a2ec0 0x7f353c1a3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.122+0000 7f353aa00640 1 -- 172.21.15.102:0/3228373880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353c10d040 msgr2=0x7f353c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.122+0000 7f353aa00640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353c10d040 0x7f353c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.122+0000 7f353aa00640 1 -- 172.21.15.102:0/3228373880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353c1a39b0 con 0x7f353c107fc0 2024-04-25T01:58:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.122+0000 7f353b400640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f353c1a2ec0 0x7f353c1a3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.122+0000 7f353aa00640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c107fc0 0x7f353c19e650 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f352c00e9f0 tx=0x7f352c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.123+0000 7f3533400640 1 -- 172.21.15.102:0/3228373880 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c00cd80 con 0x7f353c107fc0 2024-04-25T01:58:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.123+0000 7f3533400640 1 -- 172.21.15.102:0/3228373880 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f352c010910 con 0x7f353c107fc0 2024-04-25T01:58:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.123+0000 7f3533400640 1 -- 172.21.15.102:0/3228373880 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c0188f0 con 0x7f353c107fc0 2024-04-25T01:58:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.123+0000 7f3541c00640 1 -- 172.21.15.102:0/3228373880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353c1a3ca0 con 0x7f353c107fc0 2024-04-25T01:58:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.124+0000 7f3541c00640 1 -- 172.21.15.102:0/3228373880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f353c1ab440 con 0x7f353c107fc0 2024-04-25T01:58:47.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.128+0000 7f3544a82740 1 Processor -- start 2024-04-25T01:58:47.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.129+0000 7f3544a82740 1 -- start start 2024-04-25T01:58:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.129+0000 7f3544a82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd3b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.129+0000 7f3544a82740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 0x55595fd46070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.129+0000 7f3544a82740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd465b0 0x55595fd489a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.129+0000 7f3544a82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55595fca04e0 con 0x55595fd3b5c0 2024-04-25T01:58:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.129+0000 7f3544a82740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55595fc48a70 con 0x55595fd3bee0 2024-04-25T01:58:47.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.129+0000 7f3544a82740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55595fc9c4a0 con 0x55595fd465b0 2024-04-25T01:58:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.130+0000 7f353a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd465b0 0x55595fd489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.130+0000 7f353a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd465b0 0x55595fd489a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58700/0 (socket says 172.21.15.102:58700) 2024-04-25T01:58:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.130+0000 7f353a000640 1 -- 172.21.15.102:0/2613100139 learned_addr learned my addr 172.21.15.102:0/2613100139 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.130+0000 7f3533400640 1 -- 172.21.15.102:0/3228373880 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f352c010430 con 0x7f353c107fc0 2024-04-25T01:58:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.130+0000 7f353b400640 1 --2- 172.21.15.102:0/2613100139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd3b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.131+0000 7f353a000640 1 -- 172.21.15.102:0/2613100139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 msgr2=0x55595fd46070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:58:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.131+0000 7f353a000640 1 --2- 172.21.15.102:0/2613100139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 0x55595fd46070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.131+0000 7f353a000640 1 -- 172.21.15.102:0/2613100139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 msgr2=0x55595fd3b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.131+0000 7f353a000640 1 --2- 172.21.15.102:0/2613100139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd3b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.132+0000 7f353a000640 1 -- 172.21.15.102:0/2613100139 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55595f9b43f0 con 0x55595fd465b0 2024-04-25T01:58:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.132+0000 7f353b400640 1 --2- 172.21.15.102:0/2613100139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd3b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.132+0000 7f353a000640 1 --2- 172.21.15.102:0/2613100139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd465b0 0x55595fd489a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3528009c00 tx=0x7f3528011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cebc5c8ea5d3113d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.133+0000 7f3533400640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f35200a2e50 0x7f35200a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.133+0000 7f3533400640 1 -- 172.21.15.102:0/3228373880 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f352c099d20 con 0x7f353c107fc0 2024-04-25T01:58:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.133+0000 7f3527e00640 1 -- 172.21.15.102:0/2613100139 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528004230 con 0x55595fd465b0 2024-04-25T01:58:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.133+0000 7f3527e00640 1 -- 172.21.15.102:0/2613100139 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35280043d0 con 0x55595fd465b0 2024-04-25T01:58:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.133+0000 7f3527e00640 1 -- 172.21.15.102:0/2613100139 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528004230 con 0x55595fd465b0 2024-04-25T01:58:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.133+0000 7f353b400640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f35200a2e50 0x7f35200a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.134+0000 7f3544a82740 1 -- 172.21.15.102:0/2613100139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd465b0 msgr2=0x55595fd489a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.134+0000 7f3544a82740 1 --2- 172.21.15.102:0/2613100139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd465b0 0x55595fd489a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3528009c00 tx=0x7f3528011810 comp rx=0 tx=0).stop 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.134+0000 7f353b400640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f35200a2e50 0x7f35200a5310 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3534006010 tx=0x7f3534005d80 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.135+0000 7f3544a82740 1 -- 172.21.15.102:0/2613100139 shutdown_connections 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.135+0000 7f3544a82740 1 --2- 172.21.15.102:0/2613100139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd465b0 0x55595fd489a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.135+0000 7f3544a82740 1 --2- 172.21.15.102:0/2613100139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 0x55595fd46070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.135+0000 7f3544a82740 1 --2- 172.21.15.102:0/2613100139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd3b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.135+0000 7f3544a82740 1 -- 172.21.15.102:0/2613100139 >> 172.21.15.102:0/2613100139 conn(0x55595fd378e0 msgr2=0x55595fd37cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.135+0000 7f3544a82740 1 -- 172.21.15.102:0/2613100139 shutdown_connections 2024-04-25T01:58:47.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.135+0000 7f3544a82740 1 -- 172.21.15.102:0/2613100139 wait complete. 2024-04-25T01:58:47.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.136+0000 7f3544a82740 1 -- start start 2024-04-25T01:58:47.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.138+0000 7f3544a82740 1 Processor -- start 2024-04-25T01:58:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f3544a82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd43570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f3544a82740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd43ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f3544a82740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd465b0 0x55595fd43ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f3544a82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55595fd493a0 con 0x55595fd3b5c0 2024-04-25T01:58:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f3544a82740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55595fc5bec0 con 0x55595fd465b0 2024-04-25T01:58:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f3544a82740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55595fd49520 con 0x55595fd3bee0 2024-04-25T01:58:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f3544a82740 10 client.0 _ll_register_callbacks cb 0x55595fc983f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f353a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd43ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f353b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd465b0 0x55595fd43ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f353aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd43570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f353b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd465b0 0x55595fd43ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42664/0 (socket says 172.21.15.102:42664) 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.139+0000 7f353b400640 1 -- 172.21.15.102:0/4178510261 learned_addr learned my addr 172.21.15.102:0/4178510261 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f353b400640 1 -- 172.21.15.102:0/4178510261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 msgr2=0x55595fd43ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f353b400640 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd43ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f353b400640 1 -- 172.21.15.102:0/4178510261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 msgr2=0x55595fd43570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f353b400640 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd43570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f353b400640 1 -- 172.21.15.102:0/4178510261 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55595fa7e7d0 con 0x55595fd465b0 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f353a000640 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd43ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f353aa00640 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd43570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f353b400640 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd465b0 0x55595fd43ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3534003fa0 tx=0x7f353402a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dbb7f334a4cef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:47.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f3524c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f3524c00640 1 -- 172.21.15.102:0/4178510261 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35340319a0 con 0x55595fd465b0 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f3524c00640 1 -- 172.21.15.102:0/4178510261 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3534004250 con 0x55595fd465b0 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f3544a82740 1 -- 172.21.15.102:0/4178510261 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35280043d0 con 0x55595fd465b0 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.140+0000 7f3524c00640 1 -- 172.21.15.102:0/4178510261 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3534004450 con 0x55595fd465b0 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3544a82740 10 client.26252 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3544a82740 1 -- 172.21.15.102:0/4178510261 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55595fd55670 con 0x55595fd465b0 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3524c00640 1 -- 172.21.15.102:0/4178510261 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3534036040 con 0x55595fd465b0 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3524c00640 1 -- 172.21.15.102:0/4178510261 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f3534052020 con 0x55595fd465b0 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3544a82740 10 client.26252 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:47.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3544a82740 10 client.26252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3544a82740 20 client.26252 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3544a82740 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55595fd56ad0 0x55595fd58f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3544a82740 4 client.26252 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f3544a82740 1 -- 172.21.15.102:0/4178510261 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f35280043d0 con 0x55595fd56ad0 2024-04-25T01:58:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.141+0000 7f353aa00640 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55595fd56ad0 0x55595fd58f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.142+0000 7f353aa00640 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55595fd56ad0 0x55595fd58f10 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e347ba6b67245f32 server_cookie=49be181917491477 in_seq=0 out_seq=0 2024-04-25T01:58:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.142+0000 7f3524c00640 10 client.26252 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.143+0000 7f3524c00640 1 -- 172.21.15.102:0/4178510261 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f35280043d0 con 0x55595fd56ad0 2024-04-25T01:58:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.143+0000 7f3524c00640 10 client.26252 handle_command_reply: tid=0 2024-04-25T01:58:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.143+0000 7f3544a82740 1 client.26252 shutdown 2024-04-25T01:58:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.144+0000 7f3544a82740 1 -- 172.21.15.102:0/4178510261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd465b0 msgr2=0x55595fd43ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.144+0000 7f3544a82740 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd465b0 0x55595fd43ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3534003fa0 tx=0x7f353402a8c0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.144+0000 7f3544a82740 1 -- 172.21.15.102:0/4178510261 shutdown_connections 2024-04-25T01:58:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.144+0000 7f3544a82740 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55595fd56ad0 0x55595fd58f10 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.144+0000 7f3544a82740 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd465b0 0x55595fd43ff0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.144+0000 7f3544a82740 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd43ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.144+0000 7f3524c00640 0 client.26252 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.144+0000 7f3544a82740 1 --2- 172.21.15.102:0/4178510261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd3b5c0 0x55595fd43570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.144+0000 7f3544a82740 1 -- 172.21.15.102:0/4178510261 >> 172.21.15.102:0/4178510261 conn(0x55595fd378e0 msgr2=0x55595fd36b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.145+0000 7f3544a82740 1 -- 172.21.15.102:0/4178510261 shutdown_connections 2024-04-25T01:58:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.145+0000 7f3544a82740 1 -- 172.21.15.102:0/4178510261 wait complete. 2024-04-25T01:58:47.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.145+0000 7f3544a82740 20 client.26252 trim_cache size 0 max 16384 2024-04-25T01:58:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.166+0000 7f3544a82740 1 Processor -- start 2024-04-25T01:58:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.166+0000 7f3544a82740 1 -- start start 2024-04-25T01:58:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.166+0000 7f3544a82740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3b5c0 0x55595fd52260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.166+0000 7f3544a82740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 0x55595fd32a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.166+0000 7f3544a82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd465b0 0x55595fd3de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.166+0000 7f3544a82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1b6260 con 0x55595fd465b0 2024-04-25T01:58:47.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.166+0000 7f3544a82740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1b3b00 con 0x55595fd3bee0 2024-04-25T01:58:47.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.166+0000 7f3544a82740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1b1970 con 0x55595fd3b5c0 2024-04-25T01:58:47.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 0x55595fd32a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3b5c0 0x55595fd52260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3b5c0 0x55595fd52260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58730/0 (socket says 172.21.15.102:58730) 2024-04-25T01:58:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353aa00640 1 -- 172.21.15.102:0/3846876401 learned_addr learned my addr 172.21.15.102:0/3846876401 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 0x55595fd32a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42680/0 (socket says 172.21.15.102:42680) 2024-04-25T01:58:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353aa00640 1 -- 172.21.15.102:0/3846876401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 msgr2=0x55595fd32a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353aa00640 1 --2- 172.21.15.102:0/3846876401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 0x55595fd32a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353aa00640 1 -- 172.21.15.102:0/3846876401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd465b0 msgr2=0x55595fd3de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353aa00640 1 --2- 172.21.15.102:0/3846876401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd465b0 0x55595fd3de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.167+0000 7f353aa00640 1 -- 172.21.15.102:0/3846876401 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55595fd42520 con 0x55595fd3b5c0 2024-04-25T01:58:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.168+0000 7f353aa00640 1 --2- 172.21.15.102:0/3846876401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3b5c0 0x55595fd52260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f352c098b90 tx=0x7f352c0a0220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2a6516fbfc29ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.168+0000 7f3527400640 1 -- 172.21.15.102:0/3846876401 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c0986b0 con 0x55595fd3b5c0 2024-04-25T01:58:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.168+0000 7f3527400640 1 -- 172.21.15.102:0/3846876401 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f352c005620 con 0x55595fd3b5c0 2024-04-25T01:58:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.168+0000 7f3527400640 1 -- 172.21.15.102:0/3846876401 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c098850 con 0x55595fd3b5c0 2024-04-25T01:58:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.169+0000 7f3544a82740 1 -- 172.21.15.102:0/3846876401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3b5c0 msgr2=0x55595fd52260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.169+0000 7f3544a82740 1 --2- 172.21.15.102:0/3846876401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3b5c0 0x55595fd52260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f352c098b90 tx=0x7f352c0a0220 comp rx=0 tx=0).stop 2024-04-25T01:58:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.169+0000 7f3544a82740 1 -- 172.21.15.102:0/3846876401 shutdown_connections 2024-04-25T01:58:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.170+0000 7f3544a82740 1 --2- 172.21.15.102:0/3846876401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd465b0 0x55595fd3de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.170+0000 7f3544a82740 1 --2- 172.21.15.102:0/3846876401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd3bee0 0x55595fd32a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.170+0000 7f3544a82740 1 --2- 172.21.15.102:0/3846876401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3b5c0 0x55595fd52260 secure :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f352c098b90 tx=0x7f352c0a0220 comp rx=0 tx=0).stop 2024-04-25T01:58:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.170+0000 7f3544a82740 1 -- 172.21.15.102:0/3846876401 >> 172.21.15.102:0/3846876401 conn(0x55595fd378e0 msgr2=0x55595fd43dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.170+0000 7f3544a82740 1 -- 172.21.15.102:0/3846876401 shutdown_connections 2024-04-25T01:58:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.170+0000 7f3544a82740 1 -- 172.21.15.102:0/3846876401 wait complete. 2024-04-25T01:58:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.170+0000 7f3544a82740 1 -- start start 2024-04-25T01:58:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.171+0000 7f3544a82740 1 Processor -- start 2024-04-25T01:58:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.171+0000 7f3544a82740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd40950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.171+0000 7f3544a82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd465b0 0x55595fd40e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.171+0000 7f3544a82740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd56ad0 0x55595fdbc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.171+0000 7f3544a82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1b1fc0 con 0x55595fd465b0 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.171+0000 7f3544a82740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1b3430 con 0x55595fd56ad0 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.171+0000 7f3544a82740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f353c1b2690 con 0x55595fd3bee0 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.171+0000 7f3544a82740 10 client.0 _ll_register_callbacks cb 0x7f3534009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd40950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd40950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58734/0 (socket says 172.21.15.102:58734) 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353aa00640 1 -- 172.21.15.102:0/333415899 learned_addr learned my addr 172.21.15.102:0/333415899 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353b400640 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd56ad0 0x55595fdbc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353a000640 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd465b0 0x55595fd40e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353aa00640 1 -- 172.21.15.102:0/333415899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd56ad0 msgr2=0x55595fdbc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353aa00640 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd56ad0 0x55595fdbc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353aa00640 1 -- 172.21.15.102:0/333415899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd465b0 msgr2=0x55595fd40e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353aa00640 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd465b0 0x55595fd40e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353aa00640 1 -- 172.21.15.102:0/333415899 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55595fdbc640 con 0x55595fd3bee0 2024-04-25T01:58:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353b400640 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd56ad0 0x55595fdbc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f353aa00640 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd40950 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f352c09b980 tx=0x7f352c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3225d925dd3c8c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f3524c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.172+0000 7f3524c00640 1 -- 172.21.15.102:0/333415899 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c0b8070 con 0x55595fd3bee0 2024-04-25T01:58:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.173+0000 7f3524c00640 1 -- 172.21.15.102:0/333415899 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f352c0979d0 con 0x55595fd3bee0 2024-04-25T01:58:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.173+0000 7f3544a82740 1 -- 172.21.15.102:0/333415899 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f352c005620 con 0x55595fd3bee0 2024-04-25T01:58:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.173+0000 7f3524c00640 1 -- 172.21.15.102:0/333415899 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f352c0989f0 con 0x55595fd3bee0 2024-04-25T01:58:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.173+0000 7f3544a82740 10 client.26227 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.173+0000 7f3544a82740 1 -- 172.21.15.102:0/333415899 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55595fdbd920 con 0x55595fd3bee0 2024-04-25T01:58:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.173+0000 7f3524c00640 1 -- 172.21.15.102:0/333415899 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f352c005620 con 0x55595fd3bee0 2024-04-25T01:58:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.173+0000 7f3524c00640 1 -- 172.21.15.102:0/333415899 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f352c097d50 con 0x55595fd3bee0 2024-04-25T01:58:47.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.174+0000 7f3544a82740 10 client.26227 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.174+0000 7f3544a82740 10 client.26227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.174+0000 7f3544a82740 20 client.26227 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.174+0000 7f3544a82740 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55595fdbdf90 0x55595fdc03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.174+0000 7f3544a82740 4 client.26227 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:58:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.174+0000 7f3544a82740 1 -- 172.21.15.102:0/333415899 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55595fcada20 con 0x55595fdbdf90 2024-04-25T01:58:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.175+0000 7f353a000640 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55595fdbdf90 0x55595fdc03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f353a000640 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55595fdbdf90 0x55595fdc03b0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0542cb2ab3798de server_cookie=8d59f64de50d7915 in_seq=0 out_seq=0 2024-04-25T01:58:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3524c00640 10 client.26227 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3524c00640 1 -- 172.21.15.102:0/333415899 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55595fcada20 con 0x55595fdbdf90 2024-04-25T01:58:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3524c00640 10 client.26227 handle_command_reply: tid=0 2024-04-25T01:58:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3544a82740 1 client.26227 shutdown 2024-04-25T01:58:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3544a82740 1 -- 172.21.15.102:0/333415899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 msgr2=0x55595fd40950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:47.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3544a82740 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd40950 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f352c09b980 tx=0x7f352c000bc0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3544a82740 1 -- 172.21.15.102:0/333415899 shutdown_connections 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3544a82740 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55595fdbdf90 0x55595fdc03b0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3544a82740 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55595fd56ad0 0x55595fdbc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3544a82740 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55595fd465b0 0x55595fd40e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3544a82740 1 --2- 172.21.15.102:0/333415899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55595fd3bee0 0x55595fd40950 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3544a82740 1 -- 172.21.15.102:0/333415899 >> 172.21.15.102:0/333415899 conn(0x55595fd378e0 msgr2=0x55595fd44da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.176+0000 7f3524c00640 0 client.26227 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.177+0000 7f3544a82740 1 -- 172.21.15.102:0/333415899 shutdown_connections 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.177+0000 7f3544a82740 1 -- 172.21.15.102:0/333415899 wait complete. 2024-04-25T01:58:47.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.177+0000 7f3544a82740 20 client.26227 trim_cache size 0 max 16384 2024-04-25T01:58:47.182 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:58:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.178+0000 7f3541c00640 1 -- 172.21.15.102:0/3228373880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f35200a2e50 msgr2=0x7f35200a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.178+0000 7f3541c00640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f35200a2e50 0x7f35200a5310 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3534006010 tx=0x7f3534005d80 comp rx=0 tx=0).stop 2024-04-25T01:58:47.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.178+0000 7f3541c00640 1 -- 172.21.15.102:0/3228373880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c107fc0 msgr2=0x7f353c19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.178+0000 7f3541c00640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c107fc0 0x7f353c19e650 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f352c00e9f0 tx=0x7f352c00eeb0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.179+0000 7f3541c00640 1 -- 172.21.15.102:0/3228373880 shutdown_connections 2024-04-25T01:58:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.179+0000 7f3541c00640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f35200a2e50 0x7f35200a5310 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.179+0000 7f3541c00640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f353c1a2ec0 0x7f353c1a3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.179+0000 7f3541c00640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f353c10d040 0x7f353c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.179+0000 7f3541c00640 1 --2- 172.21.15.102:0/3228373880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f353c107fc0 0x7f353c19e650 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.179+0000 7f3541c00640 1 -- 172.21.15.102:0/3228373880 >> 172.21.15.102:0/3228373880 conn(0x7f353c075c80 msgr2=0x7f353c077fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.179+0000 7f3541c00640 1 -- 172.21.15.102:0/3228373880 shutdown_connections 2024-04-25T01:58:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:47.179+0000 7f3541c00640 1 -- 172.21.15.102:0/3228373880 wait complete. 2024-04-25T01:58:47.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:47 smithi102 ceph-mon[27796]: pgmap v108: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.5 MiB/s wr, 586 op/s 2024-04-25T01:58:47.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:47 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1660828719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:58:47.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:47 smithi110 ceph-mon[32848]: pgmap v108: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.5 MiB/s wr, 586 op/s 2024-04-25T01:58:47.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:47 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1660828719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:58:47.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:47 smithi193 ceph-mon[32981]: pgmap v108: 97 pgs: 97 active+clean; 2.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.5 MiB/s wr, 586 op/s 2024-04-25T01:58:47.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:47 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1660828719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:58:48.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 175.0 s, 2359.8 tps, lat 42.130 ms stddev 51.096 2024-04-25T01:58:48.215 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:58:48.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:58:48.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:58:48.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.502+0000 7feccd600640 1 -- 172.21.15.102:0/1086135754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc8073af0 msgr2=0x7fecc8073ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.502+0000 7feccd600640 1 --2- 172.21.15.102:0/1086135754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc8073af0 0x7fecc8073ed0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009ca0 tx=0x7fecc002a960 comp rx=0 tx=0).stop 2024-04-25T01:58:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.502+0000 7feccd600640 1 -- 172.21.15.102:0/1086135754 shutdown_connections 2024-04-25T01:58:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.502+0000 7feccd600640 1 --2- 172.21.15.102:0/1086135754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc810e020 0x7fecc8110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.502+0000 7feccd600640 1 --2- 172.21.15.102:0/1086135754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc8074410 0x7fecc810dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.502+0000 7feccd600640 1 --2- 172.21.15.102:0/1086135754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc8073af0 0x7fecc8073ed0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.502+0000 7feccd600640 1 -- 172.21.15.102:0/1086135754 >> 172.21.15.102:0/1086135754 conn(0x7fecc80fd250 msgr2=0x7fecc80ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.503+0000 7feccd600640 1 -- 172.21.15.102:0/1086135754 shutdown_connections 2024-04-25T01:58:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.504+0000 7feccd600640 1 -- 172.21.15.102:0/1086135754 wait complete. 2024-04-25T01:58:48.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.505+0000 7feccd600640 1 Processor -- start 2024-04-25T01:58:48.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.505+0000 7feccd600640 1 -- start start 2024-04-25T01:58:48.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7feccd600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc8073af0 0x7fecc819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7feccd600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc8074410 0x7fecc819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7feccd600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc810e020 0x7fecc819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7feccd600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fecc8113df0 con 0x7fecc810e020 2024-04-25T01:58:48.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7feccd600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fecc8113c70 con 0x7fecc8074410 2024-04-25T01:58:48.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7feccd600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fecc8113f70 con 0x7fecc8073af0 2024-04-25T01:58:48.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7fecc6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc8073af0 0x7fecc819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7fecc6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc8074410 0x7fecc819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7fecc6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc8073af0 0x7fecc819a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58744/0 (socket says 172.21.15.102:58744) 2024-04-25T01:58:48.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.506+0000 7fecc6a00640 1 -- 172.21.15.102:0/127728236 learned_addr learned my addr 172.21.15.102:0/127728236 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.507+0000 7fecc6000640 1 -- 172.21.15.102:0/127728236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc8073af0 msgr2=0x7fecc819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.507+0000 7fecc7400640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc810e020 0x7fecc819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.507+0000 7fecc6000640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc8073af0 0x7fecc819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.507+0000 7fecc6000640 1 -- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc810e020 msgr2=0x7fecc819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.507+0000 7fecc6000640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc810e020 0x7fecc819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.507+0000 7fecc6000640 1 -- 172.21.15.102:0/127728236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc819f290 con 0x7fecc8074410 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.507+0000 7fecc7400640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc810e020 0x7fecc819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.507+0000 7fecc6000640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc8074410 0x7fecc819a830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009fd0 tx=0x7fecb800b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.508+0000 7fecbf400640 1 -- 172.21.15.102:0/127728236 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8015020 con 0x7fecc8074410 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.508+0000 7feccd600640 1 -- 172.21.15.102:0/127728236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc819f580 con 0x7fecc8074410 2024-04-25T01:58:48.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.508+0000 7feccd600640 1 -- 172.21.15.102:0/127728236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc8101670 con 0x7fecc8074410 2024-04-25T01:58:48.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.508+0000 7fecbf400640 1 -- 172.21.15.102:0/127728236 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb8011d10 con 0x7fecc8074410 2024-04-25T01:58:48.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.508+0000 7fecbf400640 1 -- 172.21.15.102:0/127728236 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb800f2e0 con 0x7fecc8074410 2024-04-25T01:58:48.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.512+0000 7fecbf400640 1 -- 172.21.15.102:0/127728236 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fecb8018020 con 0x7fecc8074410 2024-04-25T01:58:48.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.515+0000 7fecd045e740 1 Processor -- start 2024-04-25T01:58:48.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.515+0000 7fecd045e740 1 -- start start 2024-04-25T01:58:48.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.515+0000 7fecbf400640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecd045e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483dd60 0x56029483e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecd045e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x560294848810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecd045e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560294848d50 0x56029484b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecd045e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56029479cf40 con 0x560294848d50 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecd045e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5602947a2190 con 0x56029483e680 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecd045e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56029474aa10 con 0x56029483dd60 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecbf400640 1 -- 172.21.15.102:0/127728236 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fecb80cfb60 con 0x7fecc8074410 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecc6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483dd60 0x56029483e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecc6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483dd60 0x56029483e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58752/0 (socket says 172.21.15.102:58752) 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecc6a00640 1 -- 172.21.15.102:0/2367618104 learned_addr learned my addr 172.21.15.102:0/2367618104 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.516+0000 7fecc7400640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.517+0000 7fecc6a00640 1 -- 172.21.15.102:0/2367618104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 msgr2=0x560294848810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.517+0000 7fecc6a00640 1 --2- 172.21.15.102:0/2367618104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x560294848810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.517+0000 7fecc6a00640 1 -- 172.21.15.102:0/2367618104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560294848d50 msgr2=0x56029484b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.517+0000 7fecc6a00640 1 --2- 172.21.15.102:0/2367618104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560294848d50 0x56029484b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.517+0000 7fecc6a00640 1 -- 172.21.15.102:0/2367618104 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56029457e7e0 con 0x56029483dd60 2024-04-25T01:58:48.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.518+0000 7fecc7400640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x56029483b210 tx=0x7fecb0005e20 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.518+0000 7fecc6a00640 1 --2- 172.21.15.102:0/2367618104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483dd60 0x56029483e140 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009dc0 tx=0x7fecc002f760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f5b12796ac94e80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.518+0000 7fecb7400640 1 -- 172.21.15.102:0/2367618104 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0048070 con 0x56029483dd60 2024-04-25T01:58:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.519+0000 7fecb7400640 1 -- 172.21.15.102:0/2367618104 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc0003e50 con 0x56029483dd60 2024-04-25T01:58:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.519+0000 7fecb7400640 1 -- 172.21.15.102:0/2367618104 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0034040 con 0x56029483dd60 2024-04-25T01:58:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.519+0000 7fecd045e740 1 -- 172.21.15.102:0/2367618104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483dd60 msgr2=0x56029483e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.519+0000 7fecd045e740 1 --2- 172.21.15.102:0/2367618104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483dd60 0x56029483e140 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009dc0 tx=0x7fecc002f760 comp rx=0 tx=0).stop 2024-04-25T01:58:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.520+0000 7fecd045e740 1 -- 172.21.15.102:0/2367618104 shutdown_connections 2024-04-25T01:58:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.520+0000 7fecd045e740 1 --2- 172.21.15.102:0/2367618104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560294848d50 0x56029484b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.520+0000 7fecd045e740 1 --2- 172.21.15.102:0/2367618104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x560294848810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.520+0000 7fecd045e740 1 --2- 172.21.15.102:0/2367618104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483dd60 0x56029483e140 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.520+0000 7fecd045e740 1 -- 172.21.15.102:0/2367618104 >> 172.21.15.102:0/2367618104 conn(0x56029483a080 msgr2=0x56029483a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.520+0000 7fecd045e740 1 -- 172.21.15.102:0/2367618104 shutdown_connections 2024-04-25T01:58:48.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.520+0000 7fecd045e740 1 -- 172.21.15.102:0/2367618104 wait complete. 2024-04-25T01:58:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.521+0000 7fecd045e740 1 -- start start 2024-04-25T01:58:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.521+0000 7fecd045e740 1 Processor -- start 2024-04-25T01:58:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.521+0000 7fecd045e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x560294845cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.521+0000 7fecd045e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x5602948461f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.521+0000 7fecd045e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 0x560294846730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.521+0000 7fecd045e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56029475de70 con 0x56029483dd60 2024-04-25T01:58:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.521+0000 7fecd045e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56029479e150 con 0x56029483e680 2024-04-25T01:58:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.521+0000 7fecd045e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56029484bb80 con 0x560294848d50 2024-04-25T01:58:48.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.521+0000 7fecd045e740 10 client.0 _ll_register_callbacks cb 0x5602947991f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:48.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 0x560294846730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x5602948461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 0x560294846730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58766/0 (socket says 172.21.15.102:58766) 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x5602948461f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42730/0 (socket says 172.21.15.102:42730) 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc7400640 1 -- 172.21.15.102:0/1710477275 learned_addr learned my addr 172.21.15.102:0/1710477275 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc6a00640 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x560294845cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc6000640 1 -- 172.21.15.102:0/1710477275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 msgr2=0x560294846730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc6000640 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 0x560294846730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc6000640 1 -- 172.21.15.102:0/1710477275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 msgr2=0x560294845cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc6000640 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x560294845cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.522+0000 7fecc6000640 1 -- 172.21.15.102:0/1710477275 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602945def70 con 0x56029483e680 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.523+0000 7fecc6000640 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x5602948461f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fecb80cd390 tx=0x7fecb809bb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d38cd0563553e5fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.523+0000 7fecabe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:48.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.523+0000 7fecabe00640 1 -- 172.21.15.102:0/1710477275 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb80a4dd0 con 0x56029483e680 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.523+0000 7fecabe00640 1 -- 172.21.15.102:0/1710477275 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fecb80a0160 con 0x56029483e680 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.523+0000 7fecd045e740 1 -- 172.21.15.102:0/1710477275 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecc0003e50 con 0x56029483e680 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.523+0000 7fecabe00640 1 -- 172.21.15.102:0/1710477275 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fecb80cd500 con 0x56029483e680 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.523+0000 7fecd045e740 10 client.26267 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.523+0000 7fecd045e740 1 -- 172.21.15.102:0/1710477275 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560294476170 con 0x56029483e680 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.523+0000 7fecabe00640 1 -- 172.21.15.102:0/1710477275 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecc0003e50 con 0x56029483e680 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.524+0000 7fecabe00640 1 -- 172.21.15.102:0/1710477275 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fecb809b5b0 con 0x56029483e680 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.524+0000 7fecd045e740 10 client.26267 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.524+0000 7fecd045e740 10 client.26267 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.524+0000 7fecd045e740 20 client.26267 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.524+0000 7fecd045e740 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560294858dd0 0x56029485b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.524+0000 7fecd045e740 4 client.26267 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:48.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.524+0000 7fecd045e740 1 -- 172.21.15.102:0/1710477275 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602947af6d0 con 0x560294858dd0 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.524+0000 7fecc6a00640 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560294858dd0 0x56029485b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.525+0000 7fecc6a00640 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560294858dd0 0x56029485b1c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27aec6dce8493ee3 server_cookie=11d15b51483f6dfd in_seq=0 out_seq=0 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.525+0000 7fecabe00640 10 client.26267 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecabe00640 1 -- 172.21.15.102:0/1710477275 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602947af6d0 con 0x560294858dd0 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecabe00640 10 client.26267 handle_command_reply: tid=0 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecd045e740 1 client.26267 shutdown 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecd045e740 1 -- 172.21.15.102:0/1710477275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 msgr2=0x5602948461f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecd045e740 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x5602948461f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fecb80cd390 tx=0x7fecb809bb80 comp rx=0 tx=0).stop 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecd045e740 1 -- 172.21.15.102:0/1710477275 shutdown_connections 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecd045e740 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560294858dd0 0x56029485b1c0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecd045e740 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 0x560294846730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecd045e740 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x5602948461f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecd045e740 1 --2- 172.21.15.102:0/1710477275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x560294845cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecd045e740 1 -- 172.21.15.102:0/1710477275 >> 172.21.15.102:0/1710477275 conn(0x56029483a080 msgr2=0x56029483fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.526+0000 7fecabe00640 0 client.26267 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:48.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.527+0000 7fecd045e740 1 -- 172.21.15.102:0/1710477275 shutdown_connections 2024-04-25T01:58:48.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.527+0000 7fecd045e740 1 -- 172.21.15.102:0/1710477275 wait complete. 2024-04-25T01:58:48.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.527+0000 7fecd045e740 20 client.26267 trim_cache size 0 max 16384 2024-04-25T01:58:48.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.561+0000 7fecd045e740 1 Processor -- start 2024-04-25T01:58:48.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.561+0000 7fecd045e740 1 -- start start 2024-04-25T01:58:48.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.561+0000 7fecd045e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x560294846550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.561+0000 7fecd045e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483e680 0x560294860a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.561+0000 7fecd045e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560294848d50 0x560294860f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.561+0000 7fecd045e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feca4010f20 con 0x56029483dd60 2024-04-25T01:58:48.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.561+0000 7fecd045e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feca400e7c0 con 0x560294848d50 2024-04-25T01:58:48.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.561+0000 7fecd045e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feca400c610 con 0x56029483e680 2024-04-25T01:58:48.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc7400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560294848d50 0x560294860f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc7400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560294848d50 0x560294860f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42742/0 (socket says 172.21.15.102:42742) 2024-04-25T01:58:48.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc7400640 1 -- 172.21.15.102:0/3149363583 learned_addr learned my addr 172.21.15.102:0/3149363583 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc6000640 1 --2- 172.21.15.102:0/3149363583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483e680 0x560294860a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc6a00640 1 --2- 172.21.15.102:0/3149363583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x560294846550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc6000640 1 -- 172.21.15.102:0/3149363583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560294848d50 msgr2=0x560294860f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc6000640 1 --2- 172.21.15.102:0/3149363583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560294848d50 0x560294860f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc6000640 1 -- 172.21.15.102:0/3149363583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 msgr2=0x560294846550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc6000640 1 --2- 172.21.15.102:0/3149363583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x560294846550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.562+0000 7fecc6000640 1 -- 172.21.15.102:0/3149363583 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56029485b820 con 0x56029483e680 2024-04-25T01:58:48.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.563+0000 7fecc6000640 1 --2- 172.21.15.102:0/3149363583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483e680 0x560294860a50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fecb809c0f0 tx=0x7fecb809ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=878098b3795254ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.563+0000 7fecb6a00640 1 -- 172.21.15.102:0/3149363583 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb80b50d0 con 0x56029483e680 2024-04-25T01:58:48.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.563+0000 7fecb6a00640 1 -- 172.21.15.102:0/3149363583 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb8098000 con 0x56029483e680 2024-04-25T01:58:48.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.564+0000 7fecb6a00640 1 -- 172.21.15.102:0/3149363583 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb80bc0b0 con 0x56029483e680 2024-04-25T01:58:48.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.565+0000 7fecd045e740 1 -- 172.21.15.102:0/3149363583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483e680 msgr2=0x560294860a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.565+0000 7fecd045e740 1 --2- 172.21.15.102:0/3149363583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483e680 0x560294860a50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fecb809c0f0 tx=0x7fecb809ba10 comp rx=0 tx=0).stop 2024-04-25T01:58:48.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.565+0000 7fecd045e740 1 -- 172.21.15.102:0/3149363583 shutdown_connections 2024-04-25T01:58:48.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.565+0000 7fecd045e740 1 --2- 172.21.15.102:0/3149363583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560294848d50 0x560294860f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.565+0000 7fecd045e740 1 --2- 172.21.15.102:0/3149363583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029483e680 0x560294860a50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.565+0000 7fecd045e740 1 --2- 172.21.15.102:0/3149363583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x560294846550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.565+0000 7fecd045e740 1 -- 172.21.15.102:0/3149363583 >> 172.21.15.102:0/3149363583 conn(0x56029483a080 msgr2=0x56029483fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.565+0000 7fecd045e740 1 -- 172.21.15.102:0/3149363583 shutdown_connections 2024-04-25T01:58:48.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.565+0000 7fecd045e740 1 -- 172.21.15.102:0/3149363583 wait complete. 2024-04-25T01:58:48.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.566+0000 7fecd045e740 1 -- start start 2024-04-25T01:58:48.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.566+0000 7fecd045e740 1 Processor -- start 2024-04-25T01:58:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.566+0000 7fecd045e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x5602948512c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecd045e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x56029484d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecd045e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 0x56029484dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecd045e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feca400cc80 con 0x56029483dd60 2024-04-25T01:58:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecd045e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7feca400e0f0 con 0x56029483e680 2024-04-25T01:58:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecd045e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feca400d350 con 0x560294848d50 2024-04-25T01:58:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecd045e740 10 client.0 _ll_register_callbacks cb 0x7fecb8011ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecc6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x56029484d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecc6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x56029484d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42748/0 (socket says 172.21.15.102:42748) 2024-04-25T01:58:48.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecc6000640 1 -- 172.21.15.102:0/1267728220 learned_addr learned my addr 172.21.15.102:0/1267728220 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecc7400640 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 0x56029484dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecc6000640 1 -- 172.21.15.102:0/1267728220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 msgr2=0x56029484dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecc6000640 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 0x56029484dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecc6000640 1 -- 172.21.15.102:0/1267728220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 msgr2=0x5602948512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecc6000640 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x5602948512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.567+0000 7fecc6000640 1 -- 172.21.15.102:0/1267728220 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56029484e3f0 con 0x56029483e680 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.568+0000 7fecc6000640 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x56029484d640 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fecb809be60 tx=0x7fecb8010fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=947887fd98706b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.568+0000 7fecabe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.568+0000 7fecabe00640 1 -- 172.21.15.102:0/1267728220 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb807e020 con 0x56029483e680 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.568+0000 7fecabe00640 1 -- 172.21.15.102:0/1267728220 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fecb80bc0b0 con 0x56029483e680 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.568+0000 7fecd045e740 1 -- 172.21.15.102:0/1267728220 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb8098000 con 0x56029483e680 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.569+0000 7fecabe00640 1 -- 172.21.15.102:0/1267728220 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fecb80a4dd0 con 0x56029483e680 2024-04-25T01:58:48.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.569+0000 7fecd045e740 10 client.26273 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:48.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.569+0000 7fecd045e740 1 -- 172.21.15.102:0/1267728220 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5602948d13c0 con 0x56029483e680 2024-04-25T01:58:48.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.569+0000 7fecabe00640 1 -- 172.21.15.102:0/1267728220 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb8098000 con 0x56029483e680 2024-04-25T01:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.569+0000 7fecabe00640 1 -- 172.21.15.102:0/1267728220 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fecb809b1e0 con 0x56029483e680 2024-04-25T01:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.570+0000 7fecd045e740 10 client.26273 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.570+0000 7fecd045e740 10 client.26273 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.570+0000 7fecd045e740 20 client.26273 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.570+0000 7fecd045e740 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5602948d18f0 0x5602948d3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.570+0000 7fecc6a00640 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5602948d18f0 0x5602948d3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.570+0000 7fecd045e740 4 client.26273 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:58:48.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.570+0000 7fecd045e740 1 -- 172.21.15.102:0/1267728220 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5602947a9a20 con 0x5602948d18f0 2024-04-25T01:58:48.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.571+0000 7fecc6a00640 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5602948d18f0 0x5602948d3d10 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5771c0c87c89b3c0 server_cookie=d8e61658eff56fb3 in_seq=0 out_seq=0 2024-04-25T01:58:48.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.571+0000 7fecabe00640 10 client.26273 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:48.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.572+0000 7fecabe00640 1 -- 172.21.15.102:0/1267728220 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5602947a9a20 con 0x5602948d18f0 2024-04-25T01:58:48.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.572+0000 7fecabe00640 10 client.26273 handle_command_reply: tid=0 2024-04-25T01:58:48.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.572+0000 7fecd045e740 1 client.26273 shutdown 2024-04-25T01:58:48.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.573+0000 7fecd045e740 1 -- 172.21.15.102:0/1267728220 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 msgr2=0x56029484d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.573+0000 7fecd045e740 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x56029484d640 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fecb809be60 tx=0x7fecb8010fb0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.574+0000 7fecd045e740 1 -- 172.21.15.102:0/1267728220 shutdown_connections 2024-04-25T01:58:48.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.574+0000 7fecd045e740 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5602948d18f0 0x5602948d3d10 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.574+0000 7fecabe00640 0 client.26273 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:48.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.574+0000 7fecd045e740 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560294848d50 0x56029484dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.574+0000 7fecd045e740 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56029483e680 0x56029484d640 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.574+0000 7fecd045e740 1 --2- 172.21.15.102:0/1267728220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56029483dd60 0x5602948512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.574+0000 7fecd045e740 1 -- 172.21.15.102:0/1267728220 >> 172.21.15.102:0/1267728220 conn(0x56029483a080 msgr2=0x5602948474e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.575+0000 7fecd045e740 1 -- 172.21.15.102:0/1267728220 shutdown_connections 2024-04-25T01:58:48.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.575+0000 7fecd045e740 1 -- 172.21.15.102:0/1267728220 wait complete. 2024-04-25T01:58:48.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.576+0000 7fecd045e740 20 client.26273 trim_cache size 0 max 16384 2024-04-25T01:58:48.579 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:58:48.579 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T01:58:48.579 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "81957b67-2c8b-4414-87a0-f0c92997c295", 2024-04-25T01:58:48.579 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T01:58:48.579 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:58:48.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 -- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 msgr2=0x7fecac0a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x56029483b210 tx=0x7fecb0005e20 comp rx=0 tx=0).stop 2024-04-25T01:58:48.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 -- 172.21.15.102:0/127728236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc8074410 msgr2=0x7fecc819a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc8074410 0x7fecc819a830 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fecb8009fd0 tx=0x7fecb800b870 comp rx=0 tx=0).stop 2024-04-25T01:58:48.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 -- 172.21.15.102:0/127728236 shutdown_connections 2024-04-25T01:58:48.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fecac0a2e00 0x7fecac0a52c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fecc810e020 0x7fecc819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecc8074410 0x7fecc819a830 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 --2- 172.21.15.102:0/127728236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc8073af0 0x7fecc819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.577+0000 7feccd600640 1 -- 172.21.15.102:0/127728236 >> 172.21.15.102:0/127728236 conn(0x7fecc80fd250 msgr2=0x7fecc80fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.578+0000 7feccd600640 1 -- 172.21.15.102:0/127728236 shutdown_connections 2024-04-25T01:58:48.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.578+0000 7feccd600640 1 -- 172.21.15.102:0/127728236 wait complete. 2024-04-25T01:58:48.607 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:58:48.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:58:48.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.877+0000 7f226e600640 1 -- 172.21.15.102:0/466008954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2268101fa0 msgr2=0x7f226810e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.877+0000 7f226e600640 1 --2- 172.21.15.102:0/466008954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2268101fa0 0x7f226810e4b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f2260009ca0 tx=0x7f226002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.877+0000 7f226e600640 1 -- 172.21.15.102:0/466008954 shutdown_connections 2024-04-25T01:58:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.877+0000 7f226e600640 1 --2- 172.21.15.102:0/466008954 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2268101fa0 0x7f226810e4b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.877+0000 7f226e600640 1 --2- 172.21.15.102:0/466008954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22681015e0 0x7f2268101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.877+0000 7f226e600640 1 --2- 172.21.15.102:0/466008954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2268107680 0x7f2268107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.877+0000 7f226e600640 1 -- 172.21.15.102:0/466008954 >> 172.21.15.102:0/466008954 conn(0x7f22680fd110 msgr2=0x7f22680ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.878+0000 7f226e600640 1 -- 172.21.15.102:0/466008954 shutdown_connections 2024-04-25T01:58:48.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.878+0000 7f226e600640 1 -- 172.21.15.102:0/466008954 wait complete. 2024-04-25T01:58:48.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.880+0000 7f226e600640 1 Processor -- start 2024-04-25T01:58:48.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.880+0000 7f226e600640 1 -- start start 2024-04-25T01:58:48.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.880+0000 7f226e600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22681015e0 0x7f2268074f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.880+0000 7f226e600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2268101fa0 0x7f22680754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.880+0000 7f226e600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2268107680 0x7f226806cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.880+0000 7f226e600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f226806ba20 con 0x7f22681015e0 2024-04-25T01:58:48.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.880+0000 7f226e600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f226806b8a0 con 0x7f2268101fa0 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.880+0000 7f226e600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f226806bba0 con 0x7f2268107680 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.881+0000 7f2267e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2268101fa0 0x7f22680754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.881+0000 7f2267e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2268101fa0 0x7f22680754a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42756/0 (socket says 172.21.15.102:42756) 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.881+0000 7f226dc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2268107680 0x7f226806cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.881+0000 7f2267e00640 1 -- 172.21.15.102:0/2217260813 learned_addr learned my addr 172.21.15.102:0/2217260813 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.881+0000 7f226dc00640 1 -- 172.21.15.102:0/2217260813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2268101fa0 msgr2=0x7f22680754a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.881+0000 7f226dc00640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2268101fa0 0x7f22680754a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.881+0000 7f226dc00640 1 -- 172.21.15.102:0/2217260813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22681015e0 msgr2=0x7f2268074f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.881+0000 7f226dc00640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22681015e0 0x7f2268074f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.881+0000 7f226dc00640 1 -- 172.21.15.102:0/2217260813 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226806d830 con 0x7f2268107680 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.882+0000 7f226dc00640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2268107680 0x7f226806cf70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2260009bf0 tx=0x7f2260030d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.883+0000 7f2265600640 1 -- 172.21.15.102:0/2217260813 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2260003970 con 0x7f2268107680 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.883+0000 7f2265600640 1 -- 172.21.15.102:0/2217260813 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2260031450 con 0x7f2268107680 2024-04-25T01:58:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.883+0000 7f2265600640 1 -- 172.21.15.102:0/2217260813 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2260004640 con 0x7f2268107680 2024-04-25T01:58:48.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.883+0000 7f226e600640 1 -- 172.21.15.102:0/2217260813 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f226806dac0 con 0x7f2268107680 2024-04-25T01:58:48.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.884+0000 7f226e600640 1 -- 172.21.15.102:0/2217260813 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22681b3830 con 0x7f2268107680 2024-04-25T01:58:48.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.889+0000 7f2272a6a740 1 Processor -- start 2024-04-25T01:58:48.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.890+0000 7f2265600640 1 -- 172.21.15.102:0/2217260813 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2260003b10 con 0x7f2268107680 2024-04-25T01:58:48.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.890+0000 7f2272a6a740 1 -- start start 2024-04-25T01:58:48.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.892+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42a1e0 0x55706b42a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.892+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b434c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.892+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b4351d0 0x55706b4375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.892+0000 7f2272a6a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55706b38e560 con 0x55706b42a1e0 2024-04-25T01:58:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.892+0000 7f2272a6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55706b336930 con 0x55706b4351d0 2024-04-25T01:58:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.892+0000 7f2272a6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55706b38a520 con 0x55706b42ab00 2024-04-25T01:58:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.892+0000 7f2265600640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f22500a2dc0 0x7f22500a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.893+0000 7f226dc00640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f22500a2dc0 0x7f22500a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.893+0000 7f2267e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42a1e0 0x55706b42a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.893+0000 7f2267e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42a1e0 0x55706b42a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58220/0 (socket says 172.21.15.102:58220) 2024-04-25T01:58:48.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.893+0000 7f2267e00640 1 -- 172.21.15.102:0/2456567153 learned_addr learned my addr 172.21.15.102:0/2456567153 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.893+0000 7f2265600640 1 -- 172.21.15.102:0/2217260813 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f22600f0880 con 0x7f2268107680 2024-04-25T01:58:48.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.894+0000 7f226d200640 1 --2- 172.21.15.102:0/2456567153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b434c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.894+0000 7f2267e00640 1 -- 172.21.15.102:0/2456567153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 msgr2=0x55706b434c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.894+0000 7f2267e00640 1 --2- 172.21.15.102:0/2456567153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b434c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.894+0000 7f2267e00640 1 -- 172.21.15.102:0/2456567153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b4351d0 msgr2=0x55706b4375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.894+0000 7f2267e00640 1 --2- 172.21.15.102:0/2456567153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b4351d0 0x55706b4375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.894+0000 7f2267e00640 1 -- 172.21.15.102:0/2456567153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55706b16c7d0 con 0x55706b42a1e0 2024-04-25T01:58:48.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.894+0000 7f226dc00640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f22500a2dc0 0x7f22500a5280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f22600b98b0 tx=0x7f226002a8c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.895+0000 7f2267e00640 1 --2- 172.21.15.102:0/2456567153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42a1e0 0x55706b42a5c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f225800bde0 tx=0x7f22580107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b30e9e64411625 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.895+0000 7f2256000640 1 -- 172.21.15.102:0/2456567153 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2258004230 con 0x55706b42a1e0 2024-04-25T01:58:48.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.895+0000 7f2256000640 1 -- 172.21.15.102:0/2456567153 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22580043d0 con 0x55706b42a1e0 2024-04-25T01:58:48.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.895+0000 7f2256000640 1 -- 172.21.15.102:0/2456567153 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22580035f0 con 0x55706b42a1e0 2024-04-25T01:58:48.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.896+0000 7f2272a6a740 1 -- 172.21.15.102:0/2456567153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42a1e0 msgr2=0x55706b42a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.896+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2456567153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42a1e0 0x55706b42a5c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f225800bde0 tx=0x7f22580107a0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.896+0000 7f2272a6a740 1 -- 172.21.15.102:0/2456567153 shutdown_connections 2024-04-25T01:58:48.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.896+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2456567153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b4351d0 0x55706b4375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.896+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2456567153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b434c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.896+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2456567153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42a1e0 0x55706b42a5c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.896+0000 7f2272a6a740 1 -- 172.21.15.102:0/2456567153 >> 172.21.15.102:0/2456567153 conn(0x55706b426500 msgr2=0x55706b4268e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.897+0000 7f2272a6a740 1 -- 172.21.15.102:0/2456567153 shutdown_connections 2024-04-25T01:58:48.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.897+0000 7f2272a6a740 1 -- 172.21.15.102:0/2456567153 wait complete. 2024-04-25T01:58:48.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.897+0000 7f2272a6a740 1 -- start start 2024-04-25T01:58:48.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.897+0000 7f2272a6a740 1 Processor -- start 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.898+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b432330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.898+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b4351d0 0x55706b432870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.898+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b432db0 0x55706b4414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.898+0000 7f2272a6a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55706b438010 con 0x55706b432db0 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.898+0000 7f2272a6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55706b349d90 con 0x55706b4351d0 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.898+0000 7f2272a6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55706b438190 con 0x55706b42ab00 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.898+0000 7f2272a6a740 10 client.0 _ll_register_callbacks cb 0x55706b349780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.899+0000 7f2267e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b432330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.899+0000 7f2267e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b432330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58832/0 (socket says 172.21.15.102:58832) 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.899+0000 7f2267e00640 1 -- 172.21.15.102:0/3489853873 learned_addr learned my addr 172.21.15.102:0/3489853873 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.899+0000 7f2267e00640 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b432db0 0x55706b4414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.899+0000 7f2267e00640 1 -- 172.21.15.102:0/3489853873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b4351d0 msgr2=0x55706b432870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.899+0000 7f2267e00640 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b4351d0 0x55706b432870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.899+0000 7f2267e00640 1 -- 172.21.15.102:0/3489853873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b432db0 msgr2=0x55706b4414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.899+0000 7f2267e00640 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b432db0 0x55706b4414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.899+0000 7f2267e00640 1 -- 172.21.15.102:0/3489853873 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55706b0a23f0 con 0x55706b42ab00 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.900+0000 7f2267e00640 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b432330 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2258005940 tx=0x7f2258004b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53e5059ef215599d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.900+0000 7f224aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.900+0000 7f224aa00640 1 -- 172.21.15.102:0/3489853873 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2258026040 con 0x55706b42ab00 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.900+0000 7f224aa00640 1 -- 172.21.15.102:0/3489853873 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f2258005450 con 0x55706b42ab00 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.900+0000 7f2272a6a740 1 -- 172.21.15.102:0/3489853873 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22580043d0 con 0x55706b42ab00 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.900+0000 7f224aa00640 1 -- 172.21.15.102:0/3489853873 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2258033050 con 0x55706b42ab00 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.900+0000 7f2272a6a740 10 client.26254 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.900+0000 7f2272a6a740 1 -- 172.21.15.102:0/3489853873 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55706b0620f0 con 0x55706b42ab00 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.901+0000 7f224aa00640 1 -- 172.21.15.102:0/3489853873 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22580043d0 con 0x55706b42ab00 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.901+0000 7f224aa00640 1 -- 172.21.15.102:0/3489853873 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f225801b470 con 0x55706b42ab00 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.901+0000 7f2272a6a740 10 client.26254 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.901+0000 7f2272a6a740 10 client.26254 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.901+0000 7f2272a6a740 20 client.26254 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.901+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55706b445a60 0x55706b447e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.901+0000 7f2272a6a740 4 client.26254 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.901+0000 7f2272a6a740 1 -- 172.21.15.102:0/3489853873 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55706b3a1d90 con 0x55706b445a60 2024-04-25T01:58:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.902+0000 7f226d200640 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55706b445a60 0x55706b447e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.904+0000 7f226d200640 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55706b445a60 0x55706b447e50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7610c7c0b74cf51 server_cookie=1325ec4df0e28dd7 in_seq=0 out_seq=0 2024-04-25T01:58:48.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.904+0000 7f224aa00640 10 client.26254 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:48.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.905+0000 7f224aa00640 1 -- 172.21.15.102:0/3489853873 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55706b3a1d90 con 0x55706b445a60 2024-04-25T01:58:48.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.905+0000 7f224aa00640 10 client.26254 handle_command_reply: tid=0 2024-04-25T01:58:48.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.905+0000 7f2272a6a740 1 client.26254 shutdown 2024-04-25T01:58:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.905+0000 7f2272a6a740 1 -- 172.21.15.102:0/3489853873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 msgr2=0x55706b432330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b432330 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2258005940 tx=0x7f2258004b30 comp rx=0 tx=0).stop 2024-04-25T01:58:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 1 -- 172.21.15.102:0/3489853873 shutdown_connections 2024-04-25T01:58:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55706b445a60 0x55706b447e50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b432db0 0x55706b4414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b4351d0 0x55706b432870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3489853873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b42ab00 0x55706b432330 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 1 -- 172.21.15.102:0/3489853873 >> 172.21.15.102:0/3489853873 conn(0x55706b426500 msgr2=0x55706b425800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f224aa00640 0 client.26254 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 1 -- 172.21.15.102:0/3489853873 shutdown_connections 2024-04-25T01:58:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 1 -- 172.21.15.102:0/3489853873 wait complete. 2024-04-25T01:58:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.906+0000 7f2272a6a740 20 client.26254 trim_cache size 0 max 16384 2024-04-25T01:58:48.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.931+0000 7f2272a6a740 1 Processor -- start 2024-04-25T01:58:48.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.931+0000 7f2272a6a740 1 -- start start 2024-04-25T01:58:48.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.931+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42ab00 0x55706b44d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.931+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b432db0 0x55706b44dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.931+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b4351d0 0x55706b440f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.931+0000 7f2272a6a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f22681be870 con 0x55706b42ab00 2024-04-25T01:58:48.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.931+0000 7f2272a6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f22681bc110 con 0x55706b432db0 2024-04-25T01:58:48.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.931+0000 7f2272a6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f22681b9f60 con 0x55706b4351d0 2024-04-25T01:58:48.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.932+0000 7f2267e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42ab00 0x55706b44d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.932+0000 7f226d200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b432db0 0x55706b44dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.932+0000 7f226d200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b432db0 0x55706b44dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42790/0 (socket says 172.21.15.102:42790) 2024-04-25T01:58:48.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.932+0000 7f226d200640 1 -- 172.21.15.102:0/3314157955 learned_addr learned my addr 172.21.15.102:0/3314157955 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.932+0000 7f2267e00640 1 --2- 172.21.15.102:0/3314157955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b4351d0 0x55706b440f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.933+0000 7f2267e00640 1 -- 172.21.15.102:0/3314157955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b4351d0 msgr2=0x55706b440f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.933+0000 7f2267e00640 1 --2- 172.21.15.102:0/3314157955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b4351d0 0x55706b440f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.933+0000 7f2267e00640 1 -- 172.21.15.102:0/3314157955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b432db0 msgr2=0x55706b44dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.933+0000 7f2267e00640 1 --2- 172.21.15.102:0/3314157955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b432db0 0x55706b44dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.933+0000 7f2267e00640 1 -- 172.21.15.102:0/3314157955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55706b430880 con 0x55706b42ab00 2024-04-25T01:58:48.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.933+0000 7f2267e00640 1 --2- 172.21.15.102:0/3314157955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42ab00 0x55706b44d670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f225800bc90 tx=0x7f2258005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c94ba901ea1326 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.933+0000 7f2255600640 1 -- 172.21.15.102:0/3314157955 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225801ced0 con 0x55706b42ab00 2024-04-25T01:58:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.933+0000 7f2255600640 1 -- 172.21.15.102:0/3314157955 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2258020070 con 0x55706b42ab00 2024-04-25T01:58:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.933+0000 7f2255600640 1 -- 172.21.15.102:0/3314157955 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225801bce0 con 0x55706b42ab00 2024-04-25T01:58:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.934+0000 7f2272a6a740 1 -- 172.21.15.102:0/3314157955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42ab00 msgr2=0x55706b44d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.934+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3314157955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42ab00 0x55706b44d670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f225800bc90 tx=0x7f2258005880 comp rx=0 tx=0).stop 2024-04-25T01:58:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.934+0000 7f2272a6a740 1 -- 172.21.15.102:0/3314157955 shutdown_connections 2024-04-25T01:58:48.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.934+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3314157955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b4351d0 0x55706b440f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.934+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3314157955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b432db0 0x55706b44dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.934+0000 7f2272a6a740 1 --2- 172.21.15.102:0/3314157955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b42ab00 0x55706b44d670 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.934+0000 7f2272a6a740 1 -- 172.21.15.102:0/3314157955 >> 172.21.15.102:0/3314157955 conn(0x55706b426500 msgr2=0x55706b430c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.935+0000 7f2272a6a740 1 -- 172.21.15.102:0/3314157955 shutdown_connections 2024-04-25T01:58:48.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.935+0000 7f2272a6a740 1 -- 172.21.15.102:0/3314157955 wait complete. 2024-04-25T01:58:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.935+0000 7f2272a6a740 1 -- start start 2024-04-25T01:58:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.935+0000 7f2272a6a740 1 Processor -- start 2024-04-25T01:58:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b42ab00 0x55706b43d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b432db0 0x55706b43d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2272a6a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b4351d0 0x55706b42e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2272a6a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f22681ba5d0 con 0x55706b4351d0 2024-04-25T01:58:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2272a6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f22681bba40 con 0x55706b42ab00 2024-04-25T01:58:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2272a6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f22681baca0 con 0x55706b432db0 2024-04-25T01:58:48.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2272a6a740 10 client.0 _ll_register_callbacks cb 0x7f2258004790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:48.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2267e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b42ab00 0x55706b43d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2267e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b42ab00 0x55706b43d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42796/0 (socket says 172.21.15.102:42796) 2024-04-25T01:58:48.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f2267e00640 1 -- 172.21.15.102:0/2943311338 learned_addr learned my addr 172.21.15.102:0/2943311338 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.936+0000 7f226d200640 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b432db0 0x55706b43d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.937+0000 7f2267e00640 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b4351d0 0x55706b42e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.937+0000 7f2267e00640 1 -- 172.21.15.102:0/2943311338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b432db0 msgr2=0x55706b43d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.937+0000 7f2267e00640 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b432db0 0x55706b43d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.937+0000 7f2267e00640 1 -- 172.21.15.102:0/2943311338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b4351d0 msgr2=0x55706b42e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.937+0000 7f2267e00640 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b4351d0 0x55706b42e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.937+0000 7f2267e00640 1 -- 172.21.15.102:0/2943311338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55706b42ec20 con 0x55706b42ab00 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.937+0000 7f226d200640 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b432db0 0x55706b43d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.937+0000 7f2267e00640 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b42ab00 0x55706b43d410 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f225801d040 tx=0x7f2258030040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f629af007ca3cde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.937+0000 7f224aa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.938+0000 7f224aa00640 1 -- 172.21.15.102:0/2943311338 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225804b060 con 0x55706b42ab00 2024-04-25T01:58:48.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.938+0000 7f224aa00640 1 -- 172.21.15.102:0/2943311338 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f2258005230 con 0x55706b42ab00 2024-04-25T01:58:48.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.938+0000 7f2272a6a740 1 -- 172.21.15.102:0/2943311338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2258020070 con 0x55706b42ab00 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.938+0000 7f224aa00640 1 -- 172.21.15.102:0/2943311338 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f225801b670 con 0x55706b42ab00 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.938+0000 7f2272a6a740 10 client.26285 fetch_fsmap learned FSMap version 28 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.938+0000 7f2272a6a740 1 -- 172.21.15.102:0/2943311338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55706b4ab810 con 0x55706b42ab00 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.938+0000 7f224aa00640 1 -- 172.21.15.102:0/2943311338 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2258020070 con 0x55706b42ab00 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.939+0000 7f224aa00640 1 -- 172.21.15.102:0/2943311338 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f2258011760 con 0x55706b42ab00 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.939+0000 7f2272a6a740 10 client.26285 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.939+0000 7f2272a6a740 10 client.26285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.939+0000 7f2272a6a740 20 client.26285 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.939+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55706b4abe20 0x55706b4ae240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.939+0000 7f2272a6a740 4 client.26285 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.939+0000 7f2272a6a740 1 -- 172.21.15.102:0/2943311338 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55706b39baa0 con 0x55706b4abe20 2024-04-25T01:58:48.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.940+0000 7f226d200640 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55706b4abe20 0x55706b4ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:48.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.941+0000 7f224aa00640 10 client.26285 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:48.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.941+0000 7f226d200640 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55706b4abe20 0x55706b4ae240 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f669502b1e7afc52 server_cookie=607f77d59f2c8f71 in_seq=0 out_seq=0 2024-04-25T01:58:48.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f224aa00640 1 -- 172.21.15.102:0/2943311338 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55706b39baa0 con 0x55706b4abe20 2024-04-25T01:58:48.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f224aa00640 10 client.26285 handle_command_reply: tid=0 2024-04-25T01:58:48.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f2272a6a740 1 client.26285 shutdown 2024-04-25T01:58:48.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f2272a6a740 1 -- 172.21.15.102:0/2943311338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b42ab00 msgr2=0x55706b43d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b42ab00 0x55706b43d410 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f225801d040 tx=0x7f2258030040 comp rx=0 tx=0).stop 2024-04-25T01:58:48.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f2272a6a740 1 -- 172.21.15.102:0/2943311338 shutdown_connections 2024-04-25T01:58:48.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55706b4abe20 0x55706b4ae240 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f224aa00640 0 client.26285 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:48.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55706b4351d0 0x55706b42e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55706b432db0 0x55706b43d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f2272a6a740 1 --2- 172.21.15.102:0/2943311338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55706b42ab00 0x55706b43d410 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.942+0000 7f2272a6a740 1 -- 172.21.15.102:0/2943311338 >> 172.21.15.102:0/2943311338 conn(0x55706b426500 msgr2=0x55706b42b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.943+0000 7f2272a6a740 1 -- 172.21.15.102:0/2943311338 shutdown_connections 2024-04-25T01:58:48.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.943+0000 7f2272a6a740 1 -- 172.21.15.102:0/2943311338 wait complete. 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.943+0000 7f2272a6a740 20 client.26285 trim_cache size 0 max 16384 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stdout: "81957b67-2c8b-4414-87a0-f0c92997c295": { 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "81957b67-2c8b-4414-87a0-f0c92997c295", 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:58:48.947 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:58:48.948 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:58:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 -- 172.21.15.102:0/2217260813 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f22500a2dc0 msgr2=0x7f22500a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f22500a2dc0 0x7f22500a5280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f22600b98b0 tx=0x7f226002a8c0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 -- 172.21.15.102:0/2217260813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2268107680 msgr2=0x7f226806cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2268107680 0x7f226806cf70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2260009bf0 tx=0x7f2260030d70 comp rx=0 tx=0).stop 2024-04-25T01:58:48.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 -- 172.21.15.102:0/2217260813 shutdown_connections 2024-04-25T01:58:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f22500a2dc0 0x7f22500a5280 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2268107680 0x7f226806cf70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2268101fa0 0x7f22680754a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 --2- 172.21.15.102:0/2217260813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22681015e0 0x7f2268074f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.944+0000 7f226e600640 1 -- 172.21.15.102:0/2217260813 >> 172.21.15.102:0/2217260813 conn(0x7f22680fd110 msgr2=0x7f22680fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.945+0000 7f226e600640 1 -- 172.21.15.102:0/2217260813 shutdown_connections 2024-04-25T01:58:48.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:58:48.945+0000 7f226e600640 1 -- 172.21.15.102:0/2217260813 wait complete. 2024-04-25T01:58:48.970 INFO:tasks.cephfs.filesystem:scrub status for tag:81957b67-2c8b-4414-87a0-f0c92997c295 - {'path': '/', 'tag': '81957b67-2c8b-4414-87a0-f0c92997c295', 'options': 'recursive,force'} 2024-04-25T01:58:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: pgmap v109: 97 pgs: 97 active+clean; 2.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 10 MiB/s wr, 923 op/s 2024-04-25T01:58:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T01:58:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:49.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:49.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:58:49.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:49 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: pgmap v109: 97 pgs: 97 active+clean; 2.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 10 MiB/s wr, 923 op/s 2024-04-25T01:58:49.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T01:58:49.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:49.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:49.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:58:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:49 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: pgmap v109: 97 pgs: 97 active+clean; 2.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 10 MiB/s wr, 923 op/s 2024-04-25T01:58:49.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T01:58:49.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:49.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:49.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:49.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:49.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:49.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:58:49.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:49.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:49 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:58:51.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:51 smithi102 ceph-mon[27796]: pgmap v110: 97 pgs: 97 active+clean; 2.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.6 MiB/s wr, 619 op/s 2024-04-25T01:58:51.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:51 smithi110 ceph-mon[32848]: pgmap v110: 97 pgs: 97 active+clean; 2.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.6 MiB/s wr, 619 op/s 2024-04-25T01:58:51.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:51 smithi193 ceph-mon[32981]: pgmap v110: 97 pgs: 97 active+clean; 2.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.6 MiB/s wr, 619 op/s 2024-04-25T01:58:53.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 180.0 s, 2893.2 tps, lat 34.353 ms stddev 41.112 2024-04-25T01:58:53.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:53 smithi102 ceph-mon[27796]: pgmap v111: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 951 op/s 2024-04-25T01:58:53.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:53 smithi110 ceph-mon[32848]: pgmap v111: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 951 op/s 2024-04-25T01:58:53.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:53 smithi193 ceph-mon[32981]: pgmap v111: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 10 MiB/s wr, 951 op/s 2024-04-25T01:58:54.425 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:54 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:58:54.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:54 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:58:54.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:54 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:58:55.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:55 smithi102 ceph-mon[27796]: pgmap v112: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 8.3 MiB/s wr, 760 op/s 2024-04-25T01:58:55.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:55 smithi110 ceph-mon[32848]: pgmap v112: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 8.3 MiB/s wr, 760 op/s 2024-04-25T01:58:55.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:55 smithi193 ceph-mon[32981]: pgmap v112: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 8.3 MiB/s wr, 760 op/s 2024-04-25T01:58:57.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:57 smithi102 ceph-mon[27796]: pgmap v113: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 7.8 MiB/s wr, 709 op/s 2024-04-25T01:58:57.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:57 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:58:57.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:57 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:58:57.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:57 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:58:57.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:57 smithi110 ceph-mon[32848]: pgmap v113: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 7.8 MiB/s wr, 709 op/s 2024-04-25T01:58:57.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:57 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:58:57.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:57 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:58:57.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:57 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:58:57.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:57 smithi193 ceph-mon[32981]: pgmap v113: 97 pgs: 97 active+clean; 2.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 7.8 MiB/s wr, 709 op/s 2024-04-25T01:58:57.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:57 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:58:57.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:57 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:58:57.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:57 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:58:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 185.0 s, 3952.3 tps, lat 25.575 ms stddev 30.683 2024-04-25T01:58:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.385+0000 7fe41ea00640 1 -- 172.21.15.102:0/3139700750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4181125c0 msgr2=0x7fe4181149b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.385+0000 7fe41ea00640 1 --2- 172.21.15.102:0/3139700750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4181125c0 0x7fe4181149b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe410009c10 tx=0x7fe41002a700 comp rx=0 tx=0).stop 2024-04-25T01:58:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.387+0000 7fe41ea00640 1 -- 172.21.15.102:0/3139700750 shutdown_connections 2024-04-25T01:58:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.387+0000 7fe41ea00640 1 --2- 172.21.15.102:0/3139700750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4181125c0 0x7fe4181149b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.387+0000 7fe41ea00640 1 --2- 172.21.15.102:0/3139700750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe418076590 0x7fe418112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.387+0000 7fe41ea00640 1 --2- 172.21.15.102:0/3139700750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe418075c70 0x7fe418076050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.387+0000 7fe41ea00640 1 -- 172.21.15.102:0/3139700750 >> 172.21.15.102:0/3139700750 conn(0x7fe4180ff430 msgr2=0x7fe418101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.387+0000 7fe41ea00640 1 -- 172.21.15.102:0/3139700750 shutdown_connections 2024-04-25T01:58:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.387+0000 7fe41ea00640 1 -- 172.21.15.102:0/3139700750 wait complete. 2024-04-25T01:58:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.389+0000 7fe41ea00640 1 Processor -- start 2024-04-25T01:58:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.390+0000 7fe41ea00640 1 -- start start 2024-04-25T01:58:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.390+0000 7fe41ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe418075c70 0x7fe418072e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.390+0000 7fe41ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe418076590 0x7fe418073360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.390+0000 7fe41ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4181125c0 0x7fe41806c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.390+0000 7fe41ea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181180e0 con 0x7fe4181125c0 2024-04-25T01:58:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.390+0000 7fe41ea00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe418117f60 con 0x7fe418075c70 2024-04-25T01:58:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.390+0000 7fe41ea00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe418118260 con 0x7fe418076590 2024-04-25T01:58:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.391+0000 7fe417e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe418075c70 0x7fe418072e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.391+0000 7fe417e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe418075c70 0x7fe418072e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56364/0 (socket says 172.21.15.102:56364) 2024-04-25T01:58:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.391+0000 7fe417e00640 1 -- 172.21.15.102:0/4052102359 learned_addr learned my addr 172.21.15.102:0/4052102359 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.392+0000 7fe417e00640 1 -- 172.21.15.102:0/4052102359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe418076590 msgr2=0x7fe418073360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.392+0000 7fe417e00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe418076590 0x7fe418073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.392+0000 7fe417e00640 1 -- 172.21.15.102:0/4052102359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4181125c0 msgr2=0x7fe41806c010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:58:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.392+0000 7fe417e00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4181125c0 0x7fe41806c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.392+0000 7fe417e00640 1 -- 172.21.15.102:0/4052102359 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe41806c8d0 con 0x7fe418075c70 2024-04-25T01:58:58.397 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.395+0000 7fe417e00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe418075c70 0x7fe418072e20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe40800d950 tx=0x7fe40800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.395+0000 7fe414c00640 1 -- 172.21.15.102:0/4052102359 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe408017070 con 0x7fe418075c70 2024-04-25T01:58:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.395+0000 7fe414c00640 1 -- 172.21.15.102:0/4052102359 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe408009c30 con 0x7fe418075c70 2024-04-25T01:58:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.395+0000 7fe414c00640 1 -- 172.21.15.102:0/4052102359 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40800f6b0 con 0x7fe418075c70 2024-04-25T01:58:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.395+0000 7fe41ea00640 1 -- 172.21.15.102:0/4052102359 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe41806cbc0 con 0x7fe418075c70 2024-04-25T01:58:58.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.395+0000 7fe41ea00640 1 -- 172.21.15.102:0/4052102359 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe418070b20 con 0x7fe418075c70 2024-04-25T01:58:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.404+0000 7fe42180f740 1 Processor -- start 2024-04-25T01:58:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.406+0000 7fe42180f740 1 -- start start 2024-04-25T01:58:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.407+0000 7fe42180f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304be00 0x563fc304c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.407+0000 7fe42180f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304c720 0x563fc3056aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.407+0000 7fe42180f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30593d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.407+0000 7fe42180f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x563fc2fafab0 con 0x563fc304c720 2024-04-25T01:58:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.407+0000 7fe42180f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563fc2f58640 con 0x563fc304be00 2024-04-25T01:58:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.407+0000 7fe42180f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563fc2faba70 con 0x563fc3056fe0 2024-04-25T01:58:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.409+0000 7fe417400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.409+0000 7fe414c00640 1 -- 172.21.15.102:0/4052102359 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe40800f850 con 0x7fe418075c70 2024-04-25T01:58:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.409+0000 7fe417400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:39154/0 (socket says 172.21.15.102:39154) 2024-04-25T01:58:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.409+0000 7fe417400640 1 -- 172.21.15.102:0/3824440156 learned_addr learned my addr 172.21.15.102:0/3824440156 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.409+0000 7fe41ca00640 1 --2- 172.21.15.102:0/3824440156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304be00 0x563fc304c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.411+0000 7fe417400640 1 -- 172.21.15.102:0/3824440156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304be00 msgr2=0x563fc304c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.411+0000 7fe417400640 1 --2- 172.21.15.102:0/3824440156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304be00 0x563fc304c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.411+0000 7fe417400640 1 -- 172.21.15.102:0/3824440156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304c720 msgr2=0x563fc3056aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.411+0000 7fe417400640 1 --2- 172.21.15.102:0/3824440156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304c720 0x563fc3056aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.411+0000 7fe417400640 1 -- 172.21.15.102:0/3824440156 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc2cc4210 con 0x563fc3056fe0 2024-04-25T01:58:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.412+0000 7fe417400640 1 --2- 172.21.15.102:0/3824440156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30593d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe400005940 tx=0x7fe40000ae20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e02d8fb61dc2e1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.413+0000 7fe40d200640 1 -- 172.21.15.102:0/3824440156 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40000cd80 con 0x563fc3056fe0 2024-04-25T01:58:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.413+0000 7fe40d200640 1 -- 172.21.15.102:0/3824440156 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4000035f0 con 0x563fc3056fe0 2024-04-25T01:58:58.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.413+0000 7fe40d200640 1 -- 172.21.15.102:0/3824440156 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40000f690 con 0x563fc3056fe0 2024-04-25T01:58:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.414+0000 7fe42180f740 1 -- 172.21.15.102:0/3824440156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 msgr2=0x563fc30593d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.414+0000 7fe42180f740 1 --2- 172.21.15.102:0/3824440156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30593d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe400005940 tx=0x7fe40000ae20 comp rx=0 tx=0).stop 2024-04-25T01:58:58.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.416+0000 7fe414c00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe3f80a2f80 0x7fe3f80a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.416+0000 7fe414c00640 1 -- 172.21.15.102:0/4052102359 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe4080cf0b0 con 0x7fe418075c70 2024-04-25T01:58:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.417+0000 7fe41ca00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe3f80a2f80 0x7fe3f80a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.421+0000 7fe42180f740 1 -- 172.21.15.102:0/3824440156 shutdown_connections 2024-04-25T01:58:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.421+0000 7fe42180f740 1 --2- 172.21.15.102:0/3824440156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30593d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.421+0000 7fe42180f740 1 --2- 172.21.15.102:0/3824440156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304c720 0x563fc3056aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.421+0000 7fe42180f740 1 --2- 172.21.15.102:0/3824440156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304be00 0x563fc304c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.421+0000 7fe42180f740 1 -- 172.21.15.102:0/3824440156 >> 172.21.15.102:0/3824440156 conn(0x563fc3048310 msgr2=0x563fc30486f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.421+0000 7fe42180f740 1 -- 172.21.15.102:0/3824440156 shutdown_connections 2024-04-25T01:58:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.422+0000 7fe42180f740 1 -- 172.21.15.102:0/3824440156 wait complete. 2024-04-25T01:58:58.425 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.423+0000 7fe42180f740 1 -- start start 2024-04-25T01:58:58.426 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.423+0000 7fe42180f740 1 Processor -- start 2024-04-25T01:58:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe42180f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe42180f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3054550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe42180f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc3056fe0 0x563fc3054a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe417400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3054550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe417e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe417400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3054550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56378/0 (socket says 172.21.15.102:56378) 2024-04-25T01:58:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe417e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:39164/0 (socket says 172.21.15.102:39164) 2024-04-25T01:58:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe417400640 1 -- 172.21.15.102:0/526478303 learned_addr learned my addr 172.21.15.102:0/526478303 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x563fc3059c10 con 0x563fc3056fe0 2024-04-25T01:58:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563fc2f6b990 con 0x563fc304c720 2024-04-25T01:58:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.426+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563fc3059d90 con 0x563fc304be00 2024-04-25T01:58:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.427+0000 7fe42180f740 10 client.0 _ll_register_callbacks cb 0x563fc2f6d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.427+0000 7fe417400640 1 -- 172.21.15.102:0/526478303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 msgr2=0x563fc3054010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.427+0000 7fe417400640 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.427+0000 7fe417400640 1 -- 172.21.15.102:0/526478303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc3056fe0 msgr2=0x563fc3054a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.427+0000 7fe417400640 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc3056fe0 0x563fc3054a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.427+0000 7fe417400640 1 -- 172.21.15.102:0/526478303 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc2d90620 con 0x563fc304c720 2024-04-25T01:58:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.427+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4000035f0 con 0x563fc304c720 2024-04-25T01:58:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.427+0000 7fe417400640 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3054550 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe400006010 tx=0x7fe4000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=776b8abd0d16945f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.427+0000 7fe405600640 10 client.26294 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.428+0000 7fe405600640 1 -- 172.21.15.102:0/526478303 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400028060 con 0x563fc304c720 2024-04-25T01:58:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.428+0000 7fe405600640 1 -- 172.21.15.102:0/526478303 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe400004230 con 0x563fc304c720 2024-04-25T01:58:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.429+0000 7fe405600640 1 -- 172.21.15.102:0/526478303 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe40000a5b0 con 0x563fc304c720 2024-04-25T01:58:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.429+0000 7fe41ca00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe3f80a2f80 0x7fe3f80a5440 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe410004af0 tx=0x7fe4100315c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.430+0000 7fe42180f740 10 client.26294 fetch_fsmap learned FSMap version 29 2024-04-25T01:58:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.430+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=29}) -- 0x563fc3065840 con 0x563fc304c720 2024-04-25T01:58:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.433+0000 7fe405600640 1 -- 172.21.15.102:0/526478303 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fe40000f450 con 0x563fc304c720 2024-04-25T01:58:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.433+0000 7fe42180f740 10 client.26294 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:58:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.433+0000 7fe42180f740 10 client.26294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.433+0000 7fe42180f740 20 client.26294 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.433+0000 7fe42180f740 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563fc3066fb0 0x563fc3069430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.433+0000 7fe42180f740 4 client.26294 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:58:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.433+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fc2fbcff0 con 0x563fc3066fb0 2024-04-25T01:58:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.434+0000 7fe417e00640 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563fc3066fb0 0x563fc3069430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.434+0000 7fe417e00640 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563fc3066fb0 0x563fc3069430 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eca89071aaa79a32 server_cookie=e32776785b0b61d3 in_seq=0 out_seq=0 2024-04-25T01:58:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.434+0000 7fe405600640 10 client.26294 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.435+0000 7fe405600640 1 -- 172.21.15.102:0/526478303 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563fc2fbcff0 con 0x563fc3066fb0 2024-04-25T01:58:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.435+0000 7fe405600640 10 client.26294 handle_command_reply: tid=0 2024-04-25T01:58:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.436+0000 7fe42180f740 1 client.26294 shutdown 2024-04-25T01:58:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.437+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 msgr2=0x563fc3054550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.437+0000 7fe42180f740 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3054550 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe400006010 tx=0x7fe4000046c0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.437+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 shutdown_connections 2024-04-25T01:58:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.437+0000 7fe42180f740 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563fc3066fb0 0x563fc3069430 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.438+0000 7fe405600640 0 client.26294 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.438+0000 7fe42180f740 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc3056fe0 0x563fc3054a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.438+0000 7fe42180f740 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3054550 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.438+0000 7fe42180f740 1 --2- 172.21.15.102:0/526478303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.438+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 >> 172.21.15.102:0/526478303 conn(0x563fc3048310 msgr2=0x563fc304e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.439+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 shutdown_connections 2024-04-25T01:58:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.441+0000 7fe42180f740 1 -- 172.21.15.102:0/526478303 wait complete. 2024-04-25T01:58:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.442+0000 7fe42180f740 20 client.26294 trim_cache size 0 max 16384 2024-04-25T01:58:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.483+0000 7fe42180f740 1 Processor -- start 2024-04-25T01:58:58.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.484+0000 7fe42180f740 1 -- start start 2024-04-25T01:58:58.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.484+0000 7fe42180f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.484+0000 7fe42180f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc304e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.484+0000 7fe42180f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc3056fe0 0x563fc306ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.484+0000 7fe42180f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181beab0 con 0x563fc3056fe0 2024-04-25T01:58:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.484+0000 7fe42180f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181bc380 con 0x563fc304c720 2024-04-25T01:58:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.484+0000 7fe42180f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181ba1d0 con 0x563fc304be00 2024-04-25T01:58:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.484+0000 7fe417400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc304e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.485+0000 7fe417400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc304e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56388/0 (socket says 172.21.15.102:56388) 2024-04-25T01:58:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.485+0000 7fe417400640 1 -- 172.21.15.102:0/4223158517 learned_addr learned my addr 172.21.15.102:0/4223158517 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.485+0000 7fe417e00640 1 --2- 172.21.15.102:0/4223158517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.485+0000 7fe41ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc3056fe0 0x563fc306ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.486+0000 7fe417e00640 1 -- 172.21.15.102:0/4223158517 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 msgr2=0x563fc304e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.486+0000 7fe417e00640 1 --2- 172.21.15.102:0/4223158517 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc304e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.486+0000 7fe417e00640 1 -- 172.21.15.102:0/4223158517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc3056fe0 msgr2=0x563fc306ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.486+0000 7fe417e00640 1 --2- 172.21.15.102:0/4223158517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc3056fe0 0x563fc306ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.486+0000 7fe417e00640 1 -- 172.21.15.102:0/4223158517 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc3046890 con 0x563fc304be00 2024-04-25T01:58:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.486+0000 7fe417400640 1 --2- 172.21.15.102:0/4223158517 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc304e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:58:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.486+0000 7fe417e00640 1 --2- 172.21.15.102:0/4223158517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054860 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe4080cc7d0 tx=0x7fe408000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c951bef3865b2be1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.486+0000 7fe407e00640 1 -- 172.21.15.102:0/4223158517 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4080b3070 con 0x563fc304be00 2024-04-25T01:58:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.487+0000 7fe407e00640 1 -- 172.21.15.102:0/4223158517 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe40809f950 con 0x563fc304be00 2024-04-25T01:58:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.487+0000 7fe407e00640 1 -- 172.21.15.102:0/4223158517 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4080b3070 con 0x563fc304be00 2024-04-25T01:58:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.488+0000 7fe42180f740 1 -- 172.21.15.102:0/4223158517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 msgr2=0x563fc3054860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.488+0000 7fe42180f740 1 --2- 172.21.15.102:0/4223158517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054860 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe4080cc7d0 tx=0x7fe408000bc0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.489+0000 7fe42180f740 1 -- 172.21.15.102:0/4223158517 shutdown_connections 2024-04-25T01:58:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.489+0000 7fe42180f740 1 --2- 172.21.15.102:0/4223158517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc3056fe0 0x563fc306ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.489+0000 7fe42180f740 1 --2- 172.21.15.102:0/4223158517 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc304e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.489+0000 7fe42180f740 1 --2- 172.21.15.102:0/4223158517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc304be00 0x563fc3054860 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.489+0000 7fe42180f740 1 -- 172.21.15.102:0/4223158517 >> 172.21.15.102:0/4223158517 conn(0x563fc3048310 msgr2=0x563fc3046bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.489+0000 7fe42180f740 1 -- 172.21.15.102:0/4223158517 shutdown_connections 2024-04-25T01:58:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.489+0000 7fe42180f740 1 -- 172.21.15.102:0/4223158517 wait complete. 2024-04-25T01:58:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.490+0000 7fe42180f740 1 -- start start 2024-04-25T01:58:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.490+0000 7fe42180f740 1 Processor -- start 2024-04-25T01:58:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.492+0000 7fe42180f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304be00 0x563fc3050e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe42180f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3051370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe42180f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30518b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe42180f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181ba840 con 0x563fc304be00 2024-04-25T01:58:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe42180f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181bbcb0 con 0x563fc304c720 2024-04-25T01:58:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe42180f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181baf10 con 0x563fc3056fe0 2024-04-25T01:58:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe42180f740 10 client.0 _ll_register_callbacks cb 0x7fe410034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:58:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe417e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304be00 0x563fc3050e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe417e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304be00 0x563fc3050e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60860/0 (socket says 172.21.15.102:60860) 2024-04-25T01:58:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe417e00640 1 -- 172.21.15.102:0/4023613693 learned_addr learned my addr 172.21.15.102:0/4023613693 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:58:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.494+0000 7fe417400640 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3051370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.495+0000 7fe41ca00640 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.495+0000 7fe417400640 1 -- 172.21.15.102:0/4023613693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 msgr2=0x563fc30518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.495+0000 7fe417400640 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.495+0000 7fe417400640 1 -- 172.21.15.102:0/4023613693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304be00 msgr2=0x563fc3050e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:58:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.496+0000 7fe417400640 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304be00 0x563fc3050e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.496+0000 7fe417400640 1 -- 172.21.15.102:0/4023613693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc30cc640 con 0x563fc304c720 2024-04-25T01:58:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.496+0000 7fe417e00640 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304be00 0x563fc3050e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:58:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.496+0000 7fe405600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:58:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.496+0000 7fe417400640 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3051370 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x563fc2fabbc0 tx=0x7fe400000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecce641710e478b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:58:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.497+0000 7fe405600640 1 -- 172.21.15.102:0/4023613693 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40000ce90 con 0x563fc304c720 2024-04-25T01:58:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.497+0000 7fe42180f740 1 -- 172.21.15.102:0/4023613693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe40809f950 con 0x563fc304c720 2024-04-25T01:58:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.497+0000 7fe405600640 1 -- 172.21.15.102:0/4023613693 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe40001f040 con 0x563fc304c720 2024-04-25T01:58:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.498+0000 7fe405600640 1 -- 172.21.15.102:0/4023613693 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe40000fa20 con 0x563fc304c720 2024-04-25T01:58:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.498+0000 7fe42180f740 10 client.26300 fetch_fsmap learned FSMap version 29 2024-04-25T01:58:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.498+0000 7fe42180f740 1 -- 172.21.15.102:0/4023613693 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563fc30cd950 con 0x563fc304c720 2024-04-25T01:58:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.499+0000 7fe405600640 1 -- 172.21.15.102:0/4023613693 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe40809f950 con 0x563fc304c720 2024-04-25T01:58:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.499+0000 7fe405600640 1 -- 172.21.15.102:0/4023613693 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fe40002d420 con 0x563fc304c720 2024-04-25T01:58:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.500+0000 7fe42180f740 10 client.26300 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T01:58:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.500+0000 7fe42180f740 10 client.26300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:58:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.500+0000 7fe42180f740 20 client.26300 populate_metadata read hostname 'smithi102' 2024-04-25T01:58:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.500+0000 7fe42180f740 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563fc3066fb0 0x563fc30cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:58:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.500+0000 7fe42180f740 4 client.26300 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:58:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.500+0000 7fe42180f740 1 -- 172.21.15.102:0/4023613693 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563fc2fb7340 con 0x563fc3066fb0 2024-04-25T01:58:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.501+0000 7fe417e00640 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563fc3066fb0 0x563fc30cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:58:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.502+0000 7fe417e00640 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563fc3066fb0 0x563fc30cfff0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b86478d3b6e2339d server_cookie=3f69d068fa3cbdb2 in_seq=0 out_seq=0 2024-04-25T01:58:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.502+0000 7fe405600640 10 client.26300 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.517+0000 7fe405600640 1 -- 172.21.15.102:0/4023613693 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563fc2fb7340 con 0x563fc3066fb0 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.517+0000 7fe405600640 10 client.26300 handle_command_reply: tid=0 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 client.26300 shutdown 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 -- 172.21.15.102:0/4023613693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 msgr2=0x563fc3051370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3051370 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x563fc2fabbc0 tx=0x7fe400000bc0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 -- 172.21.15.102:0/4023613693 shutdown_connections 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563fc3066fb0 0x563fc30cfff0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563fc3056fe0 0x563fc30518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563fc304c720 0x563fc3051370 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 --2- 172.21.15.102:0/4023613693 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563fc304be00 0x563fc3050e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe405600640 0 client.26300 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:58:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 -- 172.21.15.102:0/4023613693 >> 172.21.15.102:0/4023613693 conn(0x563fc3048310 msgr2=0x563fc30546e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 -- 172.21.15.102:0/4023613693 shutdown_connections 2024-04-25T01:58:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.518+0000 7fe42180f740 1 -- 172.21.15.102:0/4023613693 wait complete. 2024-04-25T01:58:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.519+0000 7fe42180f740 20 client.26300 trim_cache size 0 max 16384 2024-04-25T01:58:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:58:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:58:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:58:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:58:58.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.520+0000 7fe41ea00640 1 -- 172.21.15.102:0/4052102359 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe3f80a2f80 msgr2=0x7fe3f80a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.520+0000 7fe41ea00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe3f80a2f80 0x7fe3f80a5440 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe410004af0 tx=0x7fe4100315c0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.520+0000 7fe41ea00640 1 -- 172.21.15.102:0/4052102359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe418075c70 msgr2=0x7fe418072e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:58:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.520+0000 7fe41ea00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe418075c70 0x7fe418072e20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe40800d950 tx=0x7fe40800de10 comp rx=0 tx=0).stop 2024-04-25T01:58:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.520+0000 7fe41ea00640 1 -- 172.21.15.102:0/4052102359 shutdown_connections 2024-04-25T01:58:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.520+0000 7fe41ea00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe3f80a2f80 0x7fe3f80a5440 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.520+0000 7fe41ea00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4181125c0 0x7fe41806c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.520+0000 7fe41ea00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe418076590 0x7fe418073360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.521+0000 7fe41ea00640 1 --2- 172.21.15.102:0/4052102359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe418075c70 0x7fe418072e20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:58:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.521+0000 7fe41ea00640 1 -- 172.21.15.102:0/4052102359 >> 172.21.15.102:0/4052102359 conn(0x7fe4180ff430 msgr2=0x7fe418101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:58:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.521+0000 7fe41ea00640 1 -- 172.21.15.102:0/4052102359 shutdown_connections 2024-04-25T01:58:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:58:58.521+0000 7fe41ea00640 1 -- 172.21.15.102:0/4052102359 wait complete. 2024-04-25T01:58:59.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:58:59 smithi110 ceph-mon[32848]: pgmap v114: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-25T01:58:59.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:58:59 smithi193 ceph-mon[32981]: pgmap v114: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-25T01:58:59.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:58:59 smithi102 ceph-mon[27796]: pgmap v114: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-25T01:59:01.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:01 smithi110 ceph-mon[32848]: pgmap v115: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 7.3 MiB/s wr, 689 op/s 2024-04-25T01:59:01.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:01 smithi193 ceph-mon[32981]: pgmap v115: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 7.3 MiB/s wr, 689 op/s 2024-04-25T01:59:01.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:01 smithi102 ceph-mon[27796]: pgmap v115: 97 pgs: 97 active+clean; 2.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 7.3 MiB/s wr, 689 op/s 2024-04-25T01:59:02.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:02 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:59:02.507 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:02 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:59:02.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:02 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:59:03.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 190.0 s, 1525.6 tps, lat 46.170 ms stddev 113.303 2024-04-25T01:59:03.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:03 smithi110 ceph-mon[32848]: pgmap v116: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.9 MiB/s wr, 931 op/s 2024-04-25T01:59:03.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:03 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:59:03.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:03 smithi193 ceph-mon[32981]: pgmap v116: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.9 MiB/s wr, 931 op/s 2024-04-25T01:59:03.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:03 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:59:03.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:03 smithi102 ceph-mon[27796]: pgmap v116: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.9 MiB/s wr, 931 op/s 2024-04-25T01:59:03.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:03 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:59:05.490 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:05 smithi110 ceph-mon[32848]: pgmap v117: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.7 MiB/s wr, 727 op/s 2024-04-25T01:59:05.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:05 smithi193 ceph-mon[32981]: pgmap v117: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.7 MiB/s wr, 727 op/s 2024-04-25T01:59:05.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:05 smithi102 ceph-mon[27796]: pgmap v117: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 8.7 MiB/s wr, 727 op/s 2024-04-25T01:59:07.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:07 smithi193 ceph-mon[32981]: pgmap v118: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.5 MiB/s wr, 687 op/s 2024-04-25T01:59:07.675 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:07 smithi102 ceph-mon[27796]: pgmap v118: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.5 MiB/s wr, 687 op/s 2024-04-25T01:59:07.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:07 smithi110 ceph-mon[32848]: pgmap v118: 97 pgs: 97 active+clean; 2.3 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.5 MiB/s wr, 687 op/s 2024-04-25T01:59:08.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 195.0 s, 2768.0 tps, lat 46.620 ms stddev 144.966 2024-04-25T01:59:09.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:09 smithi193 ceph-mon[32981]: pgmap v119: 97 pgs: 97 active+clean; 2.6 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 39 MiB/s wr, 1.75k op/s 2024-04-25T01:59:09.542 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:09 smithi110 ceph-mon[32848]: pgmap v119: 97 pgs: 97 active+clean; 2.6 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 39 MiB/s wr, 1.75k op/s 2024-04-25T01:59:09.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:09 smithi102 ceph-mon[27796]: pgmap v119: 97 pgs: 97 active+clean; 2.6 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 39 MiB/s wr, 1.75k op/s 2024-04-25T01:59:11.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:11 smithi193 ceph-mon[32981]: pgmap v120: 97 pgs: 97 active+clean; 2.6 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 35 MiB/s wr, 1.43k op/s 2024-04-25T01:59:11.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:11 smithi102 ceph-mon[27796]: pgmap v120: 97 pgs: 97 active+clean; 2.6 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 35 MiB/s wr, 1.43k op/s 2024-04-25T01:59:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.716+0000 7f032a400640 1 -- 172.21.15.102:0/1742326242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03241125e0 msgr2=0x7f03241149d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.716+0000 7f032a400640 1 --2- 172.21.15.102:0/1742326242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03241125e0 0x7f03241149d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f031c009c10 tx=0x7f031c02a6f0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.716+0000 7f032a400640 1 -- 172.21.15.102:0/1742326242 shutdown_connections 2024-04-25T01:59:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.716+0000 7f032a400640 1 --2- 172.21.15.102:0/1742326242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03241125e0 0x7f03241149d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.716+0000 7f032a400640 1 --2- 172.21.15.102:0/1742326242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0324076590 0x7f03241120a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.716+0000 7f032a400640 1 --2- 172.21.15.102:0/1742326242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0324075c70 0x7f0324076050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.716+0000 7f032a400640 1 -- 172.21.15.102:0/1742326242 >> 172.21.15.102:0/1742326242 conn(0x7f03240ff470 msgr2=0x7f0324101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.717+0000 7f032a400640 1 -- 172.21.15.102:0/1742326242 shutdown_connections 2024-04-25T01:59:11.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.717+0000 7f032a400640 1 -- 172.21.15.102:0/1742326242 wait complete. 2024-04-25T01:59:11.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.718+0000 7f032a400640 1 Processor -- start 2024-04-25T01:59:11.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.718+0000 7f032a400640 1 -- start start 2024-04-25T01:59:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.719+0000 7f032a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0324075c70 0x7f032419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.719+0000 7f032a400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0324076590 0x7f032419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.719+0000 7f032a400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03241125e0 0x7f03241a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.719+0000 7f032a400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0324118100 con 0x7f0324076590 2024-04-25T01:59:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.719+0000 7f032a400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0324117f80 con 0x7f03241125e0 2024-04-25T01:59:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.719+0000 7f032a400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0324118280 con 0x7f0324075c70 2024-04-25T01:59:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.719+0000 7f0323400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0324075c70 0x7f032419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.719+0000 7f0323400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0324075c70 0x7f032419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:39926/0 (socket says 172.21.15.102:39926) 2024-04-25T01:59:11.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.719+0000 7f0323400640 1 -- 172.21.15.102:0/1633434039 learned_addr learned my addr 172.21.15.102:0/1633434039 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.720+0000 7f0323e00640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03241125e0 0x7f03241a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.720+0000 7f0323400640 1 -- 172.21.15.102:0/1633434039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03241125e0 msgr2=0x7f03241a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.720+0000 7f0323400640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03241125e0 0x7f03241a2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.720+0000 7f0323400640 1 -- 172.21.15.102:0/1633434039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0324076590 msgr2=0x7f032419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.720+0000 7f0323400640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0324076590 0x7f032419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.720+0000 7f0323400640 1 -- 172.21.15.102:0/1633434039 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03241a35c0 con 0x7f0324075c70 2024-04-25T01:59:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.721+0000 7f0323400640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0324075c70 0x7f032419e620 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f031400e9f0 tx=0x7f031400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.721+0000 7f031be00640 1 -- 172.21.15.102:0/1633434039 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031400cd80 con 0x7f0324075c70 2024-04-25T01:59:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.721+0000 7f031be00640 1 -- 172.21.15.102:0/1633434039 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0314010910 con 0x7f0324075c70 2024-04-25T01:59:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.721+0000 7f031be00640 1 -- 172.21.15.102:0/1633434039 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0314018990 con 0x7f0324075c70 2024-04-25T01:59:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.721+0000 7f032a400640 1 -- 172.21.15.102:0/1633434039 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03241a38b0 con 0x7f0324075c70 2024-04-25T01:59:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.722+0000 7f032a400640 1 -- 172.21.15.102:0/1633434039 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f032406b160 con 0x7f0324075c70 2024-04-25T01:59:11.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.727+0000 7f031be00640 1 -- 172.21.15.102:0/1633434039 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0314010ac0 con 0x7f0324075c70 2024-04-25T01:59:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.730+0000 7f031be00640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03080a2ef0 0x7f03080a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.731+0000 7f031be00640 1 -- 172.21.15.102:0/1633434039 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f03140cf850 con 0x7f0324075c70 2024-04-25T01:59:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.731+0000 7f0322a00640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03080a2ef0 0x7f03080a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.732+0000 7f0322a00640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03080a2ef0 0x7f03080a53b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0324078800 tx=0x7f0310009520 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.733+0000 7f032d1a6740 1 Processor -- start 2024-04-25T01:59:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f032d1a6740 1 -- start start 2024-04-25T01:59:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbd0720 0x559eabbdaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbdd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f032d1a6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559eabb33ab0 con 0x559eabbcfe00 2024-04-25T01:59:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f032d1a6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559eabadc640 con 0x559eabbdafe0 2024-04-25T01:59:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f032d1a6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559eabb2fa70 con 0x559eabbd0720 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f0322a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbd0720 0x559eabbdaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f0322a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbd0720 0x559eabbdaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:39936/0 (socket says 172.21.15.102:39936) 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.734+0000 7f0322a00640 1 -- 172.21.15.102:0/2893623472 learned_addr learned my addr 172.21.15.102:0/2893623472 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0323e00640 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbdd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0322a00640 1 -- 172.21.15.102:0/2893623472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 msgr2=0x559eabbdd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0322a00640 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbdd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0322a00640 1 -- 172.21.15.102:0/2893623472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 msgr2=0x559eabbd01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0323400640 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0322a00640 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0322a00640 1 -- 172.21.15.102:0/2893623472 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559eab848210 con 0x559eabbd0720 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0323e00640 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbdd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0323400640 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.735+0000 7f0322a00640 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbd0720 0x559eabbdaaa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0310004420 tx=0x7f0310011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=589b885db3bee6a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:11.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:11 smithi110 ceph-mon[32848]: pgmap v120: 97 pgs: 97 active+clean; 2.6 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 35 MiB/s wr, 1.43k op/s 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.736+0000 7f0318c00640 1 -- 172.21.15.102:0/2893623472 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310021070 con 0x559eabbd0720 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.736+0000 7f0318c00640 1 -- 172.21.15.102:0/2893623472 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0310003b70 con 0x559eabbd0720 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.736+0000 7f0318c00640 1 -- 172.21.15.102:0/2893623472 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310010450 con 0x559eabbd0720 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.737+0000 7f032d1a6740 1 -- 172.21.15.102:0/2893623472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbd0720 msgr2=0x559eabbdaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.737+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbd0720 0x559eabbdaaa0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0310004420 tx=0x7f0310011000 comp rx=0 tx=0).stop 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.737+0000 7f032d1a6740 1 -- 172.21.15.102:0/2893623472 shutdown_connections 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.737+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbdd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.737+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbd0720 0x559eabbdaaa0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.737+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2893623472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.737+0000 7f032d1a6740 1 -- 172.21.15.102:0/2893623472 >> 172.21.15.102:0/2893623472 conn(0x559eabbcc310 msgr2=0x559eabbcc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.738+0000 7f032d1a6740 1 -- 172.21.15.102:0/2893623472 shutdown_connections 2024-04-25T01:59:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.738+0000 7f032d1a6740 1 -- 172.21.15.102:0/2893623472 wait complete. 2024-04-25T01:59:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.739+0000 7f032d1a6740 1 -- start start 2024-04-25T01:59:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.739+0000 7f032d1a6740 1 Processor -- start 2024-04-25T01:59:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.739+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbd7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.740+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbd0720 0x559eabbd84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.740+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbd8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.740+0000 7f032d1a6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559eabbddc10 con 0x559eabbd0720 2024-04-25T01:59:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.740+0000 7f0323e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbd8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.740+0000 7f0323e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbd8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:55658/0 (socket says 172.21.15.102:55658) 2024-04-25T01:59:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.740+0000 7f0323e00640 1 -- 172.21.15.102:0/1154467619 learned_addr learned my addr 172.21.15.102:0/1154467619 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.740+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559eabaef990 con 0x559eabbdafe0 2024-04-25T01:59:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.740+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559eabbddd90 con 0x559eabbcfe00 2024-04-25T01:59:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.741+0000 7f032d1a6740 10 client.0 _ll_register_callbacks cb 0x559eabaf1950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.741+0000 7f0323400640 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbd7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.741+0000 7f0322a00640 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbd0720 0x559eabbd84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.741+0000 7f0323e00640 1 -- 172.21.15.102:0/1154467619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 msgr2=0x559eabbd7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.741+0000 7f0323e00640 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbd7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.741+0000 7f0323e00640 1 -- 172.21.15.102:0/1154467619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbd0720 msgr2=0x559eabbd84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.741+0000 7f0323e00640 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbd0720 0x559eabbd84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.741+0000 7f0323e00640 1 -- 172.21.15.102:0/1154467619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559eab914620 con 0x559eabbdafe0 2024-04-25T01:59:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.741+0000 7f0323400640 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbd7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.742+0000 7f0323e00640 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbd8a10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f031c009b30 tx=0x7f031c0305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=389ecd0ea0d6b0b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.742+0000 7f030d600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.742+0000 7f030d600640 1 -- 172.21.15.102:0/1154467619 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c040070 con 0x559eabbdafe0 2024-04-25T01:59:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.742+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0310003b70 con 0x559eabbdafe0 2024-04-25T01:59:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.742+0000 7f030d600640 1 -- 172.21.15.102:0/1154467619 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f031c003970 con 0x559eabbdafe0 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.743+0000 7f030d600640 1 -- 172.21.15.102:0/1154467619 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f031c04a050 con 0x559eabbdafe0 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.743+0000 7f032d1a6740 10 client.26306 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.743+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559eabbe9aa0 con 0x559eabbdafe0 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.743+0000 7f030d600640 1 -- 172.21.15.102:0/1154467619 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0310003b70 con 0x559eabbdafe0 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.743+0000 7f030d600640 1 -- 172.21.15.102:0/1154467619 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f031c0045b0 con 0x559eabbdafe0 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.744+0000 7f032d1a6740 10 client.26306 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.744+0000 7f032d1a6740 10 client.26306 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.744+0000 7f032d1a6740 20 client.26306 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.744+0000 7f032d1a6740 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559eabbeaf10 0x559eabbed300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.744+0000 7f032d1a6740 4 client.26306 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.744+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559eabb40ff0 con 0x559eabbeaf10 2024-04-25T01:59:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.744+0000 7f0323400640 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559eabbeaf10 0x559eabbed300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.745+0000 7f0323400640 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559eabbeaf10 0x559eabbed300 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bb2f5b135754849 server_cookie=4693d1320aca6ab in_seq=0 out_seq=0 2024-04-25T01:59:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.745+0000 7f030d600640 10 client.26306 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.746+0000 7f030d600640 1 -- 172.21.15.102:0/1154467619 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559eabb40ff0 con 0x559eabbeaf10 2024-04-25T01:59:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.746+0000 7f030d600640 10 client.26306 handle_command_reply: tid=0 2024-04-25T01:59:11.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.746+0000 7f032d1a6740 1 client.26306 shutdown 2024-04-25T01:59:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.747+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 msgr2=0x559eabbd8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.747+0000 7f032d1a6740 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbd8a10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f031c009b30 tx=0x7f031c0305d0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.747+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 shutdown_connections 2024-04-25T01:59:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.747+0000 7f032d1a6740 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559eabbeaf10 0x559eabbed300 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.747+0000 7f032d1a6740 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbdafe0 0x559eabbd8a10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.747+0000 7f032d1a6740 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbd0720 0x559eabbd84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.747+0000 7f032d1a6740 1 --2- 172.21.15.102:0/1154467619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbd7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.747+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 >> 172.21.15.102:0/1154467619 conn(0x559eabbcc310 msgr2=0x559eabbd2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.748+0000 7f030d600640 0 client.26306 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:11.752 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.748+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 shutdown_connections 2024-04-25T01:59:11.752 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.748+0000 7f032d1a6740 1 -- 172.21.15.102:0/1154467619 wait complete. 2024-04-25T01:59:11.752 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.748+0000 7f032d1a6740 20 client.26306 trim_cache size 0 max 16384 2024-04-25T01:59:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.783+0000 7f032d1a6740 1 Processor -- start 2024-04-25T01:59:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.784+0000 7f032d1a6740 1 -- start start 2024-04-25T01:59:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.785+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbe6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.785+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.785+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbdafe0 0x559eabbd87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.785+0000 7f032d1a6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03241bec50 con 0x559eabbdafe0 2024-04-25T01:59:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.785+0000 7f032d1a6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03241bc4f0 con 0x559eabbd0720 2024-04-25T01:59:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.785+0000 7f032d1a6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0324071b00 con 0x559eabbcfe00 2024-04-25T01:59:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0323400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbe6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0323400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbe6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:39946/0 (socket says 172.21.15.102:39946) 2024-04-25T01:59:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0323400640 1 -- 172.21.15.102:0/3617642215 learned_addr learned my addr 172.21.15.102:0/3617642215 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0323e00640 1 --2- 172.21.15.102:0/3617642215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbdafe0 0x559eabbd87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0322a00640 1 --2- 172.21.15.102:0/3617642215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0323400640 1 -- 172.21.15.102:0/3617642215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 msgr2=0x559eabbd2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0323400640 1 --2- 172.21.15.102:0/3617642215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0323400640 1 -- 172.21.15.102:0/3617642215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbdafe0 msgr2=0x559eabbd87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0323400640 1 --2- 172.21.15.102:0/3617642215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbdafe0 0x559eabbd87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.786+0000 7f0323400640 1 -- 172.21.15.102:0/3617642215 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559eabbd6f40 con 0x559eabbcfe00 2024-04-25T01:59:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.787+0000 7f0323400640 1 --2- 172.21.15.102:0/3617642215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbe6ad0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f031409bc80 tx=0x7f0314009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9817773ff8d0a00d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.787+0000 7f030fe00640 1 -- 172.21.15.102:0/3617642215 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031409a040 con 0x559eabbcfe00 2024-04-25T01:59:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.787+0000 7f030fe00640 1 -- 172.21.15.102:0/3617642215 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03140983b0 con 0x559eabbcfe00 2024-04-25T01:59:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.787+0000 7f030fe00640 1 -- 172.21.15.102:0/3617642215 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031409b190 con 0x559eabbcfe00 2024-04-25T01:59:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.789+0000 7f032d1a6740 1 -- 172.21.15.102:0/3617642215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 msgr2=0x559eabbe6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.789+0000 7f032d1a6740 1 --2- 172.21.15.102:0/3617642215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbe6ad0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f031409bc80 tx=0x7f0314009700 comp rx=0 tx=0).stop 2024-04-25T01:59:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.790+0000 7f032d1a6740 1 -- 172.21.15.102:0/3617642215 shutdown_connections 2024-04-25T01:59:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.790+0000 7f032d1a6740 1 --2- 172.21.15.102:0/3617642215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbdafe0 0x559eabbd87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.790+0000 7f032d1a6740 1 --2- 172.21.15.102:0/3617642215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.790+0000 7f032d1a6740 1 --2- 172.21.15.102:0/3617642215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbcfe00 0x559eabbe6ad0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.790+0000 7f032d1a6740 1 -- 172.21.15.102:0/3617642215 >> 172.21.15.102:0/3617642215 conn(0x559eabbcc310 msgr2=0x559eabbcc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.790+0000 7f032d1a6740 1 -- 172.21.15.102:0/3617642215 shutdown_connections 2024-04-25T01:59:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.790+0000 7f032d1a6740 1 -- 172.21.15.102:0/3617642215 wait complete. 2024-04-25T01:59:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.791+0000 7f032d1a6740 1 -- start start 2024-04-25T01:59:11.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.792+0000 7f032d1a6740 1 Processor -- start 2024-04-25T01:59:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.793+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.793+0000 7f032d1a6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.793+0000 7f0322a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.793+0000 7f0322a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:55674/0 (socket says 172.21.15.102:55674) 2024-04-25T01:59:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.794+0000 7f0322a00640 1 -- 172.21.15.102:0/2543936411 learned_addr learned my addr 172.21.15.102:0/2543936411 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.794+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbdafe0 0x559eabc50460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.794+0000 7f0323400640 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.794+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03240721d0 con 0x559eabbcfe00 2024-04-25T01:59:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.794+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0324073640 con 0x559eabbd0720 2024-04-25T01:59:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.794+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03240728a0 con 0x559eabbdafe0 2024-04-25T01:59:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.794+0000 7f032d1a6740 10 client.0 _ll_register_callbacks cb 0x7f031c034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.795+0000 7f0323e00640 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbdafe0 0x559eabc50460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.795+0000 7f0322a00640 1 -- 172.21.15.102:0/2543936411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbdafe0 msgr2=0x559eabc50460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.795+0000 7f0322a00640 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbdafe0 0x559eabc50460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.795+0000 7f0322a00640 1 -- 172.21.15.102:0/2543936411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 msgr2=0x559eabbd5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.795+0000 7f0322a00640 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.795+0000 7f0322a00640 1 -- 172.21.15.102:0/2543936411 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559eabbd0550 con 0x559eabbd0720 2024-04-25T01:59:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.795+0000 7f0323400640 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.797+0000 7f0322a00640 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd55a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0310004420 tx=0x7f031000efd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9268f4240773bd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.797+0000 7f030d600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.797+0000 7f030d600640 1 -- 172.21.15.102:0/2543936411 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310021040 con 0x559eabbd0720 2024-04-25T01:59:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.797+0000 7f030d600640 1 -- 172.21.15.102:0/2543936411 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f031001d840 con 0x559eabbd0720 2024-04-25T01:59:11.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.797+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03140983b0 con 0x559eabbd0720 2024-04-25T01:59:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.798+0000 7f030d600640 1 -- 172.21.15.102:0/2543936411 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f03100040a0 con 0x559eabbd0720 2024-04-25T01:59:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.798+0000 7f032d1a6740 10 client.26312 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.798+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559eabc51990 con 0x559eabbd0720 2024-04-25T01:59:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.798+0000 7f030d600640 1 -- 172.21.15.102:0/2543936411 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03140983b0 con 0x559eabbd0720 2024-04-25T01:59:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.798+0000 7f030d600640 1 -- 172.21.15.102:0/2543936411 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f03100265c0 con 0x559eabbd0720 2024-04-25T01:59:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.799+0000 7f032d1a6740 10 client.26312 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.799+0000 7f032d1a6740 10 client.26312 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.799+0000 7f032d1a6740 20 client.26312 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.799+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559eabc51fa0 0x559eabc543c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.799+0000 7f032d1a6740 4 client.26312 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:59:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.799+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559eabb3b340 con 0x559eabc51fa0 2024-04-25T01:59:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.799+0000 7f0323400640 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559eabc51fa0 0x559eabc543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.800+0000 7f030d600640 10 client.26312 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:11.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.800+0000 7f0323400640 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559eabc51fa0 0x559eabc543c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d51edb2465127126 server_cookie=2e83a1becdd23139 in_seq=0 out_seq=0 2024-04-25T01:59:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.836+0000 7f030d600640 1 -- 172.21.15.102:0/2543936411 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559eabb3b340 con 0x559eabc51fa0 2024-04-25T01:59:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.836+0000 7f030d600640 10 client.26312 handle_command_reply: tid=0 2024-04-25T01:59:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.836+0000 7f032d1a6740 1 client.26312 shutdown 2024-04-25T01:59:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.837+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 msgr2=0x559eabbd55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.837+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd55a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0310004420 tx=0x7f031000efd0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.837+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 shutdown_connections 2024-04-25T01:59:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.837+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559eabc51fa0 0x559eabc543c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.837+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559eabbdafe0 0x559eabc50460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.837+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559eabbd0720 0x559eabbd55a0 secure :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0310004420 tx=0x7f031000efd0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.837+0000 7f032d1a6740 1 --2- 172.21.15.102:0/2543936411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559eabbcfe00 0x559eabbd5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.837+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 >> 172.21.15.102:0/2543936411 conn(0x559eabbcc310 msgr2=0x559eabbcb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.837+0000 7f030d600640 0 client.26312 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.838+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 shutdown_connections 2024-04-25T01:59:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.838+0000 7f032d1a6740 1 -- 172.21.15.102:0/2543936411 wait complete. 2024-04-25T01:59:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.838+0000 7f032d1a6740 20 client.26312 trim_cache size 0 max 16384 2024-04-25T01:59:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:59:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:59:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:59:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:59:11.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 -- 172.21.15.102:0/1633434039 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03080a2ef0 msgr2=0x7f03080a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:11.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03080a2ef0 0x7f03080a53b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0324078800 tx=0x7f0310009520 comp rx=0 tx=0).stop 2024-04-25T01:59:11.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 -- 172.21.15.102:0/1633434039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0324075c70 msgr2=0x7f032419e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:11.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0324075c70 0x7f032419e620 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f031400e9f0 tx=0x7f031400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 -- 172.21.15.102:0/1633434039 shutdown_connections 2024-04-25T01:59:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03080a2ef0 0x7f03080a53b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03241125e0 0x7f03241a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0324076590 0x7f032419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 --2- 172.21.15.102:0/1633434039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0324075c70 0x7f032419e620 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.839+0000 7f032a400640 1 -- 172.21.15.102:0/1633434039 >> 172.21.15.102:0/1633434039 conn(0x7f03240ff470 msgr2=0x7f0324101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.840+0000 7f032a400640 1 -- 172.21.15.102:0/1633434039 shutdown_connections 2024-04-25T01:59:11.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:11.840+0000 7f032a400640 1 -- 172.21.15.102:0/1633434039 wait complete. 2024-04-25T01:59:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 200.0 s, 2231.4 tps, lat 44.778 ms stddev 53.700 2024-04-25T01:59:13.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:13 smithi193 ceph-mon[32981]: pgmap v121: 97 pgs: 97 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 MiB/s wr, 1.67k op/s 2024-04-25T01:59:13.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:13 smithi102 ceph-mon[27796]: pgmap v121: 97 pgs: 97 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 MiB/s wr, 1.67k op/s 2024-04-25T01:59:13.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:13 smithi110 ceph-mon[32848]: pgmap v121: 97 pgs: 97 active+clean; 2.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 MiB/s wr, 1.67k op/s 2024-04-25T01:59:15.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:15 smithi102 ceph-mon[27796]: pgmap v122: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 48 MiB/s wr, 1.58k op/s 2024-04-25T01:59:15.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:15 smithi110 ceph-mon[32848]: pgmap v122: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 48 MiB/s wr, 1.58k op/s 2024-04-25T01:59:15.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:15 smithi193 ceph-mon[32981]: pgmap v122: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 48 MiB/s wr, 1.58k op/s 2024-04-25T01:59:17.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:17 smithi102 ceph-mon[27796]: pgmap v123: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 47 MiB/s wr, 1.45k op/s 2024-04-25T01:59:17.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:17 smithi110 ceph-mon[32848]: pgmap v123: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 47 MiB/s wr, 1.45k op/s 2024-04-25T01:59:17.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:17 smithi193 ceph-mon[32981]: pgmap v123: 97 pgs: 97 active+clean; 2.7 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 47 MiB/s wr, 1.45k op/s 2024-04-25T01:59:18.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 205.0 s, 2349.0 tps, lat 42.586 ms stddev 54.399 2024-04-25T01:59:18.971 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:59:18.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:59:19.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.254+0000 7f45d8200640 1 -- 172.21.15.102:0/2812828468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006b910 msgr2=0x7f45d011aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.254+0000 7f45d8200640 1 --2- 172.21.15.102:0/2812828468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006b910 0x7f45d011aa00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f45c4009900 tx=0x7f45c402a490 comp rx=0 tx=0).stop 2024-04-25T01:59:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.254+0000 7f45d8200640 1 -- 172.21.15.102:0/2812828468 shutdown_connections 2024-04-25T01:59:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.254+0000 7f45d8200640 1 --2- 172.21.15.102:0/2812828468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d011af40 0x7f45d011d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.254+0000 7f45d8200640 1 --2- 172.21.15.102:0/2812828468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006b910 0x7f45d011aa00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.254+0000 7f45d8200640 1 --2- 172.21.15.102:0/2812828468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45d006aff0 0x7f45d006b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.254+0000 7f45d8200640 1 -- 172.21.15.102:0/2812828468 >> 172.21.15.102:0/2812828468 conn(0x7f45d0072d40 msgr2=0x7f45d0073150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:19.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.255+0000 7f45d8200640 1 -- 172.21.15.102:0/2812828468 shutdown_connections 2024-04-25T01:59:19.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.255+0000 7f45d8200640 1 -- 172.21.15.102:0/2812828468 wait complete. 2024-04-25T01:59:19.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.257+0000 7f45d8200640 1 Processor -- start 2024-04-25T01:59:19.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d8200640 1 -- start start 2024-04-25T01:59:19.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d8200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006aff0 0x7f45d01a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d8200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d006b910 0x7f45d01a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d8200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45d011af40 0x7f45d01ab740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d8200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0120930 con 0x7f45d011af40 2024-04-25T01:59:19.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d8200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01207b0 con 0x7f45d006b910 2024-04-25T01:59:19.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d8200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45d0120ab0 con 0x7f45d006aff0 2024-04-25T01:59:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006aff0 0x7f45d01a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d4e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d006b910 0x7f45d01a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006aff0 0x7f45d01a6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38200/0 (socket says 172.21.15.102:38200) 2024-04-25T01:59:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d5800640 1 -- 172.21.15.102:0/311663794 learned_addr learned my addr 172.21.15.102:0/311663794 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.258+0000 7f45d6200640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45d011af40 0x7f45d01ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.259+0000 7f45d5800640 1 -- 172.21.15.102:0/311663794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d006b910 msgr2=0x7f45d01a7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.259+0000 7f45d5800640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d006b910 0x7f45d01a7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.259+0000 7f45d5800640 1 -- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45d011af40 msgr2=0x7f45d01ab740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.259+0000 7f45d5800640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45d011af40 0x7f45d01ab740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.259+0000 7f45d5800640 1 -- 172.21.15.102:0/311663794 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d01abe70 con 0x7f45d006aff0 2024-04-25T01:59:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.259+0000 7f45d5800640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006aff0 0x7f45d01a6ed0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f45c000d950 tx=0x7f45c000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.260+0000 7f45ce000640 1 -- 172.21.15.102:0/311663794 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c0017070 con 0x7f45d006aff0 2024-04-25T01:59:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.260+0000 7f45ce000640 1 -- 172.21.15.102:0/311663794 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c0009c30 con 0x7f45d006aff0 2024-04-25T01:59:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.260+0000 7f45ce000640 1 -- 172.21.15.102:0/311663794 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c000f6b0 con 0x7f45d006aff0 2024-04-25T01:59:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.260+0000 7f45d8200640 1 -- 172.21.15.102:0/311663794 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d01ac160 con 0x7f45d006aff0 2024-04-25T01:59:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.260+0000 7f45d8200640 1 -- 172.21.15.102:0/311663794 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d01b39f0 con 0x7f45d006aff0 2024-04-25T01:59:19.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.265+0000 7f45ce000640 1 -- 172.21.15.102:0/311663794 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f45c000f850 con 0x7f45d006aff0 2024-04-25T01:59:19.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.265+0000 7f45dafcb740 1 Processor -- start 2024-04-25T01:59:19.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.265+0000 7f45dafcb740 1 -- start start 2024-04-25T01:59:19.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feb73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7920 0x56050fec1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050fec1ff0 0x56050fec43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45dafcb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56050fe1b560 con 0x56050feb7920 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45dafcb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56050fdc3930 con 0x56050fec1ff0 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45dafcb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56050fe17520 con 0x56050feb7000 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45d4e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050fec1ff0 0x56050fec43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45d6200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feb73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45d4e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050fec1ff0 0x56050fec43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41366/0 (socket says 172.21.15.102:41366) 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45d4e00640 1 -- 172.21.15.102:0/593319936 learned_addr learned my addr 172.21.15.102:0/593319936 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45d5800640 1 --2- 172.21.15.102:0/593319936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7920 0x56050fec1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.266+0000 7f45ce000640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f45b80a2e40 0x7f45b80a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.267+0000 7f45d4e00640 1 -- 172.21.15.102:0/593319936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 msgr2=0x56050feb73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.267+0000 7f45d4e00640 1 --2- 172.21.15.102:0/593319936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feb73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.267+0000 7f45d4e00640 1 -- 172.21.15.102:0/593319936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7920 msgr2=0x56050fec1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.267+0000 7f45d4e00640 1 --2- 172.21.15.102:0/593319936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7920 0x56050fec1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.267+0000 7f45d4e00640 1 -- 172.21.15.102:0/593319936 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56050fbf97d0 con 0x56050fec1ff0 2024-04-25T01:59:19.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.267+0000 7f45d5800640 1 --2- 172.21.15.102:0/593319936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7920 0x56050fec1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.267+0000 7f45ce000640 1 -- 172.21.15.102:0/311663794 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f45c00ced90 con 0x7f45d006aff0 2024-04-25T01:59:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.267+0000 7f45d6200640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f45b80a2e40 0x7f45b80a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.267+0000 7f45d4e00640 1 --2- 172.21.15.102:0/593319936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050fec1ff0 0x56050fec43e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f45c4005dc0 tx=0x7f45c4004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a9e7f9bad1fcf2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.268+0000 7f45bea00640 1 -- 172.21.15.102:0/593319936 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4035030 con 0x56050fec1ff0 2024-04-25T01:59:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.268+0000 7f45bea00640 1 -- 172.21.15.102:0/593319936 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c4004430 con 0x56050fec1ff0 2024-04-25T01:59:19.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.268+0000 7f45bea00640 1 -- 172.21.15.102:0/593319936 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c40035f0 con 0x56050fec1ff0 2024-04-25T01:59:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45dafcb740 1 -- 172.21.15.102:0/593319936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050fec1ff0 msgr2=0x56050fec43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45dafcb740 1 --2- 172.21.15.102:0/593319936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050fec1ff0 0x56050fec43e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f45c4005dc0 tx=0x7f45c4004040 comp rx=0 tx=0).stop 2024-04-25T01:59:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45d6200640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f45b80a2e40 0x7f45b80a5300 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45d006ae20 tx=0x7f45c8004ab0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45dafcb740 1 -- 172.21.15.102:0/593319936 shutdown_connections 2024-04-25T01:59:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45dafcb740 1 --2- 172.21.15.102:0/593319936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050fec1ff0 0x56050fec43e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45dafcb740 1 --2- 172.21.15.102:0/593319936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7920 0x56050fec1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45dafcb740 1 --2- 172.21.15.102:0/593319936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feb73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45dafcb740 1 -- 172.21.15.102:0/593319936 >> 172.21.15.102:0/593319936 conn(0x56050feb3320 msgr2=0x56050feb3700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45dafcb740 1 -- 172.21.15.102:0/593319936 shutdown_connections 2024-04-25T01:59:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.269+0000 7f45dafcb740 1 -- 172.21.15.102:0/593319936 wait complete. 2024-04-25T01:59:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.270+0000 7f45dafcb740 1 -- start start 2024-04-25T01:59:19.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.270+0000 7f45dafcb740 1 Processor -- start 2024-04-25T01:59:19.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.270+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050febef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.270+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050febf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.270+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050febf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.270+0000 7f45dafcb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56050fec4d80 con 0x56050feb7000 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.270+0000 7f45dafcb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56050fdd6d90 con 0x56050feb7920 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.270+0000 7f45dafcb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56050fec4f00 con 0x56050fec1ff0 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.270+0000 7f45dafcb740 10 client.0 _ll_register_callbacks cb 0x56050fdd6780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d6200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050febf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d6200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050febf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38212/0 (socket says 172.21.15.102:38212) 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d6200640 1 -- 172.21.15.102:0/2903851354 learned_addr learned my addr 172.21.15.102:0/2903851354 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d4e00640 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050febf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d5800640 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050febef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d6200640 1 -- 172.21.15.102:0/2903851354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 msgr2=0x56050febf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:19.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d6200640 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050febf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d6200640 1 -- 172.21.15.102:0/2903851354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 msgr2=0x56050febef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d6200640 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050febef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.271+0000 7f45d6200640 1 -- 172.21.15.102:0/2903851354 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56050fb2f3f0 con 0x56050fec1ff0 2024-04-25T01:59:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.272+0000 7f45d6200640 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050febf9e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f45c8014330 tx=0x7f45c8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55c064323ce5f84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.272+0000 7f45b3400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:19.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.272+0000 7f45b3400640 1 -- 172.21.15.102:0/2903851354 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c8025070 con 0x56050fec1ff0 2024-04-25T01:59:19.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.272+0000 7f45b3400640 1 -- 172.21.15.102:0/2903851354 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f45c8012940 con 0x56050fec1ff0 2024-04-25T01:59:19.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.275+0000 7f45dafcb740 1 -- 172.21.15.102:0/2903851354 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45c4004430 con 0x56050fec1ff0 2024-04-25T01:59:19.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.276+0000 7f45b3400640 1 -- 172.21.15.102:0/2903851354 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45c802f050 con 0x56050fec1ff0 2024-04-25T01:59:19.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.277+0000 7f45dafcb740 10 client.26302 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:19.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.277+0000 7f45dafcb740 1 -- 172.21.15.102:0/2903851354 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56050faef0f0 con 0x56050fec1ff0 2024-04-25T01:59:19.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.278+0000 7f45b3400640 1 -- 172.21.15.102:0/2903851354 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c4004430 con 0x56050fec1ff0 2024-04-25T01:59:19.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.278+0000 7f45b3400640 1 -- 172.21.15.102:0/2903851354 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f45c8020040 con 0x56050fec1ff0 2024-04-25T01:59:19.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.281+0000 7f45dafcb740 10 client.26302 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:19.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.281+0000 7f45dafcb740 10 client.26302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:19.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.281+0000 7f45dafcb740 20 client.26302 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:19.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.281+0000 7f45dafcb740 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56050fed23a0 0x56050fed47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.281+0000 7f45dafcb740 4 client.26302 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:19.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.281+0000 7f45dafcb740 1 -- 172.21.15.102:0/2903851354 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56050fe2ed90 con 0x56050fed23a0 2024-04-25T01:59:19.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.281+0000 7f45d5800640 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56050fed23a0 0x56050fed47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.283+0000 7f45b3400640 10 client.26302 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:19.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.283+0000 7f45d5800640 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56050fed23a0 0x56050fed47d0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d8429aa69870d2f server_cookie=61a6940e77eba274 in_seq=0 out_seq=0 2024-04-25T01:59:19.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.285+0000 7f45b3400640 1 -- 172.21.15.102:0/2903851354 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56050fe2ed90 con 0x56050fed23a0 2024-04-25T01:59:19.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.285+0000 7f45b3400640 10 client.26302 handle_command_reply: tid=0 2024-04-25T01:59:19.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.285+0000 7f45dafcb740 1 client.26302 shutdown 2024-04-25T01:59:19.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.287+0000 7f45dafcb740 1 -- 172.21.15.102:0/2903851354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 msgr2=0x56050febf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.287+0000 7f45dafcb740 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050febf9e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f45c8014330 tx=0x7f45c8004a40 comp rx=0 tx=0).stop 2024-04-25T01:59:19.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.288+0000 7f45dafcb740 1 -- 172.21.15.102:0/2903851354 shutdown_connections 2024-04-25T01:59:19.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.288+0000 7f45dafcb740 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56050fed23a0 0x56050fed47d0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.288+0000 7f45dafcb740 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050febf9e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.288+0000 7f45dafcb740 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050febf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.288+0000 7f45dafcb740 1 --2- 172.21.15.102:0/2903851354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050febef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.288+0000 7f45dafcb740 1 -- 172.21.15.102:0/2903851354 >> 172.21.15.102:0/2903851354 conn(0x56050feb3320 msgr2=0x56050feb9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.288+0000 7f45b3400640 0 client.26302 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.288+0000 7f45dafcb740 1 -- 172.21.15.102:0/2903851354 shutdown_connections 2024-04-25T01:59:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.289+0000 7f45dafcb740 1 -- 172.21.15.102:0/2903851354 wait complete. 2024-04-25T01:59:19.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.289+0000 7f45dafcb740 20 client.26302 trim_cache size 0 max 16384 2024-04-25T01:59:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.338+0000 7f45dafcb740 1 Processor -- start 2024-04-25T01:59:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.338+0000 7f45dafcb740 1 -- start start 2024-04-25T01:59:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.339+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050fecdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.340+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050feb9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.340+0000 7f45d5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050fecdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.340+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050feda0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.340+0000 7f45d5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050fecdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41150/0 (socket says 172.21.15.102:41150) 2024-04-25T01:59:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45dafcb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01beb00 con 0x56050feb7000 2024-04-25T01:59:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45dafcb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01bc3a0 con 0x56050feb7920 2024-04-25T01:59:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45d4e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050feb9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.340+0000 7f45d6200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050feda0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45d6200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050feda0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38226/0 (socket says 172.21.15.102:38226) 2024-04-25T01:59:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45d6200640 1 -- 172.21.15.102:0/3557126785 learned_addr learned my addr 172.21.15.102:0/3557126785 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45dafcb740 1 -- 172.21.15.102:0/3557126785 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01ba1f0 con 0x56050fec1ff0 2024-04-25T01:59:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45d6200640 1 -- 172.21.15.102:0/3557126785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 msgr2=0x56050feb9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45d6200640 1 --2- 172.21.15.102:0/3557126785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050feb9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45d6200640 1 -- 172.21.15.102:0/3557126785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 msgr2=0x56050fecdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45d6200640 1 --2- 172.21.15.102:0/3557126785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050fecdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.341+0000 7f45d6200640 1 -- 172.21.15.102:0/3557126785 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56050feb2c00 con 0x56050fec1ff0 2024-04-25T01:59:19.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.342+0000 7f45d5800640 1 --2- 172.21.15.102:0/3557126785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050fecdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:19.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.343+0000 7f45d6200640 1 --2- 172.21.15.102:0/3557126785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050feda0a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f45c8010890 tx=0x7f45c8009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95fbca8f3372615f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.343+0000 7f45be000640 1 -- 172.21.15.102:0/3557126785 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c8038070 con 0x56050fec1ff0 2024-04-25T01:59:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.343+0000 7f45be000640 1 -- 172.21.15.102:0/3557126785 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c80214a0 con 0x56050fec1ff0 2024-04-25T01:59:19.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.343+0000 7f45be000640 1 -- 172.21.15.102:0/3557126785 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c8025070 con 0x56050fec1ff0 2024-04-25T01:59:19.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.345+0000 7f45dafcb740 1 -- 172.21.15.102:0/3557126785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 msgr2=0x56050feda0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.345+0000 7f45dafcb740 1 --2- 172.21.15.102:0/3557126785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050feda0a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f45c8010890 tx=0x7f45c8009ef0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.346+0000 7f45dafcb740 1 -- 172.21.15.102:0/3557126785 shutdown_connections 2024-04-25T01:59:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.346+0000 7f45dafcb740 1 --2- 172.21.15.102:0/3557126785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050fec1ff0 0x56050feda0a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.346+0000 7f45dafcb740 1 --2- 172.21.15.102:0/3557126785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050feb9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.346+0000 7f45dafcb740 1 --2- 172.21.15.102:0/3557126785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050feb7000 0x56050fecdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.346+0000 7f45dafcb740 1 -- 172.21.15.102:0/3557126785 >> 172.21.15.102:0/3557126785 conn(0x56050feb3320 msgr2=0x56050febf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:19.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.346+0000 7f45dafcb740 1 -- 172.21.15.102:0/3557126785 shutdown_connections 2024-04-25T01:59:19.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.346+0000 7f45dafcb740 1 -- 172.21.15.102:0/3557126785 wait complete. 2024-04-25T01:59:19.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.347+0000 7f45dafcb740 1 -- start start 2024-04-25T01:59:19.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.348+0000 7f45dafcb740 1 Processor -- start 2024-04-25T01:59:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.349+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.349+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050febb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.349+0000 7f45dafcb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050fec1ff0 0x56050febbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.349+0000 7f45d5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.349+0000 7f45dafcb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01ba860 con 0x56050fec1ff0 2024-04-25T01:59:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.349+0000 7f45dafcb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01bbcd0 con 0x56050feb7920 2024-04-25T01:59:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.349+0000 7f45dafcb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01baf30 con 0x56050feb7000 2024-04-25T01:59:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.349+0000 7f45d5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feca440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38238/0 (socket says 172.21.15.102:38238) 2024-04-25T01:59:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.350+0000 7f45d5800640 1 -- 172.21.15.102:0/336830974 learned_addr learned my addr 172.21.15.102:0/336830974 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:19.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.350+0000 7f45dafcb740 10 client.0 _ll_register_callbacks cb 0x7f45c8014ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:19.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.350+0000 7f45d4e00640 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050febb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.350+0000 7f45d6200640 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050fec1ff0 0x56050febbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.352+0000 7f45d4e00640 1 -- 172.21.15.102:0/336830974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 msgr2=0x56050feca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.352+0000 7f45d4e00640 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.352+0000 7f45d4e00640 1 -- 172.21.15.102:0/336830974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050fec1ff0 msgr2=0x56050febbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.352+0000 7f45d4e00640 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050fec1ff0 0x56050febbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.352+0000 7f45d4e00640 1 -- 172.21.15.102:0/336830974 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56050febc570 con 0x56050feb7920 2024-04-25T01:59:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.353+0000 7f45d5800640 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.353+0000 7f45d6200640 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050fec1ff0 0x56050febbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:19.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.353+0000 7f45d4e00640 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050febb8f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f45c4005ab0 tx=0x7f45c4005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d00716efc2decab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:19.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.353+0000 7f45b3400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:19.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.354+0000 7f45b3400640 1 -- 172.21.15.102:0/336830974 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4035040 con 0x56050feb7920 2024-04-25T01:59:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.354+0000 7f45b3400640 1 -- 172.21.15.102:0/336830974 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f45c40055f0 con 0x56050feb7920 2024-04-25T01:59:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.354+0000 7f45dafcb740 1 -- 172.21.15.102:0/336830974 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45c80214a0 con 0x56050feb7920 2024-04-25T01:59:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.355+0000 7f45b3400640 1 -- 172.21.15.102:0/336830974 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45c4058050 con 0x56050feb7920 2024-04-25T01:59:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.355+0000 7f45dafcb740 10 client.26330 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.355+0000 7f45dafcb740 1 -- 172.21.15.102:0/336830974 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56050ff38800 con 0x56050feb7920 2024-04-25T01:59:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.355+0000 7f45b3400640 1 -- 172.21.15.102:0/336830974 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c80214a0 con 0x56050feb7920 2024-04-25T01:59:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.355+0000 7f45b3400640 1 -- 172.21.15.102:0/336830974 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f45c4003a80 con 0x56050feb7920 2024-04-25T01:59:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.355+0000 7f45dafcb740 10 client.26330 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.355+0000 7f45dafcb740 10 client.26330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.355+0000 7f45dafcb740 20 client.26330 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.356+0000 7f45dafcb740 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56050ff38b40 0x56050ff3af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.356+0000 7f45dafcb740 4 client.26330 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:59:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.356+0000 7f45dafcb740 1 -- 172.21.15.102:0/336830974 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56050fe28aa0 con 0x56050ff38b40 2024-04-25T01:59:19.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.356+0000 7f45d5800640 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56050ff38b40 0x56050ff3af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.357+0000 7f45d5800640 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56050ff38b40 0x56050ff3af60 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a439cd6b8cd61938 server_cookie=343270e4e65b90b in_seq=0 out_seq=0 2024-04-25T01:59:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.357+0000 7f45b3400640 10 client.26330 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.357+0000 7f45b3400640 1 -- 172.21.15.102:0/336830974 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56050fe28aa0 con 0x56050ff38b40 2024-04-25T01:59:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.357+0000 7f45b3400640 10 client.26330 handle_command_reply: tid=0 2024-04-25T01:59:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.357+0000 7f45dafcb740 1 client.26330 shutdown 2024-04-25T01:59:19.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 -- 172.21.15.102:0/336830974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 msgr2=0x56050febb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050febb8f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f45c4005ab0 tx=0x7f45c4005270 comp rx=0 tx=0).stop 2024-04-25T01:59:19.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 -- 172.21.15.102:0/336830974 shutdown_connections 2024-04-25T01:59:19.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56050ff38b40 0x56050ff3af60 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56050fec1ff0 0x56050febbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56050feb7920 0x56050febb8f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 --2- 172.21.15.102:0/336830974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56050feb7000 0x56050feca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 -- 172.21.15.102:0/336830974 >> 172.21.15.102:0/336830974 conn(0x56050feb3320 msgr2=0x56050feb1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45b3400640 0 client.26330 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 -- 172.21.15.102:0/336830974 shutdown_connections 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 1 -- 172.21.15.102:0/336830974 wait complete. 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.358+0000 7f45dafcb740 20 client.26330 trim_cache size 0 max 16384 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:59:19.363 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T01:59:19.364 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T01:59:19.364 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:59:19.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.359+0000 7f45d8200640 1 -- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f45b80a2e40 msgr2=0x7f45b80a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.359+0000 7f45d8200640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f45b80a2e40 0x7f45b80a5300 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45d006ae20 tx=0x7f45c8004ab0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.359+0000 7f45d8200640 1 -- 172.21.15.102:0/311663794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006aff0 msgr2=0x7f45d01a6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.359+0000 7f45d8200640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006aff0 0x7f45d01a6ed0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f45c000d950 tx=0x7f45c000de10 comp rx=0 tx=0).stop 2024-04-25T01:59:19.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.360+0000 7f45d8200640 1 -- 172.21.15.102:0/311663794 shutdown_connections 2024-04-25T01:59:19.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.360+0000 7f45d8200640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f45b80a2e40 0x7f45b80a5300 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.360+0000 7f45d8200640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45d011af40 0x7f45d01ab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.360+0000 7f45d8200640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d006b910 0x7f45d01a7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.360+0000 7f45d8200640 1 --2- 172.21.15.102:0/311663794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45d006aff0 0x7f45d01a6ed0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.360+0000 7f45d8200640 1 -- 172.21.15.102:0/311663794 >> 172.21.15.102:0/311663794 conn(0x7f45d0072d40 msgr2=0x7f45d011b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:19.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.361+0000 7f45d8200640 1 -- 172.21.15.102:0/311663794 shutdown_connections 2024-04-25T01:59:19.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.361+0000 7f45d8200640 1 -- 172.21.15.102:0/311663794 wait complete. 2024-04-25T01:59:19.389 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:19 smithi102 ceph-mon[27796]: pgmap v124: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 62 MiB/s wr, 1.75k op/s 2024-04-25T01:59:19.394 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:59:19.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:59:19.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.640+0000 7f812a800640 1 -- 172.21.15.102:0/2045027592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f812410bd70 msgr2=0x7f812410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.640+0000 7f812a800640 1 --2- 172.21.15.102:0/2045027592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f812410bd70 0x7f812410e1a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f811c009ca0 tx=0x7f811c02ea30 comp rx=0 tx=0).stop 2024-04-25T01:59:19.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.640+0000 7f812a800640 1 -- 172.21.15.102:0/2045027592 shutdown_connections 2024-04-25T01:59:19.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.640+0000 7f812a800640 1 --2- 172.21.15.102:0/2045027592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f812410bd70 0x7f812410e1a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.640+0000 7f812a800640 1 --2- 172.21.15.102:0/2045027592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81241032c0 0x7f812410b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.640+0000 7f812a800640 1 --2- 172.21.15.102:0/2045027592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81241029a0 0x7f8124102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.640+0000 7f812a800640 1 -- 172.21.15.102:0/2045027592 >> 172.21.15.102:0/2045027592 conn(0x7f81240769c0 msgr2=0x7f8124076dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:19.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.640+0000 7f812a800640 1 -- 172.21.15.102:0/2045027592 shutdown_connections 2024-04-25T01:59:19.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.640+0000 7f812a800640 1 -- 172.21.15.102:0/2045027592 wait complete. 2024-04-25T01:59:19.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.642+0000 7f812a800640 1 Processor -- start 2024-04-25T01:59:19.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f812a800640 1 -- start start 2024-04-25T01:59:19.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f812a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81241029a0 0x7f8124195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f812a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81241032c0 0x7f81241963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f812a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f812410bd70 0x7f812419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f812a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f812410f960 con 0x7f81241029a0 2024-04-25T01:59:19.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f812a800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f812410f7e0 con 0x7f81241032c0 2024-04-25T01:59:19.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f812a800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f812410fae0 con 0x7f812410bd70 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f8122a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81241032c0 0x7f81241963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f8122a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81241032c0 0x7f81241963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41412/0 (socket says 172.21.15.102:41412) 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f8122a00640 1 -- 172.21.15.102:0/37067769 learned_addr learned my addr 172.21.15.102:0/37067769 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f8123400640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81241029a0 0x7f8124195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.643+0000 7f8123e00640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f812410bd70 0x7f812419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f8122a00640 1 -- 172.21.15.102:0/37067769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f812410bd70 msgr2=0x7f812419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f8122a00640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f812410bd70 0x7f812419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f8122a00640 1 -- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81241029a0 msgr2=0x7f8124195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f8122a00640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81241029a0 0x7f8124195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f8122a00640 1 -- 172.21.15.102:0/37067769 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f812419ae00 con 0x7f81241032c0 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f8122a00640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81241032c0 0x7f81241963a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f811000e9f0 tx=0x7f811000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:19.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f811be00640 1 -- 172.21.15.102:0/37067769 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811000cd80 con 0x7f81241032c0 2024-04-25T01:59:19.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f811be00640 1 -- 172.21.15.102:0/37067769 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8110005500 con 0x7f81241032c0 2024-04-25T01:59:19.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f811be00640 1 -- 172.21.15.102:0/37067769 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8110010690 con 0x7f81241032c0 2024-04-25T01:59:19.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.644+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f812419b0f0 con 0x7f81241032c0 2024-04-25T01:59:19.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.645+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81241a2980 con 0x7f81241032c0 2024-04-25T01:59:19.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.649+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f812406a5c0 con 0x7f81241032c0 2024-04-25T01:59:19.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.650+0000 7f811be00640 1 -- 172.21.15.102:0/37067769 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f81100046c0 con 0x7f81241032c0 2024-04-25T01:59:19.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.653+0000 7f811be00640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f81080a2d00 0x7f81080a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:19.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.653+0000 7f811be00640 1 -- 172.21.15.102:0/37067769 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f81100cf7e0 con 0x7f81241032c0 2024-04-25T01:59:19.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.653+0000 7f8123400640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f81080a2d00 0x7f81080a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:19.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.654+0000 7f8123400640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f81080a2d00 0x7f81080a51c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f81140095f0 tx=0x7f8114005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:19.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:19.656+0000 7f811be00640 1 -- 172.21.15.102:0/37067769 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f811009c430 con 0x7f81241032c0 2024-04-25T01:59:19.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:19 smithi110 ceph-mon[32848]: pgmap v124: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 62 MiB/s wr, 1.75k op/s 2024-04-25T01:59:19.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:19 smithi193 ceph-mon[32981]: pgmap v124: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 62 MiB/s wr, 1.75k op/s 2024-04-25T01:59:20.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.063+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8124197160 con 0x7f81241032c0 2024-04-25T01:59:20.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.064+0000 7f811be00640 1 -- 172.21.15.102:0/37067769 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11897 (secure 0 0 0) 0x7f811009bd30 con 0x7f81241032c0 2024-04-25T01:59:20.069 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:59:20.069 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:59:01.795132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T01:59:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 31 2024-04-25T01:59:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f81080a2d00 msgr2=0x7f81080a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f81080a2d00 0x7f81080a51c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f81140095f0 tx=0x7f8114005f90 comp rx=0 tx=0).stop 2024-04-25T01:59:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81241032c0 msgr2=0x7f81241963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81241032c0 0x7f81241963a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f811000e9f0 tx=0x7f811000edf0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 shutdown_connections 2024-04-25T01:59:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f81080a2d00 0x7f81080a51c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f812410bd70 0x7f812419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81241032c0 0x7f81241963a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 --2- 172.21.15.102:0/37067769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81241029a0 0x7f8124195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 >> 172.21.15.102:0/37067769 conn(0x7f81240769c0 msgr2=0x7f81240fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 shutdown_connections 2024-04-25T01:59:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.068+0000 7f812a800640 1 -- 172.21.15.102:0/37067769 wait complete. 2024-04-25T01:59:20.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:59:20.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.350+0000 7fa777200640 1 -- 172.21.15.102:0/885460303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 msgr2=0x7fa770098550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.350+0000 7fa777200640 1 --2- 172.21.15.102:0/885460303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 0x7fa770098550 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa768009860 tx=0x7fa76802a340 comp rx=0 tx=0).stop 2024-04-25T01:59:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.352+0000 7fa777200640 1 -- 172.21.15.102:0/885460303 shutdown_connections 2024-04-25T01:59:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.352+0000 7fa777200640 1 --2- 172.21.15.102:0/885460303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77009d240 0x7fa7700a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.352+0000 7fa777200640 1 --2- 172.21.15.102:0/885460303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa770098b20 0x7fa77009cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.352+0000 7fa777200640 1 --2- 172.21.15.102:0/885460303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 0x7fa770098550 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.352+0000 7fa777200640 1 -- 172.21.15.102:0/885460303 >> 172.21.15.102:0/885460303 conn(0x7fa770093a10 msgr2=0x7fa770095e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.352+0000 7fa777200640 1 -- 172.21.15.102:0/885460303 shutdown_connections 2024-04-25T01:59:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.352+0000 7fa777200640 1 -- 172.21.15.102:0/885460303 wait complete. 2024-04-25T01:59:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.353+0000 7fa777200640 1 Processor -- start 2024-04-25T01:59:20.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.354+0000 7fa777200640 1 -- start start 2024-04-25T01:59:20.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa777200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 0x7fa7700a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa777200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77009d240 0x7fa7700af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa777200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7700a3940 0x7fa7700a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa775e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 0x7fa7700a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa775e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 0x7fa7700a3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41426/0 (socket says 172.21.15.102:41426) 2024-04-25T01:59:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa77000a6a0 con 0x7fa7700a3940 2024-04-25T01:59:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa776800640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7700a3940 0x7fa7700a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa775e00640 1 -- 172.21.15.102:0/440549652 learned_addr learned my addr 172.21.15.102:0/440549652 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa77000a520 con 0x7fa770098170 2024-04-25T01:59:20.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.356+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa77000a820 con 0x7fa77009d240 2024-04-25T01:59:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.356+0000 7fa776800640 1 -- 172.21.15.102:0/440549652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77009d240 msgr2=0x7fa7700af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.355+0000 7fa775400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77009d240 0x7fa7700af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.357+0000 7fa776800640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77009d240 0x7fa7700af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.357+0000 7fa776800640 1 -- 172.21.15.102:0/440549652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 msgr2=0x7fa7700a3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.357+0000 7fa776800640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 0x7fa7700a3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.357+0000 7fa776800640 1 -- 172.21.15.102:0/440549652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa770147ba0 con 0x7fa7700a3940 2024-04-25T01:59:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.357+0000 7fa775400640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77009d240 0x7fa7700af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:20.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.357+0000 7fa775e00640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 0x7fa7700a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:20.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.357+0000 7fa776800640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7700a3940 0x7fa7700a3df0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa778067580 tx=0x7fa77807ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.358+0000 7fa76ea00640 1 -- 172.21.15.102:0/440549652 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa778084020 con 0x7fa7700a3940 2024-04-25T01:59:20.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.358+0000 7fa76ea00640 1 -- 172.21.15.102:0/440549652 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa778078e60 con 0x7fa7700a3940 2024-04-25T01:59:20.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.358+0000 7fa76ea00640 1 -- 172.21.15.102:0/440549652 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa778069940 con 0x7fa7700a3940 2024-04-25T01:59:20.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.358+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa770147e30 con 0x7fa7700a3940 2024-04-25T01:59:20.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.358+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7701482a0 con 0x7fa7700a3940 2024-04-25T01:59:20.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.366+0000 7fa77ff7c740 1 Processor -- start 2024-04-25T01:59:20.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.369+0000 7fa77ff7c740 1 -- start start 2024-04-25T01:59:20.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.369+0000 7fa76ea00640 1 -- 172.21.15.102:0/440549652 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa778087070 con 0x7fa7700a3940 2024-04-25T01:59:20.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.369+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c5eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.369+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c69250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.369+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c6bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.369+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555a33bc34e0 con 0x555a33c5f0c0 2024-04-25T01:59:20.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.369+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555a33b6ba70 con 0x555a33c69790 2024-04-25T01:59:20.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.369+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555a33bbf4a0 con 0x555a33c5e7a0 2024-04-25T01:59:20.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.370+0000 7fa775400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c6bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.370+0000 7fa775400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c6bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41432/0 (socket says 172.21.15.102:41432) 2024-04-25T01:59:20.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.370+0000 7fa775400640 1 -- 172.21.15.102:0/3039099634 learned_addr learned my addr 172.21.15.102:0/3039099634 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.370+0000 7fa775e00640 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c69250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.370+0000 7fa775400640 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c5eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.371+0000 7fa775400640 1 -- 172.21.15.102:0/3039099634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 msgr2=0x555a33c5eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.371+0000 7fa775400640 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c5eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.371+0000 7fa775400640 1 -- 172.21.15.102:0/3039099634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 msgr2=0x555a33c69250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.371+0000 7fa775400640 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c69250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.371+0000 7fa775400640 1 -- 172.21.15.102:0/3039099634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a338d73f0 con 0x555a33c69790 2024-04-25T01:59:20.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.372+0000 7fa775e00640 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c69250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:20.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.372+0000 7fa775400640 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c6bb80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa7640148b0 tx=0x7fa764009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b1908a884ac0dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.373+0000 7fa76ea00640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa75c0a2f00 0x7fa75c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.373+0000 7fa763400640 1 -- 172.21.15.102:0/3039099634 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7640142d0 con 0x555a33c69790 2024-04-25T01:59:20.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.373+0000 7fa763400640 1 -- 172.21.15.102:0/3039099634 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa764014e30 con 0x555a33c69790 2024-04-25T01:59:20.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.373+0000 7fa763400640 1 -- 172.21.15.102:0/3039099634 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7640142d0 con 0x555a33c69790 2024-04-25T01:59:20.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.373+0000 7fa76ea00640 1 -- 172.21.15.102:0/440549652 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa77813a7a0 con 0x7fa7700a3940 2024-04-25T01:59:20.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.374+0000 7fa775e00640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa75c0a2f00 0x7fa75c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.375+0000 7fa77ff7c740 1 -- 172.21.15.102:0/3039099634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 msgr2=0x555a33c6bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.375+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c6bb80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa7640148b0 tx=0x7fa764009600 comp rx=0 tx=0).stop 2024-04-25T01:59:20.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.377+0000 7fa775e00640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa75c0a2f00 0x7fa75c0a53c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x555a33c5fd60 tx=0x7fa7680056d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.377+0000 7fa77ff7c740 1 -- 172.21.15.102:0/3039099634 shutdown_connections 2024-04-25T01:59:20.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.377+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c6bb80 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.377+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c69250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.377+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/3039099634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c5eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.377+0000 7fa77ff7c740 1 -- 172.21.15.102:0/3039099634 >> 172.21.15.102:0/3039099634 conn(0x555a33c5aac0 msgr2=0x555a33c5aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.378+0000 7fa77ff7c740 1 -- 172.21.15.102:0/3039099634 shutdown_connections 2024-04-25T01:59:20.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.378+0000 7fa77ff7c740 1 -- 172.21.15.102:0/3039099634 wait complete. 2024-04-25T01:59:20.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.379+0000 7fa77ff7c740 1 -- start start 2024-04-25T01:59:20.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.379+0000 7fa77ff7c740 1 Processor -- start 2024-04-25T01:59:20.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.379+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c66740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.379+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5f0c0 0x555a33c66c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.379+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c69790 0x555a33c671c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa775e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c66740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555a33c6c520 con 0x555a33c69790 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555a33b7eec0 con 0x555a33c5f0c0 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555a33c6c6a0 con 0x555a33c5e7a0 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa775e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c66740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38288/0 (socket says 172.21.15.102:38288) 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa775e00640 1 -- 172.21.15.102:0/4218466814 learned_addr learned my addr 172.21.15.102:0/4218466814 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa775400640 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5f0c0 0x555a33c66c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa776800640 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c69790 0x555a33c671c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa77ff7c740 10 client.0 _ll_register_callbacks cb 0x555a33bbb3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa775e00640 1 -- 172.21.15.102:0/4218466814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5f0c0 msgr2=0x555a33c66c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa775e00640 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5f0c0 0x555a33c66c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa775e00640 1 -- 172.21.15.102:0/4218466814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c69790 msgr2=0x555a33c671c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa775e00640 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c69790 0x555a33c671c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa775e00640 1 -- 172.21.15.102:0/4218466814 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a339a17d0 con 0x555a33c5e7a0 2024-04-25T01:59:20.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.380+0000 7fa776800640 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c69790 0x555a33c671c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:20.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.381+0000 7fa775e00640 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c66740 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa768035040 tx=0x7fa76802a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd65e4a6837afdb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.381+0000 7fa757e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:20.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.381+0000 7fa757e00640 1 -- 172.21.15.102:0/4218466814 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768037040 con 0x555a33c5e7a0 2024-04-25T01:59:20.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.381+0000 7fa77ff7c740 1 -- 172.21.15.102:0/4218466814 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa764014e30 con 0x555a33c5e7a0 2024-04-25T01:59:20.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.381+0000 7fa757e00640 1 -- 172.21.15.102:0/4218466814 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa768003b80 con 0x555a33c5e7a0 2024-04-25T01:59:20.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.381+0000 7fa757e00640 1 -- 172.21.15.102:0/4218466814 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa768057050 con 0x555a33c5e7a0 2024-04-25T01:59:20.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.381+0000 7fa77ff7c740 10 client.26323 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:20.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.381+0000 7fa77ff7c740 1 -- 172.21.15.102:0/4218466814 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a33c78670 con 0x555a33c5e7a0 2024-04-25T01:59:20.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.382+0000 7fa757e00640 1 -- 172.21.15.102:0/4218466814 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa764014e30 con 0x555a33c5e7a0 2024-04-25T01:59:20.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.382+0000 7fa757e00640 1 -- 172.21.15.102:0/4218466814 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fa76803d570 con 0x555a33c5e7a0 2024-04-25T01:59:20.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.382+0000 7fa77ff7c740 10 client.26323 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:20.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.382+0000 7fa77ff7c740 10 client.26323 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:59:20.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.382+0000 7fa77ff7c740 20 client.26323 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:20.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.382+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555a33c79ad0 0x555a33c7bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.383+0000 7fa77ff7c740 4 client.26323 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:20.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.383+0000 7fa77ff7c740 1 -- 172.21.15.102:0/4218466814 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a33bd0a20 con 0x555a33c79ad0 2024-04-25T01:59:20.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.383+0000 7fa775400640 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555a33c79ad0 0x555a33c7bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.384+0000 7fa775400640 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555a33c79ad0 0x555a33c7bf70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a760759cbe13cd9d server_cookie=b9aed332229f74c in_seq=0 out_seq=0 2024-04-25T01:59:20.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.384+0000 7fa757e00640 10 client.26323 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:59:20.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.385+0000 7fa757e00640 1 -- 172.21.15.102:0/4218466814 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a33bd0a20 con 0x555a33c79ad0 2024-04-25T01:59:20.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.385+0000 7fa757e00640 10 client.26323 handle_command_reply: tid=0 2024-04-25T01:59:20.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.385+0000 7fa77ff7c740 1 client.26323 shutdown 2024-04-25T01:59:20.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 -- 172.21.15.102:0/4218466814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 msgr2=0x555a33c66740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c66740 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa768035040 tx=0x7fa76802a340 comp rx=0 tx=0).stop 2024-04-25T01:59:20.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 -- 172.21.15.102:0/4218466814 shutdown_connections 2024-04-25T01:59:20.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c69790 0x555a33c671c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555a33c79ad0 0x555a33c7bf70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5f0c0 0x555a33c66c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/4218466814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c66740 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 -- 172.21.15.102:0/4218466814 >> 172.21.15.102:0/4218466814 conn(0x555a33c5aac0 msgr2=0x555a33c607c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa757e00640 0 client.26323 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:59:20.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 -- 172.21.15.102:0/4218466814 shutdown_connections 2024-04-25T01:59:20.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 1 -- 172.21.15.102:0/4218466814 wait complete. 2024-04-25T01:59:20.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.386+0000 7fa77ff7c740 20 client.26323 trim_cache size 0 max 16384 2024-04-25T01:59:20.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.407+0000 7fa77ff7c740 1 Processor -- start 2024-04-25T01:59:20.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.408+0000 7fa77ff7c740 1 -- start start 2024-04-25T01:59:20.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.408+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c5a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.408+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c66fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.408+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c60600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.408+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa770153160 con 0x555a33c5f0c0 2024-04-25T01:59:20.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.408+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa770150a00 con 0x555a33c69790 2024-04-25T01:59:20.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.408+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa77014e850 con 0x555a33c5e7a0 2024-04-25T01:59:20.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.408+0000 7fa776800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c60600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa776800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c60600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41460/0 (socket says 172.21.15.102:41460) 2024-04-25T01:59:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa776800640 1 -- 172.21.15.102:0/298723609 learned_addr learned my addr 172.21.15.102:0/298723609 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa776800640 1 -- 172.21.15.102:0/298723609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 msgr2=0x555a33c5a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa775400640 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c66fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa775e00640 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c5a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa776800640 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c5a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa776800640 1 -- 172.21.15.102:0/298723609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 msgr2=0x555a33c66fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa776800640 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c66fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa776800640 1 -- 172.21.15.102:0/298723609 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a33c5eef0 con 0x555a33c69790 2024-04-25T01:59:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.409+0000 7fa775400640 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c66fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.410+0000 7fa776800640 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c60600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa778103660 tx=0x7fa778107450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a47062ea0b1d0b81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.410+0000 7fa762a00640 1 -- 172.21.15.102:0/298723609 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77810f900 con 0x555a33c69790 2024-04-25T01:59:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.410+0000 7fa762a00640 1 -- 172.21.15.102:0/298723609 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa778104f90 con 0x555a33c69790 2024-04-25T01:59:20.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.410+0000 7fa762a00640 1 -- 172.21.15.102:0/298723609 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7781060e0 con 0x555a33c69790 2024-04-25T01:59:20.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.411+0000 7fa77ff7c740 1 -- 172.21.15.102:0/298723609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 msgr2=0x555a33c60600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.412+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c60600 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa778103660 tx=0x7fa778107450 comp rx=0 tx=0).stop 2024-04-25T01:59:20.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.412+0000 7fa77ff7c740 1 -- 172.21.15.102:0/298723609 shutdown_connections 2024-04-25T01:59:20.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.412+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c69790 0x555a33c60600 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.412+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33c66fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.412+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/298723609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c5e7a0 0x555a33c5a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.412+0000 7fa77ff7c740 1 -- 172.21.15.102:0/298723609 >> 172.21.15.102:0/298723609 conn(0x555a33c5aac0 msgr2=0x555a33c5fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.413+0000 7fa77ff7c740 1 -- 172.21.15.102:0/298723609 shutdown_connections 2024-04-25T01:59:20.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.413+0000 7fa77ff7c740 1 -- 172.21.15.102:0/298723609 wait complete. 2024-04-25T01:59:20.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.413+0000 7fa77ff7c740 1 -- start start 2024-04-25T01:59:20.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.414+0000 7fa77ff7c740 1 Processor -- start 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.414+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5e7a0 0x555a33cd8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.414+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33cd8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.414+0000 7fa77ff7c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c69790 0x555a33c6cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.414+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa77014eec0 con 0x555a33c5f0c0 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.414+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa770150330 con 0x555a33c5e7a0 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.414+0000 7fa77ff7c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa77014f590 con 0x555a33c69790 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.414+0000 7fa77ff7c740 10 client.0 _ll_register_callbacks cb 0x7fa778067a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa776800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c69790 0x555a33c6cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa775e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5e7a0 0x555a33cd8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa776800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c69790 0x555a33c6cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38314/0 (socket says 172.21.15.102:38314) 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa776800640 1 -- 172.21.15.102:0/579079515 learned_addr learned my addr 172.21.15.102:0/579079515 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa775e00640 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5e7a0 0x555a33cd8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41474/0 (socket says 172.21.15.102:41474) 2024-04-25T01:59:20.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa775400640 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33cd8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa775e00640 1 -- 172.21.15.102:0/579079515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c69790 msgr2=0x555a33c6cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa775e00640 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c69790 0x555a33c6cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa775e00640 1 -- 172.21.15.102:0/579079515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 msgr2=0x555a33cd8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa775e00640 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33cd8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa775e00640 1 -- 172.21.15.102:0/579079515 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a33c6d440 con 0x555a33c5e7a0 2024-04-25T01:59:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.415+0000 7fa775400640 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33cd8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.416+0000 7fa775e00640 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5e7a0 0x555a33cd8410 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa7680057c0 tx=0x7fa76803d700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=571d35ad2113f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.416+0000 7fa757e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.416+0000 7fa757e00640 1 -- 172.21.15.102:0/579079515 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768037040 con 0x555a33c5e7a0 2024-04-25T01:59:20.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.416+0000 7fa757e00640 1 -- 172.21.15.102:0/579079515 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fa768003980 con 0x555a33c5e7a0 2024-04-25T01:59:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.417+0000 7fa77ff7c740 1 -- 172.21.15.102:0/579079515 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa778104f90 con 0x555a33c5e7a0 2024-04-25T01:59:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.417+0000 7fa757e00640 1 -- 172.21.15.102:0/579079515 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa768003b80 con 0x555a33c5e7a0 2024-04-25T01:59:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.417+0000 7fa77ff7c740 10 client.26354 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.417+0000 7fa77ff7c740 1 -- 172.21.15.102:0/579079515 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a33cdfb20 con 0x555a33c5e7a0 2024-04-25T01:59:20.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.417+0000 7fa757e00640 1 -- 172.21.15.102:0/579079515 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa778104f90 con 0x555a33c5e7a0 2024-04-25T01:59:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.418+0000 7fa757e00640 1 -- 172.21.15.102:0/579079515 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7fa768032440 con 0x555a33c5e7a0 2024-04-25T01:59:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.418+0000 7fa77ff7c740 10 client.26354 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.418+0000 7fa77ff7c740 10 client.26354 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:59:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.418+0000 7fa77ff7c740 20 client.26354 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.418+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555a33ce0180 0x555a33ce2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.418+0000 7fa77ff7c740 4 client.26354 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:59:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.418+0000 7fa77ff7c740 1 -- 172.21.15.102:0/579079515 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a33bcad70 con 0x555a33ce0180 2024-04-25T01:59:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.418+0000 7fa775400640 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555a33ce0180 0x555a33ce2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.419+0000 7fa775400640 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555a33ce0180 0x555a33ce2570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b3ff912dcf2d471f server_cookie=164703b8476ed556 in_seq=0 out_seq=0 2024-04-25T01:59:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.419+0000 7fa757e00640 10 client.26354 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:59:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.419+0000 7fa757e00640 1 -- 172.21.15.102:0/579079515 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a33bcad70 con 0x555a33ce0180 2024-04-25T01:59:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.419+0000 7fa757e00640 10 client.26354 handle_command_reply: tid=0 2024-04-25T01:59:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.419+0000 7fa77ff7c740 1 client.26354 shutdown 2024-04-25T01:59:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.419+0000 7fa77ff7c740 1 -- 172.21.15.102:0/579079515 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5e7a0 msgr2=0x555a33cd8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.419+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5e7a0 0x555a33cd8410 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa7680057c0 tx=0x7fa76803d700 comp rx=0 tx=0).stop 2024-04-25T01:59:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.420+0000 7fa77ff7c740 1 -- 172.21.15.102:0/579079515 shutdown_connections 2024-04-25T01:59:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.420+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a33c69790 0x555a33c6cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.420+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555a33c5f0c0 0x555a33cd8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.420+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x555a33ce0180 0x555a33ce2570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.420+0000 7fa77ff7c740 1 --2- 172.21.15.102:0/579079515 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555a33c5e7a0 0x555a33cd8410 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.420+0000 7fa77ff7c740 1 -- 172.21.15.102:0/579079515 >> 172.21.15.102:0/579079515 conn(0x555a33c5aac0 msgr2=0x555a33c60780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.420+0000 7fa757e00640 0 client.26354 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:59:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.421+0000 7fa77ff7c740 1 -- 172.21.15.102:0/579079515 shutdown_connections 2024-04-25T01:59:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.421+0000 7fa77ff7c740 1 -- 172.21.15.102:0/579079515 wait complete. 2024-04-25T01:59:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.421+0000 7fa77ff7c740 20 client.26354 trim_cache size 0 max 16384 2024-04-25T01:59:20.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:20 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/37067769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:59:20.426 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:59:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.422+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa75c0a2f00 msgr2=0x7fa75c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.422+0000 7fa777200640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa75c0a2f00 0x7fa75c0a53c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x555a33c5fd60 tx=0x7fa7680056d0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.422+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7700a3940 msgr2=0x7fa7700a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.422+0000 7fa777200640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7700a3940 0x7fa7700a3df0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa778067580 tx=0x7fa77807ae70 comp rx=0 tx=0).stop 2024-04-25T01:59:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.423+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 shutdown_connections 2024-04-25T01:59:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.423+0000 7fa777200640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa75c0a2f00 0x7fa75c0a53c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.423+0000 7fa777200640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7700a3940 0x7fa7700a3df0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.423+0000 7fa777200640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa77009d240 0x7fa7700af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.423+0000 7fa777200640 1 --2- 172.21.15.102:0/440549652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa770098170 0x7fa7700a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.423+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 >> 172.21.15.102:0/440549652 conn(0x7fa770093a10 msgr2=0x7fa770094dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.423+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 shutdown_connections 2024-04-25T01:59:20.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.423+0000 7fa777200640 1 -- 172.21.15.102:0/440549652 wait complete. 2024-04-25T01:59:20.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:59:20.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.709+0000 7f79a8200640 1 -- 172.21.15.102:0/23944481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a0103980 msgr2=0x7f79a0103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.709+0000 7f79a8200640 1 --2- 172.21.15.102:0/23944481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a0103980 0x7f79a0103e00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f798c009930 tx=0x7f798c02a440 comp rx=0 tx=0).stop 2024-04-25T01:59:20.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.709+0000 7f79a8200640 1 -- 172.21.15.102:0/23944481 shutdown_connections 2024-04-25T01:59:20.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.709+0000 7f79a8200640 1 --2- 172.21.15.102:0/23944481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79a0104340 0x7f79a0110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.709+0000 7f79a8200640 1 --2- 172.21.15.102:0/23944481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a0103980 0x7f79a0103e00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.709+0000 7f79a8200640 1 --2- 172.21.15.102:0/23944481 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79a0109990 0x7f79a0109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.709+0000 7f79a8200640 1 -- 172.21.15.102:0/23944481 >> 172.21.15.102:0/23944481 conn(0x7f79a00ff4d0 msgr2=0x7f79a01018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.710+0000 7f79a8200640 1 -- 172.21.15.102:0/23944481 shutdown_connections 2024-04-25T01:59:20.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.710+0000 7f79a8200640 1 -- 172.21.15.102:0/23944481 wait complete. 2024-04-25T01:59:20.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.711+0000 7f79a8200640 1 Processor -- start 2024-04-25T01:59:20.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.712+0000 7f79a8200640 1 -- start start 2024-04-25T01:59:20.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.712+0000 7f79a8200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79a0103980 0x7f79a0075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.712+0000 7f79a8200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79a0104340 0x7f79a0076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.712+0000 7f79a8200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a0109990 0x7f79a00766d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.712+0000 7f79a8200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f79a0113df0 con 0x7f79a0109990 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a8200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f79a0113c70 con 0x7f79a0104340 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a8200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f79a0113f70 con 0x7f79a0103980 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79a0103980 0x7f79a0075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79a0103980 0x7f79a0075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38322/0 (socket says 172.21.15.102:38322) 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a5800640 1 -- 172.21.15.102:0/3275326027 learned_addr learned my addr 172.21.15.102:0/3275326027 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a4e00640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79a0104340 0x7f79a0076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a6200640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a0109990 0x7f79a00766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a5800640 1 -- 172.21.15.102:0/3275326027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79a0104340 msgr2=0x7f79a0076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a5800640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79a0104340 0x7f79a0076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a5800640 1 -- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a0109990 msgr2=0x7f79a00766d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a5800640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a0109990 0x7f79a00766d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.713+0000 7f79a5800640 1 -- 172.21.15.102:0/3275326027 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a019a100 con 0x7f79a0103980 2024-04-25T01:59:20.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.714+0000 7f79a5800640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79a0103980 0x7f79a0075c50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f799400ea60 tx=0x7f799400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.714+0000 7f799e000640 1 -- 172.21.15.102:0/3275326027 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799400cde0 con 0x7f79a0103980 2024-04-25T01:59:20.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.715+0000 7f799e000640 1 -- 172.21.15.102:0/3275326027 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79940035f0 con 0x7f79a0103980 2024-04-25T01:59:20.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.715+0000 7f799e000640 1 -- 172.21.15.102:0/3275326027 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994010670 con 0x7f79a0103980 2024-04-25T01:59:20.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.715+0000 7f79a8200640 1 -- 172.21.15.102:0/3275326027 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a019a3f0 con 0x7f79a0103980 2024-04-25T01:59:20.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.715+0000 7f79a8200640 1 -- 172.21.15.102:0/3275326027 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a019a8c0 con 0x7f79a0103980 2024-04-25T01:59:20.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.719+0000 7f799e000640 1 -- 172.21.15.102:0/3275326027 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f79940037a0 con 0x7f79a0103980 2024-04-25T01:59:20.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.720+0000 7f79ab009740 1 Processor -- start 2024-04-25T01:59:20.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.720+0000 7f79ab009740 1 -- start start 2024-04-25T01:59:20.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79ab009740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb3a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79ab009740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb45070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79ab009740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb479a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79ab009740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x556f6fa9f4e0 con 0x556f6fb3aee0 2024-04-25T01:59:20.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79ab009740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556f6fa47a70 con 0x556f6fb3a5c0 2024-04-25T01:59:20.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79ab009740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556f6fa9b4a0 con 0x556f6fb455b0 2024-04-25T01:59:20.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79a4e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79a5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb45070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79a4e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38338/0 (socket says 172.21.15.102:38338) 2024-04-25T01:59:20.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79a4e00640 1 -- 172.21.15.102:0/3593748635 learned_addr learned my addr 172.21.15.102:0/3593748635 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.721+0000 7f79a6200640 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb3a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.722+0000 7f79a4e00640 1 -- 172.21.15.102:0/3593748635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 msgr2=0x556f6fb3a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.722+0000 7f79a4e00640 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb3a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.722+0000 7f79a4e00640 1 -- 172.21.15.102:0/3593748635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 msgr2=0x556f6fb45070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.722+0000 7f79a4e00640 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb45070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.722+0000 7f79a4e00640 1 -- 172.21.15.102:0/3593748635 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f6f7b33f0 con 0x556f6fb455b0 2024-04-25T01:59:20.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.722+0000 7f799e000640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f79880a2e00 0x7f79880a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.722+0000 7f799e000640 1 -- 172.21.15.102:0/3275326027 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f79940cf7e0 con 0x7f79a0103980 2024-04-25T01:59:20.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.722+0000 7f79a6200640 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb3a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:20.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.722+0000 7f79a5800640 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb45070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:20.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.723+0000 7f79a6200640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f79880a2e00 0x7f79880a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.723+0000 7f79a4e00640 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb479a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f798c02aa30 tx=0x7f798c005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3da2abb50ee4457d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.723+0000 7f7992a00640 1 -- 172.21.15.102:0/3593748635 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c035030 con 0x556f6fb455b0 2024-04-25T01:59:20.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.724+0000 7f7992a00640 1 -- 172.21.15.102:0/3593748635 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f798c0043c0 con 0x556f6fb455b0 2024-04-25T01:59:20.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.724+0000 7f7992a00640 1 -- 172.21.15.102:0/3593748635 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c0035f0 con 0x556f6fb455b0 2024-04-25T01:59:20.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.725+0000 7f79a6200640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f79880a2e00 0x7f79880a52c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7998005fd0 tx=0x7f7998009500 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.725+0000 7f79ab009740 1 -- 172.21.15.102:0/3593748635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 msgr2=0x556f6fb479a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.726+0000 7f79ab009740 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb479a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f798c02aa30 tx=0x7f798c005e60 comp rx=0 tx=0).stop 2024-04-25T01:59:20.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.727+0000 7f79ab009740 1 -- 172.21.15.102:0/3593748635 shutdown_connections 2024-04-25T01:59:20.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.727+0000 7f79ab009740 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb479a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.727+0000 7f79ab009740 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb45070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.727+0000 7f79ab009740 1 --2- 172.21.15.102:0/3593748635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb3a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.727+0000 7f79ab009740 1 -- 172.21.15.102:0/3593748635 >> 172.21.15.102:0/3593748635 conn(0x556f6fb368e0 msgr2=0x556f6fb36cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.728+0000 7f79ab009740 1 -- 172.21.15.102:0/3593748635 shutdown_connections 2024-04-25T01:59:20.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.728+0000 7f79ab009740 1 -- 172.21.15.102:0/3593748635 wait complete. 2024-04-25T01:59:20.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.729+0000 7f79ab009740 1 -- start start 2024-04-25T01:59:20.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.729+0000 7f79ab009740 1 Processor -- start 2024-04-25T01:59:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.731+0000 7f79ab009740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb42530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.731+0000 7f79ab009740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3aee0 0x556f6fb42a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.731+0000 7f79ab009740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb455b0 0x556f6fb42fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.731+0000 7f79ab009740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x556f6fb48340 con 0x556f6fb455b0 2024-04-25T01:59:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.731+0000 7f79ab009740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556f6fa5aec0 con 0x556f6fb3aee0 2024-04-25T01:59:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.731+0000 7f79ab009740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556f6fb484c0 con 0x556f6fb3a5c0 2024-04-25T01:59:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.731+0000 7f79ab009740 10 client.0 _ll_register_callbacks cb 0x556f6fa973f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:20.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.732+0000 7f79a5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb42530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.732+0000 7f79a5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb42530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38352/0 (socket says 172.21.15.102:38352) 2024-04-25T01:59:20.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.732+0000 7f79a5800640 1 -- 172.21.15.102:0/2608579697 learned_addr learned my addr 172.21.15.102:0/2608579697 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.732+0000 7f79a6200640 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb455b0 0x556f6fb42fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.732+0000 7f79a4e00640 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3aee0 0x556f6fb42a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.732+0000 7f79a4e00640 1 -- 172.21.15.102:0/2608579697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 msgr2=0x556f6fb42530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.732+0000 7f79a4e00640 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb42530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.733+0000 7f79a4e00640 1 -- 172.21.15.102:0/2608579697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb455b0 msgr2=0x556f6fb42fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.733+0000 7f79a4e00640 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb455b0 0x556f6fb42fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.733+0000 7f79a4e00640 1 -- 172.21.15.102:0/2608579697 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f6f87d7d0 con 0x556f6fb3aee0 2024-04-25T01:59:20.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.734+0000 7f79a4e00640 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3aee0 0x556f6fb42a70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f798c02a960 tx=0x7f798c0045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77157219e5980be9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:20 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/37067769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:59:20.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.736+0000 7f7987400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.736+0000 7f7987400640 1 -- 172.21.15.102:0/2608579697 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c035070 con 0x556f6fb3aee0 2024-04-25T01:59:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.736+0000 7f7987400640 1 -- 172.21.15.102:0/2608579697 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f798c003b10 con 0x556f6fb3aee0 2024-04-25T01:59:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.739+0000 7f79ab009740 1 -- 172.21.15.102:0/2608579697 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f798c0043c0 con 0x556f6fb3aee0 2024-04-25T01:59:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.740+0000 7f7987400640 1 -- 172.21.15.102:0/2608579697 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f798c009520 con 0x556f6fb3aee0 2024-04-25T01:59:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.740+0000 7f79ab009740 10 client.26366 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.740+0000 7f79ab009740 1 -- 172.21.15.102:0/2608579697 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556f6fb545f0 con 0x556f6fb3aee0 2024-04-25T01:59:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.741+0000 7f7987400640 1 -- 172.21.15.102:0/2608579697 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f798c0043c0 con 0x556f6fb3aee0 2024-04-25T01:59:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.741+0000 7f7987400640 1 -- 172.21.15.102:0/2608579697 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f798c004dd0 con 0x556f6fb3aee0 2024-04-25T01:59:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.743+0000 7f79ab009740 10 client.26366 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.743+0000 7f79ab009740 10 client.26366 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:59:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.743+0000 7f79ab009740 20 client.26366 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.743+0000 7f79ab009740 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x556f6fb55940 0x556f6fb57d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.743+0000 7f79ab009740 4 client.26366 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.743+0000 7f79ab009740 1 -- 172.21.15.102:0/2608579697 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f6faaca20 con 0x556f6fb55940 2024-04-25T01:59:20.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.744+0000 7f79a5800640 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x556f6fb55940 0x556f6fb57d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.745+0000 7f79a5800640 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x556f6fb55940 0x556f6fb57d30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b7eb170676d9287 server_cookie=83ec4cf58db23254 in_seq=0 out_seq=0 2024-04-25T01:59:20.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.745+0000 7f7987400640 10 client.26366 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:59:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.746+0000 7f7987400640 1 -- 172.21.15.102:0/2608579697 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556f6faaca20 con 0x556f6fb55940 2024-04-25T01:59:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.746+0000 7f7987400640 10 client.26366 handle_command_reply: tid=0 2024-04-25T01:59:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.747+0000 7f79ab009740 1 client.26366 shutdown 2024-04-25T01:59:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.750+0000 7f79ab009740 1 -- 172.21.15.102:0/2608579697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3aee0 msgr2=0x556f6fb42a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.750+0000 7f79ab009740 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3aee0 0x556f6fb42a70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f798c02a960 tx=0x7f798c0045e0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.750+0000 7f79ab009740 1 -- 172.21.15.102:0/2608579697 shutdown_connections 2024-04-25T01:59:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.750+0000 7f79ab009740 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x556f6fb55940 0x556f6fb57d30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.750+0000 7f79ab009740 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb455b0 0x556f6fb42fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.750+0000 7f79ab009740 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3aee0 0x556f6fb42a70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.750+0000 7f79ab009740 1 --2- 172.21.15.102:0/2608579697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb42530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.750+0000 7f79ab009740 1 -- 172.21.15.102:0/2608579697 >> 172.21.15.102:0/2608579697 conn(0x556f6fb368e0 msgr2=0x556f6fb35b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.750+0000 7f7987400640 0 client.26366 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:59:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.752+0000 7f79ab009740 1 -- 172.21.15.102:0/2608579697 shutdown_connections 2024-04-25T01:59:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.752+0000 7f79ab009740 1 -- 172.21.15.102:0/2608579697 wait complete. 2024-04-25T01:59:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.752+0000 7f79ab009740 20 client.26366 trim_cache size 0 max 16384 2024-04-25T01:59:20.757 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:20 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/37067769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:59:20.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.785+0000 7f79ab009740 1 Processor -- start 2024-04-25T01:59:20.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.785+0000 7f79ab009740 1 -- start start 2024-04-25T01:59:20.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79ab009740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79ab009740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb40e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79ab009740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb5a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79ab009740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01becb0 con 0x556f6fb3aee0 2024-04-25T01:59:20.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79ab009740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01bc550 con 0x556f6fb3a5c0 2024-04-25T01:59:20.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79ab009740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01a12a0 con 0x556f6fb455b0 2024-04-25T01:59:20.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79a5800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79a5800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb362c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41508/0 (socket says 172.21.15.102:41508) 2024-04-25T01:59:20.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79a5800640 1 -- 172.21.15.102:0/239334073 learned_addr learned my addr 172.21.15.102:0/239334073 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.786+0000 7f79a6200640 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb5a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.787+0000 7f79a4e00640 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb40e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.787+0000 7f79a5800640 1 -- 172.21.15.102:0/239334073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 msgr2=0x556f6fb5a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.787+0000 7f79a5800640 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb5a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.787+0000 7f79a5800640 1 -- 172.21.15.102:0/239334073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 msgr2=0x556f6fb40e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.787+0000 7f79a5800640 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb40e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.787+0000 7f79a5800640 1 -- 172.21.15.102:0/239334073 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f6fb41520 con 0x556f6fb3a5c0 2024-04-25T01:59:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.787+0000 7f79a4e00640 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb40e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.787+0000 7f79a6200640 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb5a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.787+0000 7f79a5800640 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb362c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f799409b9d0 tx=0x7f79940d55a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2c9935b74af8198 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.788+0000 7f7992000640 1 -- 172.21.15.102:0/239334073 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799409b630 con 0x556f6fb3a5c0 2024-04-25T01:59:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.788+0000 7f7992000640 1 -- 172.21.15.102:0/239334073 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f799409c3d0 con 0x556f6fb3a5c0 2024-04-25T01:59:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.788+0000 7f7992000640 1 -- 172.21.15.102:0/239334073 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994003b30 con 0x556f6fb3a5c0 2024-04-25T01:59:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.789+0000 7f79ab009740 1 -- 172.21.15.102:0/239334073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 msgr2=0x556f6fb362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.789+0000 7f79ab009740 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb362c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f799409b9d0 tx=0x7f79940d55a0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.789+0000 7f79ab009740 1 -- 172.21.15.102:0/239334073 shutdown_connections 2024-04-25T01:59:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.789+0000 7f79ab009740 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb455b0 0x556f6fb5a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.789+0000 7f79ab009740 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb40e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.789+0000 7f79ab009740 1 --2- 172.21.15.102:0/239334073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb3a5c0 0x556f6fb362c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.789+0000 7f79ab009740 1 -- 172.21.15.102:0/239334073 >> 172.21.15.102:0/239334073 conn(0x556f6fb368e0 msgr2=0x556f6fb3c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.790+0000 7f79ab009740 1 -- 172.21.15.102:0/239334073 shutdown_connections 2024-04-25T01:59:20.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.790+0000 7f79ab009740 1 -- 172.21.15.102:0/239334073 wait complete. 2024-04-25T01:59:20.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.791+0000 7f79ab009740 1 -- start start 2024-04-25T01:59:20.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.792+0000 7f79ab009740 1 Processor -- start 2024-04-25T01:59:20.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.793+0000 7f79ab009740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb58a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.793+0000 7f79ab009740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb58fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.794+0000 7f79ab009740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb455b0 0x556f6fbbab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.794+0000 7f79a5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb58a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.794+0000 7f79a5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb58a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38366/0 (socket says 172.21.15.102:38366) 2024-04-25T01:59:20.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.794+0000 7f79a5800640 1 -- 172.21.15.102:0/1113567811 learned_addr learned my addr 172.21.15.102:0/1113567811 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:20.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.794+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01a1910 con 0x556f6fb3aee0 2024-04-25T01:59:20.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.794+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01bbe80 con 0x556f6fb455b0 2024-04-25T01:59:20.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.794+0000 7f79a6200640 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb455b0 0x556f6fbbab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.794+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01a1fe0 con 0x556f6fb3a5c0 2024-04-25T01:59:20.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.795+0000 7f79ab009740 10 client.0 _ll_register_callbacks cb 0x7f79940d5400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:20.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.795+0000 7f79a4e00640 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb58fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.795+0000 7f79a4e00640 1 -- 172.21.15.102:0/1113567811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 msgr2=0x556f6fb58a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.795+0000 7f79a4e00640 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb58a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.795+0000 7f79a4e00640 1 -- 172.21.15.102:0/1113567811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb455b0 msgr2=0x556f6fbbab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.795+0000 7f79a4e00640 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb455b0 0x556f6fbbab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.795+0000 7f79a4e00640 1 -- 172.21.15.102:0/1113567811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f6fbbb060 con 0x556f6fb3aee0 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.795+0000 7f79a4e00640 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb58fb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f798c004650 tx=0x7f798c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa0371ba083a430 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.795+0000 7f7987400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.796+0000 7f7987400640 1 -- 172.21.15.102:0/1113567811 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c035070 con 0x556f6fb3aee0 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.796+0000 7f7987400640 1 -- 172.21.15.102:0/1113567811 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f798c003b10 con 0x556f6fb3aee0 2024-04-25T01:59:20.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.796+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f799409c3d0 con 0x556f6fb3aee0 2024-04-25T01:59:20.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.797+0000 7f7987400640 1 -- 172.21.15.102:0/1113567811 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f798c065050 con 0x556f6fb3aee0 2024-04-25T01:59:20.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.797+0000 7f79ab009740 10 client.16002 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:20.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.797+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556f6fbbc240 con 0x556f6fb3aee0 2024-04-25T01:59:20.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.798+0000 7f7987400640 1 -- 172.21.15.102:0/1113567811 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f799409c3d0 con 0x556f6fb3aee0 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.798+0000 7f7987400640 1 -- 172.21.15.102:0/1113567811 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f798c0318c0 con 0x556f6fb3aee0 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.798+0000 7f79ab009740 10 client.16002 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.798+0000 7f79ab009740 10 client.16002 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.798+0000 7f79ab009740 20 client.16002 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.798+0000 7f79ab009740 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x556f6fbbc8a0 0x556f6fbbecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.798+0000 7f79ab009740 4 client.16002 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.798+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f6faa6d70 con 0x556f6fbbc8a0 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.798+0000 7f79a5800640 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x556f6fbbc8a0 0x556f6fbbecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.799+0000 7f7987400640 10 client.16002 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.799+0000 7f79a5800640 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x556f6fbbc8a0 0x556f6fbbecc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b3e1699a58355bdd server_cookie=ef6dbeda5e66c27 in_seq=0 out_seq=0 2024-04-25T01:59:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.799+0000 7f7987400640 1 -- 172.21.15.102:0/1113567811 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f6faa6d70 con 0x556f6fbbc8a0 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.799+0000 7f7987400640 10 client.16002 handle_command_reply: tid=0 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.799+0000 7f79ab009740 1 client.16002 shutdown 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.799+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 msgr2=0x556f6fb58fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.799+0000 7f79ab009740 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb58fb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f798c004650 tx=0x7f798c005fd0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 shutdown_connections 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f79ab009740 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x556f6fbbc8a0 0x556f6fbbecc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f79ab009740 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556f6fb455b0 0x556f6fbbab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f79ab009740 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556f6fb3aee0 0x556f6fb58fb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f79ab009740 1 --2- 172.21.15.102:0/1113567811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556f6fb3a5c0 0x556f6fb58a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 >> 172.21.15.102:0/1113567811 conn(0x556f6fb368e0 msgr2=0x556f6fb50740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f7987400640 0 client.16002 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:59:20.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 shutdown_connections 2024-04-25T01:59:20.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f79ab009740 1 -- 172.21.15.102:0/1113567811 wait complete. 2024-04-25T01:59:20.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.800+0000 7f79ab009740 20 client.16002 trim_cache size 0 max 16384 2024-04-25T01:59:20.806 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:59:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.801+0000 7f79a8200640 1 -- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f79880a2e00 msgr2=0x7f79880a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.801+0000 7f79a8200640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f79880a2e00 0x7f79880a52c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7998005fd0 tx=0x7f7998009500 comp rx=0 tx=0).stop 2024-04-25T01:59:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.801+0000 7f79a8200640 1 -- 172.21.15.102:0/3275326027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79a0103980 msgr2=0x7f79a0075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.801+0000 7f79a8200640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79a0103980 0x7f79a0075c50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f799400ea60 tx=0x7f799400ef20 comp rx=0 tx=0).stop 2024-04-25T01:59:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.802+0000 7f79a8200640 1 -- 172.21.15.102:0/3275326027 shutdown_connections 2024-04-25T01:59:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.802+0000 7f79a8200640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f79880a2e00 0x7f79880a52c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.802+0000 7f79a8200640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79a0109990 0x7f79a00766d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.802+0000 7f79a8200640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79a0104340 0x7f79a0076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.802+0000 7f79a8200640 1 --2- 172.21.15.102:0/3275326027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f79a0103980 0x7f79a0075c50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.802+0000 7f79a8200640 1 -- 172.21.15.102:0/3275326027 >> 172.21.15.102:0/3275326027 conn(0x7f79a00ff4d0 msgr2=0x7f79a00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.803+0000 7f79a8200640 1 -- 172.21.15.102:0/3275326027 shutdown_connections 2024-04-25T01:59:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:20.803+0000 7f79a8200640 1 -- 172.21.15.102:0/3275326027 wait complete. 2024-04-25T01:59:20.834 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:59:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.090+0000 7f2347000640 1 -- 172.21.15.102:0/2242266871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 msgr2=0x7f2340106990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.090+0000 7f2347000640 1 --2- 172.21.15.102:0/2242266871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 0x7f2340106990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2334009930 tx=0x7f233402a460 comp rx=0 tx=0).stop 2024-04-25T01:59:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.090+0000 7f2347000640 1 -- 172.21.15.102:0/2242266871 shutdown_connections 2024-04-25T01:59:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.090+0000 7f2347000640 1 --2- 172.21.15.102:0/2242266871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2340100eb0 0x7f234010d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.090+0000 7f2347000640 1 --2- 172.21.15.102:0/2242266871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23401004f0 0x7f2340100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.090+0000 7f2347000640 1 --2- 172.21.15.102:0/2242266871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 0x7f2340106990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.090+0000 7f2347000640 1 -- 172.21.15.102:0/2242266871 >> 172.21.15.102:0/2242266871 conn(0x7f23400fbfc0 msgr2=0x7f23400fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.090+0000 7f2347000640 1 -- 172.21.15.102:0/2242266871 shutdown_connections 2024-04-25T01:59:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.090+0000 7f2347000640 1 -- 172.21.15.102:0/2242266871 wait complete. 2024-04-25T01:59:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.092+0000 7f2347000640 1 Processor -- start 2024-04-25T01:59:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.092+0000 7f2347000640 1 -- start start 2024-04-25T01:59:21.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2347000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23401004f0 0x7f2340197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2347000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2340100eb0 0x7f2340198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2347000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 0x7f234019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2347000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f23401119c0 con 0x7f2340100eb0 2024-04-25T01:59:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2347000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2340111840 con 0x7f23401065b0 2024-04-25T01:59:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2347000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2340111b40 con 0x7f23401004f0 2024-04-25T01:59:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2345c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23401004f0 0x7f2340197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2346600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 0x7f234019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2346600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 0x7f234019c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41538/0 (socket says 172.21.15.102:41538) 2024-04-25T01:59:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.093+0000 7f2346600640 1 -- 172.21.15.102:0/376076313 learned_addr learned my addr 172.21.15.102:0/376076313 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.094+0000 7f2345200640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2340100eb0 0x7f2340198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.094+0000 7f2345c00640 1 -- 172.21.15.102:0/376076313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 msgr2=0x7f234019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.094+0000 7f2345c00640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 0x7f234019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.094+0000 7f2345c00640 1 -- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2340100eb0 msgr2=0x7f2340198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.094+0000 7f2345c00640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2340100eb0 0x7f2340198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.094+0000 7f2345c00640 1 -- 172.21.15.102:0/376076313 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234019cec0 con 0x7f23401004f0 2024-04-25T01:59:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.094+0000 7f2346600640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 0x7f234019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.095+0000 7f2345c00640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23401004f0 0x7f2340197f20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2334009230 tx=0x7f2334005080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.095+0000 7f233e000640 1 -- 172.21.15.102:0/376076313 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2334040030 con 0x7f23401004f0 2024-04-25T01:59:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.095+0000 7f233e000640 1 -- 172.21.15.102:0/376076313 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2334005480 con 0x7f23401004f0 2024-04-25T01:59:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.095+0000 7f233e000640 1 -- 172.21.15.102:0/376076313 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233403c9a0 con 0x7f23401004f0 2024-04-25T01:59:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.096+0000 7f2347000640 1 -- 172.21.15.102:0/376076313 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f234019d150 con 0x7f23401004f0 2024-04-25T01:59:21.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.096+0000 7f2347000640 1 -- 172.21.15.102:0/376076313 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f234019d5c0 con 0x7f23401004f0 2024-04-25T01:59:21.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.102+0000 7f233e000640 1 -- 172.21.15.102:0/376076313 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2334043030 con 0x7f23401004f0 2024-04-25T01:59:21.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.104+0000 7f233e000640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23280a2ed0 0x7f23280a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.104+0000 7f233e000640 1 -- 172.21.15.102:0/376076313 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f23340efa30 con 0x7f23401004f0 2024-04-25T01:59:21.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.104+0000 7f2345200640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23280a2ed0 0x7f23280a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.105+0000 7f234b5bb740 1 Processor -- start 2024-04-25T01:59:21.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.105+0000 7f234b5bb740 1 -- start start 2024-04-25T01:59:21.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.105+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fbdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.105+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.105+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fc8790 0x5596a5fcab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.105+0000 7f234b5bb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5596a5f224e0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.105+0000 7f234b5bb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5596a5ecaa70 con 0x5596a5fbd7a0 2024-04-25T01:59:21.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.105+0000 7f234b5bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5596a5f1e4a0 con 0x5596a5fc8790 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.106+0000 7f2346600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fc8790 0x5596a5fcab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.106+0000 7f2345c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fbdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.106+0000 7f2346600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fc8790 0x5596a5fcab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38384/0 (socket says 172.21.15.102:38384) 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.106+0000 7f2346600640 1 -- 172.21.15.102:0/1295680017 learned_addr learned my addr 172.21.15.102:0/1295680017 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.106+0000 7f2345200640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23280a2ed0 0x7f23280a5390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2338005c40 tx=0x7f2338005bd0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.106+0000 7f2345200640 1 --2- 172.21.15.102:0/1295680017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.107+0000 7f2345c00640 1 -- 172.21.15.102:0/1295680017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fc8790 msgr2=0x5596a5fcab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.107+0000 7f2345c00640 1 --2- 172.21.15.102:0/1295680017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fc8790 0x5596a5fcab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.107+0000 7f2345c00640 1 -- 172.21.15.102:0/1295680017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbe0c0 msgr2=0x5596a5fc8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:21.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.107+0000 7f2345c00640 1 --2- 172.21.15.102:0/1295680017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.107+0000 7f2345c00640 1 -- 172.21.15.102:0/1295680017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596a5c363f0 con 0x5596a5fbd7a0 2024-04-25T01:59:21.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.108+0000 7f2345c00640 1 --2- 172.21.15.102:0/1295680017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fbdb80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f23340ed150 tx=0x7f233402a460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c89284cf7453f0b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:21.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.108+0000 7f2332a00640 1 -- 172.21.15.102:0/1295680017 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233404a040 con 0x5596a5fbd7a0 2024-04-25T01:59:21.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.108+0000 7f2332a00640 1 -- 172.21.15.102:0/1295680017 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23340c4b90 con 0x5596a5fbd7a0 2024-04-25T01:59:21.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.108+0000 7f2332a00640 1 -- 172.21.15.102:0/1295680017 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23340bc180 con 0x5596a5fbd7a0 2024-04-25T01:59:21.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.109+0000 7f234b5bb740 1 -- 172.21.15.102:0/1295680017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 msgr2=0x5596a5fbdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:21.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.109+0000 7f234b5bb740 1 --2- 172.21.15.102:0/1295680017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fbdb80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f23340ed150 tx=0x7f233402a460 comp rx=0 tx=0).stop 2024-04-25T01:59:21.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.109+0000 7f234b5bb740 1 -- 172.21.15.102:0/1295680017 shutdown_connections 2024-04-25T01:59:21.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.109+0000 7f234b5bb740 1 --2- 172.21.15.102:0/1295680017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fc8790 0x5596a5fcab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.109+0000 7f234b5bb740 1 --2- 172.21.15.102:0/1295680017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.109+0000 7f234b5bb740 1 --2- 172.21.15.102:0/1295680017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fbdb80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.109+0000 7f234b5bb740 1 -- 172.21.15.102:0/1295680017 >> 172.21.15.102:0/1295680017 conn(0x5596a5fb9ac0 msgr2=0x5596a5fb9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:21.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.109+0000 7f234b5bb740 1 -- 172.21.15.102:0/1295680017 shutdown_connections 2024-04-25T01:59:21.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.109+0000 7f234b5bb740 1 -- 172.21.15.102:0/1295680017 wait complete. 2024-04-25T01:59:21.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.111+0000 7f234b5bb740 1 -- start start 2024-04-25T01:59:21.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.111+0000 7f234b5bb740 1 Processor -- start 2024-04-25T01:59:21.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbd7a0 0x5596a5fc57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 0x5596a5fc6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f234b5bb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5596a5fcb340 con 0x5596a5fbd7a0 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f234b5bb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5596a5eddec0 con 0x5596a5fc8790 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f234b5bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5596a5fcb4c0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f234b5bb740 10 client.0 _ll_register_callbacks cb 0x5596a5f1a3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f2345200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f2345200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38394/0 (socket says 172.21.15.102:38394) 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f2345200640 1 -- 172.21.15.102:0/420349496 learned_addr learned my addr 172.21.15.102:0/420349496 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.112+0000 7f2345c00640 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbd7a0 0x5596a5fc57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.113+0000 7f2346600640 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 0x5596a5fc6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.113+0000 7f2345200640 1 -- 172.21.15.102:0/420349496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 msgr2=0x5596a5fc6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:21.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.113+0000 7f2345200640 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 0x5596a5fc6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.113+0000 7f2345200640 1 -- 172.21.15.102:0/420349496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbd7a0 msgr2=0x5596a5fc57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:21.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.113+0000 7f2345200640 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbd7a0 0x5596a5fc57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.113+0000 7f2345200640 1 -- 172.21.15.102:0/420349496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596a5d007d0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.114+0000 7f2345200640 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc5d30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f23380116c0 tx=0x7f2338004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1db96163746c7167 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:21.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.114+0000 7f2327400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:21.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.115+0000 7f2327400640 1 -- 172.21.15.102:0/420349496 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233801f900 con 0x5596a5fbe0c0 2024-04-25T01:59:21.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.115+0000 7f2327400640 1 -- 172.21.15.102:0/420349496 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f233801faa0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.115+0000 7f234b5bb740 1 -- 172.21.15.102:0/420349496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23340c4b90 con 0x5596a5fbe0c0 2024-04-25T01:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.116+0000 7f2327400640 1 -- 172.21.15.102:0/420349496 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f233801fca0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.116+0000 7f234b5bb740 10 client.26353 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.116+0000 7f234b5bb740 1 -- 172.21.15.102:0/420349496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5596a5fd7690 con 0x5596a5fbe0c0 2024-04-25T01:59:21.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.116+0000 7f2327400640 1 -- 172.21.15.102:0/420349496 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23340c4b90 con 0x5596a5fbe0c0 2024-04-25T01:59:21.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.116+0000 7f2327400640 1 -- 172.21.15.102:0/420349496 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f23380104e0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.117+0000 7f234b5bb740 10 client.26353 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:21.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.117+0000 7f234b5bb740 10 client.26353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:21.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.117+0000 7f234b5bb740 20 client.26353 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:21.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.117+0000 7f234b5bb740 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5596a5fd8bf0 0x5596a5fdb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.117+0000 7f234b5bb740 4 client.26353 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:21.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.117+0000 7f234b5bb740 1 -- 172.21.15.102:0/420349496 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596a5f2fa20 con 0x5596a5fd8bf0 2024-04-25T01:59:21.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.118+0000 7f2345c00640 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5596a5fd8bf0 0x5596a5fdb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.118+0000 7f2345c00640 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5596a5fd8bf0 0x5596a5fdb010 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be7a85be41ddf524 server_cookie=7ffe64fb2d37aebc in_seq=0 out_seq=0 2024-04-25T01:59:21.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.119+0000 7f2327400640 10 client.26353 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:21.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.119+0000 7f2327400640 1 -- 172.21.15.102:0/420349496 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596a5f2fa20 con 0x5596a5fd8bf0 2024-04-25T01:59:21.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.119+0000 7f2327400640 10 client.26353 handle_command_reply: tid=0 2024-04-25T01:59:21.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.120+0000 7f234b5bb740 1 client.26353 shutdown 2024-04-25T01:59:21.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.120+0000 7f234b5bb740 1 -- 172.21.15.102:0/420349496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 msgr2=0x5596a5fc5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:21.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.120+0000 7f234b5bb740 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc5d30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f23380116c0 tx=0x7f2338004a40 comp rx=0 tx=0).stop 2024-04-25T01:59:21.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.121+0000 7f234b5bb740 1 -- 172.21.15.102:0/420349496 shutdown_connections 2024-04-25T01:59:21.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.121+0000 7f234b5bb740 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5596a5fd8bf0 0x5596a5fdb010 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.121+0000 7f234b5bb740 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 0x5596a5fc6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.121+0000 7f2327400640 0 client.26353 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:21.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.121+0000 7f234b5bb740 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fc5d30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.121+0000 7f234b5bb740 1 --2- 172.21.15.102:0/420349496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbd7a0 0x5596a5fc57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.121+0000 7f234b5bb740 1 -- 172.21.15.102:0/420349496 >> 172.21.15.102:0/420349496 conn(0x5596a5fb9ac0 msgr2=0x5596a5fbf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:21.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.122+0000 7f234b5bb740 1 -- 172.21.15.102:0/420349496 shutdown_connections 2024-04-25T01:59:21.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.122+0000 7f234b5bb740 1 -- 172.21.15.102:0/420349496 wait complete. 2024-04-25T01:59:21.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.123+0000 7f234b5bb740 20 client.26353 trim_cache size 0 max 16384 2024-04-25T01:59:21.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.150+0000 7f234b5bb740 1 Processor -- start 2024-04-25T01:59:21.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.151+0000 7f234b5bb740 1 -- start start 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.151+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbd7a0 0x5596a5fd4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.151+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fe0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.151+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 0x5596a5fe0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.152+0000 7f2345200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fe0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.152+0000 7f2345200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fe0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38402/0 (socket says 172.21.15.102:38402) 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.152+0000 7f2345200640 1 -- 172.21.15.102:0/1773752562 learned_addr learned my addr 172.21.15.102:0/1773752562 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.152+0000 7f2346600640 1 --2- 172.21.15.102:0/1773752562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 0x5596a5fe0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.153+0000 7f234b5bb740 1 -- 172.21.15.102:0/1773752562 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f23401ad900 con 0x5596a5fbd7a0 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.153+0000 7f234b5bb740 1 -- 172.21.15.102:0/1773752562 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f23401ab1a0 con 0x5596a5fc8790 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.153+0000 7f234b5bb740 1 -- 172.21.15.102:0/1773752562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f23401a8ff0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.154+0000 7f2346600640 1 -- 172.21.15.102:0/1773752562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 msgr2=0x5596a5fe0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.154+0000 7f2346600640 1 --2- 172.21.15.102:0/1773752562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fe0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.154+0000 7f2346600640 1 -- 172.21.15.102:0/1773752562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbd7a0 msgr2=0x5596a5fd4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:21.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.154+0000 7f2346600640 1 --2- 172.21.15.102:0/1773752562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbd7a0 0x5596a5fd4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.154+0000 7f2346600640 1 -- 172.21.15.102:0/1773752562 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596a5fbdef0 con 0x5596a5fc8790 2024-04-25T01:59:21.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.155+0000 7f2346600640 1 --2- 172.21.15.102:0/1773752562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 0x5596a5fe0dc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f232c015b30 tx=0x7f232c00b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ea529a0600b8afb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:21.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.156+0000 7f2332000640 1 -- 172.21.15.102:0/1773752562 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c00fc10 con 0x5596a5fc8790 2024-04-25T01:59:21.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.156+0000 7f2332000640 1 -- 172.21.15.102:0/1773752562 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f232c00fdb0 con 0x5596a5fc8790 2024-04-25T01:59:21.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.156+0000 7f2332000640 1 -- 172.21.15.102:0/1773752562 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c011aa0 con 0x5596a5fc8790 2024-04-25T01:59:21.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.157+0000 7f234b5bb740 1 -- 172.21.15.102:0/1773752562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 msgr2=0x5596a5fe0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:21.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.157+0000 7f234b5bb740 1 --2- 172.21.15.102:0/1773752562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 0x5596a5fe0dc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f232c015b30 tx=0x7f232c00b390 comp rx=0 tx=0).stop 2024-04-25T01:59:21.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.159+0000 7f234b5bb740 1 -- 172.21.15.102:0/1773752562 shutdown_connections 2024-04-25T01:59:21.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.159+0000 7f234b5bb740 1 --2- 172.21.15.102:0/1773752562 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fc8790 0x5596a5fe0dc0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.159+0000 7f234b5bb740 1 --2- 172.21.15.102:0/1773752562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fe0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.159+0000 7f234b5bb740 1 --2- 172.21.15.102:0/1773752562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fbd7a0 0x5596a5fd4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.159+0000 7f234b5bb740 1 -- 172.21.15.102:0/1773752562 >> 172.21.15.102:0/1773752562 conn(0x5596a5fb9ac0 msgr2=0x5596a5fb8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:21.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.160+0000 7f234b5bb740 1 -- 172.21.15.102:0/1773752562 shutdown_connections 2024-04-25T01:59:21.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.161+0000 7f234b5bb740 1 -- 172.21.15.102:0/1773752562 wait complete. 2024-04-25T01:59:21.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.163+0000 7f234b5bb740 1 -- start start 2024-04-25T01:59:21.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.163+0000 7f234b5bb740 1 Processor -- start 2024-04-25T01:59:21.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fcc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fcc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f234b5bb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fc8790 0x5596a5fccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f234b5bb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f23401a9660 con 0x5596a5fc8790 2024-04-25T01:59:21.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f234b5bb740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f23401aaad0 con 0x5596a5fbd7a0 2024-04-25T01:59:21.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f234b5bb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f23401a9d30 con 0x5596a5fbe0c0 2024-04-25T01:59:21.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f234b5bb740 10 client.0 _ll_register_callbacks cb 0x7f2338015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:21.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f2345c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fcc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f2345c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fcc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41576/0 (socket says 172.21.15.102:41576) 2024-04-25T01:59:21.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.164+0000 7f2345c00640 1 -- 172.21.15.102:0/3252562143 learned_addr learned my addr 172.21.15.102:0/3252562143 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:21.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.165+0000 7f2345200640 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fcc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.165+0000 7f2346600640 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fc8790 0x5596a5fccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.165+0000 7f2345200640 1 -- 172.21.15.102:0/3252562143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 msgr2=0x5596a5fcc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:21.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.165+0000 7f2345200640 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fcc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.165+0000 7f2345200640 1 -- 172.21.15.102:0/3252562143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fc8790 msgr2=0x5596a5fccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:21.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.165+0000 7f2345200640 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fc8790 0x5596a5fccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.165+0000 7f2345200640 1 -- 172.21.15.102:0/3252562143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596a603e2f0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.167+0000 7f2345200640 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fcc670 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2338000f80 tx=0x7f2338009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e82df806a35163f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:21.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.167+0000 7f2327400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:21.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.168+0000 7f2327400640 1 -- 172.21.15.102:0/3252562143 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338004ab0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.168+0000 7f2327400640 1 -- 172.21.15.102:0/3252562143 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f2338004c50 con 0x5596a5fbe0c0 2024-04-25T01:59:21.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.168+0000 7f234b5bb740 1 -- 172.21.15.102:0/3252562143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f232c00fdb0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f2327400640 1 -- 172.21.15.102:0/3252562143 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f233806b050 con 0x5596a5fbe0c0 2024-04-25T01:59:21.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f234b5bb740 10 client.26359 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:21.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f234b5bb740 1 -- 172.21.15.102:0/3252562143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5596a603f5b0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f2327400640 1 -- 172.21.15.102:0/3252562143 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f232c00fdb0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f2327400640 1 -- 172.21.15.102:0/3252562143 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f23380055d0 con 0x5596a5fbe0c0 2024-04-25T01:59:21.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f234b5bb740 10 client.26359 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:21.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f234b5bb740 10 client.26359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:21.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f234b5bb740 20 client.26359 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:21.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f234b5bb740 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5596a5fd8bf0 0x5596a6041d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:21.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f234b5bb740 4 client.26359 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:59:21.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.169+0000 7f234b5bb740 1 -- 172.21.15.102:0/3252562143 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596a5f29d70 con 0x5596a5fd8bf0 2024-04-25T01:59:21.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.170+0000 7f2345c00640 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5596a5fd8bf0 0x5596a6041d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:21.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.171+0000 7f2345c00640 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5596a5fd8bf0 0x5596a6041d60 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b903568230602c3 server_cookie=aa2fb7809cb48867 in_seq=0 out_seq=0 2024-04-25T01:59:21.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.171+0000 7f2327400640 10 client.26359 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:21.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.172+0000 7f2327400640 1 -- 172.21.15.102:0/3252562143 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596a5f29d70 con 0x5596a5fd8bf0 2024-04-25T01:59:21.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.172+0000 7f2327400640 10 client.26359 handle_command_reply: tid=0 2024-04-25T01:59:21.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.172+0000 7f234b5bb740 1 client.26359 shutdown 2024-04-25T01:59:21.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.176+0000 7f234b5bb740 1 -- 172.21.15.102:0/3252562143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 msgr2=0x5596a5fcc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:21.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.176+0000 7f234b5bb740 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fcc670 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2338000f80 tx=0x7f2338009ed0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.176+0000 7f234b5bb740 1 -- 172.21.15.102:0/3252562143 shutdown_connections 2024-04-25T01:59:21.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.177+0000 7f234b5bb740 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5596a5fd8bf0 0x5596a6041d60 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.177+0000 7f234b5bb740 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5596a5fc8790 0x5596a5fccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.177+0000 7f234b5bb740 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5596a5fbe0c0 0x5596a5fcc670 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.177+0000 7f234b5bb740 1 --2- 172.21.15.102:0/3252562143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5596a5fbd7a0 0x5596a5fcc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.177+0000 7f2327400640 0 client.26359 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:21.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.177+0000 7f234b5bb740 1 -- 172.21.15.102:0/3252562143 >> 172.21.15.102:0/3252562143 conn(0x5596a5fb9ac0 msgr2=0x5596a5fbefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:21.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.177+0000 7f234b5bb740 1 -- 172.21.15.102:0/3252562143 shutdown_connections 2024-04-25T01:59:21.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.178+0000 7f234b5bb740 1 -- 172.21.15.102:0/3252562143 wait complete. 2024-04-25T01:59:21.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.179+0000 7f234b5bb740 20 client.26359 trim_cache size 0 max 16384 2024-04-25T01:59:21.183 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:59:21.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 -- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23280a2ed0 msgr2=0x7f23280a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:21.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23280a2ed0 0x7f23280a5390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2338005c40 tx=0x7f2338005bd0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 -- 172.21.15.102:0/376076313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23401004f0 msgr2=0x7f2340197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:21.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23401004f0 0x7f2340197f20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2334009230 tx=0x7f2334005080 comp rx=0 tx=0).stop 2024-04-25T01:59:21.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 -- 172.21.15.102:0/376076313 shutdown_connections 2024-04-25T01:59:21.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23280a2ed0 0x7f23280a5390 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23401065b0 0x7f234019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2340100eb0 0x7f2340198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 --2- 172.21.15.102:0/376076313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23401004f0 0x7f2340197f20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:21.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.180+0000 7f2347000640 1 -- 172.21.15.102:0/376076313 >> 172.21.15.102:0/376076313 conn(0x7f23400fbfc0 msgr2=0x7f23400fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:21.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.181+0000 7f2347000640 1 -- 172.21.15.102:0/376076313 shutdown_connections 2024-04-25T01:59:21.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:21.181+0000 7f2347000640 1 -- 172.21.15.102:0/376076313 wait complete. 2024-04-25T01:59:21.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:21 smithi102 ceph-mon[27796]: pgmap v125: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 31 MiB/s wr, 688 op/s 2024-04-25T01:59:21.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:21 smithi110 ceph-mon[32848]: pgmap v125: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 31 MiB/s wr, 688 op/s 2024-04-25T01:59:21.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:21 smithi193 ceph-mon[32981]: pgmap v125: 97 pgs: 97 active+clean; 2.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 31 MiB/s wr, 688 op/s 2024-04-25T01:59:22.222 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:59:22.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:59:22.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:59:22.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.484+0000 7f17d3e00640 1 -- 172.21.15.102:0/1971189375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17d410bbc0 msgr2=0x7f17d410e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.484+0000 7f17d3e00640 1 --2- 172.21.15.102:0/1971189375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17d410bbc0 0x7f17d410e060 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009be0 tx=0x7f17cc02a750 comp rx=0 tx=0).stop 2024-04-25T01:59:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.484+0000 7f17d3e00640 1 -- 172.21.15.102:0/1971189375 shutdown_connections 2024-04-25T01:59:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.484+0000 7f17d3e00640 1 --2- 172.21.15.102:0/1971189375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17d410bbc0 0x7f17d410e060 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.484+0000 7f17d3e00640 1 --2- 172.21.15.102:0/1971189375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17d41030c0 0x7f17d410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.484+0000 7f17d3e00640 1 --2- 172.21.15.102:0/1971189375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17d41027a0 0x7f17d4102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.484+0000 7f17d3e00640 1 -- 172.21.15.102:0/1971189375 >> 172.21.15.102:0/1971189375 conn(0x7f17d40fbfc0 msgr2=0x7f17d40fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.484+0000 7f17d3e00640 1 -- 172.21.15.102:0/1971189375 shutdown_connections 2024-04-25T01:59:22.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.484+0000 7f17d3e00640 1 -- 172.21.15.102:0/1971189375 wait complete. 2024-04-25T01:59:22.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.486+0000 7f17d3e00640 1 Processor -- start 2024-04-25T01:59:22.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.486+0000 7f17d3e00640 1 -- start start 2024-04-25T01:59:22.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.487+0000 7f17d3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17d41027a0 0x7f17d4072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.487+0000 7f17d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17d41030c0 0x7f17d4073410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.487+0000 7f17d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17d410bbc0 0x7f17d406c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17d41030c0 0x7f17d4073410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f17d4117fc0 con 0x7f17d41027a0 2024-04-25T01:59:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d3e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17d4117e40 con 0x7f17d41030c0 2024-04-25T01:59:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d3e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17d4118140 con 0x7f17d410bbc0 2024-04-25T01:59:22.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17d41030c0 0x7f17d4073410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41584/0 (socket says 172.21.15.102:41584) 2024-04-25T01:59:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2000640 1 -- 172.21.15.102:0/484959079 learned_addr learned my addr 172.21.15.102:0/484959079 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d3400640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17d410bbc0 0x7f17d406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2a00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17d41027a0 0x7f17d4072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2000640 1 -- 172.21.15.102:0/484959079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17d410bbc0 msgr2=0x7f17d406c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2000640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17d410bbc0 0x7f17d406c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2000640 1 -- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17d41027a0 msgr2=0x7f17d4072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2000640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17d41027a0 0x7f17d4072ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2000640 1 -- 172.21.15.102:0/484959079 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d406c9d0 con 0x7f17d41030c0 2024-04-25T01:59:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.488+0000 7f17d2a00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17d41027a0 0x7f17d4072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.489+0000 7f17d2000640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17d41030c0 0x7f17d4073410 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f17c000e790 tx=0x7f17c000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.489+0000 7f17d8a00640 1 -- 172.21.15.102:0/484959079 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c000eea0 con 0x7f17d41030c0 2024-04-25T01:59:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.489+0000 7f17d3e00640 1 -- 172.21.15.102:0/484959079 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d406ccc0 con 0x7f17d41030c0 2024-04-25T01:59:22.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.489+0000 7f17d3e00640 1 -- 172.21.15.102:0/484959079 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d4070d10 con 0x7f17d41030c0 2024-04-25T01:59:22.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.490+0000 7f17d8a00640 1 -- 172.21.15.102:0/484959079 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17c0005500 con 0x7f17d41030c0 2024-04-25T01:59:22.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.490+0000 7f17d8a00640 1 -- 172.21.15.102:0/484959079 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c0010430 con 0x7f17d41030c0 2024-04-25T01:59:22.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.494+0000 7f17d8a00640 1 -- 172.21.15.102:0/484959079 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f17c00105f0 con 0x7f17d41030c0 2024-04-25T01:59:22.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.495+0000 7f17dcc0c740 1 Processor -- start 2024-04-25T01:59:22.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.495+0000 7f17dcc0c740 1 -- start start 2024-04-25T01:59:22.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.495+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139511300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.495+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139511840 0x56213951b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.495+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213951bf10 0x56213951e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.495+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56213946ff40 con 0x56213951bf10 2024-04-25T01:59:22.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.495+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562139475190 con 0x562139510f20 2024-04-25T01:59:22.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.495+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56213941da10 con 0x562139511840 2024-04-25T01:59:22.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.496+0000 7f17d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139511300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.496+0000 7f17d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139511300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41590/0 (socket says 172.21.15.102:41590) 2024-04-25T01:59:22.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.496+0000 7f17d3400640 1 -- 172.21.15.102:0/3830511979 learned_addr learned my addr 172.21.15.102:0/3830511979 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.496+0000 7f17d2a00640 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139511840 0x56213951b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.496+0000 7f17d3400640 1 -- 172.21.15.102:0/3830511979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139511840 msgr2=0x56213951b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.496+0000 7f17d3400640 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139511840 0x56213951b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.496+0000 7f17d3400640 1 -- 172.21.15.102:0/3830511979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213951bf10 msgr2=0x56213951e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.496+0000 7f17d2000640 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213951bf10 0x56213951e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.497+0000 7f17d3400640 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213951bf10 0x56213951e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.497+0000 7f17d3400640 1 -- 172.21.15.102:0/3830511979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621392517e0 con 0x562139510f20 2024-04-25T01:59:22.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.497+0000 7f17d2000640 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213951bf10 0x56213951e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:22.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.498+0000 7f17d3400640 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139511300 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009be0 tx=0x7f17cc030aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea0c1d265a6ee2d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.498+0000 7f17bfe00640 1 -- 172.21.15.102:0/3830511979 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc041030 con 0x562139510f20 2024-04-25T01:59:22.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.498+0000 7f17bfe00640 1 -- 172.21.15.102:0/3830511979 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17cc030c50 con 0x562139510f20 2024-04-25T01:59:22.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.498+0000 7f17bfe00640 1 -- 172.21.15.102:0/3830511979 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc036040 con 0x562139510f20 2024-04-25T01:59:22.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.499+0000 7f17d8a00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17b80a2de0 0x7f17b80a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.499+0000 7f17dcc0c740 1 -- 172.21.15.102:0/3830511979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 msgr2=0x562139511300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.499+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139511300 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009be0 tx=0x7f17cc030aa0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.499+0000 7f17d8a00640 1 -- 172.21.15.102:0/484959079 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f17c00d0630 con 0x7f17d41030c0 2024-04-25T01:59:22.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.500+0000 7f17dcc0c740 1 -- 172.21.15.102:0/3830511979 shutdown_connections 2024-04-25T01:59:22.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.500+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213951bf10 0x56213951e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.500+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139511840 0x56213951b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.500+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/3830511979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139511300 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.500+0000 7f17dcc0c740 1 -- 172.21.15.102:0/3830511979 >> 172.21.15.102:0/3830511979 conn(0x56213950d240 msgr2=0x56213950d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.500+0000 7f17dcc0c740 1 -- 172.21.15.102:0/3830511979 shutdown_connections 2024-04-25T01:59:22.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.500+0000 7f17dcc0c740 1 -- 172.21.15.102:0/3830511979 wait complete. 2024-04-25T01:59:22.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.501+0000 7f17d2a00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17b80a2de0 0x7f17b80a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.501+0000 7f17dcc0c740 1 -- start start 2024-04-25T01:59:22.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.501+0000 7f17dcc0c740 1 Processor -- start 2024-04-25T01:59:22.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.502+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139518e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.502+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562139511840 0x5621395193d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.502+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56213951bf10 0x562139519910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.502+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x562139430e70 con 0x562139511840 2024-04-25T01:59:22.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.502+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562139471150 con 0x562139510f20 2024-04-25T01:59:22.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.502+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56213951ed40 con 0x56213951bf10 2024-04-25T01:59:22.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.502+0000 7f17dcc0c740 10 client.0 _ll_register_callbacks cb 0x56213946c1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:22.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.502+0000 7f17d2a00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17b80a2de0 0x7f17b80a52a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5621395124e0 tx=0x7f17c4005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139518e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562139511840 0x5621395193d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139518e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41600/0 (socket says 172.21.15.102:41600) 2024-04-25T01:59:22.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d2a00640 1 -- 172.21.15.102:0/1712432217 learned_addr learned my addr 172.21.15.102:0/1712432217 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d3400640 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56213951bf10 0x562139519910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d2000640 1 -- 172.21.15.102:0/1712432217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56213951bf10 msgr2=0x562139519910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d2000640 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56213951bf10 0x562139519910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d2000640 1 -- 172.21.15.102:0/1712432217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 msgr2=0x562139518e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d2000640 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139518e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d2000640 1 -- 172.21.15.102:0/1712432217 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621392b1f70 con 0x562139511840 2024-04-25T01:59:22.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.503+0000 7f17d3400640 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56213951bf10 0x562139519910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:22.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.504+0000 7f17d2000640 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562139511840 0x5621395193d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f17c00cdd50 tx=0x7f17c009c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2f9faa9a79ba81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.504+0000 7f17bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:59:22.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.504+0000 7f17bcc00640 1 -- 172.21.15.102:0/1712432217 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c00a5790 con 0x562139511840 2024-04-25T01:59:22.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.504+0000 7f17bcc00640 1 -- 172.21.15.102:0/1712432217 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f17c00a5300 con 0x562139511840 2024-04-25T01:59:22.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.504+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1712432217 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17cc030c50 con 0x562139511840 2024-04-25T01:59:22.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17bcc00640 1 -- 172.21.15.102:0/1712432217 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f17c00bb050 con 0x562139511840 2024-04-25T01:59:22.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17dcc0c740 10 client.16017 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:22.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1712432217 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562139149170 con 0x562139511840 2024-04-25T01:59:22.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17bcc00640 1 -- 172.21.15.102:0/1712432217 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17cc030c50 con 0x562139511840 2024-04-25T01:59:22.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17bcc00640 1 -- 172.21.15.102:0/1712432217 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f17c00047e0 con 0x562139511840 2024-04-25T01:59:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17dcc0c740 10 client.16017 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17dcc0c740 10 client.16017 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17dcc0c740 20 client.16017 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56213952c090 0x56213952e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17dcc0c740 4 client.16017 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.505+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1712432217 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621394826d0 con 0x56213952c090 2024-04-25T01:59:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.506+0000 7f17d2a00640 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56213952c090 0x56213952e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.507+0000 7f17d2a00640 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56213952c090 0x56213952e4c0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecd06f5b1d3f44c server_cookie=720f14fab019fd26 in_seq=0 out_seq=0 2024-04-25T01:59:22.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.507+0000 7f17bcc00640 10 client.16017 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:22.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.508+0000 7f17bcc00640 1 -- 172.21.15.102:0/1712432217 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621394826d0 con 0x56213952c090 2024-04-25T01:59:22.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.508+0000 7f17bcc00640 10 client.16017 handle_command_reply: tid=0 2024-04-25T01:59:22.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.508+0000 7f17dcc0c740 1 client.16017 shutdown 2024-04-25T01:59:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.509+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1712432217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562139511840 msgr2=0x5621395193d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.509+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562139511840 0x5621395193d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f17c00cdd50 tx=0x7f17c009c790 comp rx=0 tx=0).stop 2024-04-25T01:59:22.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.509+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1712432217 shutdown_connections 2024-04-25T01:59:22.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.509+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56213952c090 0x56213952e4c0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.509+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56213951bf10 0x562139519910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.509+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562139511840 0x5621395193d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.509+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1712432217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562139510f20 0x562139518e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.509+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1712432217 >> 172.21.15.102:0/1712432217 conn(0x56213950d240 msgr2=0x56213950c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.510+0000 7f17bcc00640 0 client.16017 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:22.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.510+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1712432217 shutdown_connections 2024-04-25T01:59:22.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.510+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1712432217 wait complete. 2024-04-25T01:59:22.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.511+0000 7f17dcc0c740 20 client.16017 trim_cache size 0 max 16384 2024-04-25T01:59:22.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.541+0000 7f17dcc0c740 1 Processor -- start 2024-04-25T01:59:22.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.541+0000 7f17dcc0c740 1 -- start start 2024-04-25T01:59:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.541+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 0x562139527ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.541+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 0x562139533d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.541+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x5621395342d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.541+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f17b4010f20 con 0x56213950d240 2024-04-25T01:59:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.541+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17b400e7c0 con 0x56213951bf10 2024-04-25T01:59:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.541+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17b400c610 con 0x562139510f20 2024-04-25T01:59:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x5621395342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x5621395342d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41610/0 (socket says 172.21.15.102:41610) 2024-04-25T01:59:22.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17d3400640 1 -- 172.21.15.102:0/102961021 learned_addr learned my addr 172.21.15.102:0/102961021 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17d3400640 1 -- 172.21.15.102:0/102961021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 msgr2=0x562139533d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17d3400640 1 --2- 172.21.15.102:0/102961021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 0x562139533d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17d3400640 1 -- 172.21.15.102:0/102961021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 msgr2=0x562139527ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17d3400640 1 --2- 172.21.15.102:0/102961021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 0x562139527ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17d3400640 1 -- 172.21.15.102:0/102961021 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56213950cda0 con 0x56213951bf10 2024-04-25T01:59:22.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17d3400640 1 --2- 172.21.15.102:0/102961021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x5621395342d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009be0 tx=0x7f17cc003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9247838460fb3364 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.542+0000 7f17bf400640 1 -- 172.21.15.102:0/102961021 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc04e070 con 0x56213951bf10 2024-04-25T01:59:22.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.543+0000 7f17bf400640 1 -- 172.21.15.102:0/102961021 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17cc030e50 con 0x56213951bf10 2024-04-25T01:59:22.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.543+0000 7f17bf400640 1 -- 172.21.15.102:0/102961021 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc041040 con 0x56213951bf10 2024-04-25T01:59:22.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.544+0000 7f17dcc0c740 1 -- 172.21.15.102:0/102961021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 msgr2=0x5621395342d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.544+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/102961021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x5621395342d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009be0 tx=0x7f17cc003ce0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.544+0000 7f17dcc0c740 1 -- 172.21.15.102:0/102961021 shutdown_connections 2024-04-25T01:59:22.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.544+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/102961021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x5621395342d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.544+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/102961021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 0x562139533d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.544+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/102961021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 0x562139527ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.544+0000 7f17dcc0c740 1 -- 172.21.15.102:0/102961021 >> 172.21.15.102:0/102961021 conn(0x562139517b10 msgr2=0x562139512f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.545+0000 7f17dcc0c740 1 -- 172.21.15.102:0/102961021 shutdown_connections 2024-04-25T01:59:22.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.545+0000 7f17dcc0c740 1 -- 172.21.15.102:0/102961021 wait complete. 2024-04-25T01:59:22.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.546+0000 7f17dcc0c740 1 -- start start 2024-04-25T01:59:22.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.546+0000 7f17dcc0c740 1 Processor -- start 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.547+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 0x562139524280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.547+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 0x562139520600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.547+0000 7f17dcc0c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x562139520b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.547+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f17b400cc80 con 0x56213950d240 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.547+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17b400e0f0 con 0x56213951bf10 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.547+0000 7f17dcc0c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17b400d350 con 0x562139510f20 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.547+0000 7f17dcc0c740 10 client.0 _ll_register_callbacks cb 0x7f17c009c8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.548+0000 7f17d2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 0x562139520600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.548+0000 7f17d2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 0x562139520600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38488/0 (socket says 172.21.15.102:38488) 2024-04-25T01:59:22.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.548+0000 7f17d2000640 1 -- 172.21.15.102:0/1985136472 learned_addr learned my addr 172.21.15.102:0/1985136472 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.548+0000 7f17d3400640 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x562139520b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.548+0000 7f17d2a00640 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 0x562139524280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.548+0000 7f17d3400640 1 -- 172.21.15.102:0/1985136472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 msgr2=0x562139520600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.549+0000 7f17d3400640 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 0x562139520600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.549+0000 7f17d3400640 1 -- 172.21.15.102:0/1985136472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 msgr2=0x562139524280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.549+0000 7f17d3400640 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 0x562139524280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.549+0000 7f17d3400640 1 -- 172.21.15.102:0/1985136472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562139521230 con 0x56213951bf10 2024-04-25T01:59:22.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.549+0000 7f17d2a00640 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 0x562139524280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:22.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.550+0000 7f17d3400640 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x562139520b40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009b30 tx=0x7f17cc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3180c66334c0c4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.550+0000 7f17bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:22.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.551+0000 7f17bcc00640 1 -- 172.21.15.102:0/1985136472 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc041070 con 0x56213951bf10 2024-04-25T01:59:22.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.551+0000 7f17bcc00640 1 -- 172.21.15.102:0/1985136472 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f17cc03d460 con 0x56213951bf10 2024-04-25T01:59:22.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.551+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1985136472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17cc030e50 con 0x56213951bf10 2024-04-25T01:59:22.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.551+0000 7f17bcc00640 1 -- 172.21.15.102:0/1985136472 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f17cc04e070 con 0x56213951bf10 2024-04-25T01:59:22.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.551+0000 7f17dcc0c740 10 client.26408 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:22.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.551+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1985136472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562139593510 con 0x56213951bf10 2024-04-25T01:59:22.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.552+0000 7f17bcc00640 1 -- 172.21.15.102:0/1985136472 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17cc030e50 con 0x56213951bf10 2024-04-25T01:59:22.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.552+0000 7f17bcc00640 1 -- 172.21.15.102:0/1985136472 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f17cc0316b0 con 0x56213951bf10 2024-04-25T01:59:22.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.552+0000 7f17dcc0c740 10 client.26408 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:22.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.552+0000 7f17dcc0c740 10 client.26408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:22.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.552+0000 7f17dcc0c740 20 client.26408 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:22.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.552+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562139593740 0x562139595b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.552+0000 7f17dcc0c740 4 client.26408 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:59:22.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.552+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1985136472 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56213947ca20 con 0x562139593740 2024-04-25T01:59:22.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.552+0000 7f17d2a00640 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562139593740 0x562139595b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.555+0000 7f17d2a00640 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562139593740 0x562139595b30 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=518130d8ab584248 server_cookie=28dad3eb49b33022 in_seq=0 out_seq=0 2024-04-25T01:59:22.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.555+0000 7f17bcc00640 10 client.26408 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:22.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.556+0000 7f17bcc00640 1 -- 172.21.15.102:0/1985136472 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56213947ca20 con 0x562139593740 2024-04-25T01:59:22.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.556+0000 7f17bcc00640 10 client.26408 handle_command_reply: tid=0 2024-04-25T01:59:22.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.556+0000 7f17dcc0c740 1 client.26408 shutdown 2024-04-25T01:59:22.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.557+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1985136472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 msgr2=0x562139520b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.557+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x562139520b40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009b30 tx=0x7f17cc004a40 comp rx=0 tx=0).stop 2024-04-25T01:59:22.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.557+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1985136472 shutdown_connections 2024-04-25T01:59:22.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.557+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562139593740 0x562139595b30 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.557+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56213951bf10 0x562139520b40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.557+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562139510f20 0x562139520600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.557+0000 7f17dcc0c740 1 --2- 172.21.15.102:0/1985136472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56213950d240 0x562139524280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.557+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1985136472 >> 172.21.15.102:0/1985136472 conn(0x562139517b10 msgr2=0x56213950bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.557+0000 7f17bcc00640 0 client.26408 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:22.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.558+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1985136472 shutdown_connections 2024-04-25T01:59:22.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.558+0000 7f17dcc0c740 1 -- 172.21.15.102:0/1985136472 wait complete. 2024-04-25T01:59:22.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.558+0000 7f17dcc0c740 20 client.26408 trim_cache size 0 max 16384 2024-04-25T01:59:22.563 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:59:22.563 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T01:59:22.563 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "59ba533c-4f19-4bf0-ac89-b67f4b213135", 2024-04-25T01:59:22.564 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T01:59:22.564 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:59:22.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 -- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17b80a2de0 msgr2=0x7f17b80a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17b80a2de0 0x7f17b80a52a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5621395124e0 tx=0x7f17c4005f90 comp rx=0 tx=0).stop 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 -- 172.21.15.102:0/484959079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17d41030c0 msgr2=0x7f17d4073410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17d41030c0 0x7f17d4073410 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f17c000e790 tx=0x7f17c000ec50 comp rx=0 tx=0).stop 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 -- 172.21.15.102:0/484959079 shutdown_connections 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17b80a2de0 0x7f17b80a52a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17d410bbc0 0x7f17d406c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17d41030c0 0x7f17d4073410 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 --2- 172.21.15.102:0/484959079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17d41027a0 0x7f17d4072ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 -- 172.21.15.102:0/484959079 >> 172.21.15.102:0/484959079 conn(0x7f17d40fbfc0 msgr2=0x7f17d40fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.560+0000 7f17d3e00640 1 -- 172.21.15.102:0/484959079 shutdown_connections 2024-04-25T01:59:22.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.561+0000 7f17d3e00640 1 -- 172.21.15.102:0/484959079 wait complete. 2024-04-25T01:59:22.600 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:59:22.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:59:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.867+0000 7f9a3e000640 1 -- 172.21.15.102:0/1225194297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a381004a0 msgr2=0x7f9a38106480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.867+0000 7f9a3e000640 1 --2- 172.21.15.102:0/1225194297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a381004a0 0x7f9a38106480 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009890 tx=0x7f9a2802a360 comp rx=0 tx=0).stop 2024-04-25T01:59:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.867+0000 7f9a3e000640 1 -- 172.21.15.102:0/1225194297 shutdown_connections 2024-04-25T01:59:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.867+0000 7f9a3e000640 1 --2- 172.21.15.102:0/1225194297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a38106bb0 0x7f9a3810e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.867+0000 7f9a3e000640 1 --2- 172.21.15.102:0/1225194297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a381004a0 0x7f9a38106480 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.867+0000 7f9a3e000640 1 --2- 172.21.15.102:0/1225194297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a380ffaf0 0x7f9a380ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.867+0000 7f9a3e000640 1 -- 172.21.15.102:0/1225194297 >> 172.21.15.102:0/1225194297 conn(0x7f9a380767a0 msgr2=0x7f9a380fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.868+0000 7f9a3e000640 1 -- 172.21.15.102:0/1225194297 shutdown_connections 2024-04-25T01:59:22.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.868+0000 7f9a3e000640 1 -- 172.21.15.102:0/1225194297 wait complete. 2024-04-25T01:59:22.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.869+0000 7f9a3e000640 1 Processor -- start 2024-04-25T01:59:22.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.870+0000 7f9a3e000640 1 -- start start 2024-04-25T01:59:22.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.870+0000 7f9a3e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a380ffaf0 0x7f9a381a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.870+0000 7f9a3e000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a38106bb0 0x7f9a381a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.870+0000 7f9a3e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a3810d910 0x7f9a3810ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.871+0000 7f9a37400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a380ffaf0 0x7f9a381a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.871+0000 7f9a37400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a380ffaf0 0x7f9a381a4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46538/0 (socket says 172.21.15.102:46538) 2024-04-25T01:59:22.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.871+0000 7f9a2ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a38106bb0 0x7f9a381a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.871+0000 7f9a2ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a38106bb0 0x7f9a381a4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59816/0 (socket says 172.21.15.102:59816) 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a3e000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9a38111c00 con 0x7f9a3810d910 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a3e000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a38111a80 con 0x7f9a38106bb0 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a3e000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9a38111d80 con 0x7f9a380ffaf0 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a37400640 1 -- 172.21.15.102:0/4204188273 learned_addr learned my addr 172.21.15.102:0/4204188273 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a37e00640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a3810d910 0x7f9a3810ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a2ea00640 1 -- 172.21.15.102:0/4204188273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a380ffaf0 msgr2=0x7f9a381a4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a380ffaf0 0x7f9a381a4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a2ea00640 1 -- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a3810d910 msgr2=0x7f9a3810ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a3810d910 0x7f9a3810ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.872+0000 7f9a2ea00640 1 -- 172.21.15.102:0/4204188273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a3810e430 con 0x7f9a38106bb0 2024-04-25T01:59:22.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.873+0000 7f9a37400640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a380ffaf0 0x7f9a381a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:22.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.873+0000 7f9a37e00640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a3810d910 0x7f9a3810ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:22.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.873+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a38106bb0 0x7f9a381a4970 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009780 tx=0x7f9a28030480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.874+0000 7f9a34c00640 1 -- 172.21.15.102:0/4204188273 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a28037040 con 0x7f9a38106bb0 2024-04-25T01:59:22.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.874+0000 7f9a34c00640 1 -- 172.21.15.102:0/4204188273 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a28004260 con 0x7f9a38106bb0 2024-04-25T01:59:22.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.874+0000 7f9a34c00640 1 -- 172.21.15.102:0/4204188273 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a28003b90 con 0x7f9a38106bb0 2024-04-25T01:59:22.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.874+0000 7f9a3e000640 1 -- 172.21.15.102:0/4204188273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a3810e6c0 con 0x7f9a38106bb0 2024-04-25T01:59:22.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.874+0000 7f9a3e000640 1 -- 172.21.15.102:0/4204188273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a3806b3a0 con 0x7f9a38106bb0 2024-04-25T01:59:22.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.879+0000 7f9a40d8b740 1 Processor -- start 2024-04-25T01:59:22.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.880+0000 7f9a34c00640 1 -- 172.21.15.102:0/4204188273 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9a28053020 con 0x7f9a38106bb0 2024-04-25T01:59:22.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.880+0000 7f9a40d8b740 1 -- start start 2024-04-25T01:59:22.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 0x556be65d13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65dbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65dbff0 0x556be65de3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x556be6535560 con 0x556be65d1000 2024-04-25T01:59:22.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556be64dd930 con 0x556be65d1920 2024-04-25T01:59:22.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556be6531520 con 0x556be65dbff0 2024-04-25T01:59:22.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a37400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65dbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a37400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65dbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59832/0 (socket says 172.21.15.102:59832) 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a37400640 1 -- 172.21.15.102:0/1253074564 learned_addr learned my addr 172.21.15.102:0/1253074564 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.882+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65dbff0 0x556be65de3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a37e00640 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 0x556be65d13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a37400640 1 -- 172.21.15.102:0/1253074564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65dbff0 msgr2=0x556be65de3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a37400640 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65dbff0 0x556be65de3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a37400640 1 -- 172.21.15.102:0/1253074564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 msgr2=0x556be65d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a37400640 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 0x556be65d13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a37400640 1 -- 172.21.15.102:0/1253074564 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556be63137d0 con 0x556be65d1920 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a37e00640 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 0x556be65d13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a37400640 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65dbab0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9a200115c0 tx=0x7f9a20011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56a0431e8cad61c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a34c00640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a1c0a2e50 0x7f9a1c0a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a27e00640 1 -- 172.21.15.102:0/1253074564 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2000f6c0 con 0x556be65d1920 2024-04-25T01:59:22.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a27e00640 1 -- 172.21.15.102:0/1253074564 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a2000fd40 con 0x556be65d1920 2024-04-25T01:59:22.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.883+0000 7f9a27e00640 1 -- 172.21.15.102:0/1253074564 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a200183b0 con 0x556be65d1920 2024-04-25T01:59:22.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.884+0000 7f9a34c00640 1 -- 172.21.15.102:0/4204188273 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f9a280ff970 con 0x7f9a38106bb0 2024-04-25T01:59:22.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.884+0000 7f9a40d8b740 1 -- 172.21.15.102:0/1253074564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 msgr2=0x556be65dbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.884+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65dbab0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f9a200115c0 tx=0x7f9a20011a80 comp rx=0 tx=0).stop 2024-04-25T01:59:22.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.885+0000 7f9a40d8b740 1 -- 172.21.15.102:0/1253074564 shutdown_connections 2024-04-25T01:59:22.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.885+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65dbff0 0x556be65de3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.885+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65dbab0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.885+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/1253074564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 0x556be65d13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.885+0000 7f9a40d8b740 1 -- 172.21.15.102:0/1253074564 >> 172.21.15.102:0/1253074564 conn(0x556be65cd320 msgr2=0x556be65cd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.885+0000 7f9a40d8b740 1 -- 172.21.15.102:0/1253074564 shutdown_connections 2024-04-25T01:59:22.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.885+0000 7f9a40d8b740 1 -- 172.21.15.102:0/1253074564 wait complete. 2024-04-25T01:59:22.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.885+0000 7f9a40d8b740 1 -- start start 2024-04-25T01:59:22.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.886+0000 7f9a40d8b740 1 Processor -- start 2024-04-25T01:59:22.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.886+0000 7f9a37e00640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a1c0a2e50 0x7f9a1c0a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.887+0000 7f9a37e00640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a1c0a2e50 0x7f9a1c0a5310 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a30005fd0 tx=0x7f9a30009520 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65d8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65d9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65dbff0 0x556be65d9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x556be65dede0 con 0x556be65dbff0 2024-04-25T01:59:22.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x556be64f0d90 con 0x556be65d1920 2024-04-25T01:59:22.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556be65def60 con 0x556be65d1000 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a37400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65d8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a2ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65d9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a37400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65d8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46554/0 (socket says 172.21.15.102:46554) 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a37400640 1 -- 172.21.15.102:0/2016129156 learned_addr learned my addr 172.21.15.102:0/2016129156 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a40d8b740 10 client.0 _ll_register_callbacks cb 0x556be64f0780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.888+0000 7f9a37e00640 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65dbff0 0x556be65d9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a37400640 1 -- 172.21.15.102:0/2016129156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 msgr2=0x556be65d9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a37400640 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65d9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a37400640 1 -- 172.21.15.102:0/2016129156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65dbff0 msgr2=0x556be65d9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a37400640 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65dbff0 0x556be65d9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a37400640 1 -- 172.21.15.102:0/2016129156 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556be62493f0 con 0x556be65d1000 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a37e00640 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65dbff0 0x556be65d9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:22.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a37400640 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65d8f00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9a200058c0 tx=0x7f9a20019640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=473fed524eecdf62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a24c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a24c00640 1 -- 172.21.15.102:0/2016129156 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20028020 con 0x556be65d1000 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.889+0000 7f9a24c00640 1 -- 172.21.15.102:0/2016129156 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f9a20016040 con 0x556be65d1000 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.890+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2016129156 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a2000fd40 con 0x556be65d1000 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.890+0000 7f9a24c00640 1 -- 172.21.15.102:0/2016129156 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9a20004250 con 0x556be65d1000 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.890+0000 7f9a40d8b740 10 client.26386 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.890+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2016129156 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556be62090f0 con 0x556be65d1000 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.890+0000 7f9a24c00640 1 -- 172.21.15.102:0/2016129156 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a2000fd40 con 0x556be65d1000 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.891+0000 7f9a24c00640 1 -- 172.21.15.102:0/2016129156 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f9a200038d0 con 0x556be65d1000 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.891+0000 7f9a40d8b740 10 client.26386 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.891+0000 7f9a40d8b740 10 client.26386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:22.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.891+0000 7f9a40d8b740 20 client.26386 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:22.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.891+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556be65ec170 0x556be65ee560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.891+0000 7f9a40d8b740 4 client.26386 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:22.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.891+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2016129156 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556be6548d90 con 0x556be65ec170 2024-04-25T01:59:22.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.892+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556be65ec170 0x556be65ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.893+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556be65ec170 0x556be65ee560 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=225b9a1c4e5786a2 server_cookie=428d70c29d976a96 in_seq=0 out_seq=0 2024-04-25T01:59:22.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.893+0000 7f9a24c00640 10 client.26386 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:22.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.893+0000 7f9a24c00640 1 -- 172.21.15.102:0/2016129156 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556be6548d90 con 0x556be65ec170 2024-04-25T01:59:22.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.893+0000 7f9a24c00640 10 client.26386 handle_command_reply: tid=0 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.893+0000 7f9a40d8b740 1 client.26386 shutdown 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2016129156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 msgr2=0x556be65d8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65d8f00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9a200058c0 tx=0x7f9a20019640 comp rx=0 tx=0).stop 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2016129156 shutdown_connections 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556be65ec170 0x556be65ee560 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65dbff0 0x556be65d9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a24c00640 0 client.26386 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65d1920 0x556be65d9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2016129156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65d8f00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2016129156 >> 172.21.15.102:0/2016129156 conn(0x556be65cd320 msgr2=0x556be65d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2016129156 shutdown_connections 2024-04-25T01:59:22.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.894+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2016129156 wait complete. 2024-04-25T01:59:22.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.895+0000 7f9a40d8b740 20 client.26386 trim_cache size 0 max 16384 2024-04-25T01:59:22.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.924+0000 7f9a40d8b740 1 Processor -- start 2024-04-25T01:59:22.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.924+0000 7f9a40d8b740 1 -- start start 2024-04-25T01:59:22.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.924+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65e7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.924+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1920 0x556be65d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.924+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65dbff0 0x556be65f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.924+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381adb00 con 0x556be65d1920 2024-04-25T01:59:22.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.924+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a381ab3c0 con 0x556be65dbff0 2024-04-25T01:59:22.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.924+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9a380689b0 con 0x556be65d1000 2024-04-25T01:59:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a2ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1920 0x556be65d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65e7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65e7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46558/0 (socket says 172.21.15.102:46558) 2024-04-25T01:59:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37400640 1 -- 172.21.15.102:0/2136681303 learned_addr learned my addr 172.21.15.102:0/2136681303 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37e00640 1 --2- 172.21.15.102:0/2136681303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65dbff0 0x556be65f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37400640 1 -- 172.21.15.102:0/2136681303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65dbff0 msgr2=0x556be65f3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37400640 1 --2- 172.21.15.102:0/2136681303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65dbff0 0x556be65f3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37400640 1 -- 172.21.15.102:0/2136681303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1920 msgr2=0x556be65d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37400640 1 --2- 172.21.15.102:0/2136681303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1920 0x556be65d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37400640 1 -- 172.21.15.102:0/2136681303 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556be65decc0 con 0x556be65d1000 2024-04-25T01:59:22.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/2136681303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1920 0x556be65d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:22.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.925+0000 7f9a37400640 1 --2- 172.21.15.102:0/2136681303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65e7ca0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9a20005ec0 tx=0x7f9a20012f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e547deb40dc0395a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.926+0000 7f9a27400640 1 -- 172.21.15.102:0/2136681303 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20028040 con 0x556be65d1000 2024-04-25T01:59:22.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.926+0000 7f9a27400640 1 -- 172.21.15.102:0/2136681303 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a200197c0 con 0x556be65d1000 2024-04-25T01:59:22.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.926+0000 7f9a27400640 1 -- 172.21.15.102:0/2136681303 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a2000f9b0 con 0x556be65d1000 2024-04-25T01:59:22.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.927+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2136681303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 msgr2=0x556be65e7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.927+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2136681303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65e7ca0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9a20005ec0 tx=0x7f9a20012f90 comp rx=0 tx=0).stop 2024-04-25T01:59:22.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.927+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2136681303 shutdown_connections 2024-04-25T01:59:22.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.927+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2136681303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65dbff0 0x556be65f3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.927+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2136681303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1920 0x556be65d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.927+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2136681303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1000 0x556be65e7ca0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.927+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2136681303 >> 172.21.15.102:0/2136681303 conn(0x556be65cd320 msgr2=0x556be65d7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.927+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2136681303 shutdown_connections 2024-04-25T01:59:22.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.928+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2136681303 wait complete. 2024-04-25T01:59:22.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.928+0000 7f9a40d8b740 1 -- start start 2024-04-25T01:59:22.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a40d8b740 1 Processor -- start 2024-04-25T01:59:22.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 0x556be65e41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1920 0x556be65e46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a40d8b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65dbff0 0x556be664d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9a38071f60 con 0x556be65d1000 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a38073390 con 0x556be65dbff0 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a40d8b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9a380725f0 con 0x556be65d1920 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a40d8b740 10 client.0 _ll_register_callbacks cb 0x7f9a20005df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a2ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1920 0x556be65e46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a2ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1920 0x556be65e46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46570/0 (socket says 172.21.15.102:46570) 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.929+0000 7f9a2ea00640 1 -- 172.21.15.102:0/2345441188 learned_addr learned my addr 172.21.15.102:0/2345441188 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:22.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.930+0000 7f9a37400640 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 0x556be65e41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.930+0000 7f9a2ea00640 1 -- 172.21.15.102:0/2345441188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65dbff0 msgr2=0x556be664d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.930+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65dbff0 0x556be664d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.930+0000 7f9a2ea00640 1 -- 172.21.15.102:0/2345441188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 msgr2=0x556be65e41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.930+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 0x556be65e41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.930+0000 7f9a2ea00640 1 -- 172.21.15.102:0/2345441188 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556be664dd40 con 0x556be65d1920 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.930+0000 7f9a2ea00640 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1920 0x556be65e46f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9a280cbcd0 tx=0x7f9a28005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9a6fe7e5e83bbd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.930+0000 7f9a24c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.931+0000 7f9a24c00640 1 -- 172.21.15.102:0/2345441188 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a280cbfd0 con 0x556be65d1920 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.931+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2345441188 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a200197c0 con 0x556be65d1920 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.931+0000 7f9a24c00640 1 -- 172.21.15.102:0/2345441188 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f9a280ca2d0 con 0x556be65d1920 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.931+0000 7f9a24c00640 1 -- 172.21.15.102:0/2345441188 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9a280d0380 con 0x556be65d1920 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.931+0000 7f9a40d8b740 10 client.26392 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:22.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.931+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2345441188 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556be65d5de0 con 0x556be65d1920 2024-04-25T01:59:22.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.932+0000 7f9a24c00640 1 -- 172.21.15.102:0/2345441188 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a200197c0 con 0x556be65d1920 2024-04-25T01:59:22.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.932+0000 7f9a24c00640 1 -- 172.21.15.102:0/2345441188 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f9a280cc230 con 0x556be65d1920 2024-04-25T01:59:22.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.933+0000 7f9a40d8b740 10 client.26392 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:22.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.933+0000 7f9a40d8b740 10 client.26392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:22.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.933+0000 7f9a40d8b740 20 client.26392 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:22.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.933+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556be6663780 0x556be6665ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:22.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.933+0000 7f9a40d8b740 4 client.26392 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:59:22.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.933+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2345441188 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556be6542aa0 con 0x556be6663780 2024-04-25T01:59:22.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.933+0000 7f9a37400640 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556be6663780 0x556be6665ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:22.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.934+0000 7f9a37400640 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556be6663780 0x556be6665ba0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1769d6a6ed6fdaf1 server_cookie=3069b139143e1a6f in_seq=0 out_seq=0 2024-04-25T01:59:22.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a24c00640 10 client.26392 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:22.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a24c00640 1 -- 172.21.15.102:0/2345441188 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556be6542aa0 con 0x556be6663780 2024-04-25T01:59:22.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a24c00640 10 client.26392 handle_command_reply: tid=0 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 client.26392 shutdown 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2345441188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1920 msgr2=0x556be65e46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1920 0x556be65e46f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9a280cbcd0 tx=0x7f9a28005f90 comp rx=0 tx=0).stop 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2345441188 shutdown_connections 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x556be6663780 0x556be6665ba0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556be65dbff0 0x556be664d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556be65d1920 0x556be65e46f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 --2- 172.21.15.102:0/2345441188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556be65d1000 0x556be65e41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a24c00640 0 client.26392 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2345441188 >> 172.21.15.102:0/2345441188 conn(0x556be65cd320 msgr2=0x556be65da730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2345441188 shutdown_connections 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 1 -- 172.21.15.102:0/2345441188 wait complete. 2024-04-25T01:59:22.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.935+0000 7f9a40d8b740 20 client.26392 trim_cache size 0 max 16384 2024-04-25T01:59:22.941 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:59:22.941 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:59:22.941 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T01:59:22.941 INFO:teuthology.orchestra.run.smithi102.stdout: "59ba533c-4f19-4bf0-ac89-b67f4b213135": { 2024-04-25T01:59:22.941 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T01:59:22.941 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "59ba533c-4f19-4bf0-ac89-b67f4b213135", 2024-04-25T01:59:22.941 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T01:59:22.941 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:59:22.941 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:59:22.942 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:59:22.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 -- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a1c0a2e50 msgr2=0x7f9a1c0a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a1c0a2e50 0x7f9a1c0a5310 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a30005fd0 tx=0x7f9a30009520 comp rx=0 tx=0).stop 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 -- 172.21.15.102:0/4204188273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a38106bb0 msgr2=0x7f9a381a4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a38106bb0 0x7f9a381a4970 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9a28009780 tx=0x7f9a28030480 comp rx=0 tx=0).stop 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 -- 172.21.15.102:0/4204188273 shutdown_connections 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a1c0a2e50 0x7f9a1c0a5310 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a3810d910 0x7f9a3810ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a38106bb0 0x7f9a381a4970 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 --2- 172.21.15.102:0/4204188273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a380ffaf0 0x7f9a381a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.936+0000 7f9a3e000640 1 -- 172.21.15.102:0/4204188273 >> 172.21.15.102:0/4204188273 conn(0x7f9a380767a0 msgr2=0x7f9a38104ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.937+0000 7f9a3e000640 1 -- 172.21.15.102:0/4204188273 shutdown_connections 2024-04-25T01:59:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:22.937+0000 7f9a3e000640 1 -- 172.21.15.102:0/4204188273 wait complete. 2024-04-25T01:59:22.966 INFO:tasks.cephfs.filesystem:scrub status for tag:59ba533c-4f19-4bf0-ac89-b67f4b213135 - {'path': '/', 'tag': '59ba533c-4f19-4bf0-ac89-b67f4b213135', 'options': 'recursive,force'} 2024-04-25T01:59:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 210.0 s, 2392.8 tps, lat 41.541 ms stddev 51.229 2024-04-25T01:59:23.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: pgmap v126: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 41 MiB/s wr, 957 op/s 2024-04-25T01:59:23.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T01:59:23.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:23.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:23.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:23 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: pgmap v126: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 41 MiB/s wr, 957 op/s 2024-04-25T01:59:23.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T01:59:23.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:23 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: pgmap v126: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 41 MiB/s wr, 957 op/s 2024-04-25T01:59:23.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T01:59:23.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:23.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:23.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:23 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.038+0000 7f5c61800640 1 -- 172.21.15.102:0/2103379775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c5c1001d0 msgr2=0x7f5c5c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.038+0000 7f5c61800640 1 --2- 172.21.15.102:0/2103379775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c5c1001d0 0x7f5c5c10e610 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5c54009ca0 tx=0x7f5c5402a950 comp rx=0 tx=0).stop 2024-04-25T01:59:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.039+0000 7f5c61800640 1 -- 172.21.15.102:0/2103379775 shutdown_connections 2024-04-25T01:59:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.039+0000 7f5c61800640 1 --2- 172.21.15.102:0/2103379775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c5c1001d0 0x7f5c5c10e610 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.039+0000 7f5c61800640 1 --2- 172.21.15.102:0/2103379775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5c0ff830 0x7f5c5c0ffc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.039+0000 7f5c61800640 1 --2- 172.21.15.102:0/2103379775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c5c1067c0 0x7f5c5c106ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.039+0000 7f5c61800640 1 -- 172.21.15.102:0/2103379775 >> 172.21.15.102:0/2103379775 conn(0x7f5c5c076740 msgr2=0x7f5c5c0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.039+0000 7f5c61800640 1 -- 172.21.15.102:0/2103379775 shutdown_connections 2024-04-25T01:59:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.039+0000 7f5c61800640 1 -- 172.21.15.102:0/2103379775 wait complete. 2024-04-25T01:59:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.040+0000 7f5c61800640 1 Processor -- start 2024-04-25T01:59:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.040+0000 7f5c61800640 1 -- start start 2024-04-25T01:59:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.041+0000 7f5c61800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c5c0ff830 0x7f5c5c10d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.041+0000 7f5c61800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c5c1001d0 0x7f5c5c10dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.041+0000 7f5c61800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5c1067c0 0x7f5c5c1a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.041+0000 7f5c61800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c111b30 con 0x7f5c5c1067c0 2024-04-25T01:59:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.041+0000 7f5c61800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c1119b0 con 0x7f5c5c1001d0 2024-04-25T01:59:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.041+0000 7f5c61800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c111cb0 con 0x7f5c5c0ff830 2024-04-25T01:59:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.041+0000 7f5c5aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c5c0ff830 0x7f5c5c10d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.042+0000 7f5c5aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c5c0ff830 0x7f5c5c10d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46588/0 (socket says 172.21.15.102:46588) 2024-04-25T01:59:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.042+0000 7f5c5aa00640 1 -- 172.21.15.102:0/163597145 learned_addr learned my addr 172.21.15.102:0/163597145 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.042+0000 7f5c5b400640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5c1067c0 0x7f5c5c1a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.041+0000 7f5c5a000640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c5c1001d0 0x7f5c5c10dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.042+0000 7f5c5b400640 1 -- 172.21.15.102:0/163597145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c5c0ff830 msgr2=0x7f5c5c10d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.042+0000 7f5c5b400640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c5c0ff830 0x7f5c5c10d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.042+0000 7f5c5b400640 1 -- 172.21.15.102:0/163597145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c5c1001d0 msgr2=0x7f5c5c10dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.042+0000 7f5c5b400640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c5c1001d0 0x7f5c5c10dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.042+0000 7f5c5b400640 1 -- 172.21.15.102:0/163597145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5c1a3450 con 0x7f5c5c1067c0 2024-04-25T01:59:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.043+0000 7f5c5b400640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5c1067c0 0x7f5c5c1a2d20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5c54005e30 tx=0x7f5c54003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.043+0000 7f5c53400640 1 -- 172.21.15.102:0/163597145 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54034040 con 0x7f5c5c1067c0 2024-04-25T01:59:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.043+0000 7f5c53400640 1 -- 172.21.15.102:0/163597145 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c5402fe20 con 0x7f5c5c1067c0 2024-04-25T01:59:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.043+0000 7f5c53400640 1 -- 172.21.15.102:0/163597145 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54004680 con 0x7f5c5c1067c0 2024-04-25T01:59:25.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.044+0000 7f5c61800640 1 -- 172.21.15.102:0/163597145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c5c1a3740 con 0x7f5c5c1067c0 2024-04-25T01:59:25.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.044+0000 7f5c61800640 1 -- 172.21.15.102:0/163597145 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c5c1aafd0 con 0x7f5c5c1067c0 2024-04-25T01:59:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.050+0000 7f5c53400640 1 -- 172.21.15.102:0/163597145 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5c54037020 con 0x7f5c5c1067c0 2024-04-25T01:59:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.052+0000 7f5c53400640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f5c400a2ea0 0x7f5c400a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.052+0000 7f5c53400640 1 -- 172.21.15.102:0/163597145 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f5c540f00d0 con 0x7f5c5c1067c0 2024-04-25T01:59:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.053+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f5c400a2ea0 0x7f5c400a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.054+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f5c400a2ea0 0x7f5c400a5360 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009c20 tx=0x7f5c4c005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.054+0000 7f5c645bf740 1 Processor -- start 2024-04-25T01:59:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.055+0000 7f5c645bf740 1 -- start start 2024-04-25T01:59:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c7720 0x55cb9a3d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3d43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c645bf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cb9a32aab0 con 0x55cb9a3c6e00 2024-04-25T01:59:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c645bf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cb9a2d3640 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c645bf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cb9a326a70 con 0x55cb9a3c7720 2024-04-25T01:59:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c5aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c5b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3d43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c5b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3d43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59902/0 (socket says 172.21.15.102:59902) 2024-04-25T01:59:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.056+0000 7f5c5b400640 1 -- 172.21.15.102:0/3344482290 learned_addr learned my addr 172.21.15.102:0/3344482290 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c5b400640 1 -- 172.21.15.102:0/3344482290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c7720 msgr2=0x55cb9a3d1aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c5b400640 1 --2- 172.21.15.102:0/3344482290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c7720 0x55cb9a3d1aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c5b400640 1 -- 172.21.15.102:0/3344482290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 msgr2=0x55cb9a3c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c5b400640 1 --2- 172.21.15.102:0/3344482290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3c71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c5b400640 1 -- 172.21.15.102:0/3344482290 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb9a03f210 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/3344482290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c5b400640 1 --2- 172.21.15.102:0/3344482290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3d43d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5c540b8b30 tx=0x7f5c540bc2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4208d5f39c0abd38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c47e00640 1 -- 172.21.15.102:0/3344482290 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c540a70b0 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c47e00640 1 -- 172.21.15.102:0/3344482290 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c540bcdd0 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.057+0000 7f5c47e00640 1 -- 172.21.15.102:0/3344482290 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c540b8d40 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.060+0000 7f5c645bf740 1 -- 172.21.15.102:0/3344482290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3d1fe0 msgr2=0x55cb9a3d43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.060+0000 7f5c645bf740 1 --2- 172.21.15.102:0/3344482290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3d43d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5c540b8b30 tx=0x7f5c540bc2d0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.060+0000 7f5c645bf740 1 -- 172.21.15.102:0/3344482290 shutdown_connections 2024-04-25T01:59:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.060+0000 7f5c645bf740 1 --2- 172.21.15.102:0/3344482290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3d43d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.060+0000 7f5c645bf740 1 --2- 172.21.15.102:0/3344482290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c7720 0x55cb9a3d1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.060+0000 7f5c645bf740 1 --2- 172.21.15.102:0/3344482290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.060+0000 7f5c645bf740 1 -- 172.21.15.102:0/3344482290 >> 172.21.15.102:0/3344482290 conn(0x55cb9a3c3310 msgr2=0x55cb9a3c36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.061+0000 7f5c645bf740 1 -- 172.21.15.102:0/3344482290 shutdown_connections 2024-04-25T01:59:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.061+0000 7f5c645bf740 1 -- 172.21.15.102:0/3344482290 wait complete. 2024-04-25T01:59:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.061+0000 7f5c645bf740 1 -- start start 2024-04-25T01:59:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.061+0000 7f5c645bf740 1 Processor -- start 2024-04-25T01:59:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.062+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.062+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.062+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.062+0000 7f5c645bf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55cb9a2d3640 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.062+0000 7f5c645bf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55cb9a2e6990 con 0x55cb9a3c7720 2024-04-25T01:59:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.062+0000 7f5c645bf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cb9a3d49d0 con 0x55cb9a3c6e00 2024-04-25T01:59:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.062+0000 7f5c645bf740 10 client.0 _ll_register_callbacks cb 0x55cb9a2e8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.063+0000 7f5c5aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.063+0000 7f5c5aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:46608/0 (socket says 172.21.15.102:46608) 2024-04-25T01:59:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.063+0000 7f5c5aa00640 1 -- 172.21.15.102:0/1329128489 learned_addr learned my addr 172.21.15.102:0/1329128489 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.063+0000 7f5c5a000640 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.063+0000 7f5c5b400640 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.064+0000 7f5c5a000640 1 -- 172.21.15.102:0/1329128489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 msgr2=0x55cb9a3ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.064+0000 7f5c5a000640 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.064+0000 7f5c5a000640 1 -- 172.21.15.102:0/1329128489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 msgr2=0x55cb9a3cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.064+0000 7f5c5a000640 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.064+0000 7f5c5a000640 1 -- 172.21.15.102:0/1329128489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb9a10b620 con 0x55cb9a3c7720 2024-04-25T01:59:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.064+0000 7f5c5a000640 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3cf530 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c1032a0 tx=0x7f5c4800d5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9815e1f3292d39d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.064+0000 7f5c44c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.065+0000 7f5c44c00640 1 -- 172.21.15.102:0/1329128489 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c48014430 con 0x55cb9a3c7720 2024-04-25T01:59:25.070 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.065+0000 7f5c44c00640 1 -- 172.21.15.102:0/1329128489 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f5c480145d0 con 0x55cb9a3c7720 2024-04-25T01:59:25.070 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.065+0000 7f5c645bf740 1 -- 172.21.15.102:0/1329128489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c540bcdd0 con 0x55cb9a3c7720 2024-04-25T01:59:25.070 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.065+0000 7f5c44c00640 1 -- 172.21.15.102:0/1329128489 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5c48022050 con 0x55cb9a3c7720 2024-04-25T01:59:25.070 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.065+0000 7f5c645bf740 10 client.26435 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:25.070 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.065+0000 7f5c645bf740 1 -- 172.21.15.102:0/1329128489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cb9a3e0780 con 0x55cb9a3c7720 2024-04-25T01:59:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.066+0000 7f5c44c00640 1 -- 172.21.15.102:0/1329128489 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c540bcdd0 con 0x55cb9a3c7720 2024-04-25T01:59:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.067+0000 7f5c44c00640 1 -- 172.21.15.102:0/1329128489 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f5c48013070 con 0x55cb9a3c7720 2024-04-25T01:59:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.068+0000 7f5c645bf740 10 client.26435 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.068+0000 7f5c645bf740 10 client.26435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.068+0000 7f5c645bf740 20 client.26435 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.068+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cb9a3e1c40 0x55cb9a3e4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.068+0000 7f5c645bf740 4 client.26435 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.068+0000 7f5c645bf740 1 -- 172.21.15.102:0/1329128489 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb9a337ff0 con 0x55cb9a3e1c40 2024-04-25T01:59:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.069+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cb9a3e1c40 0x55cb9a3e4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.070+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cb9a3e1c40 0x55cb9a3e4030 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51777ace306243b5 server_cookie=61b4ef8e54e49583 in_seq=0 out_seq=0 2024-04-25T01:59:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.070+0000 7f5c44c00640 10 client.26435 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.071+0000 7f5c44c00640 1 -- 172.21.15.102:0/1329128489 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cb9a337ff0 con 0x55cb9a3e1c40 2024-04-25T01:59:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.071+0000 7f5c44c00640 10 client.26435 handle_command_reply: tid=0 2024-04-25T01:59:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.072+0000 7f5c645bf740 1 client.26435 shutdown 2024-04-25T01:59:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 -- 172.21.15.102:0/1329128489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 msgr2=0x55cb9a3cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3cf530 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c1032a0 tx=0x7f5c4800d5f0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 -- 172.21.15.102:0/1329128489 shutdown_connections 2024-04-25T01:59:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cb9a3e1c40 0x55cb9a3e4030 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3cf530 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c44c00640 0 client.26435 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1329128489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 -- 172.21.15.102:0/1329128489 >> 172.21.15.102:0/1329128489 conn(0x55cb9a3c3310 msgr2=0x55cb9a3c9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 -- 172.21.15.102:0/1329128489 shutdown_connections 2024-04-25T01:59:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.073+0000 7f5c645bf740 1 -- 172.21.15.102:0/1329128489 wait complete. 2024-04-25T01:59:25.079 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.074+0000 7f5c645bf740 20 client.26435 trim_cache size 0 max 16384 2024-04-25T01:59:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.102+0000 7f5c645bf740 1 Processor -- start 2024-04-25T01:59:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.102+0000 7f5c645bf740 1 -- start start 2024-04-25T01:59:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.102+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3dd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.102+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3c9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.102+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3c2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.102+0000 7f5c645bf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c1b6200 con 0x55cb9a3c6e00 2024-04-25T01:59:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.102+0000 7f5c645bf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c1b3aa0 con 0x55cb9a3c7720 2024-04-25T01:59:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.102+0000 7f5c645bf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c1b18f0 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59924/0 (socket says 172.21.15.102:59924) 2024-04-25T01:59:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5a000640 1 -- 172.21.15.102:0/1793928052 learned_addr learned my addr 172.21.15.102:0/1793928052 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3dd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5b400640 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3c2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5a000640 1 -- 172.21.15.102:0/1793928052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3d1fe0 msgr2=0x55cb9a3c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5a000640 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3c2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5a000640 1 -- 172.21.15.102:0/1793928052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 msgr2=0x55cb9a3dd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5a000640 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3dd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.103+0000 7f5c5a000640 1 -- 172.21.15.102:0/1793928052 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb9a3c1890 con 0x55cb9a3c7720 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.104+0000 7f5c5b400640 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.104+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3dd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.104+0000 7f5c5a000640 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3c9810 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5c48008010 tx=0x7f5c48004640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12d5efa4eb7909ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.104+0000 7f5c47400640 1 -- 172.21.15.102:0/1793928052 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c480123e0 con 0x55cb9a3c7720 2024-04-25T01:59:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.104+0000 7f5c47400640 1 -- 172.21.15.102:0/1793928052 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c48019070 con 0x55cb9a3c7720 2024-04-25T01:59:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.104+0000 7f5c47400640 1 -- 172.21.15.102:0/1793928052 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4801dc40 con 0x55cb9a3c7720 2024-04-25T01:59:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.106+0000 7f5c645bf740 1 -- 172.21.15.102:0/1793928052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 msgr2=0x55cb9a3c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.106+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3c9810 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5c48008010 tx=0x7f5c48004640 comp rx=0 tx=0).stop 2024-04-25T01:59:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.106+0000 7f5c645bf740 1 -- 172.21.15.102:0/1793928052 shutdown_connections 2024-04-25T01:59:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.106+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3c2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.106+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3c9810 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.106+0000 7f5c645bf740 1 --2- 172.21.15.102:0/1793928052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3dd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.106+0000 7f5c645bf740 1 -- 172.21.15.102:0/1793928052 >> 172.21.15.102:0/1793928052 conn(0x55cb9a3c3310 msgr2=0x55cb9a3c1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.106+0000 7f5c645bf740 1 -- 172.21.15.102:0/1793928052 shutdown_connections 2024-04-25T01:59:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.107+0000 7f5c645bf740 1 -- 172.21.15.102:0/1793928052 wait complete. 2024-04-25T01:59:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.108+0000 7f5c645bf740 1 -- start start 2024-04-25T01:59:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.109+0000 7f5c645bf740 1 Processor -- start 2024-04-25T01:59:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.109+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3d9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.110+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.110+0000 7f5c645bf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.110+0000 7f5c645bf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c1b1f60 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.110+0000 7f5c645bf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c1b33d0 con 0x55cb9a3c7720 2024-04-25T01:59:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.110+0000 7f5c645bf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5c1b2630 con 0x55cb9a3c6e00 2024-04-25T01:59:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.110+0000 7f5c5b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.110+0000 7f5c5b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53124/0 (socket says 172.21.15.102:53124) 2024-04-25T01:59:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.111+0000 7f5c5b400640 1 -- 172.21.15.102:0/413468759 learned_addr learned my addr 172.21.15.102:0/413468759 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:25.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.111+0000 7f5c645bf740 10 client.0 _ll_register_callbacks cb 0x7f5c4800b100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:25.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.111+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3d9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.111+0000 7f5c5b400640 1 -- 172.21.15.102:0/413468759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 msgr2=0x55cb9a3d9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.111+0000 7f5c5b400640 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3d9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.111+0000 7f5c5b400640 1 -- 172.21.15.102:0/413468759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 msgr2=0x55cb9a3cba10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.111+0000 7f5c5b400640 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3cba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.111+0000 7f5c5b400640 1 -- 172.21.15.102:0/413468759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb9a3cc750 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.111+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3d9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.112+0000 7f5c5b400640 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cbf50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5c54032000 tx=0x7f5c540041b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a57bf6a4c50e2ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.112+0000 7f5c44c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.112+0000 7f5c44c00640 1 -- 172.21.15.102:0/413468759 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c540d9070 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.112+0000 7f5c44c00640 1 -- 172.21.15.102:0/413468759 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f5c540bc930 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.113+0000 7f5c645bf740 1 -- 172.21.15.102:0/413468759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c48019070 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.113+0000 7f5c44c00640 1 -- 172.21.15.102:0/413468759 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5c540edc70 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.113+0000 7f5c645bf740 10 client.16041 fetch_fsmap learned FSMap version 31 2024-04-25T01:59:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.114+0000 7f5c645bf740 1 -- 172.21.15.102:0/413468759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cb9a448070 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.114+0000 7f5c44c00640 1 -- 172.21.15.102:0/413468759 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c48019070 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.116+0000 7f5c44c00640 1 -- 172.21.15.102:0/413468759 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 31) ==== 8342+0+0 (secure 0 0 0) 0x7f5c540c4eb0 con 0x55cb9a3d1fe0 2024-04-25T01:59:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.116+0000 7f5c645bf740 10 client.16041 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T01:59:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.116+0000 7f5c645bf740 10 client.16041 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.116+0000 7f5c645bf740 20 client.16041 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.116+0000 7f5c645bf740 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cb9a4485e0 0x55cb9a44aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.116+0000 7f5c645bf740 4 client.16041 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:59:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.116+0000 7f5c645bf740 1 -- 172.21.15.102:0/413468759 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cb9a332340 con 0x55cb9a4485e0 2024-04-25T01:59:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.116+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cb9a4485e0 0x55cb9a44aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.117+0000 7f5c44c00640 10 client.16041 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.117+0000 7f5c5aa00640 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cb9a4485e0 0x55cb9a44aa50 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74ae2b63da8e76d6 server_cookie=5d738fc5a5fb47b1 in_seq=0 out_seq=0 2024-04-25T01:59:25.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.133+0000 7f5c44c00640 1 -- 172.21.15.102:0/413468759 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cb9a332340 con 0x55cb9a4485e0 2024-04-25T01:59:25.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.133+0000 7f5c44c00640 10 client.16041 handle_command_reply: tid=0 2024-04-25T01:59:25.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.133+0000 7f5c645bf740 1 client.16041 shutdown 2024-04-25T01:59:25.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c645bf740 1 -- 172.21.15.102:0/413468759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 msgr2=0x55cb9a3cbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:25.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c645bf740 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cbf50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5c54032000 tx=0x7f5c540041b0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c645bf740 1 -- 172.21.15.102:0/413468759 shutdown_connections 2024-04-25T01:59:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c645bf740 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55cb9a4485e0 0x55cb9a44aa50 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c44c00640 0 client.16041 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c645bf740 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cb9a3d1fe0 0x55cb9a3cbf50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c645bf740 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55cb9a3c7720 0x55cb9a3cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c645bf740 1 --2- 172.21.15.102:0/413468759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb9a3c6e00 0x55cb9a3d9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.139 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c645bf740 1 -- 172.21.15.102:0/413468759 >> 172.21.15.102:0/413468759 conn(0x55cb9a3c3310 msgr2=0x55cb9a3c2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:25.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.134+0000 7f5c645bf740 1 -- 172.21.15.102:0/413468759 shutdown_connections 2024-04-25T01:59:25.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.135+0000 7f5c645bf740 1 -- 172.21.15.102:0/413468759 wait complete. 2024-04-25T01:59:25.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.135+0000 7f5c645bf740 20 client.16041 trim_cache size 0 max 16384 2024-04-25T01:59:25.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:59:25.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:59:25.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:59:25.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:59:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.136+0000 7f5c61800640 1 -- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f5c400a2ea0 msgr2=0x7f5c400a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.136+0000 7f5c61800640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f5c400a2ea0 0x7f5c400a5360 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009c20 tx=0x7f5c4c005f90 comp rx=0 tx=0).stop 2024-04-25T01:59:25.141 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.136+0000 7f5c61800640 1 -- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5c1067c0 msgr2=0x7f5c5c1a2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.136+0000 7f5c61800640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5c1067c0 0x7f5c5c1a2d20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5c54005e30 tx=0x7f5c54003c10 comp rx=0 tx=0).stop 2024-04-25T01:59:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.137+0000 7f5c61800640 1 -- 172.21.15.102:0/163597145 shutdown_connections 2024-04-25T01:59:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.137+0000 7f5c61800640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f5c400a2ea0 0x7f5c400a5360 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.137+0000 7f5c61800640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c5c1067c0 0x7f5c5c1a2d20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.137+0000 7f5c61800640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c5c1001d0 0x7f5c5c10dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.137+0000 7f5c61800640 1 --2- 172.21.15.102:0/163597145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c5c0ff830 0x7f5c5c10d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.137+0000 7f5c61800640 1 -- 172.21.15.102:0/163597145 >> 172.21.15.102:0/163597145 conn(0x7f5c5c076740 msgr2=0x7f5c5c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.137+0000 7f5c61800640 1 -- 172.21.15.102:0/163597145 shutdown_connections 2024-04-25T01:59:25.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:25.137+0000 7f5c61800640 1 -- 172.21.15.102:0/163597145 wait complete. 2024-04-25T01:59:25.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:25 smithi102 ceph-mon[27796]: pgmap v127: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 36 MiB/s wr, 880 op/s 2024-04-25T01:59:25.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:25 smithi110 ceph-mon[32848]: pgmap v127: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 36 MiB/s wr, 880 op/s 2024-04-25T01:59:25.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:25 smithi193 ceph-mon[32981]: pgmap v127: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 36 MiB/s wr, 880 op/s 2024-04-25T01:59:27.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:27 smithi102 ceph-mon[27796]: pgmap v128: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 MiB/s wr, 727 op/s 2024-04-25T01:59:27.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:27 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:59:27.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:27 smithi110 ceph-mon[32848]: pgmap v128: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 MiB/s wr, 727 op/s 2024-04-25T01:59:27.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:27 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:59:27.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:27 smithi193 ceph-mon[32981]: pgmap v128: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 MiB/s wr, 727 op/s 2024-04-25T01:59:27.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:27 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T01:59:28.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 215.0 s, 2667.2 tps, lat 37.505 ms stddev 58.363 2024-04-25T01:59:29.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:29 smithi102 ceph-mon[27796]: pgmap v129: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 40 MiB/s wr, 1.10k op/s 2024-04-25T01:59:29.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:29 smithi110 ceph-mon[32848]: pgmap v129: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 40 MiB/s wr, 1.10k op/s 2024-04-25T01:59:29.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:29 smithi193 ceph-mon[32981]: pgmap v129: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 40 MiB/s wr, 1.10k op/s 2024-04-25T01:59:31.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:31 smithi102 ceph-mon[27796]: pgmap v130: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 25 MiB/s wr, 799 op/s 2024-04-25T01:59:31.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:31 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:59:31.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:31 smithi110 ceph-mon[32848]: pgmap v130: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 25 MiB/s wr, 799 op/s 2024-04-25T01:59:31.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:31 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:59:31.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:31 smithi193 ceph-mon[32981]: pgmap v130: 97 pgs: 97 active+clean; 2.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 25 MiB/s wr, 799 op/s 2024-04-25T01:59:31.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:31 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T01:59:32.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:32 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:59:32.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:32 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:59:32.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:32 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T01:59:32.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:32 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:59:32.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:32 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:59:32.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:32 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T01:59:32.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:32 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:59:32.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:32 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:59:32.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:32 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 220.0 s, 2579.6 tps, lat 38.644 ms stddev 47.843 2024-04-25T01:59:33.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:33 smithi102 ceph-mon[27796]: pgmap v131: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 31 MiB/s wr, 1.05k op/s 2024-04-25T01:59:33.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:33 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:59:33.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:33 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:59:33.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:33 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:59:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:33 smithi110 ceph-mon[32848]: pgmap v131: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 31 MiB/s wr, 1.05k op/s 2024-04-25T01:59:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:33 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:59:33.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:33 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:59:33.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:33 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:59:33.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:33 smithi193 ceph-mon[32981]: pgmap v131: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 31 MiB/s wr, 1.05k op/s 2024-04-25T01:59:33.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:33 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T01:59:33.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:33 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T01:59:33.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:33 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T01:59:35.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:35 smithi102 ceph-mon[27796]: pgmap v132: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 26 MiB/s wr, 925 op/s 2024-04-25T01:59:35.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:35 smithi110 ceph-mon[32848]: pgmap v132: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 26 MiB/s wr, 925 op/s 2024-04-25T01:59:35.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:35 smithi193 ceph-mon[32981]: pgmap v132: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 26 MiB/s wr, 925 op/s 2024-04-25T01:59:37.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:37 smithi102 ceph-mon[27796]: pgmap v133: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 22 MiB/s wr, 770 op/s 2024-04-25T01:59:37.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:37 smithi110 ceph-mon[32848]: pgmap v133: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 22 MiB/s wr, 770 op/s 2024-04-25T01:59:37.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:37 smithi193 ceph-mon[32981]: pgmap v133: 97 pgs: 97 active+clean; 3.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 22 MiB/s wr, 770 op/s 2024-04-25T01:59:38.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 225.0 s, 2250.2 tps, lat 44.822 ms stddev 96.098 2024-04-25T01:59:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.338+0000 7fcad6a00640 1 -- 172.21.15.102:0/3243703128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0109930 msgr2=0x7fcad0109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.338+0000 7fcad6a00640 1 --2- 172.21.15.102:0/3243703128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0109930 0x7fcad0109d10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009930 tx=0x7fcac002a440 comp rx=0 tx=0).stop 2024-04-25T01:59:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.338+0000 7fcad6a00640 1 -- 172.21.15.102:0/3243703128 shutdown_connections 2024-04-25T01:59:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.338+0000 7fcad6a00640 1 --2- 172.21.15.102:0/3243703128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcad01042e0 0x7fcad01107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.338+0000 7fcad6a00640 1 --2- 172.21.15.102:0/3243703128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcad0103920 0x7fcad0103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.338+0000 7fcad6a00640 1 --2- 172.21.15.102:0/3243703128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0109930 0x7fcad0109d10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.338+0000 7fcad6a00640 1 -- 172.21.15.102:0/3243703128 >> 172.21.15.102:0/3243703128 conn(0x7fcad00ff430 msgr2=0x7fcad0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.338+0000 7fcad6a00640 1 -- 172.21.15.102:0/3243703128 shutdown_connections 2024-04-25T01:59:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.338+0000 7fcad6a00640 1 -- 172.21.15.102:0/3243703128 wait complete. 2024-04-25T01:59:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.340+0000 7fcad6a00640 1 Processor -- start 2024-04-25T01:59:38.343 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.340+0000 7fcad6a00640 1 -- start start 2024-04-25T01:59:38.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.341+0000 7fcad6a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcad0103920 0x7fcad0110600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.341+0000 7fcad6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcad01042e0 0x7fcad010d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.341+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcad0103920 0x7fcad0110600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.341+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcad0103920 0x7fcad0110600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:39766/0 (socket says 172.21.15.102:39766) 2024-04-25T01:59:38.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.341+0000 7fcacfe00640 1 -- 172.21.15.102:0/150781783 learned_addr learned my addr 172.21.15.102:0/150781783 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:38.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.342+0000 7fcad6a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0109930 0x7fcad010dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.342+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcad0113d90 con 0x7fcad01042e0 2024-04-25T01:59:38.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.342+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcad0113c10 con 0x7fcad0103920 2024-04-25T01:59:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.342+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcad0113f10 con 0x7fcad0109930 2024-04-25T01:59:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.342+0000 7fcacf400640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcad01042e0 0x7fcad010d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.342+0000 7fcacfe00640 1 -- 172.21.15.102:0/150781783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0109930 msgr2=0x7fcad010dbe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T01:59:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.342+0000 7fcad4a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0109930 0x7fcad010dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.343+0000 7fcacfe00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0109930 0x7fcad010dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.343+0000 7fcacfe00640 1 -- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcad01042e0 msgr2=0x7fcad010d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.343+0000 7fcacfe00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcad01042e0 0x7fcad010d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.343+0000 7fcacfe00640 1 -- 172.21.15.102:0/150781783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad010e470 con 0x7fcad0103920 2024-04-25T01:59:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.343+0000 7fcacf400640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcad01042e0 0x7fcad010d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.343+0000 7fcad4a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0109930 0x7fcad010dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.343+0000 7fcacfe00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcad0103920 0x7fcad0110600 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcac002a960 tx=0x7fcac0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.344+0000 7fcaccc00640 1 -- 172.21.15.102:0/150781783 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac003f070 con 0x7fcad0103920 2024-04-25T01:59:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.344+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad01a02b0 con 0x7fcad0103920 2024-04-25T01:59:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.344+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad01a0850 con 0x7fcad0103920 2024-04-25T01:59:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.344+0000 7fcaccc00640 1 -- 172.21.15.102:0/150781783 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcac0031c90 con 0x7fcad0103920 2024-04-25T01:59:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.344+0000 7fcaccc00640 1 -- 172.21.15.102:0/150781783 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0035030 con 0x7fcad0103920 2024-04-25T01:59:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.349+0000 7fcad997b740 1 Processor -- start 2024-04-25T01:59:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.350+0000 7fcad997b740 1 -- start start 2024-04-25T01:59:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.350+0000 7fcaccc00640 1 -- 172.21.15.102:0/150781783 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcac0042070 con 0x7fcad0103920 2024-04-25T01:59:38.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcad997b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed661e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcad997b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed70aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcad997b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed733d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcad997b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f45ecc9ab0 con 0x55f45ed70fe0 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcad997b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f45ec72640 con 0x55f45ed65e00 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcad997b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f45ecc5a70 con 0x55f45ed66720 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcad4a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcad4a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed661e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:39778/0 (socket says 172.21.15.102:39778) 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcad4a00640 1 -- 172.21.15.102:0/894852211 learned_addr learned my addr 172.21.15.102:0/894852211 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcacfe00640 1 --2- 172.21.15.102:0/894852211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed70aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.351+0000 7fcacf400640 1 --2- 172.21.15.102:0/894852211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed733d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.352+0000 7fcaccc00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcab40a2e20 0x7fcab40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.352+0000 7fcad4a00640 1 -- 172.21.15.102:0/894852211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 msgr2=0x55f45ed70aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.352+0000 7fcad4a00640 1 --2- 172.21.15.102:0/894852211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed70aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.352+0000 7fcad4a00640 1 -- 172.21.15.102:0/894852211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 msgr2=0x55f45ed733d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.352+0000 7fcad4a00640 1 --2- 172.21.15.102:0/894852211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed733d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.352+0000 7fcad4a00640 1 -- 172.21.15.102:0/894852211 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f45e9de210 con 0x55f45ed65e00 2024-04-25T01:59:38.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.352+0000 7fcaccc00640 1 -- 172.21.15.102:0/150781783 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fcac00ef6c0 con 0x7fcad0103920 2024-04-25T01:59:38.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.353+0000 7fcad4a00640 1 --2- 172.21.15.102:0/894852211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed661e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009f10 tx=0x7fcac800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7f89e2d06d21d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:38.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.353+0000 7fcad4a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcab40a2e20 0x7fcab40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.361 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.358+0000 7fcad4a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcab40a2e20 0x7fcab40a52e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcac8019000 tx=0x7fcac801d040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:38.361 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.358+0000 7fcac5200640 1 -- 172.21.15.102:0/894852211 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8011640 con 0x55f45ed65e00 2024-04-25T01:59:38.361 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.358+0000 7fcac5200640 1 -- 172.21.15.102:0/894852211 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcac801bdc0 con 0x55f45ed65e00 2024-04-25T01:59:38.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.358+0000 7fcac5200640 1 -- 172.21.15.102:0/894852211 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac80103d0 con 0x55f45ed65e00 2024-04-25T01:59:38.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.360+0000 7fcad997b740 1 -- 172.21.15.102:0/894852211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 msgr2=0x55f45ed661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:38.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.360+0000 7fcad997b740 1 --2- 172.21.15.102:0/894852211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed661e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009f10 tx=0x7fcac800f730 comp rx=0 tx=0).stop 2024-04-25T01:59:38.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.360+0000 7fcad997b740 1 -- 172.21.15.102:0/894852211 shutdown_connections 2024-04-25T01:59:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.360+0000 7fcad997b740 1 --2- 172.21.15.102:0/894852211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed733d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.360+0000 7fcad997b740 1 --2- 172.21.15.102:0/894852211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed70aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.360+0000 7fcad997b740 1 --2- 172.21.15.102:0/894852211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed661e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.360+0000 7fcad997b740 1 -- 172.21.15.102:0/894852211 >> 172.21.15.102:0/894852211 conn(0x55f45ed62310 msgr2=0x55f45ed626f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.361+0000 7fcad997b740 1 -- 172.21.15.102:0/894852211 shutdown_connections 2024-04-25T01:59:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.362+0000 7fcad997b740 1 -- 172.21.15.102:0/894852211 wait complete. 2024-04-25T01:59:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.362+0000 7fcad997b740 1 -- start start 2024-04-25T01:59:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.362+0000 7fcad997b740 1 Processor -- start 2024-04-25T01:59:38.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.365+0000 7fcad997b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed6df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.365+0000 7fcad997b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed6e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.365+0000 7fcad997b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed6e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcad997b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f45ed73c10 con 0x55f45ed70fe0 2024-04-25T01:59:38.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcad997b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f45ec85990 con 0x55f45ed65e00 2024-04-25T01:59:38.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcad997b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f45ed73d90 con 0x55f45ed66720 2024-04-25T01:59:38.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed6df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcacf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed6e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcacf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed6e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35044/0 (socket says 172.21.15.102:35044) 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcacf400640 1 -- 172.21.15.102:0/1355316333 learned_addr learned my addr 172.21.15.102:0/1355316333 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcad997b740 10 client.0 _ll_register_callbacks cb 0x55f45ec87950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcad4a00640 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed6e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcacfe00640 1 -- 172.21.15.102:0/1355316333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 msgr2=0x55f45ed6e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcacfe00640 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed6e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcacfe00640 1 -- 172.21.15.102:0/1355316333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 msgr2=0x55f45ed6e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcacfe00640 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed6e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.366+0000 7fcacfe00640 1 -- 172.21.15.102:0/1355316333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f45eaaa620 con 0x55f45ed65e00 2024-04-25T01:59:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.367+0000 7fcacfe00640 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed6df60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcac00ece40 tx=0x7fcac002a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed0e0656334f9c27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.367+0000 7fcab9600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.367+0000 7fcab9600640 1 -- 172.21.15.102:0/1355316333 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0049070 con 0x55f45ed65e00 2024-04-25T01:59:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.368+0000 7fcab9600640 1 -- 172.21.15.102:0/1355316333 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fcac00c43f0 con 0x55f45ed65e00 2024-04-25T01:59:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.369+0000 7fcad997b740 1 -- 172.21.15.102:0/1355316333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcac801bdc0 con 0x55f45ed65e00 2024-04-25T01:59:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.370+0000 7fcab9600640 1 -- 172.21.15.102:0/1355316333 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcac00b7eb0 con 0x55f45ed65e00 2024-04-25T01:59:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.370+0000 7fcad997b740 10 client.26450 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.370+0000 7fcad997b740 1 -- 172.21.15.102:0/1355316333 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f45ed7fb60 con 0x55f45ed65e00 2024-04-25T01:59:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.371+0000 7fcab9600640 1 -- 172.21.15.102:0/1355316333 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcac801bdc0 con 0x55f45ed65e00 2024-04-25T01:59:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.371+0000 7fcab9600640 1 -- 172.21.15.102:0/1355316333 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fcac004c060 con 0x55f45ed65e00 2024-04-25T01:59:38.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.371+0000 7fcad997b740 10 client.26450 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.371+0000 7fcad997b740 10 client.26450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.371+0000 7fcad997b740 20 client.26450 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.371+0000 7fcad997b740 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f45ed81230 0x55f45ed836d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.371+0000 7fcad997b740 4 client.26450 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.371+0000 7fcad997b740 1 -- 172.21.15.102:0/1355316333 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f45ecd6ff0 con 0x55f45ed81230 2024-04-25T01:59:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.371+0000 7fcacf400640 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f45ed81230 0x55f45ed836d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.372+0000 7fcacf400640 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f45ed81230 0x55f45ed836d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97e475f4d068df2b server_cookie=aa2405c29d5eed6f in_seq=0 out_seq=0 2024-04-25T01:59:38.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.372+0000 7fcab9600640 10 client.26450 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.373+0000 7fcab9600640 1 -- 172.21.15.102:0/1355316333 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f45ecd6ff0 con 0x55f45ed81230 2024-04-25T01:59:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.373+0000 7fcab9600640 10 client.26450 handle_command_reply: tid=0 2024-04-25T01:59:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.373+0000 7fcad997b740 1 client.26450 shutdown 2024-04-25T01:59:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.374+0000 7fcad997b740 1 -- 172.21.15.102:0/1355316333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 msgr2=0x55f45ed6df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:38.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.374+0000 7fcad997b740 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed6df60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcac00ece40 tx=0x7fcac002a440 comp rx=0 tx=0).stop 2024-04-25T01:59:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.374+0000 7fcad997b740 1 -- 172.21.15.102:0/1355316333 shutdown_connections 2024-04-25T01:59:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.374+0000 7fcad997b740 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f45ed81230 0x55f45ed836d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.374+0000 7fcad997b740 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed6e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.374+0000 7fcad997b740 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed6e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.374+0000 7fcad997b740 1 --2- 172.21.15.102:0/1355316333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed65e00 0x55f45ed6df60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.378 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.374+0000 7fcad997b740 1 -- 172.21.15.102:0/1355316333 >> 172.21.15.102:0/1355316333 conn(0x55f45ed62310 msgr2=0x55f45ed68090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.374+0000 7fcab9600640 0 client.26450 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.375+0000 7fcad997b740 1 -- 172.21.15.102:0/1355316333 shutdown_connections 2024-04-25T01:59:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.376+0000 7fcad997b740 1 -- 172.21.15.102:0/1355316333 wait complete. 2024-04-25T01:59:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.377+0000 7fcad997b740 20 client.26450 trim_cache size 0 max 16384 2024-04-25T01:59:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.402+0000 7fcad997b740 1 Processor -- start 2024-04-25T01:59:38.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.402+0000 7fcad997b740 1 -- start start 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad997b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed7cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad997b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed6c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad997b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed6e830 0x55f45ed88e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad997b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcab00110a0 con 0x55f45ed70fe0 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad997b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcab000e910 con 0x55f45ed6e830 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad997b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcab000c760 con 0x55f45ed66720 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad4a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed6e830 0x55f45ed88e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad4a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed6e830 0x55f45ed88e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:39786/0 (socket says 172.21.15.102:39786) 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad4a00640 1 -- 172.21.15.102:0/4164022362 learned_addr learned my addr 172.21.15.102:0/4164022362 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed7cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcacf400640 1 --2- 172.21.15.102:0/4164022362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed6c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad4a00640 1 -- 172.21.15.102:0/4164022362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 msgr2=0x55f45ed7cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad4a00640 1 --2- 172.21.15.102:0/4164022362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed7cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad4a00640 1 -- 172.21.15.102:0/4164022362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 msgr2=0x55f45ed6c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad4a00640 1 --2- 172.21.15.102:0/4164022362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed6c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcad4a00640 1 -- 172.21.15.102:0/4164022362 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f45ed6cf10 con 0x55f45ed6e830 2024-04-25T01:59:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.403+0000 7fcacf400640 1 --2- 172.21.15.102:0/4164022362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed6c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.404+0000 7fcad4a00640 1 --2- 172.21.15.102:0/4164022362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed6e830 0x55f45ed88e30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcac8005c90 tx=0x7fcac801ad90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcb794467df4c53d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.405+0000 7fcabbe00640 1 -- 172.21.15.102:0/4164022362 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8004a00 con 0x55f45ed6e830 2024-04-25T01:59:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.405+0000 7fcabbe00640 1 -- 172.21.15.102:0/4164022362 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcac801bd40 con 0x55f45ed6e830 2024-04-25T01:59:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.405+0000 7fcabbe00640 1 -- 172.21.15.102:0/4164022362 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac801a8f0 con 0x55f45ed6e830 2024-04-25T01:59:38.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.406+0000 7fcad997b740 1 -- 172.21.15.102:0/4164022362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed6e830 msgr2=0x55f45ed88e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.406+0000 7fcad997b740 1 --2- 172.21.15.102:0/4164022362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed6e830 0x55f45ed88e30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcac8005c90 tx=0x7fcac801ad90 comp rx=0 tx=0).stop 2024-04-25T01:59:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.406+0000 7fcad997b740 1 -- 172.21.15.102:0/4164022362 shutdown_connections 2024-04-25T01:59:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.406+0000 7fcad997b740 1 --2- 172.21.15.102:0/4164022362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed6e830 0x55f45ed88e30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.406+0000 7fcad997b740 1 --2- 172.21.15.102:0/4164022362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed70fe0 0x55f45ed6c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.406+0000 7fcad997b740 1 --2- 172.21.15.102:0/4164022362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed66720 0x55f45ed7cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.406+0000 7fcad997b740 1 -- 172.21.15.102:0/4164022362 >> 172.21.15.102:0/4164022362 conn(0x55f45ed62310 msgr2=0x55f45ed61480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.406+0000 7fcad997b740 1 -- 172.21.15.102:0/4164022362 shutdown_connections 2024-04-25T01:59:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.406+0000 7fcad997b740 1 -- 172.21.15.102:0/4164022362 wait complete. 2024-04-25T01:59:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.408+0000 7fcad997b740 1 -- start start 2024-04-25T01:59:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.408+0000 7fcad997b740 1 Processor -- start 2024-04-25T01:59:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcad997b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed66720 0x55f45ed79060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcad997b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed6e830 0x55f45ed795a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcad997b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed70fe0 0x55f45ed76030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcad997b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcab000cdd0 con 0x55f45ed66720 2024-04-25T01:59:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcad997b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcab000e240 con 0x55f45ed70fe0 2024-04-25T01:59:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcad997b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcab000d4a0 con 0x55f45ed6e830 2024-04-25T01:59:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcad997b740 10 client.0 _ll_register_callbacks cb 0x7fcac00f5240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcad4a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed70fe0 0x55f45ed76030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcad4a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed70fe0 0x55f45ed76030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:39802/0 (socket says 172.21.15.102:39802) 2024-04-25T01:59:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.410+0000 7fcad4a00640 1 -- 172.21.15.102:0/380310838 learned_addr learned my addr 172.21.15.102:0/380310838 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.409+0000 7fcacf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed6e830 0x55f45ed795a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.410+0000 7fcad4a00640 1 -- 172.21.15.102:0/380310838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed6e830 msgr2=0x55f45ed795a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.410+0000 7fcad4a00640 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed6e830 0x55f45ed795a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.410+0000 7fcad4a00640 1 -- 172.21.15.102:0/380310838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed66720 msgr2=0x55f45ed79060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:59:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.410+0000 7fcad4a00640 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed66720 0x55f45ed79060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.410+0000 7fcad4a00640 1 -- 172.21.15.102:0/380310838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f45ed767a0 con 0x55f45ed70fe0 2024-04-25T01:59:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.411+0000 7fcad4a00640 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed70fe0 0x55f45ed76030 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcac8033040 tx=0x7fcac8009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d24458fdc4a71b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.411+0000 7fcab9600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:38.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.412+0000 7fcab9600640 1 -- 172.21.15.102:0/380310838 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac803f020 con 0x55f45ed70fe0 2024-04-25T01:59:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.412+0000 7fcab9600640 1 -- 172.21.15.102:0/380310838 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fcac8004a00 con 0x55f45ed70fe0 2024-04-25T01:59:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.412+0000 7fcad997b740 1 -- 172.21.15.102:0/380310838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcac801bd40 con 0x55f45ed70fe0 2024-04-25T01:59:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcab9600640 1 -- 172.21.15.102:0/380310838 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcac801bb70 con 0x55f45ed70fe0 2024-04-25T01:59:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcad997b740 10 client.26456 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcad997b740 1 -- 172.21.15.102:0/380310838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f45ede8120 con 0x55f45ed70fe0 2024-04-25T01:59:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcab9600640 1 -- 172.21.15.102:0/380310838 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcac801bd40 con 0x55f45ed70fe0 2024-04-25T01:59:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcab9600640 1 -- 172.21.15.102:0/380310838 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fcac8030070 con 0x55f45ed70fe0 2024-04-25T01:59:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcad997b740 10 client.26456 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcad997b740 10 client.26456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcad997b740 20 client.26456 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcad997b740 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f45ede8720 0x55f45edeab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcad997b740 4 client.26456 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:59:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.413+0000 7fcad997b740 1 -- 172.21.15.102:0/380310838 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f45ecd1340 con 0x55f45ede8720 2024-04-25T01:59:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.414+0000 7fcacfe00640 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f45ede8720 0x55f45edeab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.414+0000 7fcab9600640 10 client.26456 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.414+0000 7fcacfe00640 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f45ede8720 0x55f45edeab40 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb79a98a52f0a536 server_cookie=fddeecd1f5a3c13f in_seq=0 out_seq=0 2024-04-25T01:59:38.474 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcab9600640 1 -- 172.21.15.102:0/380310838 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f45ecd1340 con 0x55f45ede8720 2024-04-25T01:59:38.475 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcab9600640 10 client.26456 handle_command_reply: tid=0 2024-04-25T01:59:38.475 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcad997b740 1 client.26456 shutdown 2024-04-25T01:59:38.475 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcad997b740 1 -- 172.21.15.102:0/380310838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed70fe0 msgr2=0x55f45ed76030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:38.475 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcad997b740 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed70fe0 0x55f45ed76030 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcac8033040 tx=0x7fcac8009cd0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.475 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcad997b740 1 -- 172.21.15.102:0/380310838 shutdown_connections 2024-04-25T01:59:38.475 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcad997b740 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f45ede8720 0x55f45edeab40 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.475 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcad997b740 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f45ed70fe0 0x55f45ed76030 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.476 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcad997b740 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f45ed6e830 0x55f45ed795a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.476 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcab9600640 0 client.26456 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:38.476 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.471+0000 7fcad997b740 1 --2- 172.21.15.102:0/380310838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f45ed66720 0x55f45ed79060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.476 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.472+0000 7fcad997b740 1 -- 172.21.15.102:0/380310838 >> 172.21.15.102:0/380310838 conn(0x55f45ed62310 msgr2=0x55f45ed6d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:38.476 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.472+0000 7fcad997b740 1 -- 172.21.15.102:0/380310838 shutdown_connections 2024-04-25T01:59:38.476 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.472+0000 7fcad997b740 1 -- 172.21.15.102:0/380310838 wait complete. 2024-04-25T01:59:38.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.473+0000 7fcad997b740 20 client.26456 trim_cache size 0 max 16384 2024-04-25T01:59:38.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:59:38.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:59:38.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:59:38.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:59:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcab40a2e20 msgr2=0x7fcab40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcab40a2e20 0x7fcab40a52e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcac8019000 tx=0x7fcac801d040 comp rx=0 tx=0).stop 2024-04-25T01:59:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcad0103920 msgr2=0x7fcad0110600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcad0103920 0x7fcad0110600 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcac002a960 tx=0x7fcac0003b90 comp rx=0 tx=0).stop 2024-04-25T01:59:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 shutdown_connections 2024-04-25T01:59:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcab40a2e20 0x7fcab40a52e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0109930 0x7fcad010dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcad01042e0 0x7fcad010d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 --2- 172.21.15.102:0/150781783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcad0103920 0x7fcad0110600 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:38.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.474+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 >> 172.21.15.102:0/150781783 conn(0x7fcad00ff430 msgr2=0x7fcad00ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:38.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.475+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 shutdown_connections 2024-04-25T01:59:38.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:38.475+0000 7fcad6a00640 1 -- 172.21.15.102:0/150781783 wait complete. 2024-04-25T01:59:39.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:39 smithi102 ceph-mon[27796]: pgmap v134: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 50 MiB/s wr, 2.15k op/s 2024-04-25T01:59:39.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:39 smithi110 ceph-mon[32848]: pgmap v134: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 50 MiB/s wr, 2.15k op/s 2024-04-25T01:59:39.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:39 smithi193 ceph-mon[32981]: pgmap v134: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 50 MiB/s wr, 2.15k op/s 2024-04-25T01:59:41.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:41 smithi102 ceph-mon[27796]: pgmap v135: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 40 MiB/s wr, 1.77k op/s 2024-04-25T01:59:41.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:41 smithi110 ceph-mon[32848]: pgmap v135: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 40 MiB/s wr, 1.77k op/s 2024-04-25T01:59:41.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:41 smithi193 ceph-mon[32981]: pgmap v135: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 40 MiB/s wr, 1.77k op/s 2024-04-25T01:59:43.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 230.0 s, 2657.6 tps, lat 37.426 ms stddev 44.137 2024-04-25T01:59:43.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:43 smithi102 ceph-mon[27796]: pgmap v136: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 43 MiB/s wr, 1.98k op/s 2024-04-25T01:59:43.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:43 smithi110 ceph-mon[32848]: pgmap v136: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 43 MiB/s wr, 1.98k op/s 2024-04-25T01:59:43.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:43 smithi193 ceph-mon[32981]: pgmap v136: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 43 MiB/s wr, 1.98k op/s 2024-04-25T01:59:45.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:45 smithi102 ceph-mon[27796]: pgmap v137: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 38 MiB/s wr, 1.84k op/s 2024-04-25T01:59:45.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:45 smithi110 ceph-mon[32848]: pgmap v137: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 38 MiB/s wr, 1.84k op/s 2024-04-25T01:59:45.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:45 smithi193 ceph-mon[32981]: pgmap v137: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 38 MiB/s wr, 1.84k op/s 2024-04-25T01:59:47.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:47 smithi102 ceph-mon[27796]: pgmap v138: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 34 MiB/s wr, 1.69k op/s 2024-04-25T01:59:47.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:47 smithi110 ceph-mon[32848]: pgmap v138: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 34 MiB/s wr, 1.69k op/s 2024-04-25T01:59:47.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:47 smithi193 ceph-mon[32981]: pgmap v138: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 34 MiB/s wr, 1.69k op/s 2024-04-25T01:59:48.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 235.0 s, 3042.6 tps, lat 32.437 ms stddev 37.565 2024-04-25T01:59:49.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:49 smithi102 ceph-mon[27796]: pgmap v139: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 38 MiB/s wr, 2.00k op/s 2024-04-25T01:59:49.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:49 smithi110 ceph-mon[32848]: pgmap v139: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 38 MiB/s wr, 2.00k op/s 2024-04-25T01:59:49.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:49 smithi193 ceph-mon[32981]: pgmap v139: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 38 MiB/s wr, 2.00k op/s 2024-04-25T01:59:51.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:51 smithi102 ceph-mon[27796]: pgmap v140: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.7 MiB/s wr, 618 op/s 2024-04-25T01:59:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.676+0000 7fe6d6000640 1 -- 172.21.15.102:0/3274908685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6d01042e0 msgr2=0x7fe6d01107f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.676+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3274908685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6d01042e0 0x7fe6d01107f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009ca0 tx=0x7fe6c802a8e0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.676+0000 7fe6d6000640 1 -- 172.21.15.102:0/3274908685 shutdown_connections 2024-04-25T01:59:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.676+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3274908685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6d01042e0 0x7fe6d01107f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.676+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3274908685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0103920 0x7fe6d0103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.676+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3274908685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6d0109930 0x7fe6d0109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.676+0000 7fe6d6000640 1 -- 172.21.15.102:0/3274908685 >> 172.21.15.102:0/3274908685 conn(0x7fe6d00ff430 msgr2=0x7fe6d0101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.677+0000 7fe6d6000640 1 -- 172.21.15.102:0/3274908685 shutdown_connections 2024-04-25T01:59:51.682 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.677+0000 7fe6d6000640 1 -- 172.21.15.102:0/3274908685 wait complete. 2024-04-25T01:59:51.682 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.678+0000 7fe6d6000640 1 Processor -- start 2024-04-25T01:59:51.682 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.679+0000 7fe6d6000640 1 -- start start 2024-04-25T01:59:51.683 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.679+0000 7fe6d6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6d0103920 0x7fe6d0072e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.683 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.679+0000 7fe6d6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6d01042e0 0x7fe6d0073390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.683 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.679+0000 7fe6d6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0109930 0x7fe6d006c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.679+0000 7fe6d6000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d0113e30 con 0x7fe6d0103920 2024-04-25T01:59:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.679+0000 7fe6d6000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d0113cb0 con 0x7fe6d01042e0 2024-04-25T01:59:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.679+0000 7fe6d6000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d0113fb0 con 0x7fe6d0109930 2024-04-25T01:59:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.680+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0109930 0x7fe6d006c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.680+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0109930 0x7fe6d006c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:45324/0 (socket says 172.21.15.102:45324) 2024-04-25T01:59:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.680+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3816248184 learned_addr learned my addr 172.21.15.102:0/3816248184 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.680+0000 7fe6c7400640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6d0103920 0x7fe6d0072e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.684 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.680+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6d01042e0 0x7fe6d0073390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.681+0000 7fe6c7400640 1 -- 172.21.15.102:0/3816248184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0109930 msgr2=0x7fe6d006c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.681+0000 7fe6c7400640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0109930 0x7fe6d006c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.681+0000 7fe6c7400640 1 -- 172.21.15.102:0/3816248184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6d01042e0 msgr2=0x7fe6d0073390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.681+0000 7fe6c7400640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6d01042e0 0x7fe6d0073390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.681+0000 7fe6c7400640 1 -- 172.21.15.102:0/3816248184 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d006c550 con 0x7fe6d0103920 2024-04-25T01:59:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.681+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6d01042e0 0x7fe6d0073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.681+0000 7fe6c7400640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6d0103920 0x7fe6d0072e50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000b3f0 tx=0x7fe6c000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.682+0000 7fe6ccc00640 1 -- 172.21.15.102:0/3816248184 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0015020 con 0x7fe6d0103920 2024-04-25T01:59:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.682+0000 7fe6ccc00640 1 -- 172.21.15.102:0/3816248184 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6c0011d10 con 0x7fe6d0103920 2024-04-25T01:59:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.682+0000 7fe6d6000640 1 -- 172.21.15.102:0/3816248184 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d006c840 con 0x7fe6d0103920 2024-04-25T01:59:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.682+0000 7fe6ccc00640 1 -- 172.21.15.102:0/3816248184 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c000f2e0 con 0x7fe6d0103920 2024-04-25T01:59:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.683+0000 7fe6d6000640 1 -- 172.21.15.102:0/3816248184 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d0070970 con 0x7fe6d0103920 2024-04-25T01:59:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.687+0000 7fe6d8fd6740 1 Processor -- start 2024-04-25T01:59:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.688+0000 7fe6d8fd6740 1 -- start start 2024-04-25T01:59:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.689+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d951e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.689+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59d9faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.689+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.689+0000 7fe6d8fd6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x558f59cf8ab0 con 0x558f59d95720 2024-04-25T01:59:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.689+0000 7fe6d8fd6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558f59ca1640 con 0x558f59d94e00 2024-04-25T01:59:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.689+0000 7fe6d8fd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558f59cf4a70 con 0x558f59d9ffe0 2024-04-25T01:59:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.690+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.690+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d951e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:49470/0 (socket says 172.21.15.102:49470) 2024-04-25T01:59:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.690+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3432622082 learned_addr learned my addr 172.21.15.102:0/3432622082 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.690+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3432622082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.691+0000 7fe6ccc00640 1 -- 172.21.15.102:0/3816248184 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe6c0018020 con 0x7fe6d0103920 2024-04-25T01:59:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.691+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3432622082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 msgr2=0x558f59da23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.691+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/3432622082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.691+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3432622082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 msgr2=0x558f59d9faa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:59:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.691+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/3432622082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59d9faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.691+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3432622082 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f59a0d210 con 0x558f59d94e00 2024-04-25T01:59:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.692+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/3432622082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d951e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe6c80317b0 tx=0x7fe6c80317e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c94a8bd15538750 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.692+0000 7fe6bfe00640 1 -- 172.21.15.102:0/3432622082 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8031bf0 con 0x558f59d94e00 2024-04-25T01:59:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.693+0000 7fe6bfe00640 1 -- 172.21.15.102:0/3432622082 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6c8036030 con 0x558f59d94e00 2024-04-25T01:59:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.693+0000 7fe6bfe00640 1 -- 172.21.15.102:0/3432622082 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8004860 con 0x558f59d94e00 2024-04-25T01:59:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.693+0000 7fe6ccc00640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe6b40a2da0 0x7fe6b40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.693+0000 7fe6ccc00640 1 -- 172.21.15.102:0/3816248184 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe6c00ce7a0 con 0x7fe6d0103920 2024-04-25T01:59:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.693+0000 7fe6c7400640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe6b40a2da0 0x7fe6b40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.694+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3432622082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 msgr2=0x558f59d951e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.694+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3432622082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d951e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe6c80317b0 tx=0x7fe6c80317e0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.694+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3432622082 shutdown_connections 2024-04-25T01:59:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.694+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3432622082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.694+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3432622082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59d9faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.694+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3432622082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d951e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.694+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3432622082 >> 172.21.15.102:0/3432622082 conn(0x558f59d91310 msgr2=0x558f59d916f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.695+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3432622082 shutdown_connections 2024-04-25T01:59:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.695+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3432622082 wait complete. 2024-04-25T01:59:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.695+0000 7fe6c7400640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe6b40a2da0 0x7fe6b40a5260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x558f59d963a0 tx=0x7fe6c00046c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.695+0000 7fe6d8fd6740 1 -- start start 2024-04-25T01:59:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.696+0000 7fe6d8fd6740 1 Processor -- start 2024-04-25T01:59:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d9cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59d9d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d9cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d9cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:49472/0 (socket says 172.21.15.102:49472) 2024-04-25T01:59:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59d9d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x558f59da2bb0 con 0x558f59d95720 2024-04-25T01:59:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6cfe00640 1 -- 172.21.15.102:0/4148374107 learned_addr learned my addr 172.21.15.102:0/4148374107 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558f59cb4990 con 0x558f59d94e00 2024-04-25T01:59:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558f59da2d30 con 0x558f59d9ffe0 2024-04-25T01:59:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.697+0000 7fe6cf400640 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59d9d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.698+0000 7fe6d8fd6740 10 client.0 _ll_register_callbacks cb 0x558f59cb6950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:51.702 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.698+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59d9d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.699+0000 7fe6cf400640 1 -- 172.21.15.102:0/4148374107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 msgr2=0x558f59d9d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:51.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.699+0000 7fe6cf400640 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59d9d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.700+0000 7fe6cf400640 1 -- 172.21.15.102:0/4148374107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 msgr2=0x558f59d9cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:51.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.700+0000 7fe6cf400640 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d9cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.700+0000 7fe6cf400640 1 -- 172.21.15.102:0/4148374107 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f59ad9620 con 0x558f59d95720 2024-04-25T01:59:51.704 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.700+0000 7fe6cf400640 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59d9d460 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe6d006d690 tx=0x7fe6b8011d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920f4297877f27a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:51.704 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.701+0000 7fe6bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:59:51.704 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.701+0000 7fe6bcc00640 1 -- 172.21.15.102:0/4148374107 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800f9a0 con 0x558f59d95720 2024-04-25T01:59:51.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.702+0000 7fe6bcc00640 1 -- 172.21.15.102:0/4148374107 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe6b800fb40 con 0x558f59d95720 2024-04-25T01:59:51.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.702+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6c8036030 con 0x558f59d95720 2024-04-25T01:59:51.706 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.702+0000 7fe6bcc00640 1 -- 172.21.15.102:0/4148374107 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe6b800fd40 con 0x558f59d95720 2024-04-25T01:59:51.706 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.703+0000 7fe6d8fd6740 10 client.16062 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:51.706 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.703+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558f59daeb10 con 0x558f59d95720 2024-04-25T01:59:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.703+0000 7fe6bcc00640 1 -- 172.21.15.102:0/4148374107 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6c8036030 con 0x558f59d95720 2024-04-25T01:59:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.704+0000 7fe6bcc00640 1 -- 172.21.15.102:0/4148374107 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fe6b80274b0 con 0x558f59d95720 2024-04-25T01:59:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.704+0000 7fe6d8fd6740 10 client.16062 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.704+0000 7fe6d8fd6740 10 client.16062 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.704+0000 7fe6d8fd6740 20 client.16062 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.704+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558f59db0090 0x558f59db2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.704+0000 7fe6d8fd6740 4 client.16062 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.704+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f59d05ff0 con 0x558f59db0090 2024-04-25T01:59:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.704+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558f59db0090 0x558f59db2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.705+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558f59db0090 0x558f59db2550 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8c3fbd49287433a server_cookie=56f789d44f16f4bb in_seq=0 out_seq=0 2024-04-25T01:59:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.705+0000 7fe6bcc00640 10 client.16062 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.706+0000 7fe6bcc00640 1 -- 172.21.15.102:0/4148374107 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f59d05ff0 con 0x558f59db0090 2024-04-25T01:59:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.706+0000 7fe6bcc00640 10 client.16062 handle_command_reply: tid=0 2024-04-25T01:59:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.706+0000 7fe6d8fd6740 1 client.16062 shutdown 2024-04-25T01:59:51.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.707+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 msgr2=0x558f59d9d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:51.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.707+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59d9d460 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe6d006d690 tx=0x7fe6b8011d40 comp rx=0 tx=0).stop 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.707+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 shutdown_connections 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.707+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558f59db0090 0x558f59db2550 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.707+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59d9d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.707+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59d9d460 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.707+0000 7fe6bcc00640 0 client.16062 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.707+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/4148374107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59d9cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.707+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 >> 172.21.15.102:0/4148374107 conn(0x558f59d91310 msgr2=0x558f59d97090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.708+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 shutdown_connections 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.708+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/4148374107 wait complete. 2024-04-25T01:59:51.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.708+0000 7fe6d8fd6740 20 client.16062 trim_cache size 0 max 16384 2024-04-25T01:59:51.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:51 smithi110 ceph-mon[32848]: pgmap v140: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.7 MiB/s wr, 618 op/s 2024-04-25T01:59:51.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.742+0000 7fe6d8fd6740 1 Processor -- start 2024-04-25T01:59:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.742+0000 7fe6d8fd6740 1 -- start start 2024-04-25T01:59:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.743+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d94e00 0x558f59dabaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.743+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d95720 0x558f59d90300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.743+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59db7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.743+0000 7fe6d8fd6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01bebe0 con 0x558f59d94e00 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.743+0000 7fe6d8fd6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01bc450 con 0x558f59d95720 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.743+0000 7fe6d8fd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01ba2a0 con 0x558f59d9ffe0 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.744+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59db7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.744+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d94e00 0x558f59dabaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.744+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59db7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:45338/0 (socket says 172.21.15.102:45338) 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.744+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3517587616 learned_addr learned my addr 172.21.15.102:0/3517587616 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.744+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3517587616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d95720 0x558f59d90300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.745+0000 7fe6cf400640 1 -- 172.21.15.102:0/3517587616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 msgr2=0x558f59db7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.745+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3517587616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59db7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.745+0000 7fe6cf400640 1 -- 172.21.15.102:0/3517587616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d94e00 msgr2=0x558f59dabaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.745+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3517587616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d94e00 0x558f59dabaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.745+0000 7fe6cf400640 1 -- 172.21.15.102:0/3517587616 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f59d8f890 con 0x558f59d95720 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.745+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/3517587616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d94e00 0x558f59dabaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.745+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3517587616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d95720 0x558f59d90300 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800b010 tx=0x7fe6b800f1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e413003da6bb6e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:51.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.746+0000 7fe6bf400640 1 -- 172.21.15.102:0/3517587616 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800fba0 con 0x558f59d95720 2024-04-25T01:59:51.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.746+0000 7fe6bf400640 1 -- 172.21.15.102:0/3517587616 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b8004520 con 0x558f59d95720 2024-04-25T01:59:51.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.746+0000 7fe6bf400640 1 -- 172.21.15.102:0/3517587616 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8003690 con 0x558f59d95720 2024-04-25T01:59:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.746+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3517587616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d95720 msgr2=0x558f59d90300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.746+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3517587616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d95720 0x558f59d90300 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800b010 tx=0x7fe6b800f1d0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.747+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3517587616 shutdown_connections 2024-04-25T01:59:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.747+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3517587616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59db7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.747+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3517587616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d95720 0x558f59d90300 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.747+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3517587616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d94e00 0x558f59dabaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.747+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3517587616 >> 172.21.15.102:0/3517587616 conn(0x558f59d91310 msgr2=0x558f59d8fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.747+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3517587616 shutdown_connections 2024-04-25T01:59:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.747+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3517587616 wait complete. 2024-04-25T01:59:51.752 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.747+0000 7fe6d8fd6740 1 -- start start 2024-04-25T01:59:51.752 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.748+0000 7fe6d8fd6740 1 Processor -- start 2024-04-25T01:59:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59da8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59da8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6d8fd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6d8fd6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01ba910 con 0x558f59d95720 2024-04-25T01:59:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6d8fd6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01bbd80 con 0x558f59d94e00 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:45346/0 (socket says 172.21.15.102:45346) 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3061727950 learned_addr learned my addr 172.21.15.102:0/3061727950 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3061727950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01bafe0 con 0x558f59d9ffe0 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6d8fd6740 10 client.0 _ll_register_callbacks cb 0x7fe6b80098d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.749+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59da8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.750+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59da8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.750+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3061727950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 msgr2=0x558f59da8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.750+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59da8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.750+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3061727950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 msgr2=0x558f59da8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:51.754 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.750+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59da8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.750+0000 7fe6cfe00640 1 -- 172.21.15.102:0/3061727950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f59da5780 con 0x558f59d9ffe0 2024-04-25T01:59:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.750+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59da8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.751+0000 7fe6cfe00640 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da4fd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8031610 tx=0x7fe6c80317e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dda0381f8d0c0d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.751+0000 7fe6bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.752+0000 7fe6bcc00640 1 -- 172.21.15.102:0/3061727950 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8032eb0 con 0x558f59d9ffe0 2024-04-25T01:59:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.752+0000 7fe6bcc00640 1 -- 172.21.15.102:0/3061727950 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe6c80319f0 con 0x558f59d9ffe0 2024-04-25T01:59:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.752+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3061727950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6b8004520 con 0x558f59d9ffe0 2024-04-25T01:59:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.753+0000 7fe6bcc00640 1 -- 172.21.15.102:0/3061727950 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe6c8031bf0 con 0x558f59d9ffe0 2024-04-25T01:59:51.757 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.753+0000 7fe6d8fd6740 10 client.26434 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:51.757 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.753+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3061727950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558f59e28160 con 0x558f59d9ffe0 2024-04-25T01:59:51.757 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.754+0000 7fe6bcc00640 1 -- 172.21.15.102:0/3061727950 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6b8004520 con 0x558f59d9ffe0 2024-04-25T01:59:51.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:51 smithi193 ceph-mon[32981]: pgmap v140: 97 pgs: 97 active+clean; 3.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 9.7 MiB/s wr, 618 op/s 2024-04-25T01:59:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.755+0000 7fe6bcc00640 1 -- 172.21.15.102:0/3061727950 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fe6c8042070 con 0x558f59d9ffe0 2024-04-25T01:59:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.755+0000 7fe6d8fd6740 10 client.26434 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.755+0000 7fe6d8fd6740 10 client.26434 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.755+0000 7fe6d8fd6740 20 client.26434 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.755+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558f59e28450 0x558f59e2a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.755+0000 7fe6d8fd6740 4 client.26434 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T01:59:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.755+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3061727950 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558f59d00340 con 0x558f59e28450 2024-04-25T01:59:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.756+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558f59e28450 0x558f59e2a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:51.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.757+0000 7fe6cf400640 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558f59e28450 0x558f59e2a840 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4f1343f9b7955d4 server_cookie=827be7869f4d11c4 in_seq=0 out_seq=0 2024-04-25T01:59:51.760 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.757+0000 7fe6bcc00640 10 client.26434 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.789+0000 7fe6bcc00640 1 -- 172.21.15.102:0/3061727950 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558f59d00340 con 0x558f59e28450 2024-04-25T01:59:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.789+0000 7fe6bcc00640 10 client.26434 handle_command_reply: tid=0 2024-04-25T01:59:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.790+0000 7fe6d8fd6740 1 client.26434 shutdown 2024-04-25T01:59:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.790+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3061727950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 msgr2=0x558f59da4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.790+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da4fd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8031610 tx=0x7fe6c80317e0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.790+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3061727950 shutdown_connections 2024-04-25T01:59:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.791+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558f59e28450 0x558f59e2a840 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.791+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558f59d9ffe0 0x558f59da4fd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.791+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558f59d95720 0x558f59da8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.791+0000 7fe6d8fd6740 1 --2- 172.21.15.102:0/3061727950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558f59d94e00 0x558f59da8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.791+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3061727950 >> 172.21.15.102:0/3061727950 conn(0x558f59d91310 msgr2=0x558f59d90480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:51.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.791+0000 7fe6bcc00640 0 client.26434 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.793+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3061727950 shutdown_connections 2024-04-25T01:59:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.793+0000 7fe6d8fd6740 1 -- 172.21.15.102:0/3061727950 wait complete. 2024-04-25T01:59:51.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.793+0000 7fe6d8fd6740 20 client.26434 trim_cache size 0 max 16384 2024-04-25T01:59:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T01:59:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T01:59:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T01:59:51.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.795+0000 7fe6d6000640 1 -- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe6b40a2da0 msgr2=0x7fe6b40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.795+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe6b40a2da0 0x7fe6b40a5260 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x558f59d963a0 tx=0x7fe6c00046c0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.795+0000 7fe6d6000640 1 -- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6d0103920 msgr2=0x7fe6d0072e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.795+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6d0103920 0x7fe6d0072e50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000b3f0 tx=0x7fe6c000b8b0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.796+0000 7fe6d6000640 1 -- 172.21.15.102:0/3816248184 shutdown_connections 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.796+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe6b40a2da0 0x7fe6b40a5260 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x558f59d963a0 tx=0x7fe6c00046c0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.796+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d0109930 0x7fe6d006c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.796+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6d01042e0 0x7fe6d0073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.796+0000 7fe6d6000640 1 --2- 172.21.15.102:0/3816248184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6d0103920 0x7fe6d0072e50 secure :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000b3f0 tx=0x7fe6c000b8b0 comp rx=0 tx=0).stop 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.796+0000 7fe6d6000640 1 -- 172.21.15.102:0/3816248184 >> 172.21.15.102:0/3816248184 conn(0x7fe6d00ff430 msgr2=0x7fe6d00ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.797+0000 7fe6d6000640 1 -- 172.21.15.102:0/3816248184 shutdown_connections 2024-04-25T01:59:51.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T01:59:51.798+0000 7fe6d6000640 1 -- 172.21.15.102:0/3816248184 wait complete. 2024-04-25T01:59:52.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:59:52.968 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:59:53.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 240.0 s, 2625.0 tps, lat 38.187 ms stddev 47.476 2024-04-25T01:59:53.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.238+0000 7f879aa00640 1 -- 172.21.15.102:0/2427581008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794109270 msgr2=0x7f8794114350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.238+0000 7f879aa00640 1 --2- 172.21.15.102:0/2427581008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794109270 0x7f8794114350 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8780009900 tx=0x7f878002a440 comp rx=0 tx=0).stop 2024-04-25T01:59:53.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.238+0000 7f879aa00640 1 -- 172.21.15.102:0/2427581008 shutdown_connections 2024-04-25T01:59:53.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.238+0000 7f879aa00640 1 --2- 172.21.15.102:0/2427581008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8794114890 0x7f8794116c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.238+0000 7f879aa00640 1 --2- 172.21.15.102:0/2427581008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794109270 0x7f8794114350 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.238+0000 7f879aa00640 1 --2- 172.21.15.102:0/2427581008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8794108950 0x7f8794108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.238+0000 7f879aa00640 1 -- 172.21.15.102:0/2427581008 >> 172.21.15.102:0/2427581008 conn(0x7f879406c850 msgr2=0x7f879406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:53.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.238+0000 7f879aa00640 1 -- 172.21.15.102:0/2427581008 shutdown_connections 2024-04-25T01:59:53.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.239+0000 7f879aa00640 1 -- 172.21.15.102:0/2427581008 wait complete. 2024-04-25T01:59:53.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.242+0000 7f879aa00640 1 Processor -- start 2024-04-25T01:59:53.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.242+0000 7f879aa00640 1 -- start start 2024-04-25T01:59:53.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.243+0000 7f879aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794108950 0x7f879419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.243+0000 7f879aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8794109270 0x7f879419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.243+0000 7f879aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8794114890 0x7f87941a3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.243+0000 7f879aa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f87941181c0 con 0x7f8794109270 2024-04-25T01:59:53.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.243+0000 7f879aa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8794118040 con 0x7f8794114890 2024-04-25T01:59:53.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.243+0000 7f879aa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8794118340 con 0x7f8794108950 2024-04-25T01:59:53.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.243+0000 7f8793e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794108950 0x7f879419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.243+0000 7f8793e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794108950 0x7f879419e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:37962/0 (socket says 172.21.15.102:37962) 2024-04-25T01:59:53.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.243+0000 7f8793e00640 1 -- 172.21.15.102:0/891519854 learned_addr learned my addr 172.21.15.102:0/891519854 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:53.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.244+0000 7f8793e00640 1 -- 172.21.15.102:0/891519854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8794114890 msgr2=0x7f87941a3020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T01:59:53.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.244+0000 7f8793e00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8794114890 0x7f87941a3020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.244+0000 7f8793e00640 1 -- 172.21.15.102:0/891519854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8794109270 msgr2=0x7f879419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.244+0000 7f8793e00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8794109270 0x7f879419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.244+0000 7f8793e00640 1 -- 172.21.15.102:0/891519854 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87941a3750 con 0x7f8794108950 2024-04-25T01:59:53.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.244+0000 7f8793e00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794108950 0x7f879419e760 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f878c00ba50 tx=0x7f878c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:53.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.245+0000 7f8790c00640 1 -- 172.21.15.102:0/891519854 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c019070 con 0x7f8794108950 2024-04-25T01:59:53.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.245+0000 7f8790c00640 1 -- 172.21.15.102:0/891519854 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878c00cbe0 con 0x7f8794108950 2024-04-25T01:59:53.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.245+0000 7f8790c00640 1 -- 172.21.15.102:0/891519854 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c0076e0 con 0x7f8794108950 2024-04-25T01:59:53.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.245+0000 7f879aa00640 1 -- 172.21.15.102:0/891519854 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87941a3a40 con 0x7f8794108950 2024-04-25T01:59:53.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.245+0000 7f879aa00640 1 -- 172.21.15.102:0/891519854 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87941ab2d0 con 0x7f8794108950 2024-04-25T01:59:53.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.254+0000 7f879d3fd740 1 Processor -- start 2024-04-25T01:59:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.256+0000 7f879d3fd740 1 -- start start 2024-04-25T01:59:53.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.256+0000 7f8790c00640 1 -- 172.21.15.102:0/891519854 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f878c01c070 con 0x7f8794108950 2024-04-25T01:59:53.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.258+0000 7f8790c00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87780a2cf0 0x7f87780a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.259+0000 7f8790c00640 1 -- 172.21.15.102:0/891519854 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f878c0ce9b0 con 0x7f8794108950 2024-04-25T01:59:53.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.259+0000 7f8798a00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87780a2cf0 0x7f87780a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb544810 0x55a8bb544bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 0x55a8bb54f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb54f800 0x55a8bb551bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f879d3fd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb4a0560 con 0x55a8bb54f800 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f879d3fd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb448930 con 0x55a8bb544810 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f879d3fd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb49c520 con 0x55a8bb545130 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f8793e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb544810 0x55a8bb544bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f8793e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb544810 0x55a8bb544bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:50884/0 (socket says 172.21.15.102:50884) 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f8793e00640 1 -- 172.21.15.102:0/1626314497 learned_addr learned my addr 172.21.15.102:0/1626314497 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f8793e00640 1 -- 172.21.15.102:0/1626314497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 msgr2=0x55a8bb54f2c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f8793e00640 1 --2- 172.21.15.102:0/1626314497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 0x55a8bb54f2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f8793e00640 1 -- 172.21.15.102:0/1626314497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb54f800 msgr2=0x55a8bb551bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f8793e00640 1 --2- 172.21.15.102:0/1626314497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb54f800 0x55a8bb551bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.260+0000 7f8793e00640 1 -- 172.21.15.102:0/1626314497 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8bb27e7d0 con 0x55a8bb544810 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.261+0000 7f8793e00640 1 --2- 172.21.15.102:0/1626314497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb544810 0x55a8bb544bf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f878c0cc2a0 tx=0x7f878c09b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10545bf4e3714639 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.261+0000 7f8789200640 1 -- 172.21.15.102:0/1626314497 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c09ab50 con 0x55a8bb544810 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.261+0000 7f8789200640 1 -- 172.21.15.102:0/1626314497 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f878c09f3c0 con 0x55a8bb544810 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.261+0000 7f8789200640 1 -- 172.21.15.102:0/1626314497 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c0a3a50 con 0x55a8bb544810 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.261+0000 7f8798a00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87780a2cf0 0x7f87780a51b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8784009b80 tx=0x7f87840095e0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.262+0000 7f879d3fd740 1 -- 172.21.15.102:0/1626314497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb544810 msgr2=0x55a8bb544bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.262+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1626314497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb544810 0x55a8bb544bf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f878c0cc2a0 tx=0x7f878c09b7d0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.262+0000 7f879d3fd740 1 -- 172.21.15.102:0/1626314497 shutdown_connections 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.262+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1626314497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb54f800 0x55a8bb551bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.262+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1626314497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 0x55a8bb54f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.262+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1626314497 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb544810 0x55a8bb544bf0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.262+0000 7f879d3fd740 1 -- 172.21.15.102:0/1626314497 >> 172.21.15.102:0/1626314497 conn(0x55a8bb540b30 msgr2=0x55a8bb540f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:53.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.262+0000 7f879d3fd740 1 -- 172.21.15.102:0/1626314497 shutdown_connections 2024-04-25T01:59:53.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.262+0000 7f879d3fd740 1 -- 172.21.15.102:0/1626314497 wait complete. 2024-04-25T01:59:53.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.263+0000 7f879d3fd740 1 -- start start 2024-04-25T01:59:53.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.263+0000 7f879d3fd740 1 Processor -- start 2024-04-25T01:59:53.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 0x55a8bb54c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb545130 0x55a8bb54cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb54d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f879d3fd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb5525f0 con 0x55a8bb545130 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f879d3fd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb45bd90 con 0x55a8bb54f800 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f879d3fd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a8bb552770 con 0x55a8bb544810 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 0x55a8bb54c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 0x55a8bb54c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:37988/0 (socket says 172.21.15.102:37988) 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793e00640 1 -- 172.21.15.102:0/4262599904 learned_addr learned my addr 172.21.15.102:0/4262599904 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8798a00640 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb54d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793400640 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb545130 0x55a8bb54cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f879d3fd740 10 client.0 _ll_register_callbacks cb 0x55a8bb45b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793e00640 1 -- 172.21.15.102:0/4262599904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 msgr2=0x55a8bb54d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793e00640 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb54d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793e00640 1 -- 172.21.15.102:0/4262599904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb545130 msgr2=0x55a8bb54cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793e00640 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb545130 0x55a8bb54cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793e00640 1 -- 172.21.15.102:0/4262599904 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8bb1b43f0 con 0x55a8bb544810 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f8793400640 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb545130 0x55a8bb54cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.264+0000 7f879d3fd740 1 -- 172.21.15.102:0/4262599904 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f878c09f3c0 con 0x55a8bb544810 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.265+0000 7f8793e00640 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 0x55a8bb54c860 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f878c0b4180 tx=0x7f878c09abc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1a0450b5bab9b37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.265+0000 7f877d600640 10 client.26443 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.265+0000 7f877d600640 1 -- 172.21.15.102:0/4262599904 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878c0977f0 con 0x55a8bb544810 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.265+0000 7f877d600640 1 -- 172.21.15.102:0/4262599904 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f878c095b80 con 0x55a8bb544810 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.265+0000 7f877d600640 1 -- 172.21.15.102:0/4262599904 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f878c095d80 con 0x55a8bb544810 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.265+0000 7f879d3fd740 10 client.26443 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.265+0000 7f879d3fd740 1 -- 172.21.15.102:0/4262599904 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55a8bb1740f0 con 0x55a8bb544810 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.265+0000 7f877d600640 1 -- 172.21.15.102:0/4262599904 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f878c09a460 con 0x55a8bb544810 2024-04-25T01:59:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.266+0000 7f879d3fd740 10 client.26443 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:53.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.266+0000 7f879d3fd740 10 client.26443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:53.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.266+0000 7f879d3fd740 20 client.26443 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:53.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.266+0000 7f879d3fd740 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a8bb55fa10 0x55a8bb561e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.266+0000 7f879d3fd740 4 client.26443 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:53.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.266+0000 7f879d3fd740 1 -- 172.21.15.102:0/4262599904 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8bb4b3d90 con 0x55a8bb55fa10 2024-04-25T01:59:53.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.267+0000 7f8793400640 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a8bb55fa10 0x55a8bb561e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.269+0000 7f8793400640 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a8bb55fa10 0x55a8bb561e00 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcdd593c5ca88fe8 server_cookie=c0d626f4d3d9b677 in_seq=0 out_seq=0 2024-04-25T01:59:53.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.269+0000 7f877d600640 10 client.26443 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:53.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f877d600640 1 -- 172.21.15.102:0/4262599904 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8bb4b3d90 con 0x55a8bb55fa10 2024-04-25T01:59:53.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f877d600640 10 client.26443 handle_command_reply: tid=0 2024-04-25T01:59:53.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f879d3fd740 1 client.26443 shutdown 2024-04-25T01:59:53.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f879d3fd740 1 -- 172.21.15.102:0/4262599904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 msgr2=0x55a8bb54c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f879d3fd740 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 0x55a8bb54c860 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f878c0b4180 tx=0x7f878c09abc0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f879d3fd740 1 -- 172.21.15.102:0/4262599904 shutdown_connections 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f879d3fd740 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a8bb55fa10 0x55a8bb561e00 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f879d3fd740 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb54d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f879d3fd740 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb545130 0x55a8bb54cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f879d3fd740 1 --2- 172.21.15.102:0/4262599904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 0x55a8bb54c860 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f879d3fd740 1 -- 172.21.15.102:0/4262599904 >> 172.21.15.102:0/4262599904 conn(0x55a8bb540b30 msgr2=0x55a8bb546930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.270+0000 7f877d600640 0 client.26443 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.271+0000 7f879d3fd740 1 -- 172.21.15.102:0/4262599904 shutdown_connections 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.271+0000 7f879d3fd740 1 -- 172.21.15.102:0/4262599904 wait complete. 2024-04-25T01:59:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.271+0000 7f879d3fd740 20 client.26443 trim_cache size 0 max 16384 2024-04-25T01:59:53.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.303+0000 7f879d3fd740 1 Processor -- start 2024-04-25T01:59:53.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.304+0000 7f879d3fd740 1 -- start start 2024-04-25T01:59:53.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.304+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 0x55a8bb55b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.304+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb545130 0x55a8bb547080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.304+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb54f800 0x55a8bb54b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f879d3fd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f87941b6450 con 0x55a8bb54f800 2024-04-25T01:59:53.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f879d3fd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f87941b3cc0 con 0x55a8bb545130 2024-04-25T01:59:53.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f879d3fd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f87941b1b10 con 0x55a8bb544810 2024-04-25T01:59:53.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f8793400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb545130 0x55a8bb547080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f8793400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb545130 0x55a8bb547080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:50900/0 (socket says 172.21.15.102:50900) 2024-04-25T01:59:53.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f8793400640 1 -- 172.21.15.102:0/401618354 learned_addr learned my addr 172.21.15.102:0/401618354 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:53.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f8798a00640 1 --2- 172.21.15.102:0/401618354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb54f800 0x55a8bb54b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f8793400640 1 -- 172.21.15.102:0/401618354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 msgr2=0x55a8bb55b250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:59:53.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f8793400640 1 --2- 172.21.15.102:0/401618354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 0x55a8bb55b250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f8793400640 1 -- 172.21.15.102:0/401618354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb54f800 msgr2=0x55a8bb54b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:53.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f8793400640 1 --2- 172.21.15.102:0/401618354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb54f800 0x55a8bb54b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.305+0000 7f8793400640 1 -- 172.21.15.102:0/401618354 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8bb551bf0 con 0x55a8bb545130 2024-04-25T01:59:53.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.306+0000 7f8793400640 1 --2- 172.21.15.102:0/401618354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb545130 0x55a8bb547080 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8780009850 tx=0x7f8780000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edaafd30330085cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:53.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.306+0000 7f877fe00640 1 -- 172.21.15.102:0/401618354 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8780044b50 con 0x55a8bb545130 2024-04-25T01:59:53.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.306+0000 7f877fe00640 1 -- 172.21.15.102:0/401618354 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8780044cf0 con 0x55a8bb545130 2024-04-25T01:59:53.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.306+0000 7f877fe00640 1 -- 172.21.15.102:0/401618354 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87800055e0 con 0x55a8bb545130 2024-04-25T01:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.307+0000 7f879d3fd740 1 -- 172.21.15.102:0/401618354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb545130 msgr2=0x55a8bb547080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.307+0000 7f879d3fd740 1 --2- 172.21.15.102:0/401618354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb545130 0x55a8bb547080 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8780009850 tx=0x7f8780000bc0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.307+0000 7f879d3fd740 1 -- 172.21.15.102:0/401618354 shutdown_connections 2024-04-25T01:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.307+0000 7f879d3fd740 1 --2- 172.21.15.102:0/401618354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb54f800 0x55a8bb54b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.307+0000 7f879d3fd740 1 --2- 172.21.15.102:0/401618354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb545130 0x55a8bb547080 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.307+0000 7f879d3fd740 1 --2- 172.21.15.102:0/401618354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb544810 0x55a8bb55b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.307+0000 7f879d3fd740 1 -- 172.21.15.102:0/401618354 >> 172.21.15.102:0/401618354 conn(0x55a8bb540b30 msgr2=0x55a8bb54b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.307+0000 7f879d3fd740 1 -- 172.21.15.102:0/401618354 shutdown_connections 2024-04-25T01:59:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.307+0000 7f879d3fd740 1 -- 172.21.15.102:0/401618354 wait complete. 2024-04-25T01:59:53.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.309+0000 7f879d3fd740 1 -- start start 2024-04-25T01:59:53.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.309+0000 7f879d3fd740 1 Processor -- start 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb544810 0x55a8bb564690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 0x55a8bb564bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f879d3fd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb561b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f879d3fd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f87941b2180 con 0x55a8bb544810 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f879d3fd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f87941b35f0 con 0x55a8bb54f800 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f879d3fd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f87941b2850 con 0x55a8bb545130 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f879d3fd740 10 client.0 _ll_register_callbacks cb 0x7f878c0099a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f8798a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb561b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f8798a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb561b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:50906/0 (socket says 172.21.15.102:50906) 2024-04-25T01:59:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.311+0000 7f8798a00640 1 -- 172.21.15.102:0/1040180217 learned_addr learned my addr 172.21.15.102:0/1040180217 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.312+0000 7f8793400640 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 0x55a8bb564bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.312+0000 7f8793e00640 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb544810 0x55a8bb564690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.312+0000 7f8798a00640 1 -- 172.21.15.102:0/1040180217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 msgr2=0x55a8bb564bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.312+0000 7f8798a00640 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 0x55a8bb564bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.312+0000 7f8798a00640 1 -- 172.21.15.102:0/1040180217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb544810 msgr2=0x55a8bb564690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.312+0000 7f8798a00640 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb544810 0x55a8bb564690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.313+0000 7f8798a00640 1 -- 172.21.15.102:0/1040180217 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8bb562340 con 0x55a8bb54f800 2024-04-25T01:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.313+0000 7f8793400640 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 0x55a8bb564bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.313+0000 7f8798a00640 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb561b40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f87840126a0 tx=0x7f8784004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f06fce1eb6d1065 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.313+0000 7f877d600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.313+0000 7f877d600640 1 -- 172.21.15.102:0/1040180217 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8784020aa0 con 0x55a8bb54f800 2024-04-25T01:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.313+0000 7f877d600640 1 -- 172.21.15.102:0/1040180217 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8784020c40 con 0x55a8bb54f800 2024-04-25T01:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.313+0000 7f879d3fd740 1 -- 172.21.15.102:0/1040180217 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8780044cf0 con 0x55a8bb54f800 2024-04-25T01:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.314+0000 7f877d600640 1 -- 172.21.15.102:0/1040180217 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8784020e40 con 0x55a8bb54f800 2024-04-25T01:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.314+0000 7f879d3fd740 10 client.26489 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.314+0000 7f879d3fd740 1 -- 172.21.15.102:0/1040180217 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a8bb5c5680 con 0x55a8bb54f800 2024-04-25T01:59:53.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.314+0000 7f877d600640 1 -- 172.21.15.102:0/1040180217 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8780044cf0 con 0x55a8bb54f800 2024-04-25T01:59:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.315+0000 7f877d600640 1 -- 172.21.15.102:0/1040180217 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f8784027030 con 0x55a8bb54f800 2024-04-25T01:59:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.315+0000 7f879d3fd740 10 client.26489 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.315+0000 7f879d3fd740 10 client.26489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.315+0000 7f879d3fd740 20 client.26489 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:53.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.315+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a8bb5c5df0 0x55a8bb5c8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.315+0000 7f879d3fd740 4 client.26489 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.315+0000 7f879d3fd740 1 -- 172.21.15.102:0/1040180217 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a8bb4adaa0 con 0x55a8bb5c5df0 2024-04-25T01:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.316+0000 7f8793e00640 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a8bb5c5df0 0x55a8bb5c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.317+0000 7f8793e00640 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a8bb5c5df0 0x55a8bb5c8210 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=738c8aebd1b8aafd server_cookie=55c67f495e84c654 in_seq=0 out_seq=0 2024-04-25T01:59:53.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.317+0000 7f877d600640 10 client.26489 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:53.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.317+0000 7f877d600640 1 -- 172.21.15.102:0/1040180217 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a8bb4adaa0 con 0x55a8bb5c5df0 2024-04-25T01:59:53.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.317+0000 7f877d600640 10 client.26489 handle_command_reply: tid=0 2024-04-25T01:59:53.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.317+0000 7f879d3fd740 1 client.26489 shutdown 2024-04-25T01:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.317+0000 7f879d3fd740 1 -- 172.21.15.102:0/1040180217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 msgr2=0x55a8bb561b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.317+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb561b40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f87840126a0 tx=0x7f8784004730 comp rx=0 tx=0).stop 2024-04-25T01:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f879d3fd740 1 -- 172.21.15.102:0/1040180217 shutdown_connections 2024-04-25T01:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a8bb5c5df0 0x55a8bb5c8210 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8bb54f800 0x55a8bb561b40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8bb545130 0x55a8bb564bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f879d3fd740 1 --2- 172.21.15.102:0/1040180217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a8bb544810 0x55a8bb564690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f879d3fd740 1 -- 172.21.15.102:0/1040180217 >> 172.21.15.102:0/1040180217 conn(0x55a8bb540b30 msgr2=0x55a8bb54d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:53.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f877d600640 0 client.26489 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f879d3fd740 1 -- 172.21.15.102:0/1040180217 shutdown_connections 2024-04-25T01:59:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f879d3fd740 1 -- 172.21.15.102:0/1040180217 wait complete. 2024-04-25T01:59:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.318+0000 7f879d3fd740 20 client.26489 trim_cache size 0 max 16384 2024-04-25T01:59:53.323 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:59:53.323 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T01:59:53.323 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T01:59:53.323 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 -- 172.21.15.102:0/891519854 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87780a2cf0 msgr2=0x7f87780a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87780a2cf0 0x7f87780a51b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8784009b80 tx=0x7f87840095e0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 -- 172.21.15.102:0/891519854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794108950 msgr2=0x7f879419e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794108950 0x7f879419e760 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f878c00ba50 tx=0x7f878c00bf10 comp rx=0 tx=0).stop 2024-04-25T01:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 -- 172.21.15.102:0/891519854 shutdown_connections 2024-04-25T01:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87780a2cf0 0x7f87780a51b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8794114890 0x7f87941a3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8794109270 0x7f879419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 --2- 172.21.15.102:0/891519854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8794108950 0x7f879419e760 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 -- 172.21.15.102:0/891519854 >> 172.21.15.102:0/891519854 conn(0x7f879406c850 msgr2=0x7f87940704b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 -- 172.21.15.102:0/891519854 shutdown_connections 2024-04-25T01:59:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.319+0000 7f879aa00640 1 -- 172.21.15.102:0/891519854 wait complete. 2024-04-25T01:59:53.348 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T01:59:53.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T01:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.599+0000 7f3f7e000640 1 -- 172.21.15.102:0/2495470420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 msgr2=0x7f3f78110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.599+0000 7f3f7e000640 1 --2- 172.21.15.102:0/2495470420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 0x7f3f78110630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3f70009cd0 tx=0x7f3f7002a8f0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.600+0000 7f3f7e000640 1 -- 172.21.15.102:0/2495470420 shutdown_connections 2024-04-25T01:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.600+0000 7f3f7e000640 1 --2- 172.21.15.102:0/2495470420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 0x7f3f78110630 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.600+0000 7f3f7e000640 1 --2- 172.21.15.102:0/2495470420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f78103760 0x7f3f78103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.600+0000 7f3f7e000640 1 --2- 172.21.15.102:0/2495470420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f78109770 0x7f3f78109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.600+0000 7f3f7e000640 1 -- 172.21.15.102:0/2495470420 >> 172.21.15.102:0/2495470420 conn(0x7f3f780ff210 msgr2=0x7f3f78101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.600+0000 7f3f7e000640 1 -- 172.21.15.102:0/2495470420 shutdown_connections 2024-04-25T01:59:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.600+0000 7f3f7e000640 1 -- 172.21.15.102:0/2495470420 wait complete. 2024-04-25T01:59:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.602+0000 7f3f7e000640 1 Processor -- start 2024-04-25T01:59:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.602+0000 7f3f7e000640 1 -- start start 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.602+0000 7f3f7e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f78103760 0x7f3f78199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.602+0000 7f3f7e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 0x7f3f7819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.602+0000 7f3f7e000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f78109770 0x7f3f7819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.602+0000 7f3f7e000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3f78113b90 con 0x7f3f78103760 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.602+0000 7f3f7e000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3f78113a10 con 0x7f3f78109770 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.602+0000 7f3f7e000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3f78113d10 con 0x7f3f78104120 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f77e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 0x7f3f7819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f77e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 0x7f3f7819a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38012/0 (socket says 172.21.15.102:38012) 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f77e00640 1 -- 172.21.15.102:0/4247325326 learned_addr learned my addr 172.21.15.102:0/4247325326 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f7d600640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f78109770 0x7f3f7819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f77e00640 1 -- 172.21.15.102:0/4247325326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f78109770 msgr2=0x7f3f7819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f77e00640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f78109770 0x7f3f7819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f77e00640 1 -- 172.21.15.102:0/4247325326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f78103760 msgr2=0x7f3f78199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f77e00640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f78103760 0x7f3f78199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f77e00640 1 -- 172.21.15.102:0/4247325326 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f7819ef60 con 0x7f3f78104120 2024-04-25T01:59:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.603+0000 7f3f77e00640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 0x7f3f7819a4b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400e9f0 tx=0x7f3f6400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.604+0000 7f3f75600640 1 -- 172.21.15.102:0/4247325326 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400cd80 con 0x7f3f78104120 2024-04-25T01:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.604+0000 7f3f75600640 1 -- 172.21.15.102:0/4247325326 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f64010910 con 0x7f3f78104120 2024-04-25T01:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.604+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f7819f250 con 0x7f3f78104120 2024-04-25T01:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.604+0000 7f3f75600640 1 -- 172.21.15.102:0/4247325326 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f64018990 con 0x7f3f78104120 2024-04-25T01:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.604+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f78075b80 con 0x7f3f78104120 2024-04-25T01:59:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.605+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f78068480 con 0x7f3f78104120 2024-04-25T01:59:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.610+0000 7f3f75600640 1 -- 172.21.15.102:0/4247325326 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3f64010ac0 con 0x7f3f78104120 2024-04-25T01:59:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.613+0000 7f3f75600640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f600a2d80 0x7f3f600a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.614+0000 7f3f75600640 1 -- 172.21.15.102:0/4247325326 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f3f640cf7f0 con 0x7f3f78104120 2024-04-25T01:59:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.614+0000 7f3f7cc00640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f600a2d80 0x7f3f600a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:53.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.615+0000 7f3f7cc00640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f600a2d80 0x7f3f600a5240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00a930 tx=0x7f3f6c005c00 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:53.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:53.616+0000 7f3f75600640 1 -- 172.21.15.102:0/4247325326 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3f6409c440 con 0x7f3f78104120 2024-04-25T01:59:53.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:53 smithi102 ceph-mon[27796]: pgmap v141: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 13 MiB/s wr, 868 op/s 2024-04-25T01:59:53.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:53 smithi110 ceph-mon[32848]: pgmap v141: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 13 MiB/s wr, 868 op/s 2024-04-25T01:59:53.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:53 smithi193 ceph-mon[32981]: pgmap v141: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 13 MiB/s wr, 868 op/s 2024-04-25T01:59:54.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.041+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f78075eb0 con 0x7f3f78104120 2024-04-25T01:59:54.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.042+0000 7f3f75600640 1 -- 172.21.15.102:0/4247325326 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11899 (secure 0 0 0) 0x7f3f6409bd40 con 0x7f3f78104120 2024-04-25T01:59:54.046 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T01:59:54.046 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T01:59:25.931531+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587,24533,24541]},"id":1}]} 2024-04-25T01:59:54.047 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 32 2024-04-25T01:59:54.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f600a2d80 msgr2=0x7f3f600a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f600a2d80 0x7f3f600a5240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c00a930 tx=0x7f3f6c005c00 comp rx=0 tx=0).stop 2024-04-25T01:59:54.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 msgr2=0x7f3f7819a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 0x7f3f7819a4b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400e9f0 tx=0x7f3f6400eeb0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 shutdown_connections 2024-04-25T01:59:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3f600a2d80 0x7f3f600a5240 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f78109770 0x7f3f7819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78104120 0x7f3f7819a4b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 --2- 172.21.15.102:0/4247325326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f78103760 0x7f3f78199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.047+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 >> 172.21.15.102:0/4247325326 conn(0x7f3f780ff210 msgr2=0x7f3f780ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.048+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 shutdown_connections 2024-04-25T01:59:54.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.048+0000 7f3f7e000640 1 -- 172.21.15.102:0/4247325326 wait complete. 2024-04-25T01:59:54.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T01:59:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.357+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2371226101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc103980 msgr2=0x7ff8bc103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.357+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2371226101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc103980 0x7ff8bc103e00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009930 tx=0x7ff8a802a420 comp rx=0 tx=0).stop 2024-04-25T01:59:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.358+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2371226101 shutdown_connections 2024-04-25T01:59:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.358+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2371226101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8bc104340 0x7ff8bc110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.358+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2371226101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc103980 0x7ff8bc103e00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.358+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2371226101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8bc109990 0x7ff8bc109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.358+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2371226101 >> 172.21.15.102:0/2371226101 conn(0x7ff8bc0ff4d0 msgr2=0x7ff8bc1018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.358+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2371226101 shutdown_connections 2024-04-25T01:59:54.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.358+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2371226101 wait complete. 2024-04-25T01:59:54.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.359+0000 7ff8c0e00640 1 Processor -- start 2024-04-25T01:59:54.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.360+0000 7ff8c0e00640 1 -- start start 2024-04-25T01:59:54.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.360+0000 7ff8c0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8bc103980 0x7ff8bc10f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.360+0000 7ff8c0e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc104340 0x7ff8bc10fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.360+0000 7ff8c0e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8bc109990 0x7ff8bc1a30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.360+0000 7ff8c0e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc076e80 con 0x7ff8bc103980 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.360+0000 7ff8c0e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc076d00 con 0x7ff8bc109990 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.360+0000 7ff8c0e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc077000 con 0x7ff8bc104340 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.361+0000 7ff8b9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc104340 0x7ff8bc10fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.361+0000 7ff8b9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc104340 0x7ff8bc10fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38022/0 (socket says 172.21.15.102:38022) 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.361+0000 7ff8b9400640 1 -- 172.21.15.102:0/2096187855 learned_addr learned my addr 172.21.15.102:0/2096187855 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.361+0000 7ff8ba800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8bc109990 0x7ff8bc1a30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.361+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8bc103980 0x7ff8bc10f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.361+0000 7ff8ba800640 1 -- 172.21.15.102:0/2096187855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc104340 msgr2=0x7ff8bc10fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.361+0000 7ff8ba800640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc104340 0x7ff8bc10fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.361+0000 7ff8ba800640 1 -- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8bc103980 msgr2=0x7ff8bc10f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.361+0000 7ff8ba800640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8bc103980 0x7ff8bc10f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.362+0000 7ff8ba800640 1 -- 172.21.15.102:0/2096187855 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc1a3810 con 0x7ff8bc109990 2024-04-25T01:59:54.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.362+0000 7ff8b9400640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc104340 0x7ff8bc10fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:54.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.362+0000 7ff8ba800640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8bc109990 0x7ff8bc1a30e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400e940 tx=0x7ff8b400ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.362+0000 7ff8b2a00640 1 -- 172.21.15.102:0/2096187855 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4004d20 con 0x7ff8bc109990 2024-04-25T01:59:54.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.362+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2096187855 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8bc1a3b00 con 0x7ff8bc109990 2024-04-25T01:59:54.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.363+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2096187855 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8bc1ab3d0 con 0x7ff8bc109990 2024-04-25T01:59:54.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.363+0000 7ff8b2a00640 1 -- 172.21.15.102:0/2096187855 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8b4005020 con 0x7ff8bc109990 2024-04-25T01:59:54.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.363+0000 7ff8b2a00640 1 -- 172.21.15.102:0/2096187855 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b40116b0 con 0x7ff8bc109990 2024-04-25T01:59:54.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.368+0000 7ff8b2a00640 1 -- 172.21.15.102:0/2096187855 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff8b400cb20 con 0x7ff8bc109990 2024-04-25T01:59:54.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.368+0000 7ff8c3c22740 1 Processor -- start 2024-04-25T01:59:54.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.368+0000 7ff8c3c22740 1 -- start start 2024-04-25T01:59:54.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.368+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a86d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.368+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a878070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.369+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a87a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.369+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f29a7d24e0 con 0x55f29a8785b0 2024-04-25T01:59:54.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.369+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f29a77aa70 con 0x55f29a86dee0 2024-04-25T01:59:54.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.369+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f29a7ce4a0 con 0x55f29a86d5c0 2024-04-25T01:59:54.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.369+0000 7ff8b9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a86d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.369+0000 7ff8b9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a86d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38024/0 (socket says 172.21.15.102:38024) 2024-04-25T01:59:54.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.369+0000 7ff8b9400640 1 -- 172.21.15.102:0/571391388 learned_addr learned my addr 172.21.15.102:0/571391388 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.369+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/571391388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a878070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.370+0000 7ff8b9400640 1 -- 172.21.15.102:0/571391388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 msgr2=0x55f29a878070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.370+0000 7ff8b2a00640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff8a00a2d80 0x7ff8a00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.370+0000 7ff8b9400640 1 --2- 172.21.15.102:0/571391388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a878070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.370+0000 7ff8b9400640 1 -- 172.21.15.102:0/571391388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 msgr2=0x55f29a87a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.370+0000 7ff8b9400640 1 --2- 172.21.15.102:0/571391388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a87a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.370+0000 7ff8b9400640 1 -- 172.21.15.102:0/571391388 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29a4e63f0 con 0x55f29a86d5c0 2024-04-25T01:59:54.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.370+0000 7ff8b2a00640 1 -- 172.21.15.102:0/2096187855 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff8b40cf3a0 con 0x7ff8bc109990 2024-04-25T01:59:54.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.370+0000 7ff8b9400640 1 --2- 172.21.15.102:0/571391388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a86d9a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005e20 tx=0x7ff8a8005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0d03a21a9f3bb55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.370+0000 7ff8ba800640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff8a00a2d80 0x7ff8a00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.371+0000 7ff8a7400640 1 -- 172.21.15.102:0/571391388 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8035070 con 0x55f29a86d5c0 2024-04-25T01:59:54.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.371+0000 7ff8a7400640 1 -- 172.21.15.102:0/571391388 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a8041030 con 0x55f29a86d5c0 2024-04-25T01:59:54.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.371+0000 7ff8a7400640 1 -- 172.21.15.102:0/571391388 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a803c470 con 0x55f29a86d5c0 2024-04-25T01:59:54.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.372+0000 7ff8c3c22740 1 -- 172.21.15.102:0/571391388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 msgr2=0x55f29a86d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.372+0000 7ff8ba800640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff8a00a2d80 0x7ff8a00a5240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4005a80 tx=0x7ff8b4004a40 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.372+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/571391388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a86d9a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005e20 tx=0x7ff8a8005e50 comp rx=0 tx=0).stop 2024-04-25T01:59:54.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.372+0000 7ff8c3c22740 1 -- 172.21.15.102:0/571391388 shutdown_connections 2024-04-25T01:59:54.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.372+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/571391388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a87a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.372+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/571391388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a878070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.372+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/571391388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a86d9a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.373+0000 7ff8c3c22740 1 -- 172.21.15.102:0/571391388 >> 172.21.15.102:0/571391388 conn(0x55f29a8698e0 msgr2=0x55f29a869cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.373+0000 7ff8c3c22740 1 -- 172.21.15.102:0/571391388 shutdown_connections 2024-04-25T01:59:54.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.373+0000 7ff8c3c22740 1 -- 172.21.15.102:0/571391388 wait complete. 2024-04-25T01:59:54.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.373+0000 7ff8c3c22740 1 -- start start 2024-04-25T01:59:54.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.374+0000 7ff8c3c22740 1 Processor -- start 2024-04-25T01:59:54.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a8755d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a875b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a876050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f29a87b340 con 0x55f29a8785b0 2024-04-25T01:59:54.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f29a78dec0 con 0x55f29a86dee0 2024-04-25T01:59:54.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f29a87b4c0 con 0x55f29a86d5c0 2024-04-25T01:59:54.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8c3c22740 10 client.0 _ll_register_callbacks cb 0x55f29a7ca3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:54.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8b9e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a875b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8b9e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a875b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:50960/0 (socket says 172.21.15.102:50960) 2024-04-25T01:59:54.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.375+0000 7ff8b9e00640 1 -- 172.21.15.102:0/1574315254 learned_addr learned my addr 172.21.15.102:0/1574315254 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.376+0000 7ff8b9400640 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a8755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.376+0000 7ff8b9e00640 1 -- 172.21.15.102:0/1574315254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 msgr2=0x55f29a8755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.376+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a8755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.376+0000 7ff8b9e00640 1 -- 172.21.15.102:0/1574315254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 msgr2=0x55f29a876050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.376+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a876050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.376+0000 7ff8b9e00640 1 -- 172.21.15.102:0/1574315254 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f29a5b07d0 con 0x55f29a86dee0 2024-04-25T01:59:54.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.376+0000 7ff8b9400640 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a8755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:54.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.377+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a875b10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0104d0 tx=0x7ff8ac010990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a45351e2a2ff48b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.377+0000 7ff89be00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:54.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.377+0000 7ff89be00640 1 -- 172.21.15.102:0/1574315254 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac01a020 con 0x55f29a86dee0 2024-04-25T01:59:54.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.378+0000 7ff89be00640 1 -- 172.21.15.102:0/1574315254 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff8ac016690 con 0x55f29a86dee0 2024-04-25T01:59:54.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.378+0000 7ff8c3c22740 1 -- 172.21.15.102:0/1574315254 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8a8041030 con 0x55f29a86dee0 2024-04-25T01:59:54.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.378+0000 7ff89be00640 1 -- 172.21.15.102:0/1574315254 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff8ac016890 con 0x55f29a86dee0 2024-04-25T01:59:54.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.378+0000 7ff8c3c22740 10 client.26507 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:54.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.378+0000 7ff8c3c22740 1 -- 172.21.15.102:0/1574315254 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f29a887760 con 0x55f29a86dee0 2024-04-25T01:59:54.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.379+0000 7ff89be00640 1 -- 172.21.15.102:0/1574315254 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a8041030 con 0x55f29a86dee0 2024-04-25T01:59:54.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.379+0000 7ff89be00640 1 -- 172.21.15.102:0/1574315254 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7ff8ac027020 con 0x55f29a86dee0 2024-04-25T01:59:54.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.379+0000 7ff8c3c22740 10 client.26507 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:54.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.379+0000 7ff8c3c22740 10 client.26507 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:59:54.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.379+0000 7ff8c3c22740 20 client.26507 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:54.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.379+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55f29a888d40 0x55f29a88b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.379+0000 7ff8c3c22740 4 client.26507 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:54.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.379+0000 7ff8c3c22740 1 -- 172.21.15.102:0/1574315254 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f29a7dfa20 con 0x55f29a888d40 2024-04-25T01:59:54.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.380+0000 7ff8b9400640 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55f29a888d40 0x55f29a88b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.380+0000 7ff8b9400640 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55f29a888d40 0x55f29a88b160 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=89d71689e8b77ad3 server_cookie=7a00840887b84958 in_seq=0 out_seq=0 2024-04-25T01:59:54.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.380+0000 7ff89be00640 10 client.26507 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:59:54.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.381+0000 7ff89be00640 1 -- 172.21.15.102:0/1574315254 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f29a7dfa20 con 0x55f29a888d40 2024-04-25T01:59:54.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.381+0000 7ff89be00640 10 client.26507 handle_command_reply: tid=0 2024-04-25T01:59:54.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.381+0000 7ff8c3c22740 1 client.26507 shutdown 2024-04-25T01:59:54.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.381+0000 7ff8c3c22740 1 -- 172.21.15.102:0/1574315254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 msgr2=0x55f29a875b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.381+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a875b10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0104d0 tx=0x7ff8ac010990 comp rx=0 tx=0).stop 2024-04-25T01:59:54.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.382+0000 7ff8c3c22740 1 -- 172.21.15.102:0/1574315254 shutdown_connections 2024-04-25T01:59:54.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.382+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a876050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.382+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55f29a888d40 0x55f29a88b160 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.382+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86dee0 0x55f29a875b10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.382+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/1574315254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86d5c0 0x55f29a8755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.382+0000 7ff8c3c22740 1 -- 172.21.15.102:0/1574315254 >> 172.21.15.102:0/1574315254 conn(0x55f29a8698e0 msgr2=0x55f29a868b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.382+0000 7ff89be00640 0 client.26507 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:59:54.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.383+0000 7ff8c3c22740 1 -- 172.21.15.102:0/1574315254 shutdown_connections 2024-04-25T01:59:54.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.383+0000 7ff8c3c22740 1 -- 172.21.15.102:0/1574315254 wait complete. 2024-04-25T01:59:54.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.383+0000 7ff8c3c22740 20 client.26507 trim_cache size 0 max 16384 2024-04-25T01:59:54.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.434+0000 7ff8c3c22740 1 Processor -- start 2024-04-25T01:59:54.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.434+0000 7ff8c3c22740 1 -- start start 2024-04-25T01:59:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.436+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86d5c0 0x55f29a8692c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.436+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a875e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.436+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a890a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.436+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c010ff0 con 0x55f29a8785b0 2024-04-25T01:59:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.436+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c00e860 con 0x55f29a86d5c0 2024-04-25T01:59:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.436+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c00c6b0 con 0x55f29a86dee0 2024-04-25T01:59:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.436+0000 7ff8b9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a875e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.436+0000 7ff8b9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a875e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38038/0 (socket says 172.21.15.102:38038) 2024-04-25T01:59:54.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.436+0000 7ff8b9e00640 1 -- 172.21.15.102:0/336620347 learned_addr learned my addr 172.21.15.102:0/336620347 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.437+0000 7ff8b9400640 1 --2- 172.21.15.102:0/336620347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a890a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.437+0000 7ff8b9400640 1 --2- 172.21.15.102:0/336620347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86d5c0 0x55f29a8692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.437+0000 7ff8b9e00640 1 -- 172.21.15.102:0/336620347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86d5c0 msgr2=0x55f29a8692c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.437+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/336620347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86d5c0 0x55f29a8692c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.437+0000 7ff8b9e00640 1 -- 172.21.15.102:0/336620347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 msgr2=0x55f29a890a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.437+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/336620347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a890a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.437+0000 7ff8b9e00640 1 -- 172.21.15.102:0/336620347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29a86cbd0 con 0x55f29a86dee0 2024-04-25T01:59:54.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.438+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/336620347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a875e70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac014db0 tx=0x7ff8ac006a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a384bc4630bd0d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.438+0000 7ff8a6a00640 1 -- 172.21.15.102:0/336620347 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac02c070 con 0x55f29a86dee0 2024-04-25T01:59:54.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.438+0000 7ff8a6a00640 1 -- 172.21.15.102:0/336620347 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ac00be00 con 0x55f29a86dee0 2024-04-25T01:59:54.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.438+0000 7ff8a6a00640 1 -- 172.21.15.102:0/336620347 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac01a040 con 0x55f29a86dee0 2024-04-25T01:59:54.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.440+0000 7ff8c3c22740 1 -- 172.21.15.102:0/336620347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 msgr2=0x55f29a875e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.440+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/336620347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a875e70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac014db0 tx=0x7ff8ac006a30 comp rx=0 tx=0).stop 2024-04-25T01:59:54.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.441+0000 7ff8c3c22740 1 -- 172.21.15.102:0/336620347 shutdown_connections 2024-04-25T01:59:54.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.441+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/336620347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a8785b0 0x55f29a890a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.441+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/336620347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a875e70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.441+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/336620347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a86d5c0 0x55f29a8692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.441+0000 7ff8c3c22740 1 -- 172.21.15.102:0/336620347 >> 172.21.15.102:0/336620347 conn(0x55f29a8698e0 msgr2=0x55f29a86f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.441+0000 7ff8c3c22740 1 -- 172.21.15.102:0/336620347 shutdown_connections 2024-04-25T01:59:54.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.441+0000 7ff8c3c22740 1 -- 172.21.15.102:0/336620347 wait complete. 2024-04-25T01:59:54.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.442+0000 7ff8c3c22740 1 -- start start 2024-04-25T01:59:54.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.443+0000 7ff8c3c22740 1 Processor -- start 2024-04-25T01:59:54.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.444+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a86d5c0 0x55f29a8807a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a880ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8c3c22740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a8785b0 0x55f29a871920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c00cd20 con 0x55f29a86d5c0 2024-04-25T01:59:54.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c00e190 con 0x55f29a8785b0 2024-04-25T01:59:54.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8c3c22740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c00d3f0 con 0x55f29a86dee0 2024-04-25T01:59:54.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8c3c22740 10 client.0 _ll_register_callbacks cb 0x7ff8ac00b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:54.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8b9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a880ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8b9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a880ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38042/0 (socket says 172.21.15.102:38042) 2024-04-25T01:59:54.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8b9e00640 1 -- 172.21.15.102:0/2092319686 learned_addr learned my addr 172.21.15.102:0/2092319686 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8b9400640 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a86d5c0 0x55f29a8807a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8b9e00640 1 -- 172.21.15.102:0/2092319686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a8785b0 msgr2=0x55f29a871920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:59:54.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a8785b0 0x55f29a871920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.445+0000 7ff8b9e00640 1 -- 172.21.15.102:0/2092319686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a86d5c0 msgr2=0x55f29a8807a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.446+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a86d5c0 0x55f29a8807a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.446+0000 7ff8b9e00640 1 -- 172.21.15.102:0/2092319686 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f29a871f90 con 0x55f29a86dee0 2024-04-25T01:59:54.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.447+0000 7ff8b9400640 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a86d5c0 0x55f29a8807a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:54.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.447+0000 7ff8b9e00640 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a880ce0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac016f50 tx=0x7ff8ac005bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4dcabe953c546c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.448+0000 7ff89be00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:54.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.448+0000 7ff89be00640 1 -- 172.21.15.102:0/2092319686 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac01a040 con 0x55f29a86dee0 2024-04-25T01:59:54.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.448+0000 7ff89be00640 1 -- 172.21.15.102:0/2092319686 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff8ac01fb50 con 0x55f29a86dee0 2024-04-25T01:59:54.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.449+0000 7ff8c3c22740 1 -- 172.21.15.102:0/2092319686 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8ac00be00 con 0x55f29a86dee0 2024-04-25T01:59:54.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.449+0000 7ff89be00640 1 -- 172.21.15.102:0/2092319686 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff8ac02c040 con 0x55f29a86dee0 2024-04-25T01:59:54.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.449+0000 7ff8c3c22740 10 client.26470 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:54.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.450+0000 7ff8c3c22740 1 -- 172.21.15.102:0/2092319686 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f29a8eec00 con 0x55f29a86dee0 2024-04-25T01:59:54.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.450+0000 7ff89be00640 1 -- 172.21.15.102:0/2092319686 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ac00be00 con 0x55f29a86dee0 2024-04-25T01:59:54.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.450+0000 7ff89be00640 1 -- 172.21.15.102:0/2092319686 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7ff8ac039030 con 0x55f29a86dee0 2024-04-25T01:59:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.451+0000 7ff8c3c22740 10 client.26470 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.451+0000 7ff8c3c22740 10 client.26470 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T01:59:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.451+0000 7ff8c3c22740 20 client.26470 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.451+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55f29a8ef300 0x55f29a8f1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.451+0000 7ff8c3c22740 4 client.26470 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:59:54.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.451+0000 7ff8c3c22740 1 -- 172.21.15.102:0/2092319686 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f29a7d9d70 con 0x55f29a8ef300 2024-04-25T01:59:54.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.451+0000 7ff8b9400640 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55f29a8ef300 0x55f29a8f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.454+0000 7ff8b9400640 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55f29a8ef300 0x55f29a8f1740 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=31c6cc801e1d535b server_cookie=a333c5d32b34dc96 in_seq=0 out_seq=0 2024-04-25T01:59:54.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.454+0000 7ff89be00640 10 client.26470 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T01:59:54.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.454+0000 7ff89be00640 1 -- 172.21.15.102:0/2092319686 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f29a7d9d70 con 0x55f29a8ef300 2024-04-25T01:59:54.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.454+0000 7ff89be00640 10 client.26470 handle_command_reply: tid=0 2024-04-25T01:59:54.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.454+0000 7ff8c3c22740 1 client.26470 shutdown 2024-04-25T01:59:54.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.455+0000 7ff8c3c22740 1 -- 172.21.15.102:0/2092319686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 msgr2=0x55f29a880ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.455+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a880ce0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac016f50 tx=0x7ff8ac005bf0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.456+0000 7ff8c3c22740 1 -- 172.21.15.102:0/2092319686 shutdown_connections 2024-04-25T01:59:54.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.456+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55f29a8ef300 0x55f29a8f1740 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.456+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f29a8785b0 0x55f29a871920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.456+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f29a86dee0 0x55f29a880ce0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.456+0000 7ff8c3c22740 1 --2- 172.21.15.102:0/2092319686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f29a86d5c0 0x55f29a8807a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.456+0000 7ff8c3c22740 1 -- 172.21.15.102:0/2092319686 >> 172.21.15.102:0/2092319686 conn(0x55f29a8698e0 msgr2=0x55f29a876e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.456+0000 7ff89be00640 0 client.26470 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T01:59:54.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.457+0000 7ff8c3c22740 1 -- 172.21.15.102:0/2092319686 shutdown_connections 2024-04-25T01:59:54.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.457+0000 7ff8c3c22740 1 -- 172.21.15.102:0/2092319686 wait complete. 2024-04-25T01:59:54.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.458+0000 7ff8c3c22740 20 client.26470 trim_cache size 0 max 16384 2024-04-25T01:59:54.462 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff8a00a2d80 msgr2=0x7ff8a00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff8a00a2d80 0x7ff8a00a5240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4005a80 tx=0x7ff8b4004a40 comp rx=0 tx=0).stop 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2096187855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8bc109990 msgr2=0x7ff8bc1a30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8bc109990 0x7ff8bc1a30e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400e940 tx=0x7ff8b400ee00 comp rx=0 tx=0).stop 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2096187855 shutdown_connections 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff8a00a2d80 0x7ff8a00a5240 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8bc109990 0x7ff8bc1a30e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc104340 0x7ff8bc10fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 --2- 172.21.15.102:0/2096187855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8bc103980 0x7ff8bc10f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2096187855 >> 172.21.15.102:0/2096187855 conn(0x7ff8bc0ff4d0 msgr2=0x7ff8bc0ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2096187855 shutdown_connections 2024-04-25T01:59:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.459+0000 7ff8c0e00640 1 -- 172.21.15.102:0/2096187855 wait complete. 2024-04-25T01:59:54.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T01:59:54.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:54 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/4247325326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:59:54.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:54 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/4247325326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:59:54.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.754+0000 7f25daa00640 1 -- 172.21.15.102:0/172223851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25d410d110 msgr2=0x7f25d4114ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.754+0000 7f25daa00640 1 --2- 172.21.15.102:0/172223851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25d410d110 0x7f25d4114ca0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009ca0 tx=0x7f25cc02a8c0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.755+0000 7f25daa00640 1 -- 172.21.15.102:0/172223851 shutdown_connections 2024-04-25T01:59:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.755+0000 7f25daa00640 1 --2- 172.21.15.102:0/172223851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25d410d110 0x7f25d4114ca0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.755+0000 7f25daa00640 1 --2- 172.21.15.102:0/172223851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25d41091a0 0x7f25d410ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.755+0000 7f25daa00640 1 --2- 172.21.15.102:0/172223851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d41087f0 0x7f25d4108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.755+0000 7f25daa00640 1 -- 172.21.15.102:0/172223851 >> 172.21.15.102:0/172223851 conn(0x7f25d40775b0 msgr2=0x7f25d40799d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.755+0000 7f25daa00640 1 -- 172.21.15.102:0/172223851 shutdown_connections 2024-04-25T01:59:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.755+0000 7f25daa00640 1 -- 172.21.15.102:0/172223851 wait complete. 2024-04-25T01:59:54.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:54 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/4247325326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T01:59:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.756+0000 7f25daa00640 1 Processor -- start 2024-04-25T01:59:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.756+0000 7f25daa00640 1 -- start start 2024-04-25T01:59:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.756+0000 7f25daa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25d41087f0 0x7f25d419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.756+0000 7f25daa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d410d110 0x7f25d419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.756+0000 7f25daa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25d41a2e60 0x7f25d41a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.756+0000 7f25daa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25d4118190 con 0x7f25d41087f0 2024-04-25T01:59:54.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.756+0000 7f25daa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25d4118010 con 0x7f25d410d110 2024-04-25T01:59:54.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.756+0000 7f25daa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25d4118310 con 0x7f25d41a2e60 2024-04-25T01:59:54.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.757+0000 7f25d8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25d41a2e60 0x7f25d41a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.757+0000 7f25d8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25d41a2e60 0x7f25d41a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38064/0 (socket says 172.21.15.102:38064) 2024-04-25T01:59:54.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.757+0000 7f25d8a00640 1 -- 172.21.15.102:0/1098179690 learned_addr learned my addr 172.21.15.102:0/1098179690 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.757+0000 7f25d3400640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d410d110 0x7f25d419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.758+0000 7f25d3e00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25d41087f0 0x7f25d419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.758+0000 7f25d3400640 1 -- 172.21.15.102:0/1098179690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25d41a2e60 msgr2=0x7f25d41a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.758+0000 7f25d3400640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25d41a2e60 0x7f25d41a3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.758+0000 7f25d3400640 1 -- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25d41087f0 msgr2=0x7f25d419e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.758+0000 7f25d3400640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25d41087f0 0x7f25d419e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.758+0000 7f25d3400640 1 -- 172.21.15.102:0/1098179690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d41a3950 con 0x7f25d410d110 2024-04-25T01:59:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.759+0000 7f25d3400640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d410d110 0x7f25d419eb30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f25d407bc50 tx=0x7f25c000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.759+0000 7f25d0c00640 1 -- 172.21.15.102:0/1098179690 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c000cc40 con 0x7f25d410d110 2024-04-25T01:59:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.759+0000 7f25d0c00640 1 -- 172.21.15.102:0/1098179690 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c000cde0 con 0x7f25d410d110 2024-04-25T01:59:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.759+0000 7f25d0c00640 1 -- 172.21.15.102:0/1098179690 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c00188e0 con 0x7f25d410d110 2024-04-25T01:59:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.759+0000 7f25daa00640 1 -- 172.21.15.102:0/1098179690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d41a3c40 con 0x7f25d410d110 2024-04-25T01:59:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.759+0000 7f25daa00640 1 -- 172.21.15.102:0/1098179690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d41ab3e0 con 0x7f25d410d110 2024-04-25T01:59:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.766+0000 7f25dd9ad740 1 Processor -- start 2024-04-25T01:59:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.766+0000 7f25dd9ad740 1 -- start start 2024-04-25T01:59:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.767+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.767+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.767+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd55b0 0x559ffddd79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.767+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559ffdd2f4e0 con 0x559ffddd55b0 2024-04-25T01:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.767+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559ffdcd7a70 con 0x559ffddcaee0 2024-04-25T01:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.767+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559ffdd2b4a0 con 0x559ffddca5c0 2024-04-25T01:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.767+0000 7f25d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.767+0000 7f25d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51006/0 (socket says 172.21.15.102:51006) 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.767+0000 7f25d3e00640 1 -- 172.21.15.102:0/2995920121 learned_addr learned my addr 172.21.15.102:0/2995920121 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.768+0000 7f25d8a00640 1 --2- 172.21.15.102:0/2995920121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.768+0000 7f25d3e00640 1 -- 172.21.15.102:0/2995920121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 msgr2=0x559ffddca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.768+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2995920121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddca9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.768+0000 7f25d3e00640 1 -- 172.21.15.102:0/2995920121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd55b0 msgr2=0x559ffddd79a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.768+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2995920121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd55b0 0x559ffddd79a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.768+0000 7f25d3e00640 1 -- 172.21.15.102:0/2995920121 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ffda433f0 con 0x559ffddcaee0 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.768+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2995920121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd5070 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f25c400cc60 tx=0x7f25c400e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcc776c36371d1f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.768+0000 7f25c9200640 1 -- 172.21.15.102:0/2995920121 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4014070 con 0x559ffddcaee0 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.769+0000 7f25c9200640 1 -- 172.21.15.102:0/2995920121 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c40047a0 con 0x559ffddcaee0 2024-04-25T01:59:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.769+0000 7f25c9200640 1 -- 172.21.15.102:0/2995920121 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4010790 con 0x559ffddcaee0 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.769+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2995920121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 msgr2=0x559ffddd5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.769+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2995920121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd5070 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f25c400cc60 tx=0x7f25c400e720 comp rx=0 tx=0).stop 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.770+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2995920121 shutdown_connections 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.770+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2995920121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd55b0 0x559ffddd79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.770+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2995920121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd5070 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.770+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2995920121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.770+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2995920121 >> 172.21.15.102:0/2995920121 conn(0x559ffddc68e0 msgr2=0x559ffddc6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.771+0000 7f25d0c00640 1 -- 172.21.15.102:0/1098179690 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f25c0018a80 con 0x7f25d410d110 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.771+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2995920121 shutdown_connections 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.771+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2995920121 wait complete. 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.771+0000 7f25dd9ad740 1 -- start start 2024-04-25T01:59:54.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.772+0000 7f25dd9ad740 1 Processor -- start 2024-04-25T01:59:54.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.773+0000 7f25d0c00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25b80a2c90 0x7f25b80a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.774+0000 7f25d0c00640 1 -- 172.21.15.102:0/1098179690 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f25c00cf7b0 con 0x7f25d410d110 2024-04-25T01:59:54.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.774+0000 7f25d8a00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25b80a2c90 0x7f25b80a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddd26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 0x559ffdde1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559ffddd83a0 con 0x559ffddd3130 2024-04-25T01:59:54.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559ffdceaec0 con 0x559ffddcaee0 2024-04-25T01:59:54.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559ffddd8520 con 0x559ffddca5c0 2024-04-25T01:59:54.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25dd9ad740 10 client.0 _ll_register_callbacks cb 0x559ffdd273f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:54.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddd26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddd26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38082/0 (socket says 172.21.15.102:38082) 2024-04-25T01:59:54.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.775+0000 7f25d3e00640 1 -- 172.21.15.102:0/2216142922 learned_addr learned my addr 172.21.15.102:0/2216142922 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.776+0000 7f25d8a00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25b80a2c90 0x7f25b80a5150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009430 tx=0x7f25cc005c30 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.776+0000 7f25d8a00640 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 0x559ffdde1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.776+0000 7f25d3e00640 1 -- 172.21.15.102:0/2216142922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 msgr2=0x559ffddd2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.777+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.777+0000 7f25d3e00640 1 -- 172.21.15.102:0/2216142922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 msgr2=0x559ffdde1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.777+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 0x559ffdde1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.777+0000 7f25d3e00640 1 -- 172.21.15.102:0/2216142922 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ffdb0d7d0 con 0x559ffddca5c0 2024-04-25T01:59:54.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.777+0000 7f25d3400640 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:54.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.777+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddd26b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f25c400c7b0 tx=0x7f25c4005210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=602833b6de170121 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.777+0000 7f25bd600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:54.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.778+0000 7f25bd600640 1 -- 172.21.15.102:0/2216142922 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4014070 con 0x559ffddca5c0 2024-04-25T01:59:54.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.778+0000 7f25bd600640 1 -- 172.21.15.102:0/2216142922 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f25c4004230 con 0x559ffddca5c0 2024-04-25T01:59:54.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.778+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2216142922 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25c40047a0 con 0x559ffddca5c0 2024-04-25T01:59:54.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.779+0000 7f25bd600640 1 -- 172.21.15.102:0/2216142922 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f25c4004430 con 0x559ffddca5c0 2024-04-25T01:59:54.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.779+0000 7f25dd9ad740 10 client.26482 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:54.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.779+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2216142922 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559ffdde4bb0 con 0x559ffddca5c0 2024-04-25T01:59:54.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.780+0000 7f25bd600640 1 -- 172.21.15.102:0/2216142922 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c40047a0 con 0x559ffddca5c0 2024-04-25T01:59:54.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.780+0000 7f25bd600640 1 -- 172.21.15.102:0/2216142922 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f25c4010510 con 0x559ffddca5c0 2024-04-25T01:59:54.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.780+0000 7f25dd9ad740 10 client.26482 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:54.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.780+0000 7f25dd9ad740 10 client.26482 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:59:54.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.780+0000 7f25dd9ad740 20 client.26482 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:54.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.780+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x559ffdde6010 0x559ffdde8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.781+0000 7f25dd9ad740 4 client.26482 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:54.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.781+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2216142922 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ffdd3ca20 con 0x559ffdde6010 2024-04-25T01:59:54.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.782+0000 7f25d3400640 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x559ffdde6010 0x559ffdde8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.783+0000 7f25d3400640 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x559ffdde6010 0x559ffdde8400 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=71fa556b8c1470bb server_cookie=aea1cd7146e937a1 in_seq=0 out_seq=0 2024-04-25T01:59:54.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.783+0000 7f25bd600640 10 client.26482 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:59:54.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.783+0000 7f25bd600640 1 -- 172.21.15.102:0/2216142922 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ffdd3ca20 con 0x559ffdde6010 2024-04-25T01:59:54.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.783+0000 7f25bd600640 10 client.26482 handle_command_reply: tid=0 2024-04-25T01:59:54.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 client.26482 shutdown 2024-04-25T01:59:54.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2216142922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 msgr2=0x559ffddd26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddd26b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f25c400c7b0 tx=0x7f25c4005210 comp rx=0 tx=0).stop 2024-04-25T01:59:54.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2216142922 shutdown_connections 2024-04-25T01:59:54.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x559ffdde6010 0x559ffdde8400 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 0x559ffdde1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffddd2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2216142922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddd26b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2216142922 >> 172.21.15.102:0/2216142922 conn(0x559ffddc68e0 msgr2=0x559ffddcc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25bd600640 0 client.26482 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:59:54.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2216142922 shutdown_connections 2024-04-25T01:59:54.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2216142922 wait complete. 2024-04-25T01:59:54.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.784+0000 7f25dd9ad740 20 client.26482 trim_cache size 0 max 16384 2024-04-25T01:59:54.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.808+0000 7f25dd9ad740 1 Processor -- start 2024-04-25T01:59:54.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25dd9ad740 1 -- start start 2024-04-25T01:59:54.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddcc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffdded9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 0x559ffddedf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41b63e0 con 0x559ffddd3130 2024-04-25T01:59:54.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41b3c50 con 0x559ffddcaee0 2024-04-25T01:59:54.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41b1aa0 con 0x559ffddca5c0 2024-04-25T01:59:54.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffdded9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffdded9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51030/0 (socket says 172.21.15.102:51030) 2024-04-25T01:59:54.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.809+0000 7f25d3400640 1 -- 172.21.15.102:0/2880283297 learned_addr learned my addr 172.21.15.102:0/2880283297 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.810+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddcc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.810+0000 7f25d8a00640 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 0x559ffddedf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.810+0000 7f25d3e00640 1 -- 172.21.15.102:0/2880283297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 msgr2=0x559ffdded9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.811+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffdded9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.811+0000 7f25d3e00640 1 -- 172.21.15.102:0/2880283297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 msgr2=0x559ffddedf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.811+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 0x559ffddedf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.811+0000 7f25d3e00640 1 -- 172.21.15.102:0/2880283297 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ffddd79a0 con 0x559ffddca5c0 2024-04-25T01:59:54.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.811+0000 7f25d8a00640 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 0x559ffddedf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:54.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.811+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddcc4e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f25c400ec20 tx=0x7f25c400cab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d78e4bb21fed30e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.812+0000 7f25bfe00640 1 -- 172.21.15.102:0/2880283297 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4014040 con 0x559ffddca5c0 2024-04-25T01:59:54.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.812+0000 7f25bfe00640 1 -- 172.21.15.102:0/2880283297 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c4010c80 con 0x559ffddca5c0 2024-04-25T01:59:54.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.812+0000 7f25bfe00640 1 -- 172.21.15.102:0/2880283297 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c40104a0 con 0x559ffddca5c0 2024-04-25T01:59:54.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.813+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2880283297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 msgr2=0x559ffddcc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.813+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddcc4e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f25c400ec20 tx=0x7f25c400cab0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.813+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2880283297 shutdown_connections 2024-04-25T01:59:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.813+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddd3130 0x559ffddedf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.813+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddcaee0 0x559ffdded9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.813+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2880283297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddca5c0 0x559ffddcc4e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.813+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2880283297 >> 172.21.15.102:0/2880283297 conn(0x559ffddc68e0 msgr2=0x559ffddcbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.813+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2880283297 shutdown_connections 2024-04-25T01:59:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.813+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2880283297 wait complete. 2024-04-25T01:59:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.814+0000 7f25dd9ad740 1 -- start start 2024-04-25T01:59:54.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.814+0000 7f25dd9ad740 1 Processor -- start 2024-04-25T01:59:54.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.815+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddca5c0 0x559ffde44540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.815+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddcaee0 0x559ffde44a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.815+0000 7f25dd9ad740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddd3130 0x559ffddd8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.815+0000 7f25dd9ad740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41b2110 con 0x559ffddcaee0 2024-04-25T01:59:54.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.815+0000 7f25d8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddd3130 0x559ffddd8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25d8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddd3130 0x559ffddd8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38106/0 (socket says 172.21.15.102:38106) 2024-04-25T01:59:54.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25d8a00640 1 -- 172.21.15.102:0/2858298669 learned_addr learned my addr 172.21.15.102:0/2858298669 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:54.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddca5c0 0x559ffde44540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41b3580 con 0x559ffddca5c0 2024-04-25T01:59:54.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25d41b27e0 con 0x559ffddd3130 2024-04-25T01:59:54.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25dd9ad740 10 client.0 _ll_register_callbacks cb 0x7f25c0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25d8a00640 1 -- 172.21.15.102:0/2858298669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddca5c0 msgr2=0x559ffde44540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25d8a00640 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddca5c0 0x559ffde44540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25d8a00640 1 -- 172.21.15.102:0/2858298669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddcaee0 msgr2=0x559ffde44a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25d8a00640 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddcaee0 0x559ffde44a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.816+0000 7f25d8a00640 1 -- 172.21.15.102:0/2858298669 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ffddd9230 con 0x559ffddd3130 2024-04-25T01:59:54.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.817+0000 7f25d8a00640 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddd3130 0x559ffddd8b40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f25cc006010 tx=0x7f25cc0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6800a982e6f0bbcb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:54.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.817+0000 7f25bd600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:54.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.817+0000 7f25bd600640 1 -- 172.21.15.102:0/2858298669 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc058060 con 0x559ffddd3130 2024-04-25T01:59:54.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.817+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25c4010c80 con 0x559ffddd3130 2024-04-25T01:59:54.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.817+0000 7f25bd600640 1 -- 172.21.15.102:0/2858298669 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f25cc003d00 con 0x559ffddd3130 2024-04-25T01:59:54.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.817+0000 7f25bd600640 1 -- 172.21.15.102:0/2858298669 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f25cc031450 con 0x559ffddd3130 2024-04-25T01:59:54.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.817+0000 7f25dd9ad740 10 client.26488 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:54.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.817+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559ffde4bd50 con 0x559ffddd3130 2024-04-25T01:59:54.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.818+0000 7f25bd600640 1 -- 172.21.15.102:0/2858298669 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c4010c80 con 0x559ffddd3130 2024-04-25T01:59:54.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.818+0000 7f25bd600640 1 -- 172.21.15.102:0/2858298669 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f25cc05d480 con 0x559ffddd3130 2024-04-25T01:59:54.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.818+0000 7f25dd9ad740 10 client.26488 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:54.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.818+0000 7f25dd9ad740 10 client.26488 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.818+0000 7f25dd9ad740 20 client.26488 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.818+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x559ffde4c4e0 0x559ffde4e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.818+0000 7f25dd9ad740 4 client.26488 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.818+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ffdd36d70 con 0x559ffde4c4e0 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.818+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x559ffde4c4e0 0x559ffde4e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.819+0000 7f25bd600640 10 client.26488 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.819+0000 7f25d3e00640 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x559ffde4c4e0 0x559ffde4e960 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=811bf505a078f411 server_cookie=fe909d9796762d32 in_seq=0 out_seq=0 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.819+0000 7f25bd600640 1 -- 172.21.15.102:0/2858298669 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ffdd36d70 con 0x559ffde4c4e0 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.819+0000 7f25bd600640 10 client.26488 handle_command_reply: tid=0 2024-04-25T01:59:54.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.819+0000 7f25dd9ad740 1 client.26488 shutdown 2024-04-25T01:59:54.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.820+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddd3130 msgr2=0x559ffddd8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.820+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddd3130 0x559ffddd8b40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f25cc006010 tx=0x7f25cc0095c0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.820+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 shutdown_connections 2024-04-25T01:59:54.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.820+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x559ffde4c4e0 0x559ffde4e960 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.820+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ffddd3130 0x559ffddd8b40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.820+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ffddcaee0 0x559ffde44a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.820+0000 7f25dd9ad740 1 --2- 172.21.15.102:0/2858298669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ffddca5c0 0x559ffde44540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.820+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 >> 172.21.15.102:0/2858298669 conn(0x559ffddc68e0 msgr2=0x559ffddcbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.820+0000 7f25bd600640 0 client.26488 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T01:59:54.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.821+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 shutdown_connections 2024-04-25T01:59:54.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.821+0000 7f25dd9ad740 1 -- 172.21.15.102:0/2858298669 wait complete. 2024-04-25T01:59:54.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.821+0000 7f25dd9ad740 20 client.26488 trim_cache size 0 max 16384 2024-04-25T01:59:54.826 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 -- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25b80a2c90 msgr2=0x7f25b80a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25b80a2c90 0x7f25b80a5150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009430 tx=0x7f25cc005c30 comp rx=0 tx=0).stop 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 -- 172.21.15.102:0/1098179690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d410d110 msgr2=0x7f25d419eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d410d110 0x7f25d419eb30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f25d407bc50 tx=0x7f25c000ee50 comp rx=0 tx=0).stop 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 -- 172.21.15.102:0/1098179690 shutdown_connections 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f25b80a2c90 0x7f25b80a5150 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25d41a2e60 0x7f25d41a3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25d410d110 0x7f25d419eb30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 --2- 172.21.15.102:0/1098179690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25d41087f0 0x7f25d419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:54.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.822+0000 7f25daa00640 1 -- 172.21.15.102:0/1098179690 >> 172.21.15.102:0/1098179690 conn(0x7f25d40775b0 msgr2=0x7f25d410b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:54.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.823+0000 7f25daa00640 1 -- 172.21.15.102:0/1098179690 shutdown_connections 2024-04-25T01:59:54.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:54.823+0000 7f25daa00640 1 -- 172.21.15.102:0/1098179690 wait complete. 2024-04-25T01:59:54.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T01:59:55.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.135+0000 7f707aa00640 1 -- 172.21.15.102:0/2321571707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741027a0 msgr2=0x7f7074102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.135+0000 7f707aa00640 1 --2- 172.21.15.102:0/2321571707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741027a0 0x7f7074102b80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f706c009ca0 tx=0x7f706c02a950 comp rx=0 tx=0).stop 2024-04-25T01:59:55.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.135+0000 7f707aa00640 1 -- 172.21.15.102:0/2321571707 shutdown_connections 2024-04-25T01:59:55.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.135+0000 7f707aa00640 1 --2- 172.21.15.102:0/2321571707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f707410bbc0 0x7f707410e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.135+0000 7f707aa00640 1 --2- 172.21.15.102:0/2321571707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70741030c0 0x7f707410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.135+0000 7f707aa00640 1 --2- 172.21.15.102:0/2321571707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741027a0 0x7f7074102b80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.135+0000 7f707aa00640 1 -- 172.21.15.102:0/2321571707 >> 172.21.15.102:0/2321571707 conn(0x7f70740fbfc0 msgr2=0x7f70740fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.137+0000 7f707aa00640 1 -- 172.21.15.102:0/2321571707 shutdown_connections 2024-04-25T01:59:55.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.137+0000 7f707aa00640 1 -- 172.21.15.102:0/2321571707 wait complete. 2024-04-25T01:59:55.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.138+0000 7f707aa00640 1 Processor -- start 2024-04-25T01:59:55.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.140+0000 7f707aa00640 1 -- start start 2024-04-25T01:59:55.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.140+0000 7f707aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70741027a0 0x7f7074198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.140+0000 7f707aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70741030c0 0x7f70741985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.140+0000 7f707aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f707410bbc0 0x7f707419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.140+0000 7f707aa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7074111ad0 con 0x7f70741030c0 2024-04-25T01:59:55.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.140+0000 7f707aa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7074111950 con 0x7f70741027a0 2024-04-25T01:59:55.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.140+0000 7f707aa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7074111c50 con 0x7f707410bbc0 2024-04-25T01:59:55.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.141+0000 7f7078c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70741030c0 0x7f70741985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.141+0000 7f7078c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70741030c0 0x7f70741985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54360/0 (socket says 172.21.15.102:54360) 2024-04-25T01:59:55.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.141+0000 7f7078c00640 1 -- 172.21.15.102:0/2861583044 learned_addr learned my addr 172.21.15.102:0/2861583044 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:55.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.141+0000 7f7079600640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70741027a0 0x7f7074198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.141+0000 7f707a000640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f707410bbc0 0x7f707419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.142+0000 7f7078c00640 1 -- 172.21.15.102:0/2861583044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f707410bbc0 msgr2=0x7f707419c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.142+0000 7f7078c00640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f707410bbc0 0x7f707419c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.142+0000 7f7078c00640 1 -- 172.21.15.102:0/2861583044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70741027a0 msgr2=0x7f7074198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.142+0000 7f7078c00640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70741027a0 0x7f7074198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.142+0000 7f7078c00640 1 -- 172.21.15.102:0/2861583044 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707419d030 con 0x7f70741030c0 2024-04-25T01:59:55.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.142+0000 7f707a000640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f707410bbc0 0x7f707419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:55.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.143+0000 7f7078c00640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70741030c0 0x7f70741985b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f706800b3f0 tx=0x7f706800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.144+0000 7f7072000640 1 -- 172.21.15.102:0/2861583044 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068015020 con 0x7f70741030c0 2024-04-25T01:59:55.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.144+0000 7f7072000640 1 -- 172.21.15.102:0/2861583044 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7068011d10 con 0x7f70741030c0 2024-04-25T01:59:55.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.144+0000 7f7072000640 1 -- 172.21.15.102:0/2861583044 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706800f2e0 con 0x7f70741030c0 2024-04-25T01:59:55.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.145+0000 7f707aa00640 1 -- 172.21.15.102:0/2861583044 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f707419d320 con 0x7f70741030c0 2024-04-25T01:59:55.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.145+0000 7f707aa00640 1 -- 172.21.15.102:0/2861583044 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7074100320 con 0x7f70741030c0 2024-04-25T01:59:55.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.152+0000 7f707ed7a740 1 Processor -- start 2024-04-25T01:59:55.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.153+0000 7f707ed7a740 1 -- start start 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.153+0000 7f7072000640 1 -- 172.21.15.102:0/2861583044 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7068018020 con 0x7f70741030c0 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.155+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.155+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.155+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 0x55c9516fbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.155+0000 7f707ed7a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c9516534e0 con 0x55c9516ee7a0 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.155+0000 7f707ed7a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c9515fba70 con 0x55c9516f9790 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.155+0000 7f707ed7a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c95164f4a0 con 0x55c9516ef0c0 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.156+0000 7f7079600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.156+0000 7f7079600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38142/0 (socket says 172.21.15.102:38142) 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.156+0000 7f7079600640 1 -- 172.21.15.102:0/1459477463 learned_addr learned my addr 172.21.15.102:0/1459477463 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:55.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.156+0000 7f707a000640 1 --2- 172.21.15.102:0/1459477463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.156+0000 7f7079600640 1 -- 172.21.15.102:0/1459477463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 msgr2=0x55c9516fbb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:59:55.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.156+0000 7f7079600640 1 --2- 172.21.15.102:0/1459477463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 0x55c9516fbb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.156+0000 7f7079600640 1 -- 172.21.15.102:0/1459477463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 msgr2=0x55c9516eeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.157+0000 7f7079600640 1 --2- 172.21.15.102:0/1459477463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516eeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.157+0000 7f7079600640 1 -- 172.21.15.102:0/1459477463 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9513673f0 con 0x55c9516ef0c0 2024-04-25T01:59:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.158+0000 7f7079600640 1 --2- 172.21.15.102:0/1459477463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f9250 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f706c02ae70 tx=0x7f706c009980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0fc5f9d8de4a2e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.158+0000 7f707a000640 1 --2- 172.21.15.102:0/1459477463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516eeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.159+0000 7f7066600640 1 -- 172.21.15.102:0/1459477463 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c048070 con 0x55c9516ef0c0 2024-04-25T01:59:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.159+0000 7f7066600640 1 -- 172.21.15.102:0/1459477463 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f706c003e50 con 0x55c9516ef0c0 2024-04-25T01:59:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.159+0000 7f7066600640 1 -- 172.21.15.102:0/1459477463 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c034040 con 0x55c9516ef0c0 2024-04-25T01:59:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.160+0000 7f707ed7a740 1 -- 172.21.15.102:0/1459477463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 msgr2=0x55c9516f9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.160+0000 7f707ed7a740 1 --2- 172.21.15.102:0/1459477463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f9250 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f706c02ae70 tx=0x7f706c009980 comp rx=0 tx=0).stop 2024-04-25T01:59:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.160+0000 7f707ed7a740 1 -- 172.21.15.102:0/1459477463 shutdown_connections 2024-04-25T01:59:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.160+0000 7f707ed7a740 1 --2- 172.21.15.102:0/1459477463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 0x55c9516fbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.160+0000 7f707ed7a740 1 --2- 172.21.15.102:0/1459477463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f9250 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.160+0000 7f707ed7a740 1 --2- 172.21.15.102:0/1459477463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516eeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.160+0000 7f707ed7a740 1 -- 172.21.15.102:0/1459477463 >> 172.21.15.102:0/1459477463 conn(0x55c9516eaac0 msgr2=0x55c9516eaea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.161+0000 7f707ed7a740 1 -- 172.21.15.102:0/1459477463 shutdown_connections 2024-04-25T01:59:55.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.161+0000 7f707ed7a740 1 -- 172.21.15.102:0/1459477463 wait complete. 2024-04-25T01:59:55.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.162+0000 7f707ed7a740 1 -- start start 2024-04-25T01:59:55.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.162+0000 7f7072000640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f705c0a2c90 0x7f705c0a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.162+0000 7f707ed7a740 1 Processor -- start 2024-04-25T01:59:55.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.162+0000 7f7072000640 1 -- 172.21.15.102:0/2861583044 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f70680cead0 con 0x7f70741030c0 2024-04-25T01:59:55.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.163+0000 7f707a000640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f705c0a2c90 0x7f705c0a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.163+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.163+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.163+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 0x55c9516f7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.163+0000 7f707ed7a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c9516fc520 con 0x55c9516ee7a0 2024-04-25T01:59:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.164+0000 7f707a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 0x55c9516f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.164+0000 7f707a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 0x55c9516f7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51068/0 (socket says 172.21.15.102:51068) 2024-04-25T01:59:55.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.164+0000 7f707a000640 1 -- 172.21.15.102:0/2929157467 learned_addr learned my addr 172.21.15.102:0/2929157467 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:55.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.164+0000 7f7078c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.165+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c95160eec0 con 0x55c9516f9790 2024-04-25T01:59:55.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.165+0000 7f7079600640 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.165+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c9516fc6a0 con 0x55c9516ef0c0 2024-04-25T01:59:55.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.165+0000 7f707ed7a740 10 client.0 _ll_register_callbacks cb 0x55c95164b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:55.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.165+0000 7f707a000640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f705c0a2c90 0x7f705c0a5150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f706000b820 tx=0x7f7060008040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.166+0000 7f707a000640 1 -- 172.21.15.102:0/2929157467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 msgr2=0x55c9516f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.166+0000 7f707a000640 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.166+0000 7f707a000640 1 -- 172.21.15.102:0/2929157467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 msgr2=0x55c9516f6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.166+0000 7f707a000640 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.166+0000 7f707a000640 1 -- 172.21.15.102:0/2929157467 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9514317d0 con 0x55c9516f9790 2024-04-25T01:59:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.166+0000 7f707a000640 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 0x55c9516f7200 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7060006f90 tx=0x7f70600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dba8c9dfde1b6c8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.166+0000 7f705aa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.166+0000 7f705aa00640 1 -- 172.21.15.102:0/2929157467 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060023020 con 0x55c9516f9790 2024-04-25T01:59:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.166+0000 7f705aa00640 1 -- 172.21.15.102:0/2929157467 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7060012910 con 0x55c9516f9790 2024-04-25T01:59:55.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.167+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f706c003e50 con 0x55c9516f9790 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.167+0000 7f705aa00640 1 -- 172.21.15.102:0/2929157467 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7060012b10 con 0x55c9516f9790 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.167+0000 7f707ed7a740 10 client.26531 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.167+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c951710ff0 con 0x55c9516f9790 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.167+0000 7f705aa00640 1 -- 172.21.15.102:0/2929157467 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f706c003e50 con 0x55c9516f9790 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.167+0000 7f705aa00640 1 -- 172.21.15.102:0/2929157467 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f7060030020 con 0x55c9516f9790 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.167+0000 7f707ed7a740 10 client.26531 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.167+0000 7f707ed7a740 10 client.26531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.167+0000 7f707ed7a740 20 client.26531 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.168+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c951712500 0x55c951714960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.168+0000 7f707ed7a740 4 client.26531 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.168+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c951660a20 con 0x55c951712500 2024-04-25T01:59:55.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.168+0000 7f7079600640 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c951712500 0x55c951714960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.169+0000 7f7079600640 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c951712500 0x55c951714960 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e02c7de1914a7fc4 server_cookie=a2911611bc3e0980 in_seq=0 out_seq=0 2024-04-25T01:59:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.169+0000 7f705aa00640 10 client.26531 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.170+0000 7f705aa00640 1 -- 172.21.15.102:0/2929157467 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c951660a20 con 0x55c951712500 2024-04-25T01:59:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.170+0000 7f705aa00640 10 client.26531 handle_command_reply: tid=0 2024-04-25T01:59:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.170+0000 7f707ed7a740 1 client.26531 shutdown 2024-04-25T01:59:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 msgr2=0x55c9516f7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 0x55c9516f7200 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7060006f90 tx=0x7f70600046c0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 shutdown_connections 2024-04-25T01:59:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c951712500 0x55c951714960 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516f9790 0x55c9516f7200 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f705aa00640 0 client.26531 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2929157467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 >> 172.21.15.102:0/2929157467 conn(0x55c9516eaac0 msgr2=0x55c9516f07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.171+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 shutdown_connections 2024-04-25T01:59:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.172+0000 7f707ed7a740 1 -- 172.21.15.102:0/2929157467 wait complete. 2024-04-25T01:59:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.172+0000 7f707ed7a740 20 client.26531 trim_cache size 0 max 16384 2024-04-25T01:59:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.194+0000 7f707ed7a740 1 Processor -- start 2024-04-25T01:59:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.194+0000 7f707ed7a740 1 -- start start 2024-04-25T01:59:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ee7a0 0x55c95170dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516f9790 0x55c95171a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f707ed7a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70741b6240 con 0x55c9516f9790 2024-04-25T01:59:55.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f707ed7a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7074071830 con 0x55c9516ee7a0 2024-04-25T01:59:55.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f707ed7a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f707406f6a0 con 0x55c9516ef0c0 2024-04-25T01:59:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f7079600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ee7a0 0x55c95170dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f7079600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ee7a0 0x55c95170dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51074/0 (socket says 172.21.15.102:51074) 2024-04-25T01:59:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f7079600640 1 -- 172.21.15.102:0/2793583878 learned_addr learned my addr 172.21.15.102:0/2793583878 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.195+0000 7f707a000640 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516f9790 0x55c95171a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7079600640 1 -- 172.21.15.102:0/2793583878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 msgr2=0x55c9516f7020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T01:59:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7078c00640 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7079600640 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7079600640 1 -- 172.21.15.102:0/2793583878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516f9790 msgr2=0x55c95171a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7079600640 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516f9790 0x55c95171a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7079600640 1 -- 172.21.15.102:0/2793583878 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9516fbef0 con 0x55c9516ee7a0 2024-04-25T01:59:55.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7078c00640 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:55.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7079600640 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ee7a0 0x55c95170dca0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f706c02ae70 tx=0x7f706c0093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e78cf3fc0fb206a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7065c00640 1 -- 172.21.15.102:0/2793583878 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c054070 con 0x55c9516ee7a0 2024-04-25T01:59:55.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7065c00640 1 -- 172.21.15.102:0/2793583878 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c951660a20 con 0x55c9516ee7a0 2024-04-25T01:59:55.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.196+0000 7f7065c00640 1 -- 172.21.15.102:0/2793583878 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c048040 con 0x55c9516ee7a0 2024-04-25T01:59:55.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.197+0000 7f707ed7a740 1 -- 172.21.15.102:0/2793583878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ee7a0 msgr2=0x55c95170dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.197+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ee7a0 0x55c95170dca0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f706c02ae70 tx=0x7f706c0093f0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.198+0000 7f707ed7a740 1 -- 172.21.15.102:0/2793583878 shutdown_connections 2024-04-25T01:59:55.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.198+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516f9790 0x55c95171a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.198+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516ef0c0 0x55c9516f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.198+0000 7f707ed7a740 1 --2- 172.21.15.102:0/2793583878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ee7a0 0x55c95170dca0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.198+0000 7f707ed7a740 1 -- 172.21.15.102:0/2793583878 >> 172.21.15.102:0/2793583878 conn(0x55c9516eaac0 msgr2=0x55c9516e9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.198+0000 7f707ed7a740 1 -- 172.21.15.102:0/2793583878 shutdown_connections 2024-04-25T01:59:55.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.198+0000 7f707ed7a740 1 -- 172.21.15.102:0/2793583878 wait complete. 2024-04-25T01:59:55.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.199+0000 7f707ed7a740 1 -- start start 2024-04-25T01:59:55.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.199+0000 7f707ed7a740 1 Processor -- start 2024-04-25T01:59:55.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.200+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.200+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ef0c0 0x55c9516f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.200+0000 7f707ed7a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516f9790 0x55c9516f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.200+0000 7f707ed7a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f707406fcf0 con 0x55c9516ee7a0 2024-04-25T01:59:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.200+0000 7f707ed7a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7074071160 con 0x55c9516ef0c0 2024-04-25T01:59:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.200+0000 7f707ed7a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70740703c0 con 0x55c9516f9790 2024-04-25T01:59:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.200+0000 7f707ed7a740 10 client.0 _ll_register_callbacks cb 0x7f706000bee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.201+0000 7f7079600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.201+0000 7f7078c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ef0c0 0x55c9516f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.201+0000 7f7079600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54414/0 (socket says 172.21.15.102:54414) 2024-04-25T01:59:55.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.201+0000 7f7079600640 1 -- 172.21.15.102:0/413646384 learned_addr learned my addr 172.21.15.102:0/413646384 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.201+0000 7f707a000640 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516f9790 0x55c9516f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.201+0000 7f7078c00640 1 -- 172.21.15.102:0/413646384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516f9790 msgr2=0x55c9516f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.201+0000 7f7078c00640 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516f9790 0x55c9516f3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.201+0000 7f7078c00640 1 -- 172.21.15.102:0/413646384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 msgr2=0x55c9516f32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.201+0000 7f7078c00640 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.202+0000 7f7078c00640 1 -- 172.21.15.102:0/413646384 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c951777aa0 con 0x55c9516ef0c0 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.202+0000 7f707a000640 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516f9790 0x55c9516f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.202+0000 7f7079600640 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.202+0000 7f7078c00640 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ef0c0 0x55c9516f37f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f70680cc360 tx=0x7f706809b270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85b95a5100cde320 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.202+0000 7f705aa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.202+0000 7f705aa00640 1 -- 172.21.15.102:0/413646384 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70680b4060 con 0x55c9516ef0c0 2024-04-25T01:59:55.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.203+0000 7f705aa00640 1 -- 172.21.15.102:0/413646384 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f706809f170 con 0x55c9516ef0c0 2024-04-25T01:59:55.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.203+0000 7f707ed7a740 1 -- 172.21.15.102:0/413646384 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c951660a20 con 0x55c9516ef0c0 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.203+0000 7f705aa00640 1 -- 172.21.15.102:0/413646384 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f70680bb0b0 con 0x55c9516ef0c0 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.203+0000 7f707ed7a740 10 client.26537 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.203+0000 7f707ed7a740 1 -- 172.21.15.102:0/413646384 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c951778c30 con 0x55c9516ef0c0 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.203+0000 7f705aa00640 1 -- 172.21.15.102:0/413646384 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c951660a20 con 0x55c9516ef0c0 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.204+0000 7f705aa00640 1 -- 172.21.15.102:0/413646384 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f706809a580 con 0x55c9516ef0c0 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.204+0000 7f707ed7a740 10 client.26537 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.204+0000 7f707ed7a740 10 client.26537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.204+0000 7f707ed7a740 20 client.26537 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.204+0000 7f707ed7a740 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c951712500 0x55c95177b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.204+0000 7f707ed7a740 4 client.26537 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.204+0000 7f707ed7a740 1 -- 172.21.15.102:0/413646384 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c95165ad70 con 0x55c951712500 2024-04-25T01:59:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.204+0000 7f7079600640 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c951712500 0x55c95177b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f7079600640 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c951712500 0x55c95177b3a0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a9c775dc252054f server_cookie=5c06c1bc0f189956 in_seq=0 out_seq=0 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f705aa00640 10 client.26537 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f705aa00640 1 -- 172.21.15.102:0/413646384 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c95165ad70 con 0x55c951712500 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f705aa00640 10 client.26537 handle_command_reply: tid=0 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f707ed7a740 1 client.26537 shutdown 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f707ed7a740 1 -- 172.21.15.102:0/413646384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ef0c0 msgr2=0x55c9516f37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f707ed7a740 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ef0c0 0x55c9516f37f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f70680cc360 tx=0x7f706809b270 comp rx=0 tx=0).stop 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f707ed7a740 1 -- 172.21.15.102:0/413646384 shutdown_connections 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f707ed7a740 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c951712500 0x55c95177b3a0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f705aa00640 0 client.26537 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.205+0000 7f707ed7a740 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9516f9790 0x55c9516f3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.206+0000 7f707ed7a740 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c9516ef0c0 0x55c9516f37f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.206+0000 7f707ed7a740 1 --2- 172.21.15.102:0/413646384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c9516ee7a0 0x55c9516f32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.206+0000 7f707ed7a740 1 -- 172.21.15.102:0/413646384 >> 172.21.15.102:0/413646384 conn(0x55c9516eaac0 msgr2=0x55c9516efef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.206+0000 7f707ed7a740 1 -- 172.21.15.102:0/413646384 shutdown_connections 2024-04-25T01:59:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.206+0000 7f707ed7a740 1 -- 172.21.15.102:0/413646384 wait complete. 2024-04-25T01:59:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.206+0000 7f707ed7a740 20 client.26537 trim_cache size 0 max 16384 2024-04-25T01:59:55.211 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T01:59:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.207+0000 7f707aa00640 1 -- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f705c0a2c90 msgr2=0x7f705c0a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f705c0a2c90 0x7f705c0a5150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f706000b820 tx=0x7f7060008040 comp rx=0 tx=0).stop 2024-04-25T01:59:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 -- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70741030c0 msgr2=0x7f70741985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70741030c0 0x7f70741985b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f706800b3f0 tx=0x7f706800b8b0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 -- 172.21.15.102:0/2861583044 shutdown_connections 2024-04-25T01:59:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f705c0a2c90 0x7f705c0a5150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f707410bbc0 0x7f707419c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70741030c0 0x7f70741985b0 secure :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f706800b3f0 tx=0x7f706800b8b0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 --2- 172.21.15.102:0/2861583044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70741027a0 0x7f7074198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 -- 172.21.15.102:0/2861583044 >> 172.21.15.102:0/2861583044 conn(0x7f70740fbfc0 msgr2=0x7f70740fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.208+0000 7f707aa00640 1 -- 172.21.15.102:0/2861583044 shutdown_connections 2024-04-25T01:59:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:55.209+0000 7f707aa00640 1 -- 172.21.15.102:0/2861583044 wait complete. 2024-04-25T01:59:55.676 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:55 smithi102 ceph-mon[27796]: pgmap v142: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 787 op/s 2024-04-25T01:59:55.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:55 smithi110 ceph-mon[32848]: pgmap v142: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 787 op/s 2024-04-25T01:59:55.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:55 smithi193 ceph-mon[32981]: pgmap v142: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 787 op/s 2024-04-25T01:59:56.237 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T01:59:56.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T01:59:56.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T01:59:56.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.506+0000 7fead8c00640 1 -- 172.21.15.102:0/2106860391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 msgr2=0x7fead4101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.506+0000 7fead8c00640 1 --2- 172.21.15.102:0/2106860391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 0x7fead4101a60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7feac0009890 tx=0x7feac002a360 comp rx=0 tx=0).stop 2024-04-25T01:59:56.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.506+0000 7fead8c00640 1 -- 172.21.15.102:0/2106860391 shutdown_connections 2024-04-25T01:59:56.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.506+0000 7fead8c00640 1 --2- 172.21.15.102:0/2106860391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fead4101fa0 0x7fead410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.506+0000 7fead8c00640 1 --2- 172.21.15.102:0/2106860391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 0x7fead4101a60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.506+0000 7fead8c00640 1 --2- 172.21.15.102:0/2106860391 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead4107680 0x7fead4107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.506+0000 7fead8c00640 1 -- 172.21.15.102:0/2106860391 >> 172.21.15.102:0/2106860391 conn(0x7fead40fd110 msgr2=0x7fead40ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:56.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.506+0000 7fead8c00640 1 -- 172.21.15.102:0/2106860391 shutdown_connections 2024-04-25T01:59:56.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.507+0000 7fead8c00640 1 -- 172.21.15.102:0/2106860391 wait complete. 2024-04-25T01:59:56.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.508+0000 7fead8c00640 1 Processor -- start 2024-04-25T01:59:56.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.509+0000 7fead8c00640 1 -- start start 2024-04-25T01:59:56.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead8c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 0x7fead41a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead8c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fead4101fa0 0x7fead41a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead4107680 0x7fead410b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead8c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fead4111a10 con 0x7fead4107680 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead8c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fead4111890 con 0x7fead4101fa0 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead8c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fead4111b90 con 0x7fead41015e0 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 0x7fead41a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 0x7fead41a3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38214/0 (socket says 172.21.15.102:38214) 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead3400640 1 -- 172.21.15.102:0/2999652428 learned_addr learned my addr 172.21.15.102:0/2999652428 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.510+0000 7fead2a00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fead4101fa0 0x7fead41a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.511+0000 7fead3400640 1 -- 172.21.15.102:0/2999652428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fead4101fa0 msgr2=0x7fead41a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.511+0000 7fead3400640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fead4101fa0 0x7fead41a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.511+0000 7fead3400640 1 -- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead4107680 msgr2=0x7fead410b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.511+0000 7fead3e00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead4107680 0x7fead410b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.511+0000 7fead3400640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead4107680 0x7fead410b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.511+0000 7fead3400640 1 -- 172.21.15.102:0/2999652428 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead410bc40 con 0x7fead41015e0 2024-04-25T01:59:56.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.511+0000 7fead3e00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead4107680 0x7fead410b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:56.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.511+0000 7fead3400640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 0x7fead41a3140 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7feac400d750 tx=0x7feac400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.511+0000 7feacbe00640 1 -- 172.21.15.102:0/2999652428 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4016070 con 0x7fead41015e0 2024-04-25T01:59:56.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.512+0000 7feacbe00640 1 -- 172.21.15.102:0/2999652428 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac4004460 con 0x7fead41015e0 2024-04-25T01:59:56.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.512+0000 7feacbe00640 1 -- 172.21.15.102:0/2999652428 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4003dc0 con 0x7fead41015e0 2024-04-25T01:59:56.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.513+0000 7fead8c00640 1 -- 172.21.15.102:0/2999652428 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead410bf30 con 0x7fead41015e0 2024-04-25T01:59:56.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.513+0000 7fead8c00640 1 -- 172.21.15.102:0/2999652428 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead41ab060 con 0x7fead41015e0 2024-04-25T01:59:56.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.516+0000 7feacbe00640 1 -- 172.21.15.102:0/2999652428 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feac4019070 con 0x7fead41015e0 2024-04-25T01:59:56.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.518+0000 7feadd01f740 1 Processor -- start 2024-04-25T01:59:56.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.518+0000 7feadd01f740 1 -- start start 2024-04-25T01:59:56.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7feadd01f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc25f20 0x55bfbbc26300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7feadd01f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc309d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7feadd01f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc30f10 0x55bfbbc33300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7feadd01f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55bfbbb84f40 con 0x55bfbbc30f10 2024-04-25T01:59:56.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7feadd01f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bfbbb8a190 con 0x55bfbbc25f20 2024-04-25T01:59:56.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7feadd01f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bfbbb32a10 con 0x55bfbbc26840 2024-04-25T01:59:56.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7fead3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc25f20 0x55bfbbc26300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7feacbe00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feab80a2dc0 0x7feab80a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7fead3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc25f20 0x55bfbbc26300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51104/0 (socket says 172.21.15.102:51104) 2024-04-25T01:59:56.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.519+0000 7fead3e00640 1 -- 172.21.15.102:0/691427965 learned_addr learned my addr 172.21.15.102:0/691427965 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:56.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead3400640 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7feacbe00640 1 -- 172.21.15.102:0/2999652428 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7feac40cef80 con 0x7fead41015e0 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead3e00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feab80a2dc0 0x7feab80a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead2a00640 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc30f10 0x55bfbbc33300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead3400640 1 -- 172.21.15.102:0/691427965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc25f20 msgr2=0x55bfbbc26300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead3400640 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc25f20 0x55bfbbc26300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead3400640 1 -- 172.21.15.102:0/691427965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc30f10 msgr2=0x55bfbbc33300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead3400640 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc30f10 0x55bfbbc33300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead3400640 1 -- 172.21.15.102:0/691427965 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfbb9667e0 con 0x55bfbbc26840 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead3e00640 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc25f20 0x55bfbbc26300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.520+0000 7fead2a00640 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc30f10 0x55bfbbc33300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.521+0000 7fead3400640 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc309d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feac40cc6a0 tx=0x7feac4003fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d015a49f398258e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.521+0000 7fead3e00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feab80a2dc0 0x7feab80a5280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feacc009fd0 tx=0x7feacc004ab0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.521+0000 7feabfe00640 1 -- 172.21.15.102:0/691427965 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4020040 con 0x55bfbbc26840 2024-04-25T01:59:56.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.521+0000 7feabfe00640 1 -- 172.21.15.102:0/691427965 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac409bb70 con 0x55bfbbc26840 2024-04-25T01:59:56.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.521+0000 7feabfe00640 1 -- 172.21.15.102:0/691427965 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40cc920 con 0x55bfbbc26840 2024-04-25T01:59:56.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.522+0000 7feadd01f740 1 -- 172.21.15.102:0/691427965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 msgr2=0x55bfbbc309d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.522+0000 7feadd01f740 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc309d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feac40cc6a0 tx=0x7feac4003fe0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.523+0000 7feadd01f740 1 -- 172.21.15.102:0/691427965 shutdown_connections 2024-04-25T01:59:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.523+0000 7feadd01f740 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc30f10 0x55bfbbc33300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.523+0000 7feadd01f740 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc309d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.523+0000 7feadd01f740 1 --2- 172.21.15.102:0/691427965 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc25f20 0x55bfbbc26300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.523+0000 7feadd01f740 1 -- 172.21.15.102:0/691427965 >> 172.21.15.102:0/691427965 conn(0x55bfbbc22240 msgr2=0x55bfbbc22620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:56.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.524+0000 7feadd01f740 1 -- 172.21.15.102:0/691427965 shutdown_connections 2024-04-25T01:59:56.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.524+0000 7feadd01f740 1 -- 172.21.15.102:0/691427965 wait complete. 2024-04-25T01:59:56.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.525+0000 7feadd01f740 1 -- start start 2024-04-25T01:59:56.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.526+0000 7feadd01f740 1 Processor -- start 2024-04-25T01:59:56.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.526+0000 7feadd01f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc2df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.526+0000 7feadd01f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7feadd01f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 0x55bfbbc2ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7fead3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc2df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7feadd01f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55bfbbb45e70 con 0x55bfbbc25f20 2024-04-25T01:59:56.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7fead3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc2df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54448/0 (socket says 172.21.15.102:54448) 2024-04-25T01:59:56.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7fead3400640 1 -- 172.21.15.102:0/1529927342 learned_addr learned my addr 172.21.15.102:0/1529927342 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:56.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55bfbbb86150 con 0x55bfbbc26840 2024-04-25T01:59:56.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bfbbc33ce0 con 0x55bfbbc30f10 2024-04-25T01:59:56.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7fead2a00640 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7feadd01f740 10 client.0 _ll_register_callbacks cb 0x55bfbbb811f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:56.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.527+0000 7fead3e00640 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 0x55bfbbc2ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.528+0000 7fead3400640 1 -- 172.21.15.102:0/1529927342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 msgr2=0x55bfbbc2ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.528+0000 7fead3400640 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 0x55bfbbc2ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.528+0000 7fead3400640 1 -- 172.21.15.102:0/1529927342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 msgr2=0x55bfbbc2e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.528+0000 7fead3400640 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.528+0000 7fead3400640 1 -- 172.21.15.102:0/1529927342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfbb9c6f70 con 0x55bfbbc25f20 2024-04-25T01:59:56.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.528+0000 7fead2a00640 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:56.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.528+0000 7fead3e00640 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 0x55bfbbc2ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:56.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.529+0000 7fead3400640 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc2df80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7feac409bd20 tx=0x7feac409f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88428cc77173b12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.529+0000 7feabcc00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T01:59:56.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.530+0000 7feabcc00640 1 -- 172.21.15.102:0/1529927342 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40b4060 con 0x55bfbbc25f20 2024-04-25T01:59:56.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.530+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feac409bb70 con 0x55bfbbc25f20 2024-04-25T01:59:56.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.530+0000 7feabcc00640 1 -- 172.21.15.102:0/1529927342 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7feac40b4200 con 0x55bfbbc25f20 2024-04-25T01:59:56.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.530+0000 7feabcc00640 1 -- 172.21.15.102:0/1529927342 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feac4020070 con 0x55bfbbc25f20 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.530+0000 7feadd01f740 10 client.16110 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.530+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bfbb85e170 con 0x55bfbbc25f20 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.531+0000 7feabcc00640 1 -- 172.21.15.102:0/1529927342 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac409bb70 con 0x55bfbbc25f20 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.531+0000 7feabcc00640 1 -- 172.21.15.102:0/1529927342 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7feac409a330 con 0x55bfbbc25f20 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.531+0000 7feadd01f740 10 client.16110 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.531+0000 7feadd01f740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.531+0000 7feadd01f740 20 client.16110 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.531+0000 7feadd01f740 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55bfbbc40fd0 0x55bfbbc433c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.531+0000 7feadd01f740 4 client.16110 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.531+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfbbb976d0 con 0x55bfbbc40fd0 2024-04-25T01:59:56.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.532+0000 7fead2a00640 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55bfbbc40fd0 0x55bfbbc433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.533+0000 7fead2a00640 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55bfbbc40fd0 0x55bfbbc433c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3347e0e09b1553d server_cookie=1041014a7673d15c in_seq=0 out_seq=0 2024-04-25T01:59:56.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.533+0000 7feabcc00640 10 client.16110 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:56.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.533+0000 7feabcc00640 1 -- 172.21.15.102:0/1529927342 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfbbb976d0 con 0x55bfbbc40fd0 2024-04-25T01:59:56.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.534+0000 7feabcc00640 10 client.16110 handle_command_reply: tid=0 2024-04-25T01:59:56.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.534+0000 7feadd01f740 1 client.16110 shutdown 2024-04-25T01:59:56.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.534+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 msgr2=0x55bfbbc2df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.534+0000 7feadd01f740 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc2df80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7feac409bd20 tx=0x7feac409f8a0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.534+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 shutdown_connections 2024-04-25T01:59:56.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.534+0000 7feadd01f740 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55bfbbc40fd0 0x55bfbbc433c0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.535+0000 7feadd01f740 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 0x55bfbbc2ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.535+0000 7feadd01f740 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.535+0000 7feadd01f740 1 --2- 172.21.15.102:0/1529927342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc2df80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.535+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 >> 172.21.15.102:0/1529927342 conn(0x55bfbbc22240 msgr2=0x55bfbbc28040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:56.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.535+0000 7feabcc00640 0 client.16110 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:56.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.535+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 shutdown_connections 2024-04-25T01:59:56.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.535+0000 7feadd01f740 1 -- 172.21.15.102:0/1529927342 wait complete. 2024-04-25T01:59:56.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.535+0000 7feadd01f740 20 client.16110 trim_cache size 0 max 16384 2024-04-25T01:59:56.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.565+0000 7feadd01f740 1 Processor -- start 2024-04-25T01:59:56.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.565+0000 7feadd01f740 1 -- start start 2024-04-25T01:59:56.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7feadd01f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc28680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7feadd01f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7feadd01f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 0x55bfbbc2b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7feadd01f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fead41b62a0 con 0x55bfbbc25f20 2024-04-25T01:59:56.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7feadd01f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fead41b3b10 con 0x55bfbbc26840 2024-04-25T01:59:56.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7feadd01f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fead41b1960 con 0x55bfbbc30f10 2024-04-25T01:59:56.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7fead3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc28680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7fead3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc28680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54456/0 (socket says 172.21.15.102:54456) 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7fead3400640 1 -- 172.21.15.102:0/926199976 learned_addr learned my addr 172.21.15.102:0/926199976 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7fead2a00640 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.566+0000 7fead3e00640 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 0x55bfbbc2b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.567+0000 7fead3400640 1 -- 172.21.15.102:0/926199976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 msgr2=0x55bfbbc2b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.567+0000 7fead3400640 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 0x55bfbbc2b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.567+0000 7fead3400640 1 -- 172.21.15.102:0/926199976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 msgr2=0x55bfbbc2e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.567+0000 7fead3400640 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.567+0000 7fead3400640 1 -- 172.21.15.102:0/926199976 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfbbc20750 con 0x55bfbbc25f20 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.567+0000 7fead2a00640 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:56.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.567+0000 7fead3400640 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc28680 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7feac409b400 tx=0x7feac4004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4b3a6476a0ae40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.568+0000 7feabf400640 1 -- 172.21.15.102:0/926199976 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40b4040 con 0x55bfbbc25f20 2024-04-25T01:59:56.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.568+0000 7feabf400640 1 -- 172.21.15.102:0/926199976 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac409a8c0 con 0x55bfbbc25f20 2024-04-25T01:59:56.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.568+0000 7feabf400640 1 -- 172.21.15.102:0/926199976 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac40a0b70 con 0x55bfbbc25f20 2024-04-25T01:59:56.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.569+0000 7feadd01f740 1 -- 172.21.15.102:0/926199976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 msgr2=0x55bfbbc28680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.569+0000 7feadd01f740 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc28680 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7feac409b400 tx=0x7feac4004610 comp rx=0 tx=0).stop 2024-04-25T01:59:56.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.570+0000 7feadd01f740 1 -- 172.21.15.102:0/926199976 shutdown_connections 2024-04-25T01:59:56.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.570+0000 7feadd01f740 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc30f10 0x55bfbbc2b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.570+0000 7feadd01f740 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc26840 0x55bfbbc2e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.570+0000 7feadd01f740 1 --2- 172.21.15.102:0/926199976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc28680 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.570+0000 7feadd01f740 1 -- 172.21.15.102:0/926199976 >> 172.21.15.102:0/926199976 conn(0x55bfbbc22240 msgr2=0x55bfbbc2df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:56.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.570+0000 7feadd01f740 1 -- 172.21.15.102:0/926199976 shutdown_connections 2024-04-25T01:59:56.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.571+0000 7feadd01f740 1 -- 172.21.15.102:0/926199976 wait complete. 2024-04-25T01:59:56.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.572+0000 7feadd01f740 1 -- start start 2024-04-25T01:59:56.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.572+0000 7feadd01f740 1 Processor -- start 2024-04-25T01:59:56.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.574+0000 7feadd01f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc46140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.574+0000 7feadd01f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc46680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.574+0000 7feadd01f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc30f10 0x55bfbbc446c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.574+0000 7feadd01f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fead41b1fd0 con 0x55bfbbc25f20 2024-04-25T01:59:56.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.574+0000 7feadd01f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fead41b3440 con 0x55bfbbc30f10 2024-04-25T01:59:56.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.574+0000 7feadd01f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fead41b26a0 con 0x55bfbbc26840 2024-04-25T01:59:56.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.574+0000 7feadd01f740 10 client.0 _ll_register_callbacks cb 0x7fead41a3da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:56.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.575+0000 7fead2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc46680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.575+0000 7fead2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc46680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38244/0 (socket says 172.21.15.102:38244) 2024-04-25T01:59:56.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.577+0000 7fead2a00640 1 -- 172.21.15.102:0/3707542087 learned_addr learned my addr 172.21.15.102:0/3707542087 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:56.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.577+0000 7fead3e00640 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc30f10 0x55bfbbc446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.576+0000 7fead3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc46140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.577+0000 7fead2a00640 1 -- 172.21.15.102:0/3707542087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc30f10 msgr2=0x55bfbbc446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.577+0000 7fead2a00640 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc30f10 0x55bfbbc446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.577+0000 7fead2a00640 1 -- 172.21.15.102:0/3707542087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 msgr2=0x55bfbbc46140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.578+0000 7fead2a00640 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc46140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.578+0000 7fead2a00640 1 -- 172.21.15.102:0/3707542087 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfbbc44db0 con 0x55bfbbc26840 2024-04-25T01:59:56.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.578+0000 7fead3400640 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc46140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:56.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.578+0000 7feabcc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:56.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.578+0000 7fead2a00640 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc46680 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7feac0000c00 tx=0x7feac0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc93c0971e470783 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.578+0000 7feabcc00640 1 -- 172.21.15.102:0/3707542087 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0044070 con 0x55bfbbc26840 2024-04-25T01:59:56.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.579+0000 7feabcc00640 1 -- 172.21.15.102:0/3707542087 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7feac0030580 con 0x55bfbbc26840 2024-04-25T01:59:56.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.579+0000 7feadd01f740 1 -- 172.21.15.102:0/3707542087 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feac409a8c0 con 0x55bfbbc26840 2024-04-25T01:59:56.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.582+0000 7feabcc00640 1 -- 172.21.15.102:0/3707542087 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feac004e050 con 0x55bfbbc26840 2024-04-25T01:59:56.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.582+0000 7feadd01f740 10 client.26518 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:56.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.582+0000 7feadd01f740 1 -- 172.21.15.102:0/3707542087 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bfbbca7d20 con 0x55bfbbc26840 2024-04-25T01:59:56.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.583+0000 7feabcc00640 1 -- 172.21.15.102:0/3707542087 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feac0038040 con 0x55bfbbc26840 2024-04-25T01:59:56.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.583+0000 7feabcc00640 1 -- 172.21.15.102:0/3707542087 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7feac0004920 con 0x55bfbbc26840 2024-04-25T01:59:56.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.584+0000 7feadd01f740 10 client.26518 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:56.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.584+0000 7feadd01f740 10 client.26518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:56.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.584+0000 7feadd01f740 20 client.26518 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:56.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.584+0000 7feadd01f740 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55bfbbca8520 0x55bfbbcaa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.584+0000 7feadd01f740 4 client.26518 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T01:59:56.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.584+0000 7feadd01f740 1 -- 172.21.15.102:0/3707542087 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7feac409a8c0 con 0x55bfbbca8520 2024-04-25T01:59:56.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.584+0000 7fead3400640 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55bfbbca8520 0x55bfbbcaa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.585+0000 7fead3400640 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55bfbbca8520 0x55bfbbcaa940 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1455d0844a826a50 server_cookie=4cdd906b98f41704 in_seq=0 out_seq=0 2024-04-25T01:59:56.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.585+0000 7feabcc00640 10 client.26518 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.590+0000 7feabcc00640 1 -- 172.21.15.102:0/3707542087 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7feac409a8c0 con 0x55bfbbca8520 2024-04-25T01:59:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.590+0000 7feabcc00640 10 client.26518 handle_command_reply: tid=0 2024-04-25T01:59:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.590+0000 7feadd01f740 1 client.26518 shutdown 2024-04-25T01:59:56.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.592+0000 7feadd01f740 1 -- 172.21.15.102:0/3707542087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 msgr2=0x55bfbbc46680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.592+0000 7feadd01f740 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc46680 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7feac0000c00 tx=0x7feac0004040 comp rx=0 tx=0).stop 2024-04-25T01:59:56.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.593+0000 7feadd01f740 1 -- 172.21.15.102:0/3707542087 shutdown_connections 2024-04-25T01:59:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.593+0000 7feadd01f740 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55bfbbca8520 0x55bfbbcaa940 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.593+0000 7feadd01f740 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55bfbbc30f10 0x55bfbbc446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.593+0000 7feadd01f740 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bfbbc26840 0x55bfbbc46680 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.593+0000 7feadd01f740 1 --2- 172.21.15.102:0/3707542087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bfbbc25f20 0x55bfbbc46140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.593+0000 7feadd01f740 1 -- 172.21.15.102:0/3707542087 >> 172.21.15.102:0/3707542087 conn(0x55bfbbc22240 msgr2=0x55bfbbc213b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.593+0000 7feabcc00640 0 client.26518 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.593+0000 7feadd01f740 1 -- 172.21.15.102:0/3707542087 shutdown_connections 2024-04-25T01:59:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.594+0000 7feadd01f740 1 -- 172.21.15.102:0/3707542087 wait complete. 2024-04-25T01:59:56.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.594+0000 7feadd01f740 20 client.26518 trim_cache size 0 max 16384 2024-04-25T01:59:56.598 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:59:56.598 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T01:59:56.599 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "b349f264-e097-4884-ae8b-c89419722777", 2024-04-25T01:59:56.599 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T01:59:56.599 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:59:56.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.596+0000 7fead8c00640 1 -- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feab80a2dc0 msgr2=0x7feab80a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.596+0000 7fead8c00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feab80a2dc0 0x7feab80a5280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feacc009fd0 tx=0x7feacc004ab0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.596+0000 7fead8c00640 1 -- 172.21.15.102:0/2999652428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 msgr2=0x7fead41a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.596+0000 7fead8c00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 0x7fead41a3140 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7feac400d750 tx=0x7feac400dc10 comp rx=0 tx=0).stop 2024-04-25T01:59:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.596+0000 7fead8c00640 1 -- 172.21.15.102:0/2999652428 shutdown_connections 2024-04-25T01:59:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.597+0000 7fead8c00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feab80a2dc0 0x7feab80a5280 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.597+0000 7fead8c00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fead4107680 0x7fead410b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.597+0000 7fead8c00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fead4101fa0 0x7fead41a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.597+0000 7fead8c00640 1 --2- 172.21.15.102:0/2999652428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fead41015e0 0x7fead41a3140 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.597+0000 7fead8c00640 1 -- 172.21.15.102:0/2999652428 >> 172.21.15.102:0/2999652428 conn(0x7fead40fd110 msgr2=0x7fead40fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.597+0000 7fead8c00640 1 -- 172.21.15.102:0/2999652428 shutdown_connections 2024-04-25T01:59:56.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.598+0000 7fead8c00640 1 -- 172.21.15.102:0/2999652428 wait complete. 2024-04-25T01:59:56.621 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:56 smithi102 ceph-mon[27796]: pgmap v143: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.9 MiB/s wr, 682 op/s 2024-04-25T01:59:56.627 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T01:59:56.628 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T01:59:56.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:56 smithi110 ceph-mon[32848]: pgmap v143: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.9 MiB/s wr, 682 op/s 2024-04-25T01:59:56.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:56 smithi193 ceph-mon[32981]: pgmap v143: 97 pgs: 97 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.9 MiB/s wr, 682 op/s 2024-04-25T01:59:56.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.913+0000 7f12ad000640 1 -- 172.21.15.102:0/3698079285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a80ffa10 msgr2=0x7f12a80ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.913+0000 7f12ad000640 1 --2- 172.21.15.102:0/3698079285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a80ffa10 0x7f12a80ffe90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f129800a910 tx=0x7f129802a610 comp rx=0 tx=0).stop 2024-04-25T01:59:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.914+0000 7f12ad000640 1 -- 172.21.15.102:0/3698079285 shutdown_connections 2024-04-25T01:59:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.914+0000 7f12ad000640 1 --2- 172.21.15.102:0/3698079285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a81003d0 0x7f12a810c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.914+0000 7f12ad000640 1 --2- 172.21.15.102:0/3698079285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a80ffa10 0x7f12a80ffe90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.914+0000 7f12ad000640 1 --2- 172.21.15.102:0/3698079285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12a81059d0 0x7f12a8105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.914+0000 7f12ad000640 1 -- 172.21.15.102:0/3698079285 >> 172.21.15.102:0/3698079285 conn(0x7f12a80767a0 msgr2=0x7f12a80fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.914+0000 7f12ad000640 1 -- 172.21.15.102:0/3698079285 shutdown_connections 2024-04-25T01:59:56.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.914+0000 7f12ad000640 1 -- 172.21.15.102:0/3698079285 wait complete. 2024-04-25T01:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.916+0000 7f12ad000640 1 Processor -- start 2024-04-25T01:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.916+0000 7f12ad000640 1 -- start start 2024-04-25T01:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.916+0000 7f12ad000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12a80ffa10 0x7f12a810c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.916+0000 7f12ad000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a81003d0 0x7f12a8109830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.916+0000 7f12ad000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a81059d0 0x7f12a8109dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.916+0000 7f12ad000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f12a8113da0 con 0x7f12a80ffa10 2024-04-25T01:59:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.916+0000 7f12ad000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f12a8113c20 con 0x7f12a81059d0 2024-04-25T01:59:56.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.916+0000 7f12ad000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f12a8113f20 con 0x7f12a81003d0 2024-04-25T01:59:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.917+0000 7f12a6800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a81059d0 0x7f12a8109dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.917+0000 7f129fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a81003d0 0x7f12a8109830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.917+0000 7f12a6800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a81059d0 0x7f12a8109dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51124/0 (socket says 172.21.15.102:51124) 2024-04-25T01:59:56.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.917+0000 7f12a6800640 1 -- 172.21.15.102:0/3388598726 learned_addr learned my addr 172.21.15.102:0/3388598726 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.917+0000 7f12a5e00640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12a80ffa10 0x7f12a810c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.918+0000 7f12a6800640 1 -- 172.21.15.102:0/3388598726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a81003d0 msgr2=0x7f12a8109830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.918+0000 7f12a6800640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a81003d0 0x7f12a8109830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.918+0000 7f12a6800640 1 -- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12a80ffa10 msgr2=0x7f12a810c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.918+0000 7f12a6800640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12a80ffa10 0x7f12a810c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.918+0000 7f12a6800640 1 -- 172.21.15.102:0/3388598726 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a81a02c0 con 0x7f12a81059d0 2024-04-25T01:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.918+0000 7f129fe00640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a81003d0 0x7f12a8109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.918+0000 7f12a5e00640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12a80ffa10 0x7f12a810c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.919+0000 7f12a6800640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a81059d0 0x7f12a8109dd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f12a00099e0 tx=0x7f12a0009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.919+0000 7f129ea00640 1 -- 172.21.15.102:0/3388598726 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0004b20 con 0x7f12a81059d0 2024-04-25T01:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.919+0000 7f129ea00640 1 -- 172.21.15.102:0/3388598726 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a000ed20 con 0x7f12a81059d0 2024-04-25T01:59:56.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.919+0000 7f12ad000640 1 -- 172.21.15.102:0/3388598726 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a81a05b0 con 0x7f12a81059d0 2024-04-25T01:59:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.919+0000 7f129ea00640 1 -- 172.21.15.102:0/3388598726 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a00057e0 con 0x7f12a81059d0 2024-04-25T01:59:56.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.919+0000 7f12ad000640 1 -- 172.21.15.102:0/3388598726 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a81a0ad0 con 0x7f12a81059d0 2024-04-25T01:59:56.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.924+0000 7f129ea00640 1 -- 172.21.15.102:0/3388598726 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f12a001b020 con 0x7f12a81059d0 2024-04-25T01:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.928+0000 7f129ea00640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f128c0a2ed0 0x7f128c0a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.928+0000 7f129ea00640 1 -- 172.21.15.102:0/3388598726 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f12a00cef00 con 0x7f12a81059d0 2024-04-25T01:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.928+0000 7f12a5e00640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f128c0a2ed0 0x7f128c0a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.929+0000 7f12afdee740 1 Processor -- start 2024-04-25T01:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.929+0000 7f12a5e00640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f128c0a2ed0 0x7f128c0a5390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1290009bb0 tx=0x7f1290005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.929+0000 7f12afdee740 1 -- start start 2024-04-25T01:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12afdee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e270a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12afdee740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e2714ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12afdee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 0x55f9e27173e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12a5e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e270a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12afdee740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f9e266e560 con 0x55f9e270a920 2024-04-25T01:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12afdee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f9e2616930 con 0x55f9e270a000 2024-04-25T01:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12afdee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f9e266a520 con 0x55f9e2714ff0 2024-04-25T01:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12a5e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e270a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51134/0 (socket says 172.21.15.102:51134) 2024-04-25T01:59:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12a5e00640 1 -- 172.21.15.102:0/1893135064 learned_addr learned my addr 172.21.15.102:0/1893135064 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.930+0000 7f12a6800640 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 0x55f9e27173e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.931+0000 7f12a5e00640 1 -- 172.21.15.102:0/1893135064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 msgr2=0x55f9e27173e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.931+0000 7f12a5e00640 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 0x55f9e27173e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.931+0000 7f12a5e00640 1 -- 172.21.15.102:0/1893135064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 msgr2=0x55f9e2714ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.931+0000 7f129fe00640 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e2714ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.932+0000 7f12a5e00640 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e2714ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.932+0000 7f12a5e00640 1 -- 172.21.15.102:0/1893135064 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9e244c7d0 con 0x55f9e270a000 2024-04-25T01:59:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.932+0000 7f12a6800640 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 0x55f9e27173e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T01:59:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.932+0000 7f129fe00640 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e2714ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:56.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.934+0000 7f12a5e00640 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e270a3e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1290009990 tx=0x7f1290014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93e97c44bc179e36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.935+0000 7f1297400640 1 -- 172.21.15.102:0/1893135064 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129001f620 con 0x55f9e270a000 2024-04-25T01:59:56.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.935+0000 7f1297400640 1 -- 172.21.15.102:0/1893135064 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f129001fca0 con 0x55f9e270a000 2024-04-25T01:59:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.935+0000 7f1297400640 1 -- 172.21.15.102:0/1893135064 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129001d3b0 con 0x55f9e270a000 2024-04-25T01:59:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.936+0000 7f12afdee740 1 -- 172.21.15.102:0/1893135064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 msgr2=0x55f9e270a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.936+0000 7f12afdee740 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e270a3e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1290009990 tx=0x7f1290014000 comp rx=0 tx=0).stop 2024-04-25T01:59:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.937+0000 7f12afdee740 1 -- 172.21.15.102:0/1893135064 shutdown_connections 2024-04-25T01:59:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.937+0000 7f12afdee740 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 0x55f9e27173e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.937+0000 7f12afdee740 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e2714ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.937+0000 7f12afdee740 1 --2- 172.21.15.102:0/1893135064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e270a3e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.937+0000 7f12afdee740 1 -- 172.21.15.102:0/1893135064 >> 172.21.15.102:0/1893135064 conn(0x55f9e2706320 msgr2=0x55f9e2706700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.937+0000 7f12afdee740 1 -- 172.21.15.102:0/1893135064 shutdown_connections 2024-04-25T01:59:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.938+0000 7f12afdee740 1 -- 172.21.15.102:0/1893135064 wait complete. 2024-04-25T01:59:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.938+0000 7f12afdee740 1 -- start start 2024-04-25T01:59:56.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.938+0000 7f12afdee740 1 Processor -- start 2024-04-25T01:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.939+0000 7f12afdee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a000 0x55f9e2711f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.939+0000 7f12afdee740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e27124c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.939+0000 7f12afdee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e2714ff0 0x55f9e2712a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.939+0000 7f12afdee740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f9e2717de0 con 0x55f9e270a920 2024-04-25T01:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.939+0000 7f12afdee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f9e2629d90 con 0x55f9e2714ff0 2024-04-25T01:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.939+0000 7f12afdee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f9e2717f60 con 0x55f9e270a000 2024-04-25T01:59:56.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.939+0000 7f12afdee740 10 client.0 _ll_register_callbacks cb 0x55f9e2629780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.940+0000 7f12a6800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e2714ff0 0x55f9e2712a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.940+0000 7f12a5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a000 0x55f9e2711f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.940+0000 7f12a5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a000 0x55f9e2711f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38280/0 (socket says 172.21.15.102:38280) 2024-04-25T01:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.940+0000 7f12a5e00640 1 -- 172.21.15.102:0/1648661418 learned_addr learned my addr 172.21.15.102:0/1648661418 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.941+0000 7f12a5e00640 1 -- 172.21.15.102:0/1648661418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e2714ff0 msgr2=0x55f9e2712a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.941+0000 7f12a5e00640 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e2714ff0 0x55f9e2712a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.941+0000 7f12a5e00640 1 -- 172.21.15.102:0/1648661418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 msgr2=0x55f9e27124c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T01:59:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.942+0000 7f129fe00640 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e27124c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.943+0000 7f12a5e00640 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e27124c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.943+0000 7f12a5e00640 1 -- 172.21.15.102:0/1648661418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9e23823f0 con 0x55f9e270a000 2024-04-25T01:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.943+0000 7f12a6800640 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e2714ff0 0x55f9e2712a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T01:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.943+0000 7f129fe00640 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e27124c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.943+0000 7f12a5e00640 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a000 0x55f9e2711f80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1290027990 tx=0x7f12900279f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a54d492054e920d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.943+0000 7f128be00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T01:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.944+0000 7f128be00640 1 -- 172.21.15.102:0/1648661418 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129001f420 con 0x55f9e270a000 2024-04-25T01:59:56.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.944+0000 7f128be00640 1 -- 172.21.15.102:0/1648661418 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f129001f5c0 con 0x55f9e270a000 2024-04-25T01:59:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.944+0000 7f12afdee740 1 -- 172.21.15.102:0/1648661418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f129001fca0 con 0x55f9e270a000 2024-04-25T01:59:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.944+0000 7f128be00640 1 -- 172.21.15.102:0/1648661418 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f129001f7c0 con 0x55f9e270a000 2024-04-25T01:59:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.944+0000 7f12afdee740 10 client.26530 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:56.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.944+0000 7f12afdee740 1 -- 172.21.15.102:0/1648661418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f9e23420f0 con 0x55f9e270a000 2024-04-25T01:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.945+0000 7f128be00640 1 -- 172.21.15.102:0/1648661418 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f129001fe50 con 0x55f9e270a000 2024-04-25T01:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.945+0000 7f128be00640 1 -- 172.21.15.102:0/1648661418 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f1290030070 con 0x55f9e270a000 2024-04-25T01:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.945+0000 7f12afdee740 10 client.26530 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.945+0000 7f12afdee740 10 client.26530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.945+0000 7f12afdee740 20 client.26530 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.945+0000 7f12afdee740 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f9e27251d0 0x55f9e27275c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.945+0000 7f12afdee740 4 client.26530 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T01:59:56.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.945+0000 7f12afdee740 1 -- 172.21.15.102:0/1648661418 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f129001fca0 con 0x55f9e27251d0 2024-04-25T01:59:56.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.946+0000 7f129fe00640 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f9e27251d0 0x55f9e27275c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:56.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.948+0000 7f129fe00640 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f9e27251d0 0x55f9e27275c0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c384dde701f8b2d1 server_cookie=cd33f7ade8c70546 in_seq=0 out_seq=0 2024-04-25T01:59:56.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.948+0000 7f128be00640 10 client.26530 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.949+0000 7f128be00640 1 -- 172.21.15.102:0/1648661418 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f129001fca0 con 0x55f9e27251d0 2024-04-25T01:59:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.949+0000 7f128be00640 10 client.26530 handle_command_reply: tid=0 2024-04-25T01:59:56.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.949+0000 7f12afdee740 1 client.26530 shutdown 2024-04-25T01:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.950+0000 7f12afdee740 1 -- 172.21.15.102:0/1648661418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a000 msgr2=0x55f9e2711f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.950+0000 7f12afdee740 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a000 0x55f9e2711f80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1290027990 tx=0x7f12900279f0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.950+0000 7f12afdee740 1 -- 172.21.15.102:0/1648661418 shutdown_connections 2024-04-25T01:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.950+0000 7f12afdee740 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f9e27251d0 0x55f9e27275c0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.950+0000 7f12afdee740 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e2714ff0 0x55f9e2712a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.950+0000 7f12afdee740 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a920 0x55f9e27124c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.950+0000 7f12afdee740 1 --2- 172.21.15.102:0/1648661418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a000 0x55f9e2711f80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:56.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.950+0000 7f12afdee740 1 -- 172.21.15.102:0/1648661418 >> 172.21.15.102:0/1648661418 conn(0x55f9e2706320 msgr2=0x55f9e270c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:56.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.950+0000 7f128be00640 0 client.26530 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:56.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.951+0000 7f12afdee740 1 -- 172.21.15.102:0/1648661418 shutdown_connections 2024-04-25T01:59:56.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.951+0000 7f12afdee740 1 -- 172.21.15.102:0/1648661418 wait complete. 2024-04-25T01:59:56.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.951+0000 7f12afdee740 20 client.26530 trim_cache size 0 max 16384 2024-04-25T01:59:57.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.997+0000 7f12afdee740 1 Processor -- start 2024-04-25T01:59:57.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.997+0000 7f12afdee740 1 -- start start 2024-04-25T01:59:57.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.999+0000 7f12afdee740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a000 0x55f9e2706700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:57.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.999+0000 7f12afdee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a920 0x55f9e2720ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:57.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.999+0000 7f12afdee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 0x55f9e27110e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:57.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.999+0000 7f12afdee740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f12a81beba0 con 0x55f9e270a000 2024-04-25T01:59:57.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.999+0000 7f12afdee740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f12a81bc410 con 0x55f9e270a920 2024-04-25T01:59:57.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.999+0000 7f12afdee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f12a81ba260 con 0x55f9e2714ff0 2024-04-25T01:59:57.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:56.999+0000 7f129fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a920 0x55f9e2720ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:57.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.000+0000 7f12a6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 0x55f9e27110e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:57.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.000+0000 7f129fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a920 0x55f9e2720ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51150/0 (socket says 172.21.15.102:51150) 2024-04-25T01:59:57.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.000+0000 7f129fe00640 1 -- 172.21.15.102:0/65187054 learned_addr learned my addr 172.21.15.102:0/65187054 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:57.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.000+0000 7f12a5e00640 1 --2- 172.21.15.102:0/65187054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a000 0x55f9e2706700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:57.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.000+0000 7f129fe00640 1 -- 172.21.15.102:0/65187054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 msgr2=0x55f9e27110e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:57.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.000+0000 7f129fe00640 1 --2- 172.21.15.102:0/65187054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 0x55f9e27110e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.000+0000 7f129fe00640 1 -- 172.21.15.102:0/65187054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a000 msgr2=0x55f9e2706700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:57.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.000+0000 7f129fe00640 1 --2- 172.21.15.102:0/65187054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a000 0x55f9e2706700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.000+0000 7f129fe00640 1 -- 172.21.15.102:0/65187054 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9e27104d0 con 0x55f9e270a920 2024-04-25T01:59:57.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.001+0000 7f129fe00640 1 --2- 172.21.15.102:0/65187054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a920 0x55f9e2720ca0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1298005ab0 tx=0x7f1298005560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=531fe15357863880 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:57.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.001+0000 7f1296a00640 1 -- 172.21.15.102:0/65187054 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298035040 con 0x55f9e270a920 2024-04-25T01:59:57.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.003+0000 7f1296a00640 1 -- 172.21.15.102:0/65187054 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1298048070 con 0x55f9e270a920 2024-04-25T01:59:57.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.003+0000 7f1296a00640 1 -- 172.21.15.102:0/65187054 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12980035f0 con 0x55f9e270a920 2024-04-25T01:59:57.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.004+0000 7f12afdee740 1 -- 172.21.15.102:0/65187054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a920 msgr2=0x55f9e2720ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:57.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.004+0000 7f12afdee740 1 --2- 172.21.15.102:0/65187054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a920 0x55f9e2720ca0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1298005ab0 tx=0x7f1298005560 comp rx=0 tx=0).stop 2024-04-25T01:59:57.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.004+0000 7f12afdee740 1 -- 172.21.15.102:0/65187054 shutdown_connections 2024-04-25T01:59:57.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.004+0000 7f12afdee740 1 --2- 172.21.15.102:0/65187054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e2714ff0 0x55f9e27110e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.004+0000 7f12afdee740 1 --2- 172.21.15.102:0/65187054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a920 0x55f9e2720ca0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.004+0000 7f12afdee740 1 --2- 172.21.15.102:0/65187054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e270a000 0x55f9e2706700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.004+0000 7f12afdee740 1 -- 172.21.15.102:0/65187054 >> 172.21.15.102:0/65187054 conn(0x55f9e2706320 msgr2=0x55f9e2704bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:57.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.004+0000 7f12afdee740 1 -- 172.21.15.102:0/65187054 shutdown_connections 2024-04-25T01:59:57.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.004+0000 7f12afdee740 1 -- 172.21.15.102:0/65187054 wait complete. 2024-04-25T01:59:57.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.005+0000 7f12afdee740 1 -- start start 2024-04-25T01:59:57.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.005+0000 7f12afdee740 1 Processor -- start 2024-04-25T01:59:57.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.006+0000 7f12afdee740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e271d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:57.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12afdee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a920 0x55f9e271d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:57.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12afdee740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e2714ff0 0x55f9e2718970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:57.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12a5e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e271d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:57.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12a5e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e271d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51160/0 (socket says 172.21.15.102:51160) 2024-04-25T01:59:57.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f12a81ba8d0 con 0x55f9e2714ff0 2024-04-25T01:59:57.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12a5e00640 1 -- 172.21.15.102:0/825840206 learned_addr learned my addr 172.21.15.102:0/825840206 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T01:59:57.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f12a81bbd40 con 0x55f9e270a000 2024-04-25T01:59:57.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f12a81bafa0 con 0x55f9e270a920 2024-04-25T01:59:57.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12afdee740 10 client.0 _ll_register_callbacks cb 0x7f12900056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T01:59:57.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.007+0000 7f12a6800640 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e2714ff0 0x55f9e2718970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:57.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.008+0000 7f129fe00640 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a920 0x55f9e271d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:57.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.008+0000 7f12a5e00640 1 -- 172.21.15.102:0/825840206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a920 msgr2=0x55f9e271d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:57.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.008+0000 7f12a5e00640 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a920 0x55f9e271d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.008+0000 7f12a5e00640 1 -- 172.21.15.102:0/825840206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e2714ff0 msgr2=0x55f9e2718970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T01:59:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.008+0000 7f12a5e00640 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e2714ff0 0x55f9e2718970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.008+0000 7f12a5e00640 1 -- 172.21.15.102:0/825840206 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9e2719170 con 0x55f9e270a000 2024-04-25T01:59:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.009+0000 7f12a6800640 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e2714ff0 0x55f9e2718970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T01:59:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.009+0000 7f128be00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T01:59:57.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.009+0000 7f12a5e00640 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e271d1c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f12900279c0 tx=0x7f1290031810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5ce675c6cbcf8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T01:59:57.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.010+0000 7f128be00640 1 -- 172.21.15.102:0/825840206 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290027680 con 0x55f9e270a000 2024-04-25T01:59:57.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.011+0000 7f128be00640 1 -- 172.21.15.102:0/825840206 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f1290027c70 con 0x55f9e270a000 2024-04-25T01:59:57.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.013+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1298048070 con 0x55f9e270a000 2024-04-25T01:59:57.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.013+0000 7f128be00640 1 -- 172.21.15.102:0/825840206 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1290049050 con 0x55f9e270a000 2024-04-25T01:59:57.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.013+0000 7f12afdee740 10 client.26573 fetch_fsmap learned FSMap version 32 2024-04-25T01:59:57.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.013+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f9e278b0f0 con 0x55f9e270a000 2024-04-25T01:59:57.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.014+0000 7f128be00640 1 -- 172.21.15.102:0/825840206 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1298048070 con 0x55f9e270a000 2024-04-25T01:59:57.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.014+0000 7f128be00640 1 -- 172.21.15.102:0/825840206 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f1290030070 con 0x55f9e270a000 2024-04-25T01:59:57.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.014+0000 7f12afdee740 10 client.26573 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T01:59:57.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.014+0000 7f12afdee740 10 client.26573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T01:59:57.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.014+0000 7f12afdee740 20 client.26573 populate_metadata read hostname 'smithi102' 2024-04-25T01:59:57.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.014+0000 7f12afdee740 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f9e278b8a0 0x55f9e278dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T01:59:57.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.014+0000 7f12afdee740 4 client.26573 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T01:59:57.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.014+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f9e2681d90 con 0x55f9e278b8a0 2024-04-25T01:59:57.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.015+0000 7f129fe00640 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f9e278b8a0 0x55f9e278dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.015+0000 7f129fe00640 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f9e278b8a0 0x55f9e278dcc0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e71443d22a93cfde server_cookie=677ba9e531b0fd96 in_seq=0 out_seq=0 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.016+0000 7f128be00640 10 client.26573 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.016+0000 7f128be00640 1 -- 172.21.15.102:0/825840206 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f9e2681d90 con 0x55f9e278b8a0 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.016+0000 7f128be00640 10 client.26573 handle_command_reply: tid=0 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.016+0000 7f12afdee740 1 client.26573 shutdown 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.016+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 msgr2=0x55f9e271d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.016+0000 7f12afdee740 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e271d1c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f12900279c0 tx=0x7f1290031810 comp rx=0 tx=0).stop 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.016+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 shutdown_connections 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.016+0000 7f12afdee740 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f9e278b8a0 0x55f9e278dcc0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.017+0000 7f12afdee740 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f9e2714ff0 0x55f9e2718970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.017+0000 7f12afdee740 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9e270a920 0x55f9e271d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.017+0000 7f12afdee740 1 --2- 172.21.15.102:0/825840206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f9e270a000 0x55f9e271d1c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.017+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 >> 172.21.15.102:0/825840206 conn(0x55f9e2706320 msgr2=0x55f9e2711f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:57.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.017+0000 7f128be00640 0 client.26573 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T01:59:57.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.017+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 shutdown_connections 2024-04-25T01:59:57.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.017+0000 7f12afdee740 1 -- 172.21.15.102:0/825840206 wait complete. 2024-04-25T01:59:57.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.017+0000 7f12afdee740 20 client.26573 trim_cache size 0 max 16384 2024-04-25T01:59:57.022 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T01:59:57.022 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T01:59:57.022 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T01:59:57.022 INFO:teuthology.orchestra.run.smithi102.stdout: "b349f264-e097-4884-ae8b-c89419722777": { 2024-04-25T01:59:57.022 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T01:59:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "b349f264-e097-4884-ae8b-c89419722777", 2024-04-25T01:59:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T01:59:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:59:57.023 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T01:59:57.023 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T01:59:57.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 -- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f128c0a2ed0 msgr2=0x7f128c0a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:57.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f128c0a2ed0 0x7f128c0a5390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1290009bb0 tx=0x7f1290005f90 comp rx=0 tx=0).stop 2024-04-25T01:59:57.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 -- 172.21.15.102:0/3388598726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a81059d0 msgr2=0x7f12a8109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T01:59:57.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a81059d0 0x7f12a8109dd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f12a00099e0 tx=0x7f12a0009ea0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 -- 172.21.15.102:0/3388598726 shutdown_connections 2024-04-25T01:59:57.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f128c0a2ed0 0x7f128c0a5390 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12a81059d0 0x7f12a8109dd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a81003d0 0x7f12a8109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 --2- 172.21.15.102:0/3388598726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12a80ffa10 0x7f12a810c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T01:59:57.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.020+0000 7f12ad000640 1 -- 172.21.15.102:0/3388598726 >> 172.21.15.102:0/3388598726 conn(0x7f12a80767a0 msgr2=0x7f12a81027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T01:59:57.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.021+0000 7f12ad000640 1 -- 172.21.15.102:0/3388598726 shutdown_connections 2024-04-25T01:59:57.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T01:59:57.021+0000 7f12ad000640 1 -- 172.21.15.102:0/3388598726 wait complete. 2024-04-25T01:59:57.050 INFO:tasks.cephfs.filesystem:scrub status for tag:b349f264-e097-4884-ae8b-c89419722777 - {'path': '/', 'tag': 'b349f264-e097-4884-ae8b-c89419722777', 'options': 'recursive,force'} 2024-04-25T01:59:57.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:57 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T01:59:57.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:57 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T01:59:57.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:57.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:57.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:57.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:57.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T01:59:57.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:57.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:57 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T01:59:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 245.0 s, 2523.2 tps, lat 40.077 ms stddev 47.165 2024-04-25T01:59:58.740 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 01:59:58 smithi110 ceph-mon[32848]: pgmap v144: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 13 MiB/s wr, 1.02k op/s 2024-04-25T01:59:58.758 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 01:59:58 smithi193 ceph-mon[32981]: pgmap v144: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 13 MiB/s wr, 1.02k op/s 2024-04-25T01:59:58.926 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 01:59:58 smithi102 ceph-mon[27796]: pgmap v144: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 13 MiB/s wr, 1.02k op/s 2024-04-25T02:00:01.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:00 smithi102 ceph-mon[27796]: pgmap v145: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.5 MiB/s wr, 710 op/s 2024-04-25T02:00:01.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:00 smithi102 ceph-mon[27796]: overall HEALTH_OK 2024-04-25T02:00:01.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:00 smithi110 ceph-mon[32848]: pgmap v145: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.5 MiB/s wr, 710 op/s 2024-04-25T02:00:01.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:00 smithi110 ceph-mon[32848]: overall HEALTH_OK 2024-04-25T02:00:01.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:00 smithi193 ceph-mon[32981]: pgmap v145: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.5 MiB/s wr, 710 op/s 2024-04-25T02:00:01.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:00 smithi193 ceph-mon[32981]: overall HEALTH_OK 2024-04-25T02:00:03.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 250.0 s, 1353.2 tps, lat 71.356 ms stddev 132.336 2024-04-25T02:00:03.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:02 smithi102 ceph-mon[27796]: pgmap v146: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 11 MiB/s wr, 986 op/s 2024-04-25T02:00:03.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:02 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:00:03.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:02 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:00:03.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:02 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:00:03.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:02 smithi110 ceph-mon[32848]: pgmap v146: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 11 MiB/s wr, 986 op/s 2024-04-25T02:00:03.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:02 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:00:03.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:02 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:00:03.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:02 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:00:03.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:02 smithi193 ceph-mon[32981]: pgmap v146: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 11 MiB/s wr, 986 op/s 2024-04-25T02:00:03.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:02 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:00:03.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:02 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:00:03.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:02 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:00:04.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.949+0000 7f7abc200640 1 -- 172.21.15.102:0/1680655896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 msgr2=0x7f7ab4109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:04.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.949+0000 7f7abc200640 1 --2- 172.21.15.102:0/1680655896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 0x7f7ab4109d10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009900 tx=0x7f7aa802a410 comp rx=0 tx=0).stop 2024-04-25T02:00:04.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.949+0000 7f7abc200640 1 -- 172.21.15.102:0/1680655896 shutdown_connections 2024-04-25T02:00:04.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.949+0000 7f7abc200640 1 --2- 172.21.15.102:0/1680655896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ab41042e0 0x7f7ab41107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.949+0000 7f7abc200640 1 --2- 172.21.15.102:0/1680655896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab4103920 0x7f7ab4103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.949+0000 7f7abc200640 1 --2- 172.21.15.102:0/1680655896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 0x7f7ab4109d10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.949+0000 7f7abc200640 1 -- 172.21.15.102:0/1680655896 >> 172.21.15.102:0/1680655896 conn(0x7f7ab40ff430 msgr2=0x7f7ab4101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.949+0000 7f7abc200640 1 -- 172.21.15.102:0/1680655896 shutdown_connections 2024-04-25T02:00:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.949+0000 7f7abc200640 1 -- 172.21.15.102:0/1680655896 wait complete. 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.950+0000 7f7abc200640 1 Processor -- start 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7abc200640 1 -- start start 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7abc200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab4103920 0x7f7ab419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7abc200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ab41042e0 0x7f7ab419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7abc200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 0x7f7ab419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7abc200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab4113d40 con 0x7f7ab4103920 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7abc200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab4113bc0 con 0x7f7ab4109930 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7abc200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab4113ec0 con 0x7f7ab41042e0 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7ab9800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab4103920 0x7f7ab419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7aba200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 0x7f7ab419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ab41042e0 0x7f7ab419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7aba200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 0x7f7ab419e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33198/0 (socket says 172.21.15.102:33198) 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.951+0000 7f7aba200640 1 -- 172.21.15.102:0/467266137 learned_addr learned my addr 172.21.15.102:0/467266137 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7aba200640 1 -- 172.21.15.102:0/467266137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ab41042e0 msgr2=0x7f7ab419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7aba200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ab41042e0 0x7f7ab419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7aba200640 1 -- 172.21.15.102:0/467266137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab4103920 msgr2=0x7f7ab419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7aba200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab4103920 0x7f7ab419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7aba200640 1 -- 172.21.15.102:0/467266137 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab419f110 con 0x7f7ab4109930 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7aba200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 0x7f7ab419e9e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00eb10 tx=0x7f7aac00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7ab2000640 1 -- 172.21.15.102:0/467266137 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac004b20 con 0x7f7ab4109930 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7ab2000640 1 -- 172.21.15.102:0/467266137 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aac004cc0 con 0x7f7ab4109930 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7abc200640 1 -- 172.21.15.102:0/467266137 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab419f400 con 0x7f7ab4109930 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7ab2000640 1 -- 172.21.15.102:0/467266137 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac010760 con 0x7f7ab4109930 2024-04-25T02:00:04.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.952+0000 7f7abc200640 1 -- 172.21.15.102:0/467266137 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab41a6c90 con 0x7f7ab4109930 2024-04-25T02:00:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.955+0000 7f7ab2000640 1 -- 172.21.15.102:0/467266137 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7aac005020 con 0x7f7ab4109930 2024-04-25T02:00:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.955+0000 7f7abf16a740 1 Processor -- start 2024-04-25T02:00:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7abf16a740 1 -- start start 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef214e00 0x55aaef2151e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 0x55aaef21faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 0x55aaef2223d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7abf16a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55aaef178ab0 con 0x55aaef215720 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7abf16a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aaef121640 con 0x55aaef214e00 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7abf16a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55aaef174a70 con 0x55aaef21ffe0 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef214e00 0x55aaef2151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef214e00 0x55aaef2151e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33212/0 (socket says 172.21.15.102:33212) 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.956+0000 7f7ab8e00640 1 -- 172.21.15.102:0/2445947695 learned_addr learned my addr 172.21.15.102:0/2445947695 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/2445947695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 0x55aaef2223d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab9800640 1 --2- 172.21.15.102:0/2445947695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 0x55aaef21faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab2000640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7a9c0a2da0 0x7f7a9c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab8e00640 1 -- 172.21.15.102:0/2445947695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 msgr2=0x55aaef2223d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/2445947695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 0x55aaef2223d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab8e00640 1 -- 172.21.15.102:0/2445947695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 msgr2=0x55aaef21faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/2445947695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 0x55aaef21faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab8e00640 1 -- 172.21.15.102:0/2445947695 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aaeee8d210 con 0x55aaef214e00 2024-04-25T02:00:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab2000640 1 -- 172.21.15.102:0/467266137 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7aac0cfab0 con 0x7f7ab4109930 2024-04-25T02:00:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/2445947695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef214e00 0x55aaef2151e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009fd0 tx=0x7f7aa0005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b357db3a4c971390 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.957+0000 7f7aba200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7a9c0a2da0 0x7f7a9c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.958+0000 7f7aa6a00640 1 -- 172.21.15.102:0/2445947695 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0004030 con 0x55aaef214e00 2024-04-25T02:00:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.958+0000 7f7aa6a00640 1 -- 172.21.15.102:0/2445947695 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa0005500 con 0x55aaef214e00 2024-04-25T02:00:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.958+0000 7f7aa6a00640 1 -- 172.21.15.102:0/2445947695 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0004d00 con 0x55aaef214e00 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7abf16a740 1 -- 172.21.15.102:0/2445947695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef214e00 msgr2=0x55aaef2151e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7aba200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7a9c0a2da0 0x7f7a9c0a5260 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0049d0 tx=0x7f7aac005b40 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7abf16a740 1 --2- 172.21.15.102:0/2445947695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef214e00 0x55aaef2151e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009fd0 tx=0x7f7aa0005850 comp rx=0 tx=0).stop 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7abf16a740 1 -- 172.21.15.102:0/2445947695 shutdown_connections 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7abf16a740 1 --2- 172.21.15.102:0/2445947695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 0x55aaef2223d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7abf16a740 1 --2- 172.21.15.102:0/2445947695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 0x55aaef21faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7abf16a740 1 --2- 172.21.15.102:0/2445947695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef214e00 0x55aaef2151e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7abf16a740 1 -- 172.21.15.102:0/2445947695 >> 172.21.15.102:0/2445947695 conn(0x55aaef211310 msgr2=0x55aaef2116f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7abf16a740 1 -- 172.21.15.102:0/2445947695 shutdown_connections 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.959+0000 7f7abf16a740 1 -- 172.21.15.102:0/2445947695 wait complete. 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.960+0000 7f7abf16a740 1 -- start start 2024-04-25T02:00:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.960+0000 7f7abf16a740 1 Processor -- start 2024-04-25T02:00:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.960+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef21cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.960+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 0x55aaef21d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.960+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef21ffe0 0x55aaef21d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.960+0000 7f7abf16a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55aaef222c10 con 0x55aaef215720 2024-04-25T02:00:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.960+0000 7f7abf16a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aaef134990 con 0x55aaef21ffe0 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.960+0000 7f7abf16a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55aaef222d90 con 0x55aaef214e00 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.960+0000 7f7abf16a740 10 client.0 _ll_register_callbacks cb 0x55aaef136950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef21cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef21cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33076/0 (socket says 172.21.15.102:33076) 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab8e00640 1 -- 172.21.15.102:0/1281233072 learned_addr learned my addr 172.21.15.102:0/1281233072 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab9800640 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 0x55aaef21d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef21ffe0 0x55aaef21d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab9800640 1 -- 172.21.15.102:0/1281233072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 msgr2=0x55aaef21cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab9800640 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef21cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab9800640 1 -- 172.21.15.102:0/1281233072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef21ffe0 msgr2=0x55aaef21d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab9800640 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef21ffe0 0x55aaef21d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab9800640 1 -- 172.21.15.102:0/1281233072 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aaeef59620 con 0x55aaef215720 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.961+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef21cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.962+0000 7f7ab9800640 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 0x55aaef21d480 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8005940 tx=0x7f7aa80042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d271faba2843101e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.962+0000 7f7a9b400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.962+0000 7f7a9b400640 1 -- 172.21.15.102:0/1281233072 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8035040 con 0x55aaef215720 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.962+0000 7f7a9b400640 1 -- 172.21.15.102:0/1281233072 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7aa8003a00 con 0x55aaef215720 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.962+0000 7f7abf16a740 1 -- 172.21.15.102:0/1281233072 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7aa0005500 con 0x55aaef215720 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.962+0000 7f7a9b400640 1 -- 172.21.15.102:0/1281233072 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7aa8003c00 con 0x55aaef215720 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.962+0000 7f7abf16a740 10 client.16137 fetch_fsmap learned FSMap version 32 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.962+0000 7f7abf16a740 1 -- 172.21.15.102:0/1281233072 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55aaef22eae0 con 0x55aaef215720 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.963+0000 7f7a9b400640 1 -- 172.21.15.102:0/1281233072 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa0005500 con 0x55aaef215720 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.963+0000 7f7a9b400640 1 -- 172.21.15.102:0/1281233072 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f7aa804f450 con 0x55aaef215720 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.963+0000 7f7abf16a740 10 client.16137 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.963+0000 7f7abf16a740 10 client.16137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.963+0000 7f7abf16a740 20 client.16137 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:04.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.963+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aaef22ffa0 0x55aaef232390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.963+0000 7f7abf16a740 4 client.16137 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.963+0000 7f7abf16a740 1 -- 172.21.15.102:0/1281233072 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aaef185ff0 con 0x55aaef22ffa0 2024-04-25T02:00:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.963+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aaef22ffa0 0x55aaef232390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.964+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aaef22ffa0 0x55aaef232390 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6661d27da725bca server_cookie=2db22803dffa4f54 in_seq=0 out_seq=0 2024-04-25T02:00:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.964+0000 7f7a9b400640 10 client.16137 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.964+0000 7f7a9b400640 1 -- 172.21.15.102:0/1281233072 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aaef185ff0 con 0x55aaef22ffa0 2024-04-25T02:00:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.964+0000 7f7a9b400640 10 client.16137 handle_command_reply: tid=0 2024-04-25T02:00:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.964+0000 7f7abf16a740 1 client.16137 shutdown 2024-04-25T02:00:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 -- 172.21.15.102:0/1281233072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 msgr2=0x55aaef21d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 0x55aaef21d480 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8005940 tx=0x7f7aa80042e0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 -- 172.21.15.102:0/1281233072 shutdown_connections 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aaef22ffa0 0x55aaef232390 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef21ffe0 0x55aaef21d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef215720 0x55aaef21d480 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1281233072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef21cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 -- 172.21.15.102:0/1281233072 >> 172.21.15.102:0/1281233072 conn(0x55aaef211310 msgr2=0x55aaef217090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7a9b400640 0 client.16137 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 -- 172.21.15.102:0/1281233072 shutdown_connections 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 1 -- 172.21.15.102:0/1281233072 wait complete. 2024-04-25T02:00:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.965+0000 7f7abf16a740 20 client.16137 trim_cache size 0 max 16384 2024-04-25T02:00:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.983+0000 7f7abf16a740 1 Processor -- start 2024-04-25T02:00:04.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.983+0000 7f7abf16a740 1 -- start start 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef214e00 0x55aaef210c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef21d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 0x55aaef237c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7abf16a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41b1ea0 con 0x55aaef214e00 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7abf16a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41af710 con 0x55aaef215720 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7abf16a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41ad560 con 0x55aaef21ffe0 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef214e00 0x55aaef210c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7ab9800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef21d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 0x55aaef237c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7ab9800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef21d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33234/0 (socket says 172.21.15.102:33234) 2024-04-25T02:00:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 0x55aaef237c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33078/0 (socket says 172.21.15.102:33078) 2024-04-25T02:00:04.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.984+0000 7f7ab9800640 1 -- 172.21.15.102:0/1951367376 learned_addr learned my addr 172.21.15.102:0/1951367376 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:04.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.985+0000 7f7ab9800640 1 -- 172.21.15.102:0/1951367376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 msgr2=0x55aaef237c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:04.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.985+0000 7f7ab9800640 1 --2- 172.21.15.102:0/1951367376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 0x55aaef237c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.985+0000 7f7ab9800640 1 -- 172.21.15.102:0/1951367376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef214e00 msgr2=0x55aaef210c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:04.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.985+0000 7f7ab9800640 1 --2- 172.21.15.102:0/1951367376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef214e00 0x55aaef210c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.985+0000 7f7ab9800640 1 -- 172.21.15.102:0/1951367376 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aaef211140 con 0x55aaef215720 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.985+0000 7f7ab9800640 1 --2- 172.21.15.102:0/1951367376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef21d610 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80314e0 tx=0x7f7aa8003fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b20bf1bb942fffc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.985+0000 7f7aa6000640 1 -- 172.21.15.102:0/1951367376 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8054070 con 0x55aaef215720 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.985+0000 7f7aa6000640 1 -- 172.21.15.102:0/1951367376 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa8005510 con 0x55aaef215720 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.985+0000 7f7aa6000640 1 -- 172.21.15.102:0/1951367376 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8035030 con 0x55aaef215720 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.986+0000 7f7abf16a740 1 -- 172.21.15.102:0/1951367376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 msgr2=0x55aaef21d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.986+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1951367376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef21d610 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80314e0 tx=0x7f7aa8003fb0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.986+0000 7f7abf16a740 1 -- 172.21.15.102:0/1951367376 shutdown_connections 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.986+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1951367376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef21ffe0 0x55aaef237c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.986+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1951367376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef21d610 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.986+0000 7f7abf16a740 1 --2- 172.21.15.102:0/1951367376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef214e00 0x55aaef210c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.986+0000 7f7abf16a740 1 -- 172.21.15.102:0/1951367376 >> 172.21.15.102:0/1951367376 conn(0x55aaef211310 msgr2=0x55aaef21cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.986+0000 7f7abf16a740 1 -- 172.21.15.102:0/1951367376 shutdown_connections 2024-04-25T02:00:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.986+0000 7f7abf16a740 1 -- 172.21.15.102:0/1951367376 wait complete. 2024-04-25T02:00:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.987+0000 7f7abf16a740 1 -- start start 2024-04-25T02:00:04.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.987+0000 7f7abf16a740 1 Processor -- start 2024-04-25T02:00:04.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.987+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef227fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.987+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef228520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.987+0000 7f7abf16a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef21ffe0 0x55aaef225030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.987+0000 7f7abf16a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41adbd0 con 0x55aaef21ffe0 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.987+0000 7f7abf16a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41af040 con 0x55aaef215720 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7abf16a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41ae2a0 con 0x55aaef214e00 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7abf16a740 10 client.0 _ll_register_callbacks cb 0x7f7aa8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef227fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab8e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef227fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33086/0 (socket says 172.21.15.102:33086) 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab8e00640 1 -- 172.21.15.102:0/611779520 learned_addr learned my addr 172.21.15.102:0/611779520 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef21ffe0 0x55aaef225030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab9800640 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef228520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab8e00640 1 -- 172.21.15.102:0/611779520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 msgr2=0x55aaef227fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef227fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab8e00640 1 -- 172.21.15.102:0/611779520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 msgr2=0x55aaef228520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:04.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef228520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab8e00640 1 -- 172.21.15.102:0/611779520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aaef225860 con 0x55aaef21ffe0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.988+0000 7f7ab9800640 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef228520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.989+0000 7f7ab8e00640 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef21ffe0 0x55aaef225030 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00049b0 tx=0x7f7aa0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c045010e637fcdc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.989+0000 7f7a9b400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.989+0000 7f7a9b400640 1 -- 172.21.15.102:0/611779520 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0031520 con 0x55aaef21ffe0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.989+0000 7f7abf16a740 1 -- 172.21.15.102:0/611779520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7aa8005510 con 0x55aaef21ffe0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.989+0000 7f7a9b400640 1 -- 172.21.15.102:0/611779520 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7aa0019040 con 0x55aaef21ffe0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.989+0000 7f7a9b400640 1 -- 172.21.15.102:0/611779520 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7aa0044050 con 0x55aaef21ffe0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.989+0000 7f7abf16a740 10 client.16143 fetch_fsmap learned FSMap version 32 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.989+0000 7f7abf16a740 1 -- 172.21.15.102:0/611779520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55aaef297090 con 0x55aaef21ffe0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.990+0000 7f7a9b400640 1 -- 172.21.15.102:0/611779520 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa8005510 con 0x55aaef21ffe0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.990+0000 7f7a9b400640 1 -- 172.21.15.102:0/611779520 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f7aa0030040 con 0x55aaef21ffe0 2024-04-25T02:00:04.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.990+0000 7f7abf16a740 10 client.16143 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T02:00:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.990+0000 7f7abf16a740 10 client.16143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.990+0000 7f7abf16a740 20 client.16143 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.990+0000 7f7abf16a740 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aaef2975d0 0x55aaef2999c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.990+0000 7f7abf16a740 4 client.16143 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:00:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.990+0000 7f7abf16a740 1 -- 172.21.15.102:0/611779520 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aaef180340 con 0x55aaef2975d0 2024-04-25T02:00:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.990+0000 7f7ab9800640 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aaef2975d0 0x55aaef2999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.991+0000 7f7ab9800640 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aaef2975d0 0x55aaef2999c0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36700c5112b1bd8d server_cookie=cd0178a4832c6fd3 in_seq=0 out_seq=0 2024-04-25T02:00:04.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:04.991+0000 7f7a9b400640 10 client.16143 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:05.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.013+0000 7f7a9b400640 1 -- 172.21.15.102:0/611779520 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aaef180340 con 0x55aaef2975d0 2024-04-25T02:00:05.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.013+0000 7f7a9b400640 10 client.16143 handle_command_reply: tid=0 2024-04-25T02:00:05.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.014+0000 7f7abf16a740 1 client.16143 shutdown 2024-04-25T02:00:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.014+0000 7f7abf16a740 1 -- 172.21.15.102:0/611779520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef21ffe0 msgr2=0x55aaef225030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.014+0000 7f7abf16a740 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef21ffe0 0x55aaef225030 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00049b0 tx=0x7f7aa0005f90 comp rx=0 tx=0).stop 2024-04-25T02:00:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.014+0000 7f7abf16a740 1 -- 172.21.15.102:0/611779520 shutdown_connections 2024-04-25T02:00:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.015+0000 7f7abf16a740 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aaef2975d0 0x55aaef2999c0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.015+0000 7f7abf16a740 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aaef21ffe0 0x55aaef225030 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.015+0000 7f7abf16a740 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aaef215720 0x55aaef228520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.015+0000 7f7a9b400640 0 client.16143 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.015+0000 7f7abf16a740 1 --2- 172.21.15.102:0/611779520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaef214e00 0x55aaef227fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.015+0000 7f7abf16a740 1 -- 172.21.15.102:0/611779520 >> 172.21.15.102:0/611779520 conn(0x55aaef211310 msgr2=0x55aaef21d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.015+0000 7f7abf16a740 1 -- 172.21.15.102:0/611779520 shutdown_connections 2024-04-25T02:00:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.015+0000 7f7abf16a740 1 -- 172.21.15.102:0/611779520 wait complete. 2024-04-25T02:00:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.015+0000 7f7abf16a740 20 client.16143 trim_cache size 0 max 16384 2024-04-25T02:00:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:00:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:00:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:00:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.016+0000 7f7abc200640 1 -- 172.21.15.102:0/467266137 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7a9c0a2da0 msgr2=0x7f7a9c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.016+0000 7f7abc200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7a9c0a2da0 0x7f7a9c0a5260 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0049d0 tx=0x7f7aac005b40 comp rx=0 tx=0).stop 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.016+0000 7f7abc200640 1 -- 172.21.15.102:0/467266137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 msgr2=0x7f7ab419e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.016+0000 7f7abc200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 0x7f7ab419e9e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00eb10 tx=0x7f7aac00c2d0 comp rx=0 tx=0).stop 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.017+0000 7f7abc200640 1 -- 172.21.15.102:0/467266137 shutdown_connections 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.017+0000 7f7abc200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7a9c0a2da0 0x7f7a9c0a5260 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.017+0000 7f7abc200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ab4109930 0x7f7ab419e9e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.017+0000 7f7abc200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ab41042e0 0x7f7ab419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.017+0000 7f7abc200640 1 --2- 172.21.15.102:0/467266137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ab4103920 0x7f7ab419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.017+0000 7f7abc200640 1 -- 172.21.15.102:0/467266137 >> 172.21.15.102:0/467266137 conn(0x7f7ab40ff430 msgr2=0x7f7ab40ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.017+0000 7f7abc200640 1 -- 172.21.15.102:0/467266137 shutdown_connections 2024-04-25T02:00:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:05.017+0000 7f7abc200640 1 -- 172.21.15.102:0/467266137 wait complete. 2024-04-25T02:00:05.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:04 smithi102 ceph-mon[27796]: pgmap v147: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 8.9 MiB/s wr, 831 op/s 2024-04-25T02:00:05.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:04 smithi110 ceph-mon[32848]: pgmap v147: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 8.9 MiB/s wr, 831 op/s 2024-04-25T02:00:05.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:04 smithi193 ceph-mon[32981]: pgmap v147: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 8.9 MiB/s wr, 831 op/s 2024-04-25T02:00:07.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:06 smithi102 ceph-mon[27796]: pgmap v148: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 7.1 MiB/s wr, 706 op/s 2024-04-25T02:00:07.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:06 smithi110 ceph-mon[32848]: pgmap v148: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 7.1 MiB/s wr, 706 op/s 2024-04-25T02:00:07.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:06 smithi193 ceph-mon[32981]: pgmap v148: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 7.1 MiB/s wr, 706 op/s 2024-04-25T02:00:08.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 255.0 s, 247.8 tps, lat 367.323 ms stddev 213.267 2024-04-25T02:00:09.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:08 smithi102 ceph-mon[27796]: pgmap v149: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 8.7 MiB/s wr, 846 op/s 2024-04-25T02:00:09.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:08 smithi110 ceph-mon[32848]: pgmap v149: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 8.7 MiB/s wr, 846 op/s 2024-04-25T02:00:09.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:08 smithi193 ceph-mon[32981]: pgmap v149: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 8.7 MiB/s wr, 846 op/s 2024-04-25T02:00:11.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:10 smithi102 ceph-mon[27796]: pgmap v150: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.3 MiB/s wr, 511 op/s 2024-04-25T02:00:11.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:10 smithi110 ceph-mon[32848]: pgmap v150: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.3 MiB/s wr, 511 op/s 2024-04-25T02:00:11.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:10 smithi193 ceph-mon[32981]: pgmap v150: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.3 MiB/s wr, 511 op/s 2024-04-25T02:00:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 260.0 s, 2094.0 tps, lat 53.496 ms stddev 119.335 2024-04-25T02:00:13.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:12 smithi102 ceph-mon[27796]: pgmap v151: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-25T02:00:13.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:12 smithi110 ceph-mon[32848]: pgmap v151: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-25T02:00:13.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:12 smithi193 ceph-mon[32981]: pgmap v151: 97 pgs: 97 active+clean; 3.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-25T02:00:15.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:14 smithi102 ceph-mon[27796]: pgmap v152: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 23 MiB/s wr, 1.57k op/s 2024-04-25T02:00:15.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:14 smithi110 ceph-mon[32848]: pgmap v152: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 23 MiB/s wr, 1.57k op/s 2024-04-25T02:00:15.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:14 smithi193 ceph-mon[32981]: pgmap v152: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 23 MiB/s wr, 1.57k op/s 2024-04-25T02:00:17.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:16 smithi102 ceph-mon[27796]: pgmap v153: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 23 MiB/s wr, 1.48k op/s 2024-04-25T02:00:17.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:16 smithi110 ceph-mon[32848]: pgmap v153: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 23 MiB/s wr, 1.48k op/s 2024-04-25T02:00:17.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:16 smithi193 ceph-mon[32981]: pgmap v153: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 23 MiB/s wr, 1.48k op/s 2024-04-25T02:00:18.072 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 265.0 s, 2783.8 tps, lat 35.968 ms stddev 44.475 2024-04-25T02:00:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.208+0000 7f03b1e00640 1 -- 172.21.15.102:0/1003585366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ac101750 msgr2=0x7f03ac101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.208+0000 7f03b1e00640 1 --2- 172.21.15.102:0/1003585366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ac101750 0x7f03ac101bd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0398009890 tx=0x7f039802a360 comp rx=0 tx=0).stop 2024-04-25T02:00:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.208+0000 7f03b1e00640 1 -- 172.21.15.102:0/1003585366 shutdown_connections 2024-04-25T02:00:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.208+0000 7f03b1e00640 1 --2- 172.21.15.102:0/1003585366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac102110 0x7f03ac10e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.208+0000 7f03b1e00640 1 --2- 172.21.15.102:0/1003585366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ac101750 0x7f03ac101bd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.208+0000 7f03b1e00640 1 --2- 172.21.15.102:0/1003585366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03ac107760 0x7f03ac107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.208+0000 7f03b1e00640 1 -- 172.21.15.102:0/1003585366 >> 172.21.15.102:0/1003585366 conn(0x7f03ac078bd0 msgr2=0x7f03ac078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.208+0000 7f03b1e00640 1 -- 172.21.15.102:0/1003585366 shutdown_connections 2024-04-25T02:00:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.208+0000 7f03b1e00640 1 -- 172.21.15.102:0/1003585366 wait complete. 2024-04-25T02:00:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.210+0000 7f03b1e00640 1 Processor -- start 2024-04-25T02:00:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.210+0000 7f03b1e00640 1 -- start start 2024-04-25T02:00:18.215 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03b1e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03ac101750 0x7f03ac19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.216 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03b1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ac102110 0x7f03ac19a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.216 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03b1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac107760 0x7f03ac19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.216 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03b1e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac074710 con 0x7f03ac107760 2024-04-25T02:00:18.216 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03b1e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac074590 con 0x7f03ac101750 2024-04-25T02:00:18.216 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03b1e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac074890 con 0x7f03ac102110 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03aaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03ac101750 0x7f03ac19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03aaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03ac101750 0x7f03ac19a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54324/0 (socket says 172.21.15.102:54324) 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03aaa00640 1 -- 172.21.15.102:0/2203752980 learned_addr learned my addr 172.21.15.102:0/2203752980 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03ab400640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac107760 0x7f03ac19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.211+0000 7f03aa000640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ac102110 0x7f03ac19a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.212+0000 7f03aaa00640 1 -- 172.21.15.102:0/2203752980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ac102110 msgr2=0x7f03ac19a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.212+0000 7f03aaa00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ac102110 0x7f03ac19a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.212+0000 7f03aaa00640 1 -- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac107760 msgr2=0x7f03ac19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.212+0000 7f03aaa00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac107760 0x7f03ac19ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.212+0000 7f03aaa00640 1 -- 172.21.15.102:0/2203752980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03ac19f190 con 0x7f03ac101750 2024-04-25T02:00:18.217 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.212+0000 7f03ab400640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac107760 0x7f03ac19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:18.218 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.213+0000 7f03aaa00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03ac101750 0x7f03ac19a2b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f039c00e9c0 tx=0x7f039c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.214+0000 7f03b0a00640 1 -- 172.21.15.102:0/2203752980 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c00ccc0 con 0x7f03ac101750 2024-04-25T02:00:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.214+0000 7f03b0a00640 1 -- 172.21.15.102:0/2203752980 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f039c005500 con 0x7f03ac101750 2024-04-25T02:00:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.214+0000 7f03b0a00640 1 -- 172.21.15.102:0/2203752980 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c010430 con 0x7f03ac101750 2024-04-25T02:00:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.214+0000 7f03b1e00640 1 -- 172.21.15.102:0/2203752980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03ac19f480 con 0x7f03ac101750 2024-04-25T02:00:18.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.214+0000 7f03b1e00640 1 -- 172.21.15.102:0/2203752980 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03ac1a6d10 con 0x7f03ac101750 2024-04-25T02:00:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03b4c31740 1 Processor -- start 2024-04-25T02:00:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03b4c31740 1 -- start start 2024-04-25T02:00:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c031e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 0x55c483c0daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c0dfe0 0x55c483c103d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03b4c31740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c483b66ab0 con 0x55c483c0dfe0 2024-04-25T02:00:18.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03b4c31740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c483b0f640 con 0x55c483c03720 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03b4c31740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c483b62a70 con 0x55c483c02e00 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03ab400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03ab400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c031e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:45592/0 (socket says 172.21.15.102:45592) 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03ab400640 1 -- 172.21.15.102:0/199839520 learned_addr learned my addr 172.21.15.102:0/199839520 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.219+0000 7f03aa000640 1 --2- 172.21.15.102:0/199839520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c0dfe0 0x55c483c103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.220+0000 7f03ab400640 1 -- 172.21.15.102:0/199839520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 msgr2=0x55c483c0daa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.220+0000 7f03ab400640 1 --2- 172.21.15.102:0/199839520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 0x55c483c0daa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.220+0000 7f03ab400640 1 -- 172.21.15.102:0/199839520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c0dfe0 msgr2=0x55c483c103d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.220+0000 7f03ab400640 1 --2- 172.21.15.102:0/199839520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c0dfe0 0x55c483c103d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.220+0000 7f03ab400640 1 -- 172.21.15.102:0/199839520 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c48387b210 con 0x55c483c02e00 2024-04-25T02:00:18.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.220+0000 7f03aa000640 1 --2- 172.21.15.102:0/199839520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c0dfe0 0x55c483c103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:18.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.220+0000 7f03b0a00640 1 -- 172.21.15.102:0/2203752980 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f039c0105d0 con 0x7f03ac101750 2024-04-25T02:00:18.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.220+0000 7f03ab400640 1 --2- 172.21.15.102:0/199839520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c031e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f03a40117f0 tx=0x7f03a4011cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51935085f6cb331 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:18.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.220+0000 7f0397e00640 1 -- 172.21.15.102:0/199839520 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a40112f0 con 0x55c483c02e00 2024-04-25T02:00:18.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.221+0000 7f0397e00640 1 -- 172.21.15.102:0/199839520 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a400fe80 con 0x55c483c02e00 2024-04-25T02:00:18.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.221+0000 7f03b4c31740 1 -- 172.21.15.102:0/199839520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 msgr2=0x55c483c031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:18.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.221+0000 7f03b4c31740 1 --2- 172.21.15.102:0/199839520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c031e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f03a40117f0 tx=0x7f03a4011cb0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.222+0000 7f03b4c31740 1 -- 172.21.15.102:0/199839520 shutdown_connections 2024-04-25T02:00:18.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.222+0000 7f03b4c31740 1 --2- 172.21.15.102:0/199839520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c0dfe0 0x55c483c103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.222+0000 7f03b4c31740 1 --2- 172.21.15.102:0/199839520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 0x55c483c0daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.222+0000 7f03b4c31740 1 --2- 172.21.15.102:0/199839520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c031e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.222+0000 7f03b4c31740 1 -- 172.21.15.102:0/199839520 >> 172.21.15.102:0/199839520 conn(0x55c483bff310 msgr2=0x55c483bff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:18.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.223+0000 7f03b4c31740 1 -- 172.21.15.102:0/199839520 shutdown_connections 2024-04-25T02:00:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.223+0000 7f03b4c31740 1 -- 172.21.15.102:0/199839520 wait complete. 2024-04-25T02:00:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.223+0000 7f03b0a00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03900a2ce0 0x7f03900a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.223+0000 7f03b0a00640 1 -- 172.21.15.102:0/2203752980 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f039c0cf670 con 0x7f03ac101750 2024-04-25T02:00:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.223+0000 7f03b4c31740 1 -- start start 2024-04-25T02:00:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.223+0000 7f03aa000640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03900a2ce0 0x7f03900a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.223+0000 7f03b4c31740 1 Processor -- start 2024-04-25T02:00:18.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.224+0000 7f03aa000640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03900a2ce0 0x7f03900a51a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f039802a950 tx=0x7f03980056d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c0aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483c0b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c0dfe0 0x55c483c0b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03b4c31740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c483c10c60 con 0x55c483c03720 2024-04-25T02:00:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03b4c31740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c483b22990 con 0x55c483c0dfe0 2024-04-25T02:00:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03b4c31740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c483c10de0 con 0x55c483c02e00 2024-04-25T02:00:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03b4c31740 10 client.0 _ll_register_callbacks cb 0x55c483b24950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:18.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03ab400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c0dfe0 0x55c483c0b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03ab400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c0dfe0 0x55c483c0b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54340/0 (socket says 172.21.15.102:54340) 2024-04-25T02:00:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03ab400640 1 -- 172.21.15.102:0/780113935 learned_addr learned my addr 172.21.15.102:0/780113935 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03aaa00640 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c0aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.226+0000 7f03aa000640 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483c0b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.227+0000 7f03ab400640 1 -- 172.21.15.102:0/780113935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 msgr2=0x55c483c0aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.227+0000 7f03ab400640 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c0aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.227+0000 7f03ab400640 1 -- 172.21.15.102:0/780113935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 msgr2=0x55c483c0b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.227+0000 7f03ab400640 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483c0b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.227+0000 7f03ab400640 1 -- 172.21.15.102:0/780113935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c483947620 con 0x55c483c0dfe0 2024-04-25T02:00:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.227+0000 7f03ab400640 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c0dfe0 0x55c483c0b940 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f03a400fbb0 tx=0x7f03a400fbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=202b75220980e620 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.227+0000 7f0394c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:00:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.228+0000 7f0394c00640 1 -- 172.21.15.102:0/780113935 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a401f040 con 0x55c483c0dfe0 2024-04-25T02:00:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.228+0000 7f0394c00640 1 -- 172.21.15.102:0/780113935 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f03a40110f0 con 0x55c483c0dfe0 2024-04-25T02:00:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.228+0000 7f03b4c31740 1 -- 172.21.15.102:0/780113935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c483b73ff0 con 0x55c483c0dfe0 2024-04-25T02:00:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.229+0000 7f0394c00640 1 -- 172.21.15.102:0/780113935 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f03a401f040 con 0x55c483c0dfe0 2024-04-25T02:00:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.229+0000 7f03b4c31740 10 client.26600 fetch_fsmap learned FSMap version 32 2024-04-25T02:00:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.229+0000 7f03b4c31740 1 -- 172.21.15.102:0/780113935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c483c25310 con 0x55c483c0dfe0 2024-04-25T02:00:18.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.229+0000 7f0394c00640 1 -- 172.21.15.102:0/780113935 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c483b73ff0 con 0x55c483c0dfe0 2024-04-25T02:00:18.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.230+0000 7f0394c00640 1 -- 172.21.15.102:0/780113935 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f03a4018070 con 0x55c483c0dfe0 2024-04-25T02:00:18.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.230+0000 7f03b4c31740 10 client.26600 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T02:00:18.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.230+0000 7f03b4c31740 10 client.26600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:18.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.230+0000 7f03b4c31740 20 client.26600 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:18.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.230+0000 7f03b4c31740 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c483c26710 0x55c483c28b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.230+0000 7f03b4c31740 4 client.26600 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.230+0000 7f03b4c31740 1 -- 172.21.15.102:0/780113935 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c483b6e340 con 0x55c483c26710 2024-04-25T02:00:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.230+0000 7f03aaa00640 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c483c26710 0x55c483c28b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.231+0000 7f03aaa00640 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c483c26710 0x55c483c28b00 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a91f5bf5ef471f3 server_cookie=af62f11853fbfbad in_seq=0 out_seq=0 2024-04-25T02:00:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.231+0000 7f0394c00640 10 client.26600 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.231+0000 7f0394c00640 1 -- 172.21.15.102:0/780113935 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c483b6e340 con 0x55c483c26710 2024-04-25T02:00:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.231+0000 7f0394c00640 10 client.26600 handle_command_reply: tid=0 2024-04-25T02:00:18.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.232+0000 7f03b4c31740 1 client.26600 shutdown 2024-04-25T02:00:18.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.232+0000 7f03b4c31740 1 -- 172.21.15.102:0/780113935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c0dfe0 msgr2=0x55c483c0b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:18.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.232+0000 7f03b4c31740 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c0dfe0 0x55c483c0b940 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f03a400fbb0 tx=0x7f03a400fbe0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.233+0000 7f03b4c31740 1 -- 172.21.15.102:0/780113935 shutdown_connections 2024-04-25T02:00:18.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.233+0000 7f03b4c31740 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c483c26710 0x55c483c28b00 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.238 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.233+0000 7f03b4c31740 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c0dfe0 0x55c483c0b940 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.238 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.233+0000 7f03b4c31740 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483c0b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.238 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.233+0000 7f03b4c31740 1 --2- 172.21.15.102:0/780113935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c02e00 0x55c483c0aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.238 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.233+0000 7f0394c00640 0 client.26600 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:18.238 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.233+0000 7f03b4c31740 1 -- 172.21.15.102:0/780113935 >> 172.21.15.102:0/780113935 conn(0x55c483bff310 msgr2=0x55c483c05040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:18.238 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.233+0000 7f03b4c31740 1 -- 172.21.15.102:0/780113935 shutdown_connections 2024-04-25T02:00:18.238 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.234+0000 7f03b4c31740 1 -- 172.21.15.102:0/780113935 wait complete. 2024-04-25T02:00:18.239 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.234+0000 7f03b4c31740 20 client.26600 trim_cache size 0 max 16384 2024-04-25T02:00:18.273 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.267+0000 7f03b4c31740 1 Processor -- start 2024-04-25T02:00:18.274 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.270+0000 7f03b4c31740 1 -- start start 2024-04-25T02:00:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.270+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c02e00 0x55c483c22400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.270+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483bfec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.270+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 0x55c483c0ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.270+0000 7f03b4c31740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1adbd0 con 0x55c483c03720 2024-04-25T02:00:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.270+0000 7f03b4c31740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1ab440 con 0x55c483c02e00 2024-04-25T02:00:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.270+0000 7f03b4c31740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1a92b0 con 0x55c483c0dfe0 2024-04-25T02:00:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.272+0000 7f03aa000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483bfec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.272+0000 7f03aa000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483bfec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34262/0 (socket says 172.21.15.102:34262) 2024-04-25T02:00:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.272+0000 7f03aa000640 1 -- 172.21.15.102:0/819567852 learned_addr learned my addr 172.21.15.102:0/819567852 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.272+0000 7f03ab400640 1 --2- 172.21.15.102:0/819567852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 0x55c483c0ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.272+0000 7f03aa000640 1 -- 172.21.15.102:0/819567852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 msgr2=0x55c483c0ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.272+0000 7f03aa000640 1 --2- 172.21.15.102:0/819567852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 0x55c483c0ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.272+0000 7f03aa000640 1 -- 172.21.15.102:0/819567852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c02e00 msgr2=0x55c483c22400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.272+0000 7f03aa000640 1 --2- 172.21.15.102:0/819567852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c02e00 0x55c483c22400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.272+0000 7f03aa000640 1 -- 172.21.15.102:0/819567852 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c483bff140 con 0x55c483c03720 2024-04-25T02:00:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.273+0000 7f03aa000640 1 --2- 172.21.15.102:0/819567852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483bfec00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0398009890 tx=0x7f039803d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bcb5c20d918953 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.274+0000 7f0397400640 1 -- 172.21.15.102:0/819567852 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039804b070 con 0x55c483c03720 2024-04-25T02:00:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.276+0000 7f0397400640 1 -- 172.21.15.102:0/819567852 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0398030840 con 0x55c483c03720 2024-04-25T02:00:18.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.276+0000 7f0397400640 1 -- 172.21.15.102:0/819567852 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398041040 con 0x55c483c03720 2024-04-25T02:00:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.278+0000 7f03b4c31740 1 -- 172.21.15.102:0/819567852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 msgr2=0x55c483bfec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.278+0000 7f03b4c31740 1 --2- 172.21.15.102:0/819567852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483bfec00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0398009890 tx=0x7f039803d000 comp rx=0 tx=0).stop 2024-04-25T02:00:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.280+0000 7f03b4c31740 1 -- 172.21.15.102:0/819567852 shutdown_connections 2024-04-25T02:00:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.280+0000 7f03b4c31740 1 --2- 172.21.15.102:0/819567852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 0x55c483c0ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.280+0000 7f03b4c31740 1 --2- 172.21.15.102:0/819567852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c03720 0x55c483bfec00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.280+0000 7f03b4c31740 1 --2- 172.21.15.102:0/819567852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c02e00 0x55c483c22400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.280+0000 7f03b4c31740 1 -- 172.21.15.102:0/819567852 >> 172.21.15.102:0/819567852 conn(0x55c483bff310 msgr2=0x55c483c0c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.280+0000 7f03b4c31740 1 -- 172.21.15.102:0/819567852 shutdown_connections 2024-04-25T02:00:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.281+0000 7f03b4c31740 1 -- 172.21.15.102:0/819567852 wait complete. 2024-04-25T02:00:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.281+0000 7f03b4c31740 1 -- start start 2024-04-25T02:00:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.282+0000 7f03b4c31740 1 Processor -- start 2024-04-25T02:00:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.283+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c02e00 0x55c483c1edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.283+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 0x55c483c07980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.283+0000 7f03b4c31740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 0x55c483c07f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.283+0000 7f03aa000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 0x55c483c07980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.283+0000 7f03aa000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 0x55c483c07980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54346/0 (socket says 172.21.15.102:54346) 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.283+0000 7f03aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c02e00 0x55c483c1edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.283+0000 7f03aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c02e00 0x55c483c1edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34266/0 (socket says 172.21.15.102:34266) 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.284+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1a9900 con 0x55c483c02e00 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.284+0000 7f03aa000640 1 -- 172.21.15.102:0/3445283918 learned_addr learned my addr 172.21.15.102:0/3445283918 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.284+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1aad70 con 0x55c483c03720 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.284+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1a9fd0 con 0x55c483c0dfe0 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.284+0000 7f03b4c31740 10 client.0 _ll_register_callbacks cb 0x7f03a40116c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.284+0000 7f03ab400640 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 0x55c483c07f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.285+0000 7f03ab400640 1 -- 172.21.15.102:0/3445283918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 msgr2=0x55c483c07980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:18.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.285+0000 7f03ab400640 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 0x55c483c07980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.285+0000 7f03ab400640 1 -- 172.21.15.102:0/3445283918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c02e00 msgr2=0x55c483c1edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:18.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.285+0000 7f03ab400640 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c02e00 0x55c483c1edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.285+0000 7f03ab400640 1 -- 172.21.15.102:0/3445283918 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c483c08730 con 0x55c483c0dfe0 2024-04-25T02:00:18.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.285+0000 7f03aa000640 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 0x55c483c07980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:18.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.285+0000 7f03ab400640 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 0x55c483c07f00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009560 tx=0x7f03a4010c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6df4663a58f73571 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:18.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.285+0000 7f0394c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:00:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.287+0000 7f0394c00640 1 -- 172.21.15.102:0/3445283918 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a40107a0 con 0x55c483c0dfe0 2024-04-25T02:00:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.287+0000 7f0394c00640 1 -- 172.21.15.102:0/3445283918 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f03a4019790 con 0x55c483c0dfe0 2024-04-25T02:00:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.287+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0398030840 con 0x55c483c0dfe0 2024-04-25T02:00:18.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.288+0000 7f0394c00640 1 -- 172.21.15.102:0/3445283918 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f03a40110f0 con 0x55c483c0dfe0 2024-04-25T02:00:18.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.288+0000 7f03b4c31740 10 client.26572 fetch_fsmap learned FSMap version 32 2024-04-25T02:00:18.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.288+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c483c110c0 con 0x55c483c0dfe0 2024-04-25T02:00:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.289+0000 7f0394c00640 1 -- 172.21.15.102:0/3445283918 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0398030840 con 0x55c483c0dfe0 2024-04-25T02:00:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.290+0000 7f0394c00640 1 -- 172.21.15.102:0/3445283918 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f03a4018070 con 0x55c483c0dfe0 2024-04-25T02:00:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.290+0000 7f03b4c31740 10 client.26572 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T02:00:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.290+0000 7f03b4c31740 10 client.26572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.290+0000 7f03b4c31740 20 client.26572 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:18.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.290+0000 7f03b4c31740 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c483c11810 0x55c483c13c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.290+0000 7f03b4c31740 4 client.26572 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:00:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.290+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c483aec7a0 con 0x55c483c11810 2024-04-25T02:00:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.290+0000 7f03aaa00640 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c483c11810 0x55c483c13c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.291+0000 7f0394c00640 10 client.26572 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:18.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.291+0000 7f03aaa00640 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c483c11810 0x55c483c13c30 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a089c04ee5404cd server_cookie=e07eff5be1655b5c in_seq=0 out_seq=0 2024-04-25T02:00:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.320+0000 7f0394c00640 1 -- 172.21.15.102:0/3445283918 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c483aec7a0 con 0x55c483c11810 2024-04-25T02:00:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.320+0000 7f0394c00640 10 client.26572 handle_command_reply: tid=0 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.320+0000 7f03b4c31740 1 client.26572 shutdown 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.320+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 msgr2=0x55c483c07f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.320+0000 7f03b4c31740 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 0x55c483c07f00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009560 tx=0x7f03a4010c40 comp rx=0 tx=0).stop 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.321+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 shutdown_connections 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.321+0000 7f03b4c31740 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c483c11810 0x55c483c13c30 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.321+0000 7f03b4c31740 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c483c0dfe0 0x55c483c07f00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.321+0000 7f03b4c31740 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c483c03720 0x55c483c07980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.321+0000 7f03b4c31740 1 --2- 172.21.15.102:0/3445283918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c483c02e00 0x55c483c1edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.321+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 >> 172.21.15.102:0/3445283918 conn(0x55c483bff310 msgr2=0x55c483c0cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.321+0000 7f0394c00640 0 client.26572 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.322+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 shutdown_connections 2024-04-25T02:00:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.322+0000 7f03b4c31740 1 -- 172.21.15.102:0/3445283918 wait complete. 2024-04-25T02:00:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.322+0000 7f03b4c31740 20 client.26572 trim_cache size 0 max 16384 2024-04-25T02:00:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:00:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:00:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:00:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:00:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.324+0000 7f03b1e00640 1 -- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03900a2ce0 msgr2=0x7f03900a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.324+0000 7f03b1e00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03900a2ce0 0x7f03900a51a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f039802a950 tx=0x7f03980056d0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.324+0000 7f03b1e00640 1 -- 172.21.15.102:0/2203752980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03ac101750 msgr2=0x7f03ac19a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.324+0000 7f03b1e00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03ac101750 0x7f03ac19a2b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f039c00e9c0 tx=0x7f039c00ee80 comp rx=0 tx=0).stop 2024-04-25T02:00:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.325+0000 7f03b1e00640 1 -- 172.21.15.102:0/2203752980 shutdown_connections 2024-04-25T02:00:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.325+0000 7f03b1e00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03900a2ce0 0x7f03900a51a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.325+0000 7f03b1e00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03ac107760 0x7f03ac19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.325+0000 7f03b1e00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03ac102110 0x7f03ac19a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.325+0000 7f03b1e00640 1 --2- 172.21.15.102:0/2203752980 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03ac101750 0x7f03ac19a2b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.325+0000 7f03b1e00640 1 -- 172.21.15.102:0/2203752980 >> 172.21.15.102:0/2203752980 conn(0x7f03ac078bd0 msgr2=0x7f03ac0ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.325+0000 7f03b1e00640 1 -- 172.21.15.102:0/2203752980 shutdown_connections 2024-04-25T02:00:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:18.325+0000 7f03b1e00640 1 -- 172.21.15.102:0/2203752980 wait complete. 2024-04-25T02:00:19.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:18 smithi102 ceph-mon[27796]: pgmap v154: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 26 MiB/s wr, 1.76k op/s 2024-04-25T02:00:19.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:18 smithi110 ceph-mon[32848]: pgmap v154: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 26 MiB/s wr, 1.76k op/s 2024-04-25T02:00:19.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:18 smithi193 ceph-mon[32981]: pgmap v154: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 26 MiB/s wr, 1.76k op/s 2024-04-25T02:00:21.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:20 smithi102 ceph-mon[27796]: pgmap v155: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 MiB/s wr, 1.62k op/s 2024-04-25T02:00:21.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:20 smithi110 ceph-mon[32848]: pgmap v155: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 MiB/s wr, 1.62k op/s 2024-04-25T02:00:21.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:20 smithi193 ceph-mon[32981]: pgmap v155: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 MiB/s wr, 1.62k op/s 2024-04-25T02:00:23.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 270.0 s, 2807.3 tps, lat 35.852 ms stddev 47.865 2024-04-25T02:00:23.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:22 smithi102 ceph-mon[27796]: pgmap v156: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 27 MiB/s wr, 1.88k op/s 2024-04-25T02:00:23.240 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:22 smithi110 ceph-mon[32848]: pgmap v156: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 27 MiB/s wr, 1.88k op/s 2024-04-25T02:00:23.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:22 smithi193 ceph-mon[32981]: pgmap v156: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 27 MiB/s wr, 1.88k op/s 2024-04-25T02:00:25.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:24 smithi102 ceph-mon[27796]: pgmap v157: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 15 MiB/s wr, 1.13k op/s 2024-04-25T02:00:25.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:24 smithi110 ceph-mon[32848]: pgmap v157: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 15 MiB/s wr, 1.13k op/s 2024-04-25T02:00:25.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:24 smithi193 ceph-mon[32981]: pgmap v157: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 15 MiB/s wr, 1.13k op/s 2024-04-25T02:00:27.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T02:00:27.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T02:00:27.144 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:26 smithi102 ceph-mon[27796]: pgmap v158: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.1 MiB/s wr, 639 op/s 2024-04-25T02:00:27.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:26 smithi110 ceph-mon[32848]: pgmap v158: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.1 MiB/s wr, 639 op/s 2024-04-25T02:00:27.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:26 smithi193 ceph-mon[32981]: pgmap v158: 97 pgs: 97 active+clean; 3.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.1 MiB/s wr, 639 op/s 2024-04-25T02:00:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.330+0000 7fe31ee00640 1 -- 172.21.15.102:0/1371059258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318101760 msgr2=0x7fe318101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.330+0000 7fe31ee00640 1 --2- 172.21.15.102:0/1371059258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318101760 0x7fe318101be0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe304009900 tx=0x7fe30402a3f0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.331+0000 7fe31ee00640 1 -- 172.21.15.102:0/1371059258 shutdown_connections 2024-04-25T02:00:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.331+0000 7fe31ee00640 1 --2- 172.21.15.102:0/1371059258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe318102120 0x7fe31810e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.331+0000 7fe31ee00640 1 --2- 172.21.15.102:0/1371059258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318101760 0x7fe318101be0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.331+0000 7fe31ee00640 1 --2- 172.21.15.102:0/1371059258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe318107770 0x7fe318107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.331+0000 7fe31ee00640 1 -- 172.21.15.102:0/1371059258 >> 172.21.15.102:0/1371059258 conn(0x7fe3180fd250 msgr2=0x7fe3180ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:27.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.331+0000 7fe31ee00640 1 -- 172.21.15.102:0/1371059258 shutdown_connections 2024-04-25T02:00:27.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.331+0000 7fe31ee00640 1 -- 172.21.15.102:0/1371059258 wait complete. 2024-04-25T02:00:27.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.332+0000 7fe31ee00640 1 Processor -- start 2024-04-25T02:00:27.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.332+0000 7fe31ee00640 1 -- start start 2024-04-25T02:00:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.333+0000 7fe31ee00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318101760 0x7fe31819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.333+0000 7fe31ee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe318102120 0x7fe31819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.333+0000 7fe31ee00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe318107770 0x7fe31819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.333+0000 7fe31ee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe318111b90 con 0x7fe318102120 2024-04-25T02:00:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.333+0000 7fe31ee00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe318111a10 con 0x7fe318101760 2024-04-25T02:00:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.333+0000 7fe317e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318101760 0x7fe31819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.333+0000 7fe317e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318101760 0x7fe31819a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59464/0 (socket says 172.21.15.102:59464) 2024-04-25T02:00:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe317e00640 1 -- 172.21.15.102:0/3460769927 learned_addr learned my addr 172.21.15.102:0/3460769927 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:27.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe31ee00640 1 -- 172.21.15.102:0/3460769927 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe318111d10 con 0x7fe318107770 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe317400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe318102120 0x7fe31819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe31ce00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe318107770 0x7fe31819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe31ce00640 1 -- 172.21.15.102:0/3460769927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318101760 msgr2=0x7fe31819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe31ce00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318101760 0x7fe31819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe31ce00640 1 -- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe318102120 msgr2=0x7fe31819a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe31ce00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe318102120 0x7fe31819a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe31ce00640 1 -- 172.21.15.102:0/3460769927 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31819f2a0 con 0x7fe318107770 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe317400640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe318102120 0x7fe31819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.334+0000 7fe31ce00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe318107770 0x7fe31819eb70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe31000ec50 tx=0x7fe31000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.335+0000 7fe314c00640 1 -- 172.21.15.102:0/3460769927 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe310003b70 con 0x7fe318107770 2024-04-25T02:00:27.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.335+0000 7fe31ee00640 1 -- 172.21.15.102:0/3460769927 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31819f590 con 0x7fe318107770 2024-04-25T02:00:27.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.335+0000 7fe314c00640 1 -- 172.21.15.102:0/3460769927 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe310003d10 con 0x7fe318107770 2024-04-25T02:00:27.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.335+0000 7fe314c00640 1 -- 172.21.15.102:0/3460769927 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe310010970 con 0x7fe318107770 2024-04-25T02:00:27.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.335+0000 7fe31ee00640 1 -- 172.21.15.102:0/3460769927 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3181a6e20 con 0x7fe318107770 2024-04-25T02:00:27.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.339+0000 7fe314c00640 1 -- 172.21.15.102:0/3460769927 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe310010b10 con 0x7fe318107770 2024-04-25T02:00:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.342+0000 7fe321d7f740 1 Processor -- start 2024-04-25T02:00:27.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.343+0000 7fe321d7f740 1 -- start start 2024-04-25T02:00:27.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.343+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba5e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.343+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba68ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.343+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba6b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.343+0000 7fe321d7f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56022b9c2560 con 0x56022ba5e000 2024-04-25T02:00:27.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.343+0000 7fe321d7f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56022b96a930 con 0x56022ba5e920 2024-04-25T02:00:27.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.343+0000 7fe321d7f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56022b9be520 con 0x56022ba68ff0 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba68ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba68ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59470/0 (socket says 172.21.15.102:59470) 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317e00640 1 -- 172.21.15.102:0/2108527892 learned_addr learned my addr 172.21.15.102:0/2108527892 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317400640 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba5e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe314c00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe2fc0a2dc0 0x7fe2fc0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317400640 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba6b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317e00640 1 -- 172.21.15.102:0/2108527892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 msgr2=0x56022ba6b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317e00640 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba6b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317e00640 1 -- 172.21.15.102:0/2108527892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 msgr2=0x56022ba5e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317e00640 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba5e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317e00640 1 -- 172.21.15.102:0/2108527892 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56022b7a07d0 con 0x56022ba5e920 2024-04-25T02:00:27.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe314c00640 1 -- 172.21.15.102:0/3460769927 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe3100cf9f0 con 0x7fe318107770 2024-04-25T02:00:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.344+0000 7fe317400640 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba6b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.345+0000 7fe317e00640 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba68ab0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe308009d90 tx=0x7fe308011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de007d86ec63d23b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.345+0000 7fe30d600640 1 -- 172.21.15.102:0/2108527892 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30801e070 con 0x56022ba5e920 2024-04-25T02:00:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.345+0000 7fe30d600640 1 -- 172.21.15.102:0/2108527892 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe308019b10 con 0x56022ba5e920 2024-04-25T02:00:27.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.345+0000 7fe30d600640 1 -- 172.21.15.102:0/2108527892 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30800fcb0 con 0x56022ba5e920 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.346+0000 7fe321d7f740 1 -- 172.21.15.102:0/2108527892 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 msgr2=0x56022ba68ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.346+0000 7fe321d7f740 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba68ab0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe308009d90 tx=0x7fe308011810 comp rx=0 tx=0).stop 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.346+0000 7fe321d7f740 1 -- 172.21.15.102:0/2108527892 shutdown_connections 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.346+0000 7fe321d7f740 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba6b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.346+0000 7fe321d7f740 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba68ab0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.346+0000 7fe321d7f740 1 --2- 172.21.15.102:0/2108527892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba5e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.346+0000 7fe321d7f740 1 -- 172.21.15.102:0/2108527892 >> 172.21.15.102:0/2108527892 conn(0x56022ba5a320 msgr2=0x56022ba5a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.346+0000 7fe321d7f740 1 -- 172.21.15.102:0/2108527892 shutdown_connections 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.346+0000 7fe321d7f740 1 -- 172.21.15.102:0/2108527892 wait complete. 2024-04-25T02:00:27.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.347+0000 7fe321d7f740 1 -- start start 2024-04-25T02:00:27.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.347+0000 7fe31ce00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe2fc0a2dc0 0x7fe2fc0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.347+0000 7fe321d7f740 1 Processor -- start 2024-04-25T02:00:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.349+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e000 0x56022ba65f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.349+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 0x56022ba66480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.349+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba669c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.349+0000 7fe321d7f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56022ba6bde0 con 0x56022ba5e920 2024-04-25T02:00:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.349+0000 7fe321d7f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56022b97dd90 con 0x56022ba5e000 2024-04-25T02:00:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.349+0000 7fe321d7f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56022ba6bf60 con 0x56022ba68ff0 2024-04-25T02:00:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.349+0000 7fe321d7f740 10 client.0 _ll_register_callbacks cb 0x56022b97d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:27.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.350+0000 7fe317400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e000 0x56022ba65f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.350+0000 7fe317400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e000 0x56022ba65f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59474/0 (socket says 172.21.15.102:59474) 2024-04-25T02:00:27.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.350+0000 7fe317400640 1 -- 172.21.15.102:0/358425957 learned_addr learned my addr 172.21.15.102:0/358425957 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:27.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.350+0000 7fe317400640 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba669c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.350+0000 7fe317e00640 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 0x56022ba66480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.351+0000 7fe317400640 1 -- 172.21.15.102:0/358425957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 msgr2=0x56022ba669c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:27.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.351+0000 7fe317400640 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba669c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.351+0000 7fe317400640 1 -- 172.21.15.102:0/358425957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 msgr2=0x56022ba66480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:27.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.351+0000 7fe317400640 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 0x56022ba66480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.351+0000 7fe317400640 1 -- 172.21.15.102:0/358425957 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56022b6d63f0 con 0x56022ba5e000 2024-04-25T02:00:27.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.351+0000 7fe31ce00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe2fc0a2dc0 0x7fe2fc0a5280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe3100cd170 tx=0x7fe310004990 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:27.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.351+0000 7fe317400640 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e000 0x56022ba65f40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe304009230 tx=0x7fe30402a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feecfa6a74a8b9e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:27.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.351+0000 7fe302000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:00:27.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.352+0000 7fe302000640 1 -- 172.21.15.102:0/358425957 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304035040 con 0x56022ba5e000 2024-04-25T02:00:27.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.352+0000 7fe321d7f740 1 -- 172.21.15.102:0/358425957 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe308019b10 con 0x56022ba5e000 2024-04-25T02:00:27.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.352+0000 7fe302000640 1 -- 172.21.15.102:0/358425957 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe304003de0 con 0x56022ba5e000 2024-04-25T02:00:27.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.353+0000 7fe302000640 1 -- 172.21.15.102:0/358425957 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe304031540 con 0x56022ba5e000 2024-04-25T02:00:27.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.353+0000 7fe321d7f740 10 client.26615 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:27.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.353+0000 7fe321d7f740 1 -- 172.21.15.102:0/358425957 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56022b6960f0 con 0x56022ba5e000 2024-04-25T02:00:27.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.354+0000 7fe302000640 1 -- 172.21.15.102:0/358425957 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe308019b10 con 0x56022ba5e000 2024-04-25T02:00:27.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.354+0000 7fe302000640 1 -- 172.21.15.102:0/358425957 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7fe30403a040 con 0x56022ba5e000 2024-04-25T02:00:27.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.355+0000 7fe321d7f740 10 client.26615 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:27.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.355+0000 7fe321d7f740 10 client.26615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:27.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.355+0000 7fe321d7f740 20 client.26615 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:27.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.355+0000 7fe321d7f740 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56022ba81a40 0x56022ba83e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.355+0000 7fe321d7f740 4 client.26615 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:27.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.355+0000 7fe321d7f740 1 -- 172.21.15.102:0/358425957 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56022b9d5d90 con 0x56022ba81a40 2024-04-25T02:00:27.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.355+0000 7fe317e00640 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56022ba81a40 0x56022ba83e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.356+0000 7fe302000640 10 client.26615 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:27.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.356+0000 7fe317e00640 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56022ba81a40 0x56022ba83e30 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9976a4b812d1e11 server_cookie=3734097a15e4e730 in_seq=0 out_seq=0 2024-04-25T02:00:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.359+0000 7fe302000640 1 -- 172.21.15.102:0/358425957 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56022b9d5d90 con 0x56022ba81a40 2024-04-25T02:00:27.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.360+0000 7fe302000640 10 client.26615 handle_command_reply: tid=0 2024-04-25T02:00:27.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.360+0000 7fe321d7f740 1 client.26615 shutdown 2024-04-25T02:00:27.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 -- 172.21.15.102:0/358425957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e000 msgr2=0x56022ba65f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e000 0x56022ba65f40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe304009230 tx=0x7fe30402a3f0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 -- 172.21.15.102:0/358425957 shutdown_connections 2024-04-25T02:00:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56022ba81a40 0x56022ba83e30 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba669c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 0x56022ba66480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 --2- 172.21.15.102:0/358425957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e000 0x56022ba65f40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 -- 172.21.15.102:0/358425957 >> 172.21.15.102:0/358425957 conn(0x56022ba5a320 msgr2=0x56022ba60020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe302000640 0 client.26615 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 -- 172.21.15.102:0/358425957 shutdown_connections 2024-04-25T02:00:27.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 1 -- 172.21.15.102:0/358425957 wait complete. 2024-04-25T02:00:27.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.362+0000 7fe321d7f740 20 client.26615 trim_cache size 0 max 16384 2024-04-25T02:00:27.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.396+0000 7fe321d7f740 1 Processor -- start 2024-04-25T02:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.397+0000 7fe321d7f740 1 -- start start 2024-04-25T02:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.397+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba7d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.397+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba896c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.397+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba89c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.397+0000 7fe321d7f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181adb90 con 0x56022ba5e000 2024-04-25T02:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.397+0000 7fe321d7f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181ab400 con 0x56022ba5e920 2024-04-25T02:00:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.397+0000 7fe321d7f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181a9270 con 0x56022ba68ff0 2024-04-25T02:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.398+0000 7fe317400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba7d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.398+0000 7fe317400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba7d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47272/0 (socket says 172.21.15.102:47272) 2024-04-25T02:00:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.398+0000 7fe317400640 1 -- 172.21.15.102:0/2027270778 learned_addr learned my addr 172.21.15.102:0/2027270778 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.399+0000 7fe317e00640 1 --2- 172.21.15.102:0/2027270778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba896c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.399+0000 7fe317400640 1 --2- 172.21.15.102:0/2027270778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba89c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.400+0000 7fe317400640 1 -- 172.21.15.102:0/2027270778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 msgr2=0x56022ba89c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.400+0000 7fe317400640 1 --2- 172.21.15.102:0/2027270778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba89c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.400+0000 7fe317400640 1 -- 172.21.15.102:0/2027270778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 msgr2=0x56022ba896c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.400+0000 7fe317400640 1 --2- 172.21.15.102:0/2027270778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba896c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.400+0000 7fe317400640 1 -- 172.21.15.102:0/2027270778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56022ba54890 con 0x56022ba5e000 2024-04-25T02:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.400+0000 7fe317400640 1 --2- 172.21.15.102:0/2027270778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba7d530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe30402ab40 tx=0x7fe304009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7ba794a0817b15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.401+0000 7fe30cc00640 1 -- 172.21.15.102:0/2027270778 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304035070 con 0x56022ba5e000 2024-04-25T02:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.401+0000 7fe30cc00640 1 -- 172.21.15.102:0/2027270778 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe30403d450 con 0x56022ba5e000 2024-04-25T02:00:27.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.401+0000 7fe30cc00640 1 -- 172.21.15.102:0/2027270778 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304031bc0 con 0x56022ba5e000 2024-04-25T02:00:27.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.404+0000 7fe321d7f740 1 -- 172.21.15.102:0/2027270778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 msgr2=0x56022ba7d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.404+0000 7fe321d7f740 1 --2- 172.21.15.102:0/2027270778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba7d530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe30402ab40 tx=0x7fe304009780 comp rx=0 tx=0).stop 2024-04-25T02:00:27.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.405+0000 7fe321d7f740 1 -- 172.21.15.102:0/2027270778 shutdown_connections 2024-04-25T02:00:27.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.405+0000 7fe321d7f740 1 --2- 172.21.15.102:0/2027270778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba68ff0 0x56022ba89c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.405+0000 7fe321d7f740 1 --2- 172.21.15.102:0/2027270778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba5e920 0x56022ba896c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.405+0000 7fe321d7f740 1 --2- 172.21.15.102:0/2027270778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e000 0x56022ba7d530 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.405+0000 7fe321d7f740 1 -- 172.21.15.102:0/2027270778 >> 172.21.15.102:0/2027270778 conn(0x56022ba5a320 msgr2=0x56022ba5f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:27.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.406+0000 7fe321d7f740 1 -- 172.21.15.102:0/2027270778 shutdown_connections 2024-04-25T02:00:27.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.406+0000 7fe321d7f740 1 -- 172.21.15.102:0/2027270778 wait complete. 2024-04-25T02:00:27.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.407+0000 7fe321d7f740 1 -- start start 2024-04-25T02:00:27.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.407+0000 7fe321d7f740 1 Processor -- start 2024-04-25T02:00:27.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.408+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba5e000 0x56022ba76c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.408+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 0x56022ba771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.408+0000 7fe321d7f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba68ff0 0x56022bae5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.408+0000 7fe321d7f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181a98c0 con 0x56022ba5e920 2024-04-25T02:00:27.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.408+0000 7fe321d7f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181aad30 con 0x56022ba68ff0 2024-04-25T02:00:27.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.408+0000 7fe321d7f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3181a9f90 con 0x56022ba5e000 2024-04-25T02:00:27.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.408+0000 7fe321d7f740 10 client.0 _ll_register_callbacks cb 0x7fe304003a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:27.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.409+0000 7fe317400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba68ff0 0x56022bae5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.409+0000 7fe317400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba68ff0 0x56022bae5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59500/0 (socket says 172.21.15.102:59500) 2024-04-25T02:00:27.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.409+0000 7fe317400640 1 -- 172.21.15.102:0/907822266 learned_addr learned my addr 172.21.15.102:0/907822266 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:27.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.409+0000 7fe317400640 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba5e000 0x56022ba76c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.409+0000 7fe317e00640 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 0x56022ba771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.410+0000 7fe317e00640 1 -- 172.21.15.102:0/907822266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba5e000 msgr2=0x56022ba76c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:27.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.410+0000 7fe317e00640 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba5e000 0x56022ba76c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.410+0000 7fe317e00640 1 -- 172.21.15.102:0/907822266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba68ff0 msgr2=0x56022bae5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:27.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.410+0000 7fe317e00640 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba68ff0 0x56022bae5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.410+0000 7fe317e00640 1 -- 172.21.15.102:0/907822266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56022bae6520 con 0x56022ba5e920 2024-04-25T02:00:27.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.411+0000 7fe317e00640 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 0x56022ba771b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe308011d10 tx=0x7fe3080094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd0142e0c7cc4f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:27.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.411+0000 7fe302000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:00:27.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.411+0000 7fe302000640 1 -- 172.21.15.102:0/907822266 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308030060 con 0x56022ba5e920 2024-04-25T02:00:27.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.412+0000 7fe302000640 1 -- 172.21.15.102:0/907822266 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fe3080050c0 con 0x56022ba5e920 2024-04-25T02:00:27.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.412+0000 7fe321d7f740 1 -- 172.21.15.102:0/907822266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe30403d450 con 0x56022ba5e920 2024-04-25T02:00:27.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.412+0000 7fe302000640 1 -- 172.21.15.102:0/907822266 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe308030060 con 0x56022ba5e920 2024-04-25T02:00:27.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.412+0000 7fe321d7f740 10 client.16179 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:27.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.412+0000 7fe321d7f740 1 -- 172.21.15.102:0/907822266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56022bae7870 con 0x56022ba5e920 2024-04-25T02:00:27.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.413+0000 7fe302000640 1 -- 172.21.15.102:0/907822266 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe30403d450 con 0x56022ba5e920 2024-04-25T02:00:27.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.413+0000 7fe302000640 1 -- 172.21.15.102:0/907822266 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7fe30802b250 con 0x56022ba5e920 2024-04-25T02:00:27.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.413+0000 7fe321d7f740 10 client.16179 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:27.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.413+0000 7fe321d7f740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:27.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.413+0000 7fe321d7f740 20 client.16179 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:27.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.413+0000 7fe321d7f740 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56022bae7e30 0x56022baea250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.413+0000 7fe321d7f740 4 client.16179 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T02:00:27.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.413+0000 7fe321d7f740 1 -- 172.21.15.102:0/907822266 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56022b9cfaa0 con 0x56022bae7e30 2024-04-25T02:00:27.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.414+0000 7fe317400640 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56022bae7e30 0x56022baea250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.416+0000 7fe317400640 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56022bae7e30 0x56022baea250 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34564d55ca8615d8 server_cookie=921deb4c0c104964 in_seq=0 out_seq=0 2024-04-25T02:00:27.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.416+0000 7fe302000640 10 client.16179 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:27.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.416+0000 7fe302000640 1 -- 172.21.15.102:0/907822266 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56022b9cfaa0 con 0x56022bae7e30 2024-04-25T02:00:27.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.416+0000 7fe302000640 10 client.16179 handle_command_reply: tid=0 2024-04-25T02:00:27.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.416+0000 7fe321d7f740 1 client.16179 shutdown 2024-04-25T02:00:27.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.416+0000 7fe321d7f740 1 -- 172.21.15.102:0/907822266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 msgr2=0x56022ba771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.416+0000 7fe321d7f740 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 0x56022ba771b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe308011d10 tx=0x7fe3080094d0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.417+0000 7fe321d7f740 1 -- 172.21.15.102:0/907822266 shutdown_connections 2024-04-25T02:00:27.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.417+0000 7fe321d7f740 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56022bae7e30 0x56022baea250 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.417+0000 7fe321d7f740 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56022ba68ff0 0x56022bae5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.417+0000 7fe302000640 0 client.16179 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:27.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.417+0000 7fe321d7f740 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56022ba5e920 0x56022ba771b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.417+0000 7fe321d7f740 1 --2- 172.21.15.102:0/907822266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56022ba5e000 0x56022ba76c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.417+0000 7fe321d7f740 1 -- 172.21.15.102:0/907822266 >> 172.21.15.102:0/907822266 conn(0x56022ba5a320 msgr2=0x56022ba59480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:27.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.418+0000 7fe321d7f740 1 -- 172.21.15.102:0/907822266 shutdown_connections 2024-04-25T02:00:27.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.418+0000 7fe321d7f740 1 -- 172.21.15.102:0/907822266 wait complete. 2024-04-25T02:00:27.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.418+0000 7fe321d7f740 20 client.16179 trim_cache size 0 max 16384 2024-04-25T02:00:27.422 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:00:27.422 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T02:00:27.422 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T02:00:27.423 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:00:27.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.419+0000 7fe31ee00640 1 -- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe2fc0a2dc0 msgr2=0x7fe2fc0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.419+0000 7fe31ee00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe2fc0a2dc0 0x7fe2fc0a5280 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe3100cd170 tx=0x7fe310004990 comp rx=0 tx=0).stop 2024-04-25T02:00:27.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.420+0000 7fe31ee00640 1 -- 172.21.15.102:0/3460769927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe318107770 msgr2=0x7fe31819eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.420+0000 7fe31ee00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe318107770 0x7fe31819eb70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe31000ec50 tx=0x7fe31000c6a0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.420+0000 7fe31ee00640 1 -- 172.21.15.102:0/3460769927 shutdown_connections 2024-04-25T02:00:27.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.420+0000 7fe31ee00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe2fc0a2dc0 0x7fe2fc0a5280 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.420+0000 7fe31ee00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe318107770 0x7fe31819eb70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.420+0000 7fe31ee00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe318102120 0x7fe31819a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.420+0000 7fe31ee00640 1 --2- 172.21.15.102:0/3460769927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318101760 0x7fe31819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.420+0000 7fe31ee00640 1 -- 172.21.15.102:0/3460769927 >> 172.21.15.102:0/3460769927 conn(0x7fe3180fd250 msgr2=0x7fe31810ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:27.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.421+0000 7fe31ee00640 1 -- 172.21.15.102:0/3460769927 shutdown_connections 2024-04-25T02:00:27.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.421+0000 7fe31ee00640 1 -- 172.21.15.102:0/3460769927 wait complete. 2024-04-25T02:00:27.451 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T02:00:27.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T02:00:27.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.725+0000 7f884ac00640 1 -- 172.21.15.102:0/2148255008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88440ffc50 msgr2=0x7f88441000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.725+0000 7f884ac00640 1 --2- 172.21.15.102:0/2148255008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88440ffc50 0x7f88441000b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f882c009900 tx=0x7f882c02a3f0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.726+0000 7f884ac00640 1 -- 172.21.15.102:0/2148255008 shutdown_connections 2024-04-25T02:00:27.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.726+0000 7f884ac00640 1 --2- 172.21.15.102:0/2148255008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88441005f0 0x7f884410e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.726+0000 7f884ac00640 1 --2- 172.21.15.102:0/2148255008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88440ffc50 0x7f88441000b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.726+0000 7f884ac00640 1 --2- 172.21.15.102:0/2148255008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8844107710 0x7f8844107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.726+0000 7f884ac00640 1 -- 172.21.15.102:0/2148255008 >> 172.21.15.102:0/2148255008 conn(0x7f8844076be0 msgr2=0x7f88440fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:27.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.726+0000 7f884ac00640 1 -- 172.21.15.102:0/2148255008 shutdown_connections 2024-04-25T02:00:27.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.727+0000 7f884ac00640 1 -- 172.21.15.102:0/2148255008 wait complete. 2024-04-25T02:00:27.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.730+0000 7f884ac00640 1 Processor -- start 2024-04-25T02:00:27.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.732+0000 7f884ac00640 1 -- start start 2024-04-25T02:00:27.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.732+0000 7f884ac00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88440ffc50 0x7f884419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.732+0000 7f884ac00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88441005f0 0x7f884419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.733+0000 7f8843e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88440ffc50 0x7f884419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.733+0000 7f8843e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88440ffc50 0x7f884419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60404/0 (socket says 172.21.15.102:60404) 2024-04-25T02:00:27.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.733+0000 7f8843400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88441005f0 0x7f884419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.733+0000 7f8843400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88441005f0 0x7f884419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59514/0 (socket says 172.21.15.102:59514) 2024-04-25T02:00:27.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.733+0000 7f8843e00640 1 -- 172.21.15.102:0/2502409311 learned_addr learned my addr 172.21.15.102:0/2502409311 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:27.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.733+0000 7f884ac00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8844107710 0x7f884419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.734+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8844111b60 con 0x7f8844107710 2024-04-25T02:00:27.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.734+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f88441119e0 con 0x7f88441005f0 2024-04-25T02:00:27.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.734+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8844111ce0 con 0x7f88440ffc50 2024-04-25T02:00:27.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.735+0000 7f8848c00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8844107710 0x7f884419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.735+0000 7f8843e00640 1 -- 172.21.15.102:0/2502409311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88441005f0 msgr2=0x7f884419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.735+0000 7f8843e00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88441005f0 0x7f884419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.735+0000 7f8843e00640 1 -- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8844107710 msgr2=0x7f884419eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:27.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.735+0000 7f8843e00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8844107710 0x7f884419eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:27.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.735+0000 7f8843e00640 1 -- 172.21.15.102:0/2502409311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884419f240 con 0x7f88440ffc50 2024-04-25T02:00:27.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.735+0000 7f8848c00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8844107710 0x7f884419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:27.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.735+0000 7f8843400640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88441005f0 0x7f884419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:27.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.736+0000 7f8843e00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88440ffc50 0x7f884419a250 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f883400d950 tx=0x7f883400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:27.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.736+0000 7f8840c00640 1 -- 172.21.15.102:0/2502409311 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8834017070 con 0x7f88440ffc50 2024-04-25T02:00:27.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.736+0000 7f8840c00640 1 -- 172.21.15.102:0/2502409311 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8834009c30 con 0x7f88440ffc50 2024-04-25T02:00:27.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.736+0000 7f8840c00640 1 -- 172.21.15.102:0/2502409311 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883400f6b0 con 0x7f88440ffc50 2024-04-25T02:00:27.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.736+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884419f530 con 0x7f88440ffc50 2024-04-25T02:00:27.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.737+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f884419f940 con 0x7f88440ffc50 2024-04-25T02:00:27.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.740+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8844067cd0 con 0x7f88440ffc50 2024-04-25T02:00:27.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.745+0000 7f8840c00640 1 -- 172.21.15.102:0/2502409311 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f88340109d0 con 0x7f88440ffc50 2024-04-25T02:00:27.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.747+0000 7f8840c00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f88280a2d80 0x7f88280a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:27.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.748+0000 7f8840c00640 1 -- 172.21.15.102:0/2502409311 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f88340cefb0 con 0x7f88440ffc50 2024-04-25T02:00:27.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.748+0000 7f8843400640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f88280a2d80 0x7f88280a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:27.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.749+0000 7f8843400640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f88280a2d80 0x7f88280a5240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f882c000c00 tx=0x7f882c005b50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:27.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:27.751+0000 7f8840c00640 1 -- 172.21.15.102:0/2502409311 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f884419f940 con 0x7f88440ffc50 2024-04-25T02:00:28.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 275.0 s, 3282.3 tps, lat 30.413 ms stddev 37.223 2024-04-25T02:00:28.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:27 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:00:28.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.207+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f884419b5a0 con 0x7f88440ffc50 2024-04-25T02:00:28.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.208+0000 7f8840c00640 1 -- 172.21.15.102:0/2502409311 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11900 (secure 0 0 0) 0x7f8834009de0 con 0x7f88440ffc50 2024-04-25T02:00:28.212 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:00:28.212 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T02:00:26.797071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24541,24587,24533]},"id":1}]} 2024-04-25T02:00:28.212 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 33 2024-04-25T02:00:28.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f88280a2d80 msgr2=0x7f88280a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f88280a2d80 0x7f88280a5240 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f882c000c00 tx=0x7f882c005b50 comp rx=0 tx=0).stop 2024-04-25T02:00:28.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88440ffc50 msgr2=0x7f884419a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88440ffc50 0x7f884419a250 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f883400d950 tx=0x7f883400de10 comp rx=0 tx=0).stop 2024-04-25T02:00:28.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 shutdown_connections 2024-04-25T02:00:28.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f88280a2d80 0x7f88280a5240 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8844107710 0x7f884419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88441005f0 0x7f884419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 --2- 172.21.15.102:0/2502409311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88440ffc50 0x7f884419a250 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.212+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 >> 172.21.15.102:0/2502409311 conn(0x7f8844076be0 msgr2=0x7f88440fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.213+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 shutdown_connections 2024-04-25T02:00:28.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.213+0000 7f884ac00640 1 -- 172.21.15.102:0/2502409311 wait complete. 2024-04-25T02:00:28.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:27 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:00:28.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T02:00:28.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:27 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:00:28.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.516+0000 7f8c68a00640 1 -- 172.21.15.102:0/3344466930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 msgr2=0x7f8c64105590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.516+0000 7f8c68a00640 1 --2- 172.21.15.102:0/3344466930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 0x7f8c64105590 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8c5001a910 tx=0x7f8c5003b540 comp rx=0 tx=0).stop 2024-04-25T02:00:28.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.516+0000 7f8c68a00640 1 -- 172.21.15.102:0/3344466930 shutdown_connections 2024-04-25T02:00:28.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.516+0000 7f8c68a00640 1 --2- 172.21.15.102:0/3344466930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c64105ad0 0x7f8c64105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.516+0000 7f8c68a00640 1 --2- 172.21.15.102:0/3344466930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 0x7f8c64105590 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.516+0000 7f8c68a00640 1 --2- 172.21.15.102:0/3344466930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c641007b0 0x7f8c64100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.516+0000 7f8c68a00640 1 -- 172.21.15.102:0/3344466930 >> 172.21.15.102:0/3344466930 conn(0x7f8c64077740 msgr2=0x7f8c64077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.516+0000 7f8c68a00640 1 -- 172.21.15.102:0/3344466930 shutdown_connections 2024-04-25T02:00:28.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.516+0000 7f8c68a00640 1 -- 172.21.15.102:0/3344466930 wait complete. 2024-04-25T02:00:28.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.518+0000 7f8c68a00640 1 Processor -- start 2024-04-25T02:00:28.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.518+0000 7f8c68a00640 1 -- start start 2024-04-25T02:00:28.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c68a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c641007b0 0x7f8c64114ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c68a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 0x7f8c64110b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c68a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c64105ad0 0x7f8c641110b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c68a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641094b0 con 0x7f8c641007b0 2024-04-25T02:00:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c68a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64109330 con 0x7f8c64105ad0 2024-04-25T02:00:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c68a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8c64109630 con 0x7f8c641010d0 2024-04-25T02:00:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c61400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 0x7f8c64110b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c61400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 0x7f8c64110b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60420/0 (socket says 172.21.15.102:60420) 2024-04-25T02:00:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c61400640 1 -- 172.21.15.102:0/4180007498 learned_addr learned my addr 172.21.15.102:0/4180007498 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.519+0000 7f8c62800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c64105ad0 0x7f8c641110b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.520+0000 7f8c61e00640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c641007b0 0x7f8c64114ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.520+0000 7f8c61400640 1 -- 172.21.15.102:0/4180007498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c64105ad0 msgr2=0x7f8c641110b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.520+0000 7f8c61400640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c64105ad0 0x7f8c641110b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.520+0000 7f8c61400640 1 -- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c641007b0 msgr2=0x7f8c64114ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.520+0000 7f8c61400640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c641007b0 0x7f8c64114ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.520+0000 7f8c61400640 1 -- 172.21.15.102:0/4180007498 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c641117e0 con 0x7f8c641010d0 2024-04-25T02:00:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.520+0000 7f8c61400640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 0x7f8c64110b70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8c5003ba60 tx=0x7f8c50040ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.521+0000 7f8c5aa00640 1 -- 172.21.15.102:0/4180007498 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5004f020 con 0x7f8c641010d0 2024-04-25T02:00:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.521+0000 7f8c68a00640 1 -- 172.21.15.102:0/4180007498 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c641a2510 con 0x7f8c641010d0 2024-04-25T02:00:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.521+0000 7f8c5aa00640 1 -- 172.21.15.102:0/4180007498 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c500052a0 con 0x7f8c641010d0 2024-04-25T02:00:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.521+0000 7f8c5aa00640 1 -- 172.21.15.102:0/4180007498 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c50046070 con 0x7f8c641010d0 2024-04-25T02:00:28.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.521+0000 7f8c68a00640 1 -- 172.21.15.102:0/4180007498 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c641a2910 con 0x7f8c641010d0 2024-04-25T02:00:28.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.526+0000 7f8c6b811740 1 Processor -- start 2024-04-25T02:00:28.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.526+0000 7f8c6b811740 1 -- start start 2024-04-25T02:00:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.527+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 0x55d0809019a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.527+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d08090c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.527+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d08090e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.527+0000 7f8c6b811740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d0808664e0 con 0x55d08090c5b0 2024-04-25T02:00:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.527+0000 7f8c6b811740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d08080ea70 con 0x55d080901ee0 2024-04-25T02:00:28.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.527+0000 7f8c6b811740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d0808624a0 con 0x55d0809015c0 2024-04-25T02:00:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c61e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d08090c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c61e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d08090c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59538/0 (socket says 172.21.15.102:59538) 2024-04-25T02:00:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c61e00640 1 -- 172.21.15.102:0/2379675502 learned_addr learned my addr 172.21.15.102:0/2379675502 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c62800640 1 --2- 172.21.15.102:0/2379675502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 0x55d0809019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c5aa00640 1 -- 172.21.15.102:0/4180007498 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8c50052020 con 0x7f8c641010d0 2024-04-25T02:00:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c61e00640 1 -- 172.21.15.102:0/2379675502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 msgr2=0x55d0809019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c61e00640 1 --2- 172.21.15.102:0/2379675502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 0x55d0809019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c61e00640 1 -- 172.21.15.102:0/2379675502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 msgr2=0x55d08090e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c61e00640 1 --2- 172.21.15.102:0/2379675502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d08090e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.528+0000 7f8c61e00640 1 -- 172.21.15.102:0/2379675502 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d08057a3f0 con 0x55d080901ee0 2024-04-25T02:00:28.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.529+0000 7f8c61e00640 1 --2- 172.21.15.102:0/2379675502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d08090c070 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400baa0 tx=0x7f8c5400bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7f7858576e618e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.529+0000 7f8c4f000640 1 -- 172.21.15.102:0/2379675502 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54017420 con 0x55d080901ee0 2024-04-25T02:00:28.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.529+0000 7f8c4f000640 1 -- 172.21.15.102:0/2379675502 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c54017aa0 con 0x55d080901ee0 2024-04-25T02:00:28.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.529+0000 7f8c4f000640 1 -- 172.21.15.102:0/2379675502 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5400fbe0 con 0x55d080901ee0 2024-04-25T02:00:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.530+0000 7f8c5aa00640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f8c480a2e00 0x7f8c480a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.530+0000 7f8c6b811740 1 -- 172.21.15.102:0/2379675502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 msgr2=0x55d08090c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.530+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2379675502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d08090c070 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400baa0 tx=0x7f8c5400bf60 comp rx=0 tx=0).stop 2024-04-25T02:00:28.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.530+0000 7f8c5aa00640 1 -- 172.21.15.102:0/4180007498 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8c50100940 con 0x7f8c641010d0 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.532+0000 7f8c62800640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f8c480a2e00 0x7f8c480a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.532+0000 7f8c6b811740 1 -- 172.21.15.102:0/2379675502 shutdown_connections 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.532+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2379675502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d08090e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.532+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2379675502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d08090c070 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.532+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2379675502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 0x55d0809019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.532+0000 7f8c6b811740 1 -- 172.21.15.102:0/2379675502 >> 172.21.15.102:0/2379675502 conn(0x55d0808fd8e0 msgr2=0x55d0808fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.533+0000 7f8c6b811740 1 -- 172.21.15.102:0/2379675502 shutdown_connections 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.533+0000 7f8c62800640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f8c480a2e00 0x7f8c480a52c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c005fd0 tx=0x7f8c5c009500 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.533+0000 7f8c6b811740 1 -- 172.21.15.102:0/2379675502 wait complete. 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.534+0000 7f8c6b811740 1 -- start start 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.534+0000 7f8c6b811740 1 Processor -- start 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d0809015c0 0x55d080909690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080909bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d08090c5b0 0x55d08090a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c6b811740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d08090f3a0 con 0x55d0809015c0 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c6b811740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d080821ec0 con 0x55d080901ee0 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c6b811740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d08090f520 con 0x55d08090c5b0 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c6b811740 10 client.0 _ll_register_callbacks cb 0x55d08085e3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c61e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d0809015c0 0x55d080909690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c61e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d0809015c0 0x55d080909690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47324/0 (socket says 172.21.15.102:47324) 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c61e00640 1 -- 172.21.15.102:0/2928082920 learned_addr learned my addr 172.21.15.102:0/2928082920 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c62800640 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d08090c5b0 0x55d08090a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.535+0000 7f8c61400640 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080909bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.536+0000 7f8c61e00640 1 -- 172.21.15.102:0/2928082920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d08090c5b0 msgr2=0x55d08090a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.536+0000 7f8c61e00640 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d08090c5b0 0x55d08090a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.536+0000 7f8c61e00640 1 -- 172.21.15.102:0/2928082920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 msgr2=0x55d080909bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.536+0000 7f8c61e00640 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080909bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.536+0000 7f8c61e00640 1 -- 172.21.15.102:0/2928082920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0806447d0 con 0x55d0809015c0 2024-04-25T02:00:28.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.537+0000 7f8c62800640 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d08090c5b0 0x55d08090a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:28.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.537+0000 7f8c61e00640 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d0809015c0 0x55d080909690 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400ba70 tx=0x7f8c54017e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e45d2addb3669a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.538+0000 7f8c43400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:00:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.539+0000 7f8c43400640 1 -- 172.21.15.102:0/2928082920 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c54027020 con 0x55d0809015c0 2024-04-25T02:00:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.540+0000 7f8c43400640 1 -- 172.21.15.102:0/2928082920 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8c54017830 con 0x55d0809015c0 2024-04-25T02:00:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.540+0000 7f8c6b811740 1 -- 172.21.15.102:0/2928082920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c54017aa0 con 0x55d0809015c0 2024-04-25T02:00:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.540+0000 7f8c43400640 1 -- 172.21.15.102:0/2928082920 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8c54032050 con 0x55d0809015c0 2024-04-25T02:00:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.541+0000 7f8c6b811740 10 client.16185 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:28.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.541+0000 7f8c6b811740 1 -- 172.21.15.102:0/2928082920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d08091b7a0 con 0x55d0809015c0 2024-04-25T02:00:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.546+0000 7f8c43400640 1 -- 172.21.15.102:0/2928082920 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c54017aa0 con 0x55d0809015c0 2024-04-25T02:00:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.546+0000 7f8c43400640 1 -- 172.21.15.102:0/2928082920 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f8c54023840 con 0x55d0809015c0 2024-04-25T02:00:28.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.546+0000 7f8c6b811740 10 client.16185 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:28.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.546+0000 7f8c6b811740 10 client.16185 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T02:00:28.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.546+0000 7f8c6b811740 20 client.16185 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:28.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.546+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55d08091cc60 0x55d08091f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.547+0000 7f8c6b811740 4 client.16185 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.547+0000 7f8c6b811740 1 -- 172.21.15.102:0/2928082920 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d080873a20 con 0x55d08091cc60 2024-04-25T02:00:28.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.547+0000 7f8c61400640 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55d08091cc60 0x55d08091f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.554+0000 7f8c43400640 10 client.16185 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T02:00:28.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.554+0000 7f8c61400640 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55d08091cc60 0x55d08091f050 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=20d17fde30fe3522 server_cookie=27d12c070bfefb2a in_seq=0 out_seq=0 2024-04-25T02:00:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.555+0000 7f8c43400640 1 -- 172.21.15.102:0/2928082920 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d080873a20 con 0x55d08091cc60 2024-04-25T02:00:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.555+0000 7f8c43400640 10 client.16185 handle_command_reply: tid=0 2024-04-25T02:00:28.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.555+0000 7f8c6b811740 1 client.16185 shutdown 2024-04-25T02:00:28.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.556+0000 7f8c6b811740 1 -- 172.21.15.102:0/2928082920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d0809015c0 msgr2=0x55d080909690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.556+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d0809015c0 0x55d080909690 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400ba70 tx=0x7f8c54017e80 comp rx=0 tx=0).stop 2024-04-25T02:00:28.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.557+0000 7f8c6b811740 1 -- 172.21.15.102:0/2928082920 shutdown_connections 2024-04-25T02:00:28.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.557+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d08090c5b0 0x55d08090a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.557+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55d08091cc60 0x55d08091f050 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.557+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080909bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.557+0000 7f8c6b811740 1 --2- 172.21.15.102:0/2928082920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d0809015c0 0x55d080909690 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.557+0000 7f8c43400640 0 client.16185 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T02:00:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.557+0000 7f8c6b811740 1 -- 172.21.15.102:0/2928082920 >> 172.21.15.102:0/2928082920 conn(0x55d0808fd8e0 msgr2=0x55d0808fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.559+0000 7f8c6b811740 1 -- 172.21.15.102:0/2928082920 shutdown_connections 2024-04-25T02:00:28.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.559+0000 7f8c6b811740 1 -- 172.21.15.102:0/2928082920 wait complete. 2024-04-25T02:00:28.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.559+0000 7f8c6b811740 20 client.16185 trim_cache size 0 max 16384 2024-04-25T02:00:28.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.588+0000 7f8c6b811740 1 Processor -- start 2024-04-25T02:00:28.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.588+0000 7f8c6b811740 1 -- start start 2024-04-25T02:00:28.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 0x55d080909f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080903d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d0808fc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c6b811740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641adc10 con 0x55d08090c5b0 2024-04-25T02:00:28.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c6b811740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641ab480 con 0x55d080901ee0 2024-04-25T02:00:28.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c6b811740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641a92d0 con 0x55d0809015c0 2024-04-25T02:00:28.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c61400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080903d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c61400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080903d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59554/0 (socket says 172.21.15.102:59554) 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c61400640 1 -- 172.21.15.102:0/1015302424 learned_addr learned my addr 172.21.15.102:0/1015302424 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c61e00640 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 0x55d080909f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.589+0000 7f8c61400640 1 -- 172.21.15.102:0/1015302424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 msgr2=0x55d080909f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c62800640 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d0808fc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c61400640 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 0x55d080909f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c61400640 1 -- 172.21.15.102:0/1015302424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 msgr2=0x55d0808fc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c61400640 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d0808fc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c61400640 1 -- 172.21.15.102:0/1015302424 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d08090f4d0 con 0x55d080901ee0 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c61e00640 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 0x55d080909f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c61400640 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080903d60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8c500c9800 tx=0x7f8c5003b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=901e31861c14421b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c4e600640 1 -- 172.21.15.102:0/1015302424 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c500cd530 con 0x55d080901ee0 2024-04-25T02:00:28.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c4e600640 1 -- 172.21.15.102:0/1015302424 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c500cce30 con 0x55d080901ee0 2024-04-25T02:00:28.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.590+0000 7f8c4e600640 1 -- 172.21.15.102:0/1015302424 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c50004850 con 0x55d080901ee0 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.591+0000 7f8c6b811740 1 -- 172.21.15.102:0/1015302424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 msgr2=0x55d080903d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.591+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080903d60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8c500c9800 tx=0x7f8c5003b540 comp rx=0 tx=0).stop 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.591+0000 7f8c6b811740 1 -- 172.21.15.102:0/1015302424 shutdown_connections 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.591+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d0808fc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.591+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d080901ee0 0x55d080903d60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.591+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1015302424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d0809015c0 0x55d080909f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.591+0000 7f8c6b811740 1 -- 172.21.15.102:0/1015302424 >> 172.21.15.102:0/1015302424 conn(0x55d0808fd8e0 msgr2=0x55d0808fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.591+0000 7f8c6b811740 1 -- 172.21.15.102:0/1015302424 shutdown_connections 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.591+0000 7f8c6b811740 1 -- 172.21.15.102:0/1015302424 wait complete. 2024-04-25T02:00:28.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.592+0000 7f8c6b811740 1 -- start start 2024-04-25T02:00:28.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.592+0000 7f8c6b811740 1 Processor -- start 2024-04-25T02:00:28.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.594+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d0809015c0 0x55d080914960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.594+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d080901ee0 0x55d080905d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.594+0000 7f8c6b811740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d080906250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.594+0000 7f8c6b811740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641a9940 con 0x55d08090c5b0 2024-04-25T02:00:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.594+0000 7f8c6b811740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641aadb0 con 0x55d0809015c0 2024-04-25T02:00:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.594+0000 7f8c6b811740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8c641aa010 con 0x55d080901ee0 2024-04-25T02:00:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.594+0000 7f8c6b811740 10 client.0 _ll_register_callbacks cb 0x7f8c5400b8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.595+0000 7f8c61e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d0809015c0 0x55d080914960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.595+0000 7f8c61e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d0809015c0 0x55d080914960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59570/0 (socket says 172.21.15.102:59570) 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.595+0000 7f8c61e00640 1 -- 172.21.15.102:0/1504758109 learned_addr learned my addr 172.21.15.102:0/1504758109 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.595+0000 7f8c61400640 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d080901ee0 0x55d080905d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.595+0000 7f8c62800640 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d080906250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c61e00640 1 -- 172.21.15.102:0/1504758109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d080901ee0 msgr2=0x55d080905d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c61e00640 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d080901ee0 0x55d080905d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c61e00640 1 -- 172.21.15.102:0/1504758109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 msgr2=0x55d080906250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c61e00640 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d080906250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c61e00640 1 -- 172.21.15.102:0/1504758109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d080906790 con 0x55d0809015c0 2024-04-25T02:00:28.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c61400640 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d080901ee0 0x55d080905d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:28.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c62800640 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d080906250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:28.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c61e00640 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d0809015c0 0x55d080914960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400baa0 tx=0x7f8c5400b6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe76fe52f28dd425 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c43400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:00:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c43400640 1 -- 172.21.15.102:0/1504758109 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5403b060 con 0x55d0809015c0 2024-04-25T02:00:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.596+0000 7f8c43400640 1 -- 172.21.15.102:0/1504758109 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8c54005c90 con 0x55d0809015c0 2024-04-25T02:00:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.597+0000 7f8c6b811740 1 -- 172.21.15.102:0/1504758109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c500cce30 con 0x55d0809015c0 2024-04-25T02:00:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.597+0000 7f8c43400640 1 -- 172.21.15.102:0/1504758109 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8c54027040 con 0x55d0809015c0 2024-04-25T02:00:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.598+0000 7f8c6b811740 10 client.26645 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.598+0000 7f8c6b811740 1 -- 172.21.15.102:0/1504758109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55d080982b70 con 0x55d0809015c0 2024-04-25T02:00:28.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.598+0000 7f8c43400640 1 -- 172.21.15.102:0/1504758109 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c500cce30 con 0x55d0809015c0 2024-04-25T02:00:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.598+0000 7f8c43400640 1 -- 172.21.15.102:0/1504758109 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f8c54023470 con 0x55d0809015c0 2024-04-25T02:00:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.599+0000 7f8c6b811740 10 client.26645 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.599+0000 7f8c6b811740 10 client.26645 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T02:00:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.599+0000 7f8c6b811740 20 client.26645 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.599+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55d0809832a0 0x55d0809856c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.599+0000 7f8c6b811740 4 client.26645 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:00:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.599+0000 7f8c6b811740 1 -- 172.21.15.102:0/1504758109 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d08086dd70 con 0x55d0809832a0 2024-04-25T02:00:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.599+0000 7f8c61400640 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55d0809832a0 0x55d0809856c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.600+0000 7f8c61400640 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55d0809832a0 0x55d0809856c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5910527a98ede8dd server_cookie=a170aa97d1df8914 in_seq=0 out_seq=0 2024-04-25T02:00:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.600+0000 7f8c43400640 10 client.26645 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T02:00:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.600+0000 7f8c43400640 1 -- 172.21.15.102:0/1504758109 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d08086dd70 con 0x55d0809832a0 2024-04-25T02:00:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.600+0000 7f8c43400640 10 client.26645 handle_command_reply: tid=0 2024-04-25T02:00:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.601+0000 7f8c6b811740 1 client.26645 shutdown 2024-04-25T02:00:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.602+0000 7f8c6b811740 1 -- 172.21.15.102:0/1504758109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d0809015c0 msgr2=0x55d080914960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.602+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d0809015c0 0x55d080914960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8c5400baa0 tx=0x7f8c5400b6b0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.602+0000 7f8c6b811740 1 -- 172.21.15.102:0/1504758109 shutdown_connections 2024-04-25T02:00:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.602+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d08090c5b0 0x55d080906250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.602+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d080901ee0 0x55d080905d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.602+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55d0809832a0 0x55d0809856c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.602+0000 7f8c6b811740 1 --2- 172.21.15.102:0/1504758109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d0809015c0 0x55d080914960 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.602+0000 7f8c43400640 0 client.26645 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T02:00:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.602+0000 7f8c6b811740 1 -- 172.21.15.102:0/1504758109 >> 172.21.15.102:0/1504758109 conn(0x55d0808fd8e0 msgr2=0x55d0808fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.603+0000 7f8c6b811740 1 -- 172.21.15.102:0/1504758109 shutdown_connections 2024-04-25T02:00:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.603+0000 7f8c6b811740 1 -- 172.21.15.102:0/1504758109 wait complete. 2024-04-25T02:00:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.603+0000 7f8c6b811740 20 client.26645 trim_cache size 0 max 16384 2024-04-25T02:00:28.608 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:00:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 -- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f8c480a2e00 msgr2=0x7f8c480a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f8c480a2e00 0x7f8c480a52c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c005fd0 tx=0x7f8c5c009500 comp rx=0 tx=0).stop 2024-04-25T02:00:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 -- 172.21.15.102:0/4180007498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 msgr2=0x7f8c64110b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 0x7f8c64110b70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8c5003ba60 tx=0x7f8c50040ec0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 -- 172.21.15.102:0/4180007498 shutdown_connections 2024-04-25T02:00:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f8c480a2e00 0x7f8c480a52c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c64105ad0 0x7f8c641110b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c641010d0 0x7f8c64110b70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 --2- 172.21.15.102:0/4180007498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c641007b0 0x7f8c64114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 -- 172.21.15.102:0/4180007498 >> 172.21.15.102:0/4180007498 conn(0x7f8c64077740 msgr2=0x7f8c64103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.604+0000 7f8c68a00640 1 -- 172.21.15.102:0/4180007498 shutdown_connections 2024-04-25T02:00:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.605+0000 7f8c68a00640 1 -- 172.21.15.102:0/4180007498 wait complete. 2024-04-25T02:00:28.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T02:00:28.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.916+0000 7f9392400640 1 -- 172.21.15.102:0/2689508914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 msgr2=0x7f938c073df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.916+0000 7f9392400640 1 --2- 172.21.15.102:0/2689508914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 0x7f938c073df0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f937c009900 tx=0x7f937c02a480 comp rx=0 tx=0).stop 2024-04-25T02:00:28.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.916+0000 7f9392400640 1 -- 172.21.15.102:0/2689508914 shutdown_connections 2024-04-25T02:00:28.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.916+0000 7f9392400640 1 --2- 172.21.15.102:0/2689508914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938c10e020 0x7f938c110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.916+0000 7f9392400640 1 --2- 172.21.15.102:0/2689508914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c074330 0x7f938c10dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.916+0000 7f9392400640 1 --2- 172.21.15.102:0/2689508914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 0x7f938c073df0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.916+0000 7f9392400640 1 -- 172.21.15.102:0/2689508914 >> 172.21.15.102:0/2689508914 conn(0x7f938c078b90 msgr2=0x7f938c078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.916+0000 7f9392400640 1 -- 172.21.15.102:0/2689508914 shutdown_connections 2024-04-25T02:00:28.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.916+0000 7f9392400640 1 -- 172.21.15.102:0/2689508914 wait complete. 2024-04-25T02:00:28.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.918+0000 7f9392400640 1 Processor -- start 2024-04-25T02:00:28.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.918+0000 7f9392400640 1 -- start start 2024-04-25T02:00:28.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.918+0000 7f9392400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 0x7f938c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.918+0000 7f9392400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c074330 0x7f938c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.919+0000 7f9392400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938c10e020 0x7f938c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.919+0000 7f9392400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f938c113df0 con 0x7f938c10e020 2024-04-25T02:00:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.919+0000 7f9392400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f938c113c70 con 0x7f938c074330 2024-04-25T02:00:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.919+0000 7f9392400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f938c113f70 con 0x7f938c073a10 2024-04-25T02:00:28.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.919+0000 7f938b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 0x7f938c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.919+0000 7f938b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 0x7f938c19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60476/0 (socket says 172.21.15.102:60476) 2024-04-25T02:00:28.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.919+0000 7f938b400640 1 -- 172.21.15.102:0/520910709 learned_addr learned my addr 172.21.15.102:0/520910709 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.920+0000 7f938b400640 1 -- 172.21.15.102:0/520910709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c074330 msgr2=0x7f938c19a830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.920+0000 7f938b400640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c074330 0x7f938c19a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.920+0000 7f938b400640 1 -- 172.21.15.102:0/520910709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938c10e020 msgr2=0x7f938c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.920+0000 7f938b400640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938c10e020 0x7f938c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.920+0000 7f938b400640 1 -- 172.21.15.102:0/520910709 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938c19f2e0 con 0x7f938c073a10 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.920+0000 7f938b400640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 0x7f938c19a2f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f937c004770 tx=0x7f937c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.920+0000 7f9383e00640 1 -- 172.21.15.102:0/520910709 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c03f070 con 0x7f938c073a10 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.920+0000 7f9383e00640 1 -- 172.21.15.102:0/520910709 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937c031ca0 con 0x7f938c073a10 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.920+0000 7f9383e00640 1 -- 172.21.15.102:0/520910709 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c035030 con 0x7f938c073a10 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.921+0000 7f9392400640 1 -- 172.21.15.102:0/520910709 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f938c19f570 con 0x7f938c073a10 2024-04-25T02:00:28.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.921+0000 7f9392400640 1 -- 172.21.15.102:0/520910709 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f938c1a6e60 con 0x7f938c073a10 2024-04-25T02:00:28.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.926+0000 7f9395231740 1 Processor -- start 2024-04-25T02:00:28.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.926+0000 7f9383e00640 1 -- 172.21.15.102:0/520910709 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f937c042070 con 0x7f938c073a10 2024-04-25T02:00:28.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f9395231740 1 -- start start 2024-04-25T02:00:28.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f9395231740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x56248049b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f9395231740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 0x5624804a6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f9395231740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804a89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f9395231740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5624804004e0 con 0x56248049b5c0 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f9395231740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5624803a8a70 con 0x56248049bee0 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f9395231740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5624803fc4a0 con 0x5624804a65b0 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f938b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 0x5624804a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f938b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 0x5624804a6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59596/0 (socket says 172.21.15.102:59596) 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.927+0000 7f938b400640 1 -- 172.21.15.102:0/2739901963 learned_addr learned my addr 172.21.15.102:0/2739901963 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f938be00640 1 --2- 172.21.15.102:0/2739901963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x56248049b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f938b400640 1 -- 172.21.15.102:0/2739901963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 msgr2=0x5624804a89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f938b400640 1 --2- 172.21.15.102:0/2739901963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804a89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f938b400640 1 -- 172.21.15.102:0/2739901963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 msgr2=0x56248049b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f938b400640 1 --2- 172.21.15.102:0/2739901963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x56248049b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f938b400640 1 -- 172.21.15.102:0/2739901963 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624801143f0 con 0x56248049bee0 2024-04-25T02:00:28.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f938be00640 1 --2- 172.21.15.102:0/2739901963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x56248049b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:28.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f938b400640 1 --2- 172.21.15.102:0/2739901963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 0x5624804a6070 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f937c0ed570 tx=0x7f937c0f8320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7896aa5df4e8f70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f9380c00640 1 -- 172.21.15.102:0/2739901963 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c102470 con 0x56248049bee0 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f9380c00640 1 -- 172.21.15.102:0/2739901963 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937c0fd070 con 0x56248049bee0 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.928+0000 7f9380c00640 1 -- 172.21.15.102:0/2739901963 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c100920 con 0x56248049bee0 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.929+0000 7f9383e00640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f93700a2d80 0x7f93700a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.929+0000 7f9383e00640 1 -- 172.21.15.102:0/520910709 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f937c0efe80 con 0x7f938c073a10 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.929+0000 7f9395231740 1 -- 172.21.15.102:0/2739901963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 msgr2=0x5624804a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.929+0000 7f9395231740 1 --2- 172.21.15.102:0/2739901963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 0x5624804a6070 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f937c0ed570 tx=0x7f937c0f8320 comp rx=0 tx=0).stop 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.930+0000 7f9395231740 1 -- 172.21.15.102:0/2739901963 shutdown_connections 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.930+0000 7f9395231740 1 --2- 172.21.15.102:0/2739901963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804a89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.930+0000 7f9395231740 1 --2- 172.21.15.102:0/2739901963 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 0x5624804a6070 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.930+0000 7f9395231740 1 --2- 172.21.15.102:0/2739901963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x56248049b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.930+0000 7f9395231740 1 -- 172.21.15.102:0/2739901963 >> 172.21.15.102:0/2739901963 conn(0x5624804978e0 msgr2=0x562480497cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.930+0000 7f938be00640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f93700a2d80 0x7f93700a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.930+0000 7f9395231740 1 -- 172.21.15.102:0/2739901963 shutdown_connections 2024-04-25T02:00:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.930+0000 7f9395231740 1 -- 172.21.15.102:0/2739901963 wait complete. 2024-04-25T02:00:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.931+0000 7f9395231740 1 -- start start 2024-04-25T02:00:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.931+0000 7f938be00640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f93700a2d80 0x7f93700a5240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f938c06b010 tx=0x7f93840072c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.931+0000 7f9395231740 1 Processor -- start 2024-04-25T02:00:28.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.933+0000 7f9395231740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.933+0000 7f9395231740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 0x5624804a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.933+0000 7f9395231740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.933+0000 7f9395231740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5624803a8a70 con 0x56248049b5c0 2024-04-25T02:00:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.933+0000 7f9395231740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5624803bbec0 con 0x56248049bee0 2024-04-25T02:00:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.933+0000 7f9395231740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5624804a9160 con 0x5624804a65b0 2024-04-25T02:00:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f9395231740 10 client.0 _ll_register_callbacks cb 0x5624803f83f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804a3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60482/0 (socket says 172.21.15.102:60482) 2024-04-25T02:00:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938be00640 1 -- 172.21.15.102:0/2724304292 learned_addr learned my addr 172.21.15.102:0/2724304292 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938b400640 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938be00640 1 -- 172.21.15.102:0/2724304292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 msgr2=0x5624804a3a90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:00:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938be00640 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 0x5624804a3a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938be00640 1 -- 172.21.15.102:0/2724304292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 msgr2=0x5624804a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938be00640 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804a3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938be00640 1 -- 172.21.15.102:0/2724304292 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624801de7d0 con 0x5624804a65b0 2024-04-25T02:00:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.934+0000 7f938b400640 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.935+0000 7f938be00640 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804a3fd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f938400fdb0 tx=0x7f9384004bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9908681189eb66a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.935+0000 7f9375600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:00:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.935+0000 7f9375600640 1 -- 172.21.15.102:0/2724304292 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9384020020 con 0x5624804a65b0 2024-04-25T02:00:28.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.936+0000 7f9375600640 1 -- 172.21.15.102:0/2724304292 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f9384007d10 con 0x5624804a65b0 2024-04-25T02:00:28.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.936+0000 7f9395231740 1 -- 172.21.15.102:0/2724304292 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f937c0fd070 con 0x5624804a65b0 2024-04-25T02:00:28.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.936+0000 7f9375600640 1 -- 172.21.15.102:0/2724304292 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9384008040 con 0x5624804a65b0 2024-04-25T02:00:28.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.936+0000 7f9395231740 10 client.26614 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:28.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.937+0000 7f9395231740 1 -- 172.21.15.102:0/2724304292 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5624804bd9f0 con 0x5624804a65b0 2024-04-25T02:00:28.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.937+0000 7f9375600640 1 -- 172.21.15.102:0/2724304292 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937c0fd070 con 0x5624804a65b0 2024-04-25T02:00:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.938+0000 7f9375600640 1 -- 172.21.15.102:0/2724304292 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f938402d020 con 0x5624804a65b0 2024-04-25T02:00:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.938+0000 7f9395231740 10 client.26614 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.938+0000 7f9395231740 10 client.26614 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T02:00:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.938+0000 7f9395231740 20 client.26614 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.938+0000 7f9395231740 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624804befd0 0x5624804c1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.938+0000 7f9395231740 4 client.26614 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:28.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.938+0000 7f9395231740 1 -- 172.21.15.102:0/2724304292 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56248040da20 con 0x5624804befd0 2024-04-25T02:00:28.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.939+0000 7f938b400640 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624804befd0 0x5624804c1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.939+0000 7f938b400640 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624804befd0 0x5624804c1470 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=315a10675a6e6a57 server_cookie=ae6bad3704c5f700 in_seq=0 out_seq=0 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.939+0000 7f9375600640 10 client.26614 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.940+0000 7f9375600640 1 -- 172.21.15.102:0/2724304292 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56248040da20 con 0x5624804befd0 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.940+0000 7f9375600640 10 client.26614 handle_command_reply: tid=0 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9395231740 1 client.26614 shutdown 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9395231740 1 -- 172.21.15.102:0/2724304292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 msgr2=0x5624804a3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9395231740 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804a3fd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f938400fdb0 tx=0x7f9384004bc0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9395231740 1 -- 172.21.15.102:0/2724304292 shutdown_connections 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9395231740 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624804befd0 0x5624804c1470 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9395231740 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804a3fd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9375600640 0 client.26614 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9395231740 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049bee0 0x5624804a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9395231740 1 --2- 172.21.15.102:0/2724304292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.941+0000 7f9395231740 1 -- 172.21.15.102:0/2724304292 >> 172.21.15.102:0/2724304292 conn(0x5624804978e0 msgr2=0x562480496b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.942+0000 7f9395231740 1 -- 172.21.15.102:0/2724304292 shutdown_connections 2024-04-25T02:00:28.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.942+0000 7f9395231740 1 -- 172.21.15.102:0/2724304292 wait complete. 2024-04-25T02:00:28.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.942+0000 7f9395231740 20 client.26614 trim_cache size 0 max 16384 2024-04-25T02:00:28.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.970+0000 7f9395231740 1 Processor -- start 2024-04-25T02:00:28.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.970+0000 7f9395231740 1 -- start start 2024-04-25T02:00:28.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f9395231740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049b5c0 0x5624804972c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f9395231740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049bee0 0x5624804a3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f9395231740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f9395231740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1adbb0 con 0x56248049bee0 2024-04-25T02:00:28.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f9395231740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1ab420 con 0x56248049b5c0 2024-04-25T02:00:28.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f9395231740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1a9270 con 0x5624804a65b0 2024-04-25T02:00:28.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f938b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049b5c0 0x5624804972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f938b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049b5c0 0x5624804972c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59616/0 (socket says 172.21.15.102:59616) 2024-04-25T02:00:28.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f938b400640 1 -- 172.21.15.102:0/2839729642 learned_addr learned my addr 172.21.15.102:0/2839729642 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.971+0000 7f938be00640 1 --2- 172.21.15.102:0/2839729642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.972+0000 7f938b400640 1 -- 172.21.15.102:0/2839729642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 msgr2=0x5624804c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.972+0000 7f938b400640 1 --2- 172.21.15.102:0/2839729642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.972+0000 7f938b400640 1 -- 172.21.15.102:0/2839729642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049bee0 msgr2=0x5624804a3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.972+0000 7f938b400640 1 --2- 172.21.15.102:0/2839729642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049bee0 0x5624804a3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.972+0000 7f938b400640 1 -- 172.21.15.102:0/2839729642 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624804a1aa0 con 0x56248049b5c0 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.972+0000 7f938b400640 1 --2- 172.21.15.102:0/2839729642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049b5c0 0x5624804972c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f937c0b8830 tx=0x7f937c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db9726c7528cbde7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.973+0000 7f9377e00640 1 -- 172.21.15.102:0/2839729642 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c0b8670 con 0x56248049b5c0 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.973+0000 7f9377e00640 1 -- 172.21.15.102:0/2839729642 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937c0fd040 con 0x56248049b5c0 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.973+0000 7f9377e00640 1 -- 172.21.15.102:0/2839729642 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c0bbd30 con 0x56248049b5c0 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.974+0000 7f9395231740 1 -- 172.21.15.102:0/2839729642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049b5c0 msgr2=0x5624804972c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.974+0000 7f9395231740 1 --2- 172.21.15.102:0/2839729642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049b5c0 0x5624804972c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f937c0b8830 tx=0x7f937c009260 comp rx=0 tx=0).stop 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.974+0000 7f9395231740 1 -- 172.21.15.102:0/2839729642 shutdown_connections 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.974+0000 7f9395231740 1 --2- 172.21.15.102:0/2839729642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624804a65b0 0x5624804c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.974+0000 7f9395231740 1 --2- 172.21.15.102:0/2839729642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049bee0 0x5624804a3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.974+0000 7f9395231740 1 --2- 172.21.15.102:0/2839729642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56248049b5c0 0x5624804972c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.974+0000 7f9395231740 1 -- 172.21.15.102:0/2839729642 >> 172.21.15.102:0/2839729642 conn(0x5624804978e0 msgr2=0x5624804a3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:28.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.975+0000 7f9395231740 1 -- 172.21.15.102:0/2839729642 shutdown_connections 2024-04-25T02:00:28.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.975+0000 7f9395231740 1 -- 172.21.15.102:0/2839729642 wait complete. 2024-04-25T02:00:28.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.976+0000 7f9395231740 1 -- start start 2024-04-25T02:00:28.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.976+0000 7f9395231740 1 Processor -- start 2024-04-25T02:00:28.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.979+0000 7f9395231740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.979+0000 7f9395231740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56248049bee0 0x5624804b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.980+0000 7f9395231740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624804a65b0 0x5624805245b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.980+0000 7f938b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.980+0000 7f938aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56248049bee0 0x5624804b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.980+0000 7f938b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804b2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47392/0 (socket says 172.21.15.102:47392) 2024-04-25T02:00:28.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.980+0000 7f938b400640 1 -- 172.21.15.102:0/3819380891 learned_addr learned my addr 172.21.15.102:0/3819380891 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:28.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.981+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1a98e0 con 0x56248049b5c0 2024-04-25T02:00:28.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.981+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1aad50 con 0x5624804a65b0 2024-04-25T02:00:28.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.981+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f938c1a9fb0 con 0x56248049bee0 2024-04-25T02:00:28.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.981+0000 7f9395231740 10 client.0 _ll_register_callbacks cb 0x7f9384007900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:28.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.982+0000 7f938be00640 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624804a65b0 0x5624805245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.982+0000 7f938aa00640 1 -- 172.21.15.102:0/3819380891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624804a65b0 msgr2=0x5624805245b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.982+0000 7f938aa00640 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624804a65b0 0x5624805245b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.982+0000 7f938aa00640 1 -- 172.21.15.102:0/3819380891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 msgr2=0x5624804b2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:28.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.982+0000 7f938aa00640 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804b2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.983+0000 7f938aa00640 1 -- 172.21.15.102:0/3819380891 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562480524af0 con 0x56248049bee0 2024-04-25T02:00:28.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.983+0000 7f938be00640 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624804a65b0 0x5624805245b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:28.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.986+0000 7f938aa00640 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56248049bee0 0x5624804b2f60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x5624804a48e0 tx=0x7f937800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98dc1b6e51abe1c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:28.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.986+0000 7f9375600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:00:28.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.987+0000 7f9375600640 1 -- 172.21.15.102:0/3819380891 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9378004030 con 0x56248049bee0 2024-04-25T02:00:28.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.988+0000 7f9375600640 1 -- 172.21.15.102:0/3819380891 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f93780041d0 con 0x56248049bee0 2024-04-25T02:00:28.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.988+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f937c0fd040 con 0x56248049bee0 2024-04-25T02:00:28.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.989+0000 7f9375600640 1 -- 172.21.15.102:0/3819380891 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9378004410 con 0x56248049bee0 2024-04-25T02:00:28.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.989+0000 7f9395231740 10 client.26620 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:28.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.989+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562480525cf0 con 0x56248049bee0 2024-04-25T02:00:28.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.990+0000 7f9375600640 1 -- 172.21.15.102:0/3819380891 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937c0fd040 con 0x56248049bee0 2024-04-25T02:00:28.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.990+0000 7f9375600640 1 -- 172.21.15.102:0/3819380891 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f9378003a10 con 0x56248049bee0 2024-04-25T02:00:28.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.990+0000 7f9395231740 10 client.26620 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:28.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.990+0000 7f9395231740 10 client.26620 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T02:00:28.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.990+0000 7f9395231740 20 client.26620 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:28.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.990+0000 7f9395231740 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x562480526500 0x562480528920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:28.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.990+0000 7f9395231740 4 client.26620 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:00:28.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.990+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562480407d70 con 0x562480526500 2024-04-25T02:00:28.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.991+0000 7f938b400640 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x562480526500 0x562480528920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:28.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.992+0000 7f9375600640 10 client.26620 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T02:00:28.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.993+0000 7f938b400640 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x562480526500 0x562480528920 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3c0047a9fcc175fc server_cookie=eaa18662fa79446d in_seq=0 out_seq=0 2024-04-25T02:00:28.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.993+0000 7f9375600640 1 -- 172.21.15.102:0/3819380891 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562480407d70 con 0x562480526500 2024-04-25T02:00:28.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.993+0000 7f9375600640 10 client.26620 handle_command_reply: tid=0 2024-04-25T02:00:28.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.994+0000 7f9395231740 1 client.26620 shutdown 2024-04-25T02:00:28.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.995+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56248049bee0 msgr2=0x5624804b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.995+0000 7f9395231740 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56248049bee0 0x5624804b2f60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x5624804a48e0 tx=0x7f937800e720 comp rx=0 tx=0).stop 2024-04-25T02:00:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.995+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 shutdown_connections 2024-04-25T02:00:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.995+0000 7f9395231740 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x562480526500 0x562480528920 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.995+0000 7f9395231740 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624804a65b0 0x5624805245b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.995+0000 7f9395231740 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56248049bee0 0x5624804b2f60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.995+0000 7f9395231740 1 --2- 172.21.15.102:0/3819380891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56248049b5c0 0x5624804b2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.995+0000 7f9375600640 0 client.26620 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T02:00:28.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.995+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 >> 172.21.15.102:0/3819380891 conn(0x5624804978e0 msgr2=0x5624804a3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:29.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.996+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 shutdown_connections 2024-04-25T02:00:29.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.996+0000 7f9395231740 1 -- 172.21.15.102:0/3819380891 wait complete. 2024-04-25T02:00:29.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.997+0000 7f9395231740 20 client.26620 trim_cache size 0 max 16384 2024-04-25T02:00:29.001 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:00:29.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.998+0000 7f9392400640 1 -- 172.21.15.102:0/520910709 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f93700a2d80 msgr2=0x7f93700a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.998+0000 7f9392400640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f93700a2d80 0x7f93700a5240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f938c06b010 tx=0x7f93840072c0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.998+0000 7f9392400640 1 -- 172.21.15.102:0/520910709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 msgr2=0x7f938c19a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.998+0000 7f9392400640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 0x7f938c19a2f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f937c004770 tx=0x7f937c003c10 comp rx=0 tx=0).stop 2024-04-25T02:00:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.999+0000 7f9392400640 1 -- 172.21.15.102:0/520910709 shutdown_connections 2024-04-25T02:00:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.999+0000 7f9392400640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f93700a2d80 0x7f93700a5240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.999+0000 7f9392400640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938c10e020 0x7f938c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.999+0000 7f9392400640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c074330 0x7f938c19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.999+0000 7f9392400640 1 --2- 172.21.15.102:0/520910709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f938c073a10 0x7f938c19a2f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:28.999+0000 7f9392400640 1 -- 172.21.15.102:0/520910709 >> 172.21.15.102:0/520910709 conn(0x7f938c078b90 msgr2=0x7f938c0fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:29.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.000+0000 7f9392400640 1 -- 172.21.15.102:0/520910709 shutdown_connections 2024-04-25T02:00:29.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.000+0000 7f9392400640 1 -- 172.21.15.102:0/520910709 wait complete. 2024-04-25T02:00:29.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T02:00:29.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:28 smithi102 ceph-mon[27796]: pgmap v159: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 11 MiB/s wr, 961 op/s 2024-04-25T02:00:29.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:28 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2502409311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:00:29.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:28 smithi110 ceph-mon[32848]: pgmap v159: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 11 MiB/s wr, 961 op/s 2024-04-25T02:00:29.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:28 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2502409311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:00:29.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:28 smithi193 ceph-mon[32981]: pgmap v159: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 11 MiB/s wr, 961 op/s 2024-04-25T02:00:29.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:28 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2502409311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:00:29.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.289+0000 7f0ae9600640 1 -- 172.21.15.102:0/2367043387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 msgr2=0x7f0ae410dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.289+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2367043387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 0x7f0ae410dc10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009900 tx=0x7f0acc02a400 comp rx=0 tx=0).stop 2024-04-25T02:00:29.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.289+0000 7f0ae9600640 1 -- 172.21.15.102:0/2367043387 shutdown_connections 2024-04-25T02:00:29.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.289+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2367043387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae410e150 0x7f0ae4110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.289+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2367043387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 0x7f0ae410dc10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.289+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2367043387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ae41005a0 0x7f0ae4100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.289+0000 7f0ae9600640 1 -- 172.21.15.102:0/2367043387 >> 172.21.15.102:0/2367043387 conn(0x7f0ae40777a0 msgr2=0x7f0ae4077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:29.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.289+0000 7f0ae9600640 1 -- 172.21.15.102:0/2367043387 shutdown_connections 2024-04-25T02:00:29.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.289+0000 7f0ae9600640 1 -- 172.21.15.102:0/2367043387 wait complete. 2024-04-25T02:00:29.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.290+0000 7f0ae9600640 1 Processor -- start 2024-04-25T02:00:29.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.290+0000 7f0ae9600640 1 -- start start 2024-04-25T02:00:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.291+0000 7f0ae9600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae41005a0 0x7f0ae419a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.291+0000 7f0ae9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 0x7f0ae419a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.291+0000 7f0ae9600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ae410e150 0x7f0ae419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.291+0000 7f0ae9600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4113d80 con 0x7f0ae41005a0 2024-04-25T02:00:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.291+0000 7f0ae9600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4113c00 con 0x7f0ae4100ec0 2024-04-25T02:00:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.291+0000 7f0ae9600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4113f00 con 0x7f0ae410e150 2024-04-25T02:00:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.291+0000 7f0ae2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 0x7f0ae419a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.291+0000 7f0ae2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 0x7f0ae419a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59642/0 (socket says 172.21.15.102:59642) 2024-04-25T02:00:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.291+0000 7f0ae2000640 1 -- 172.21.15.102:0/2182903948 learned_addr learned my addr 172.21.15.102:0/2182903948 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.292+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae41005a0 0x7f0ae419a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.292+0000 7f0ae2000640 1 -- 172.21.15.102:0/2182903948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ae410e150 msgr2=0x7f0ae419ec70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T02:00:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.292+0000 7f0ae3400640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ae410e150 0x7f0ae419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.292+0000 7f0ae2000640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ae410e150 0x7f0ae419ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.292+0000 7f0ae2000640 1 -- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae41005a0 msgr2=0x7f0ae419a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.292+0000 7f0ae2000640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae41005a0 0x7f0ae419a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.292+0000 7f0ae2000640 1 -- 172.21.15.102:0/2182903948 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae419f3a0 con 0x7f0ae4100ec0 2024-04-25T02:00:29.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.292+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae41005a0 0x7f0ae419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:29.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.292+0000 7f0ae2000640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 0x7f0ae419a8f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0acc02a970 tx=0x7f0acc003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:29.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.293+0000 7f0adb400640 1 -- 172.21.15.102:0/2182903948 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc03f070 con 0x7f0ae4100ec0 2024-04-25T02:00:29.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.293+0000 7f0adb400640 1 -- 172.21.15.102:0/2182903948 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0acc031c90 con 0x7f0ae4100ec0 2024-04-25T02:00:29.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.293+0000 7f0adb400640 1 -- 172.21.15.102:0/2182903948 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc035030 con 0x7f0ae4100ec0 2024-04-25T02:00:29.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.293+0000 7f0ae9600640 1 -- 172.21.15.102:0/2182903948 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae419f630 con 0x7f0ae4100ec0 2024-04-25T02:00:29.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.294+0000 7f0ae9600640 1 -- 172.21.15.102:0/2182903948 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae41027b0 con 0x7f0ae4100ec0 2024-04-25T02:00:29.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.298+0000 7f0adb400640 1 -- 172.21.15.102:0/2182903948 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0acc042070 con 0x7f0ae4100ec0 2024-04-25T02:00:29.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.299+0000 7f0aec47d740 1 Processor -- start 2024-04-25T02:00:29.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.300+0000 7f0aec47d740 1 -- start start 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.300+0000 7f0adb400640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0ac80a2dc0 0x7f0ac80a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.300+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 0x561da54989a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da5498ee0 0x561da54a3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 0x561da54a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0aec47d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561da53fd4e0 con 0x561da54a35b0 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0aec47d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561da53a5a70 con 0x561da5498ee0 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0aec47d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561da53f94a0 con 0x561da54985c0 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0ae2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da5498ee0 0x561da54a3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0ae2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da5498ee0 0x561da54a3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59658/0 (socket says 172.21.15.102:59658) 2024-04-25T02:00:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0ae2000640 1 -- 172.21.15.102:0/390953328 learned_addr learned my addr 172.21.15.102:0/390953328 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:29.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0ae2000640 1 -- 172.21.15.102:0/390953328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 msgr2=0x561da54989a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:00:29.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0ae2000640 1 --2- 172.21.15.102:0/390953328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 0x561da54989a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0ae2000640 1 -- 172.21.15.102:0/390953328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 msgr2=0x561da54a59a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:00:29.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0ae2000640 1 --2- 172.21.15.102:0/390953328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 0x561da54a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0adb400640 1 -- 172.21.15.102:0/2182903948 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0acc0ef930 con 0x7f0ae4100ec0 2024-04-25T02:00:29.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.301+0000 7f0ae2000640 1 -- 172.21.15.102:0/390953328 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561da51113f0 con 0x561da5498ee0 2024-04-25T02:00:29.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.302+0000 7f0ae2000640 1 --2- 172.21.15.102:0/390953328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da5498ee0 0x561da54a3070 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0c4560 tx=0x7f0acc031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=486265cbc689e02b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:29.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.302+0000 7f0ad3400640 1 -- 172.21.15.102:0/390953328 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc049040 con 0x561da5498ee0 2024-04-25T02:00:29.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.302+0000 7f0ad3400640 1 -- 172.21.15.102:0/390953328 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0acc0c01d0 con 0x561da5498ee0 2024-04-25T02:00:29.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.302+0000 7f0ad3400640 1 -- 172.21.15.102:0/390953328 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc0c4600 con 0x561da5498ee0 2024-04-25T02:00:29.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.302+0000 7f0ae3400640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0ac80a2dc0 0x7f0ac80a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.304+0000 7f0ae3400640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0ac80a2dc0 0x7f0ac80a5280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x561da5495a70 tx=0x7f0adc005ed0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:29.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.304+0000 7f0aec47d740 1 -- 172.21.15.102:0/390953328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da5498ee0 msgr2=0x561da54a3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.304+0000 7f0aec47d740 1 --2- 172.21.15.102:0/390953328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da5498ee0 0x561da54a3070 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0c4560 tx=0x7f0acc031400 comp rx=0 tx=0).stop 2024-04-25T02:00:29.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.305+0000 7f0aec47d740 1 -- 172.21.15.102:0/390953328 shutdown_connections 2024-04-25T02:00:29.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.305+0000 7f0aec47d740 1 --2- 172.21.15.102:0/390953328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 0x561da54a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.305+0000 7f0aec47d740 1 --2- 172.21.15.102:0/390953328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da5498ee0 0x561da54a3070 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.305+0000 7f0aec47d740 1 --2- 172.21.15.102:0/390953328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 0x561da54989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.305+0000 7f0aec47d740 1 -- 172.21.15.102:0/390953328 >> 172.21.15.102:0/390953328 conn(0x561da54948e0 msgr2=0x561da5494cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:29.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.305+0000 7f0aec47d740 1 -- 172.21.15.102:0/390953328 shutdown_connections 2024-04-25T02:00:29.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.305+0000 7f0aec47d740 1 -- 172.21.15.102:0/390953328 wait complete. 2024-04-25T02:00:29.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.306+0000 7f0aec47d740 1 -- start start 2024-04-25T02:00:29.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.306+0000 7f0aec47d740 1 Processor -- start 2024-04-25T02:00:29.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54985c0 0x561da54a0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 0x561da54a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 0x561da54a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0aec47d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561da54b7170 con 0x561da54a35b0 2024-04-25T02:00:29.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0aec47d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561da53b8ec0 con 0x561da54985c0 2024-04-25T02:00:29.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0aec47d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561da54b72f0 con 0x561da5498ee0 2024-04-25T02:00:29.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0aec47d740 10 client.0 _ll_register_callbacks cb 0x561da53f53f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:29.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0ae2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54985c0 0x561da54a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0ae2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54985c0 0x561da54a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59666/0 (socket says 172.21.15.102:59666) 2024-04-25T02:00:29.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0ae2a00640 1 -- 172.21.15.102:0/1740765896 learned_addr learned my addr 172.21.15.102:0/1740765896 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:29.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.307+0000 7f0ae3400640 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 0x561da54a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.308+0000 7f0ae2a00640 1 -- 172.21.15.102:0/1740765896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 msgr2=0x561da54a0b60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:00:29.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.308+0000 7f0ae2000640 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 0x561da54a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.309+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 0x561da54a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.309+0000 7f0ae2a00640 1 -- 172.21.15.102:0/1740765896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 msgr2=0x561da54a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:29.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.309+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 0x561da54a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.309+0000 7f0ae2a00640 1 -- 172.21.15.102:0/1740765896 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561da51db7d0 con 0x561da54985c0 2024-04-25T02:00:29.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.309+0000 7f0ae3400640 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 0x561da54a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:29.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.309+0000 7f0ae2000640 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 0x561da54a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:29.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.310+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54985c0 0x561da54a0620 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400f610 tx=0x7f0ad400fad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ffd2af1ec29fb12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:29.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.310+0000 7f0ac7e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:00:29.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.310+0000 7f0ac7e00640 1 -- 172.21.15.102:0/1740765896 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4003d80 con 0x561da54985c0 2024-04-25T02:00:29.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.310+0000 7f0ac7e00640 1 -- 172.21.15.102:0/1740765896 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0ad4004030 con 0x561da54985c0 2024-04-25T02:00:29.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.310+0000 7f0aec47d740 1 -- 172.21.15.102:0/1740765896 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0acc0c01d0 con 0x561da54985c0 2024-04-25T02:00:29.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0ac7e00640 1 -- 172.21.15.102:0/1740765896 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0ad4021050 con 0x561da54985c0 2024-04-25T02:00:29.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0aec47d740 10 client.26669 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:29.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0aec47d740 1 -- 172.21.15.102:0/1740765896 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561da54c34c0 con 0x561da54985c0 2024-04-25T02:00:29.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0ac7e00640 1 -- 172.21.15.102:0/1740765896 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0acc0c01d0 con 0x561da54985c0 2024-04-25T02:00:29.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0ac7e00640 1 -- 172.21.15.102:0/1740765896 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f0ad4011ac0 con 0x561da54985c0 2024-04-25T02:00:29.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0aec47d740 10 client.26669 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:29.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0aec47d740 10 client.26669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:29.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0aec47d740 20 client.26669 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:29.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0aec47d740 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561da54c4a20 0x561da54c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0aec47d740 4 client.26669 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:29.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.311+0000 7f0aec47d740 1 -- 172.21.15.102:0/1740765896 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561da540aa20 con 0x561da54c4a20 2024-04-25T02:00:29.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.312+0000 7f0ae2000640 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561da54c4a20 0x561da54c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.314+0000 7f0ae2000640 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561da54c4a20 0x561da54c6ec0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c46687f23d377be6 server_cookie=69b27cf2e38a2be8 in_seq=0 out_seq=0 2024-04-25T02:00:29.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.314+0000 7f0ac7e00640 10 client.26669 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.315+0000 7f0ac7e00640 1 -- 172.21.15.102:0/1740765896 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561da540aa20 con 0x561da54c4a20 2024-04-25T02:00:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.315+0000 7f0ac7e00640 10 client.26669 handle_command_reply: tid=0 2024-04-25T02:00:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.315+0000 7f0aec47d740 1 client.26669 shutdown 2024-04-25T02:00:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.315+0000 7f0aec47d740 1 -- 172.21.15.102:0/1740765896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54985c0 msgr2=0x561da54a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.315+0000 7f0aec47d740 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54985c0 0x561da54a0620 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400f610 tx=0x7f0ad400fad0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0aec47d740 1 -- 172.21.15.102:0/1740765896 shutdown_connections 2024-04-25T02:00:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0aec47d740 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561da54c4a20 0x561da54c6ec0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0aec47d740 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54a35b0 0x561da54a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0aec47d740 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 0x561da54a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0aec47d740 1 --2- 172.21.15.102:0/1740765896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54985c0 0x561da54a0620 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0aec47d740 1 -- 172.21.15.102:0/1740765896 >> 172.21.15.102:0/1740765896 conn(0x561da54948e0 msgr2=0x561da5493b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0ac7e00640 0 client.26669 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0aec47d740 1 -- 172.21.15.102:0/1740765896 shutdown_connections 2024-04-25T02:00:29.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0aec47d740 1 -- 172.21.15.102:0/1740765896 wait complete. 2024-04-25T02:00:29.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.316+0000 7f0aec47d740 20 client.26669 trim_cache size 0 max 16384 2024-04-25T02:00:29.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.339+0000 7f0aec47d740 1 Processor -- start 2024-04-25T02:00:29.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.339+0000 7f0aec47d740 1 -- start start 2024-04-25T02:00:29.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 0x561da54bffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da5498ee0 0x561da54942c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0aec47d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae4071e00 con 0x561da5498ee0 2024-04-25T02:00:29.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0aec47d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae406f670 con 0x561da54a35b0 2024-04-25T02:00:29.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0aec47d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae406d4c0 con 0x561da54985c0 2024-04-25T02:00:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0ae3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0ae3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59680/0 (socket says 172.21.15.102:59680) 2024-04-25T02:00:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0ae3400640 1 -- 172.21.15.102:0/3291662647 learned_addr learned my addr 172.21.15.102:0/3291662647 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.340+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 0x561da54bffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ae2000640 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da5498ee0 0x561da54942c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ae3400640 1 -- 172.21.15.102:0/3291662647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 msgr2=0x561da54bffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ae3400640 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 0x561da54bffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ae3400640 1 -- 172.21.15.102:0/3291662647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da5498ee0 msgr2=0x561da54942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ae3400640 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da5498ee0 0x561da54942c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ae3400640 1 -- 172.21.15.102:0/3291662647 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561da54983c0 con 0x561da54a35b0 2024-04-25T02:00:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 0x561da54bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ae2000640 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da5498ee0 0x561da54942c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ae3400640 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54cc790 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00bd40 tx=0x7f0adc011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=239edc6c96b92d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.341+0000 7f0ad2a00640 1 -- 172.21.15.102:0/3291662647 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc022070 con 0x561da54a35b0 2024-04-25T02:00:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.342+0000 7f0ad2a00640 1 -- 172.21.15.102:0/3291662647 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0adc017040 con 0x561da54a35b0 2024-04-25T02:00:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.342+0000 7f0ad2a00640 1 -- 172.21.15.102:0/3291662647 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc0106a0 con 0x561da54a35b0 2024-04-25T02:00:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.342+0000 7f0aec47d740 1 -- 172.21.15.102:0/3291662647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 msgr2=0x561da54cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.343+0000 7f0aec47d740 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54cc790 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00bd40 tx=0x7f0adc011000 comp rx=0 tx=0).stop 2024-04-25T02:00:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.343+0000 7f0aec47d740 1 -- 172.21.15.102:0/3291662647 shutdown_connections 2024-04-25T02:00:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.343+0000 7f0aec47d740 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54cc790 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.343+0000 7f0aec47d740 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da5498ee0 0x561da54942c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.343+0000 7f0aec47d740 1 --2- 172.21.15.102:0/3291662647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da54985c0 0x561da54bffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.343+0000 7f0aec47d740 1 -- 172.21.15.102:0/3291662647 >> 172.21.15.102:0/3291662647 conn(0x561da54948e0 msgr2=0x561da549b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.343+0000 7f0aec47d740 1 -- 172.21.15.102:0/3291662647 shutdown_connections 2024-04-25T02:00:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.344+0000 7f0aec47d740 1 -- 172.21.15.102:0/3291662647 wait complete. 2024-04-25T02:00:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.344+0000 7f0aec47d740 1 -- start start 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.344+0000 7f0aec47d740 1 Processor -- start 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54985c0 0x561da54bc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 0x561da54b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0aec47d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0aec47d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae406db30 con 0x561da54985c0 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0aec47d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae406efa0 con 0x561da54a35b0 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0aec47d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae406e200 con 0x561da5498ee0 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0aec47d740 10 client.0 _ll_register_callbacks cb 0x7f0ad40098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0ae3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0ae2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 0x561da54b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0ae3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54b7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59694/0 (socket says 172.21.15.102:59694) 2024-04-25T02:00:29.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0ae3400640 1 -- 172.21.15.102:0/441625089 learned_addr learned my addr 172.21.15.102:0/441625089 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:29.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.345+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54985c0 0x561da54bc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.346+0000 7f0ae3400640 1 -- 172.21.15.102:0/441625089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 msgr2=0x561da54b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:29.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.346+0000 7f0ae3400640 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 0x561da54b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.346+0000 7f0ae3400640 1 -- 172.21.15.102:0/441625089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54985c0 msgr2=0x561da54bc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:29.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.346+0000 7f0ae3400640 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54985c0 0x561da54bc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.346+0000 7f0ae3400640 1 -- 172.21.15.102:0/441625089 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561da54b8650 con 0x561da54a35b0 2024-04-25T02:00:29.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.346+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54985c0 0x561da54bc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:29.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.347+0000 7f0ae3400640 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54b7fb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00f010 tx=0x7f0adc005860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8e403f97ff593a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:29.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.347+0000 7f0ac7e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:00:29.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.347+0000 7f0ac7e00640 1 -- 172.21.15.102:0/441625089 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc022050 con 0x561da54a35b0 2024-04-25T02:00:29.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.347+0000 7f0ac7e00640 1 -- 172.21.15.102:0/441625089 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0adc005250 con 0x561da54a35b0 2024-04-25T02:00:29.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.347+0000 7f0aec47d740 1 -- 172.21.15.102:0/441625089 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0adc017040 con 0x561da54a35b0 2024-04-25T02:00:29.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.347+0000 7f0ac7e00640 1 -- 172.21.15.102:0/441625089 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0adc005450 con 0x561da54a35b0 2024-04-25T02:00:29.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.347+0000 7f0aec47d740 10 client.26675 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:29.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.347+0000 7f0aec47d740 1 -- 172.21.15.102:0/441625089 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561da552ae20 con 0x561da54a35b0 2024-04-25T02:00:29.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.348+0000 7f0ac7e00640 1 -- 172.21.15.102:0/441625089 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0adc004ba0 con 0x561da54a35b0 2024-04-25T02:00:29.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.348+0000 7f0ac7e00640 1 -- 172.21.15.102:0/441625089 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f0adc00f040 con 0x561da54a35b0 2024-04-25T02:00:29.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.348+0000 7f0aec47d740 10 client.26675 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.348+0000 7f0aec47d740 10 client.26675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.348+0000 7f0aec47d740 20 client.26675 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.348+0000 7f0aec47d740 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561da552b540 0x561da552d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.348+0000 7f0aec47d740 4 client.26675 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.348+0000 7f0aec47d740 1 -- 172.21.15.102:0/441625089 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0adc017040 con 0x561da552b540 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.349+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561da552b540 0x561da552d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.349+0000 7f0ae2a00640 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561da552b540 0x561da552d960 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f7033a000da2e26 server_cookie=379ce7ba5df027c1 in_seq=0 out_seq=0 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.349+0000 7f0ac7e00640 10 client.26675 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.349+0000 7f0ac7e00640 1 -- 172.21.15.102:0/441625089 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0adc017040 con 0x561da552b540 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.349+0000 7f0ac7e00640 10 client.26675 handle_command_reply: tid=0 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.349+0000 7f0aec47d740 1 client.26675 shutdown 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0aec47d740 1 -- 172.21.15.102:0/441625089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 msgr2=0x561da54b7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0aec47d740 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54b7fb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00f010 tx=0x7f0adc005860 comp rx=0 tx=0).stop 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0aec47d740 1 -- 172.21.15.102:0/441625089 shutdown_connections 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0aec47d740 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561da552b540 0x561da552d960 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0aec47d740 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561da54a35b0 0x561da54b7fb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0aec47d740 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561da5498ee0 0x561da54b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0aec47d740 1 --2- 172.21.15.102:0/441625089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561da54985c0 0x561da54bc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0aec47d740 1 -- 172.21.15.102:0/441625089 >> 172.21.15.102:0/441625089 conn(0x561da54948e0 msgr2=0x561da5499e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0ac7e00640 0 client.26675 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.350+0000 7f0aec47d740 1 -- 172.21.15.102:0/441625089 shutdown_connections 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.351+0000 7f0aec47d740 1 -- 172.21.15.102:0/441625089 wait complete. 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.351+0000 7f0aec47d740 20 client.26675 trim_cache size 0 max 16384 2024-04-25T02:00:29.358 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:00:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 -- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0ac80a2dc0 msgr2=0x7f0ac80a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0ac80a2dc0 0x7f0ac80a5280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x561da5495a70 tx=0x7f0adc005ed0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 -- 172.21.15.102:0/2182903948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 msgr2=0x7f0ae419a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 0x7f0ae419a8f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0acc02a970 tx=0x7f0acc003b90 comp rx=0 tx=0).stop 2024-04-25T02:00:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 -- 172.21.15.102:0/2182903948 shutdown_connections 2024-04-25T02:00:29.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0ac80a2dc0 0x7f0ac80a5280 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ae410e150 0x7f0ae419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ae4100ec0 0x7f0ae419a8f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 --2- 172.21.15.102:0/2182903948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae41005a0 0x7f0ae419a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:29.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 -- 172.21.15.102:0/2182903948 >> 172.21.15.102:0/2182903948 conn(0x7f0ae40777a0 msgr2=0x7f0ae410f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:29.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 -- 172.21.15.102:0/2182903948 shutdown_connections 2024-04-25T02:00:29.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:29.352+0000 7f0ae9600640 1 -- 172.21.15.102:0/2182903948 wait complete. 2024-04-25T02:00:30.382 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T02:00:30.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:00:30.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T02:00:30.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.647+0000 7f537c800640 1 -- 172.21.15.102:0/3589956499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f537410bd00 msgr2=0x7f537410e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:30.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.647+0000 7f537c800640 1 --2- 172.21.15.102:0/3589956499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f537410bd00 0x7f537410e1a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5370009ca0 tx=0x7f537002a9e0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.647+0000 7f537c800640 1 -- 172.21.15.102:0/3589956499 shutdown_connections 2024-04-25T02:00:30.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.647+0000 7f537c800640 1 --2- 172.21.15.102:0/3589956499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f537410bd00 0x7f537410e1a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.647+0000 7f537c800640 1 --2- 172.21.15.102:0/3589956499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53741032e0 0x7f537410b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.647+0000 7f537c800640 1 --2- 172.21.15.102:0/3589956499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53741029c0 0x7f5374102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.647+0000 7f537c800640 1 -- 172.21.15.102:0/3589956499 >> 172.21.15.102:0/3589956499 conn(0x7f5374077740 msgr2=0x7f5374077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:30.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.647+0000 7f537c800640 1 -- 172.21.15.102:0/3589956499 shutdown_connections 2024-04-25T02:00:30.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.648+0000 7f537c800640 1 -- 172.21.15.102:0/3589956499 wait complete. 2024-04-25T02:00:30.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.650+0000 7f537c800640 1 Processor -- start 2024-04-25T02:00:30.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.653+0000 7f537c800640 1 -- start start 2024-04-25T02:00:30.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.654+0000 7f537c800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53741029c0 0x7f537419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.654+0000 7f537c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53741032e0 0x7f537419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.655+0000 7f537c800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f537410bd00 0x7f537410a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.656+0000 7f537c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5374111b50 con 0x7f53741032e0 2024-04-25T02:00:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.656+0000 7f537c800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f53741119d0 con 0x7f537410bd00 2024-04-25T02:00:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.656+0000 7f537c800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5374111cd0 con 0x7f53741029c0 2024-04-25T02:00:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.655+0000 7f537a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f537410bd00 0x7f537410a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.656+0000 7f537a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f537410bd00 0x7f537410a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59708/0 (socket says 172.21.15.102:59708) 2024-04-25T02:00:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.656+0000 7f537a800640 1 -- 172.21.15.102:0/151670750 learned_addr learned my addr 172.21.15.102:0/151670750 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:30.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.657+0000 7f5379400640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53741032e0 0x7f537419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.658+0000 7f5379e00640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53741029c0 0x7f537419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.659+0000 7f537a800640 1 -- 172.21.15.102:0/151670750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53741029c0 msgr2=0x7f537419eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:30.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.659+0000 7f537a800640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53741029c0 0x7f537419eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.659+0000 7f537a800640 1 -- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53741032e0 msgr2=0x7f537419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:30.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.659+0000 7f537a800640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53741032e0 0x7f537419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.659+0000 7f537a800640 1 -- 172.21.15.102:0/151670750 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537410b1a0 con 0x7f537410bd00 2024-04-25T02:00:30.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.661+0000 7f537a800640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f537410bd00 0x7f537410a930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5370005e30 tx=0x7f5370003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:30.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.661+0000 7f536ea00640 1 -- 172.21.15.102:0/151670750 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370047070 con 0x7f537410bd00 2024-04-25T02:00:30.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.661+0000 7f536ea00640 1 -- 172.21.15.102:0/151670750 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f537002e870 con 0x7f537410bd00 2024-04-25T02:00:30.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.661+0000 7f536ea00640 1 -- 172.21.15.102:0/151670750 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370033050 con 0x7f537410bd00 2024-04-25T02:00:30.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.662+0000 7f537c800640 1 -- 172.21.15.102:0/151670750 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537410b430 con 0x7f537410bd00 2024-04-25T02:00:30.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.662+0000 7f537c800640 1 -- 172.21.15.102:0/151670750 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f537419a650 con 0x7f537410bd00 2024-04-25T02:00:30.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.669+0000 7f536ea00640 1 -- 172.21.15.102:0/151670750 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f537004a070 con 0x7f537410bd00 2024-04-25T02:00:30.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.673+0000 7f537f7e2740 1 Processor -- start 2024-04-25T02:00:30.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.674+0000 7f536ea00640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f535c0a2d80 0x7f535c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.674+0000 7f537f7e2740 1 -- start start 2024-04-25T02:00:30.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f41d60 0x561059f42140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f42680 0x561059f4c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f4cd50 0x561059f4f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f537f7e2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561059ea0f40 con 0x561059f42680 2024-04-25T02:00:30.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f537f7e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561059ea6190 con 0x561059f4cd50 2024-04-25T02:00:30.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f537f7e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561059e4ea10 con 0x561059f41d60 2024-04-25T02:00:30.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f5379e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f41d60 0x561059f42140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f5379e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f41d60 0x561059f42140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60618/0 (socket says 172.21.15.102:60618) 2024-04-25T02:00:30.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f5379e00640 1 -- 172.21.15.102:0/3217488540 learned_addr learned my addr 172.21.15.102:0/3217488540 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:30.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.675+0000 7f537a800640 1 --2- 172.21.15.102:0/3217488540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f4cd50 0x561059f4f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.676+0000 7f537a800640 1 -- 172.21.15.102:0/3217488540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f41d60 msgr2=0x561059f42140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:30.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.676+0000 7f537a800640 1 --2- 172.21.15.102:0/3217488540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f41d60 0x561059f42140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.676+0000 7f537a800640 1 -- 172.21.15.102:0/3217488540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f42680 msgr2=0x561059f4c810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:00:30.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.676+0000 7f537a800640 1 --2- 172.21.15.102:0/3217488540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f42680 0x561059f4c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.676+0000 7f537a800640 1 -- 172.21.15.102:0/3217488540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561059c827e0 con 0x561059f4cd50 2024-04-25T02:00:30.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.676+0000 7f537a800640 1 --2- 172.21.15.102:0/3217488540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f4cd50 0x561059f4f140 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f53700ed6f0 tx=0x7f537004e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a57ad56630b7986a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:30.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.676+0000 7f536ea00640 1 -- 172.21.15.102:0/151670750 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f53700eff70 con 0x7f537410bd00 2024-04-25T02:00:30.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.677+0000 7f5363000640 1 -- 172.21.15.102:0/3217488540 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53700bc290 con 0x561059f4cd50 2024-04-25T02:00:30.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.677+0000 7f5363000640 1 -- 172.21.15.102:0/3217488540 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5370004850 con 0x561059f4cd50 2024-04-25T02:00:30.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.677+0000 7f5363000640 1 -- 172.21.15.102:0/3217488540 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53700bc290 con 0x561059f4cd50 2024-04-25T02:00:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.678+0000 7f5379400640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f535c0a2d80 0x7f535c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.679+0000 7f537f7e2740 1 -- 172.21.15.102:0/3217488540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f4cd50 msgr2=0x561059f4f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.679+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3217488540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f4cd50 0x561059f4f140 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f53700ed6f0 tx=0x7f537004e000 comp rx=0 tx=0).stop 2024-04-25T02:00:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.679+0000 7f537f7e2740 1 -- 172.21.15.102:0/3217488540 shutdown_connections 2024-04-25T02:00:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.679+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3217488540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f4cd50 0x561059f4f140 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.679+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3217488540 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f42680 0x561059f4c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.679+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3217488540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f41d60 0x561059f42140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.679+0000 7f537f7e2740 1 -- 172.21.15.102:0/3217488540 >> 172.21.15.102:0/3217488540 conn(0x561059f3e080 msgr2=0x561059f3e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:30.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.680+0000 7f537f7e2740 1 -- 172.21.15.102:0/3217488540 shutdown_connections 2024-04-25T02:00:30.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.682+0000 7f537f7e2740 1 -- 172.21.15.102:0/3217488540 wait complete. 2024-04-25T02:00:30.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.683+0000 7f537f7e2740 1 -- start start 2024-04-25T02:00:30.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.683+0000 7f5379400640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f535c0a2d80 0x7f535c0a5240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x561059f43320 tx=0x7f536400c3c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:30.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.683+0000 7f537f7e2740 1 Processor -- start 2024-04-25T02:00:30.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.684+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 0x561059f49ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.684+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f42680 0x561059f4a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.684+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f4a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.684+0000 7f537f7e2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561059e61e70 con 0x561059f4cd50 2024-04-25T02:00:30.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.684+0000 7f537f7e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561059ea2150 con 0x561059f41d60 2024-04-25T02:00:30.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.684+0000 7f537f7e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561059f4f900 con 0x561059f42680 2024-04-25T02:00:30.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.684+0000 7f537f7e2740 10 client.0 _ll_register_callbacks cb 0x561059e9d1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:30.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f42680 0x561059f4a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f42680 0x561059f4a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60634/0 (socket says 172.21.15.102:60634) 2024-04-25T02:00:30.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379400640 1 -- 172.21.15.102:0/3513905598 learned_addr learned my addr 172.21.15.102:0/3513905598 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:30.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379400640 1 -- 172.21.15.102:0/3513905598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 msgr2=0x561059f49ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:30.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f537a800640 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f4a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379e00640 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 0x561059f49ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379400640 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 0x561059f49ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379400640 1 -- 172.21.15.102:0/3513905598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 msgr2=0x561059f4a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:30.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379400640 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f4a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379400640 1 -- 172.21.15.102:0/3513905598 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561059ce2f70 con 0x561059f42680 2024-04-25T02:00:30.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f537a800640 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f4a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:30.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.685+0000 7f5379e00640 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 0x561059f49ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:30.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.686+0000 7f5379400640 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f42680 0x561059f4a1e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f53640107a0 tx=0x7f53640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d001da8893606acf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:30.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.686+0000 7f5357400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:00:30.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.686+0000 7f5357400640 1 -- 172.21.15.102:0/3513905598 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5364021070 con 0x561059f42680 2024-04-25T02:00:30.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.687+0000 7f537f7e2740 1 -- 172.21.15.102:0/3513905598 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5370004850 con 0x561059f42680 2024-04-25T02:00:30.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.687+0000 7f5357400640 1 -- 172.21.15.102:0/3513905598 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f536400fb40 con 0x561059f42680 2024-04-25T02:00:30.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.687+0000 7f5357400640 1 -- 172.21.15.102:0/3513905598 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f536402d050 con 0x561059f42680 2024-04-25T02:00:30.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.688+0000 7f537f7e2740 10 client.26641 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:30.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.688+0000 7f537f7e2740 1 -- 172.21.15.102:0/3513905598 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561059b7a170 con 0x561059f42680 2024-04-25T02:00:30.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.689+0000 7f5357400640 1 -- 172.21.15.102:0/3513905598 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5370004850 con 0x561059f42680 2024-04-25T02:00:30.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.689+0000 7f5357400640 1 -- 172.21.15.102:0/3513905598 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f5364025450 con 0x561059f42680 2024-04-25T02:00:30.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.689+0000 7f537f7e2740 10 client.26641 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:30.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.689+0000 7f537f7e2740 10 client.26641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:30.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.689+0000 7f537f7e2740 20 client.26641 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:30.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.689+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561059f5ca60 0x561059f5eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.689+0000 7f537f7e2740 4 client.26641 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:30.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.689+0000 7f537f7e2740 1 -- 172.21.15.102:0/3513905598 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561059eb36d0 con 0x561059f5ca60 2024-04-25T02:00:30.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.689+0000 7f5379e00640 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561059f5ca60 0x561059f5eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.691+0000 7f5357400640 10 client.26641 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:30.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.691+0000 7f5379e00640 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561059f5ca60 0x561059f5eeb0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6981c41a66ddf3a1 server_cookie=b0c063238a5a9541 in_seq=0 out_seq=0 2024-04-25T02:00:30.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.692+0000 7f5357400640 1 -- 172.21.15.102:0/3513905598 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561059eb36d0 con 0x561059f5ca60 2024-04-25T02:00:30.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.692+0000 7f5357400640 10 client.26641 handle_command_reply: tid=0 2024-04-25T02:00:30.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.693+0000 7f537f7e2740 1 client.26641 shutdown 2024-04-25T02:00:30.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.694+0000 7f537f7e2740 1 -- 172.21.15.102:0/3513905598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f42680 msgr2=0x561059f4a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:30.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.694+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f42680 0x561059f4a1e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f53640107a0 tx=0x7f53640046c0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.694+0000 7f537f7e2740 1 -- 172.21.15.102:0/3513905598 shutdown_connections 2024-04-25T02:00:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.694+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561059f5ca60 0x561059f5eeb0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.694+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f4a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.694+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f42680 0x561059f4a1e0 secure :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f53640107a0 tx=0x7f53640046c0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.694+0000 7f537f7e2740 1 --2- 172.21.15.102:0/3513905598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 0x561059f49ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.694+0000 7f5357400640 0 client.26641 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:30.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.694+0000 7f537f7e2740 1 -- 172.21.15.102:0/3513905598 >> 172.21.15.102:0/3513905598 conn(0x561059f3e080 msgr2=0x561059f43d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:30.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.695+0000 7f537f7e2740 1 -- 172.21.15.102:0/3513905598 shutdown_connections 2024-04-25T02:00:30.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.695+0000 7f537f7e2740 1 -- 172.21.15.102:0/3513905598 wait complete. 2024-04-25T02:00:30.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.695+0000 7f537f7e2740 20 client.26641 trim_cache size 0 max 16384 2024-04-25T02:00:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.736+0000 7f537f7e2740 1 Processor -- start 2024-04-25T02:00:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.736+0000 7f537f7e2740 1 -- start start 2024-04-25T02:00:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.736+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 0x561059f58640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f44500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f647c0 0x561059f64c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f537f7e2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5358010fe0 con 0x561059f4cd50 2024-04-25T02:00:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f537f7e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f535800e850 con 0x561059f41d60 2024-04-25T02:00:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f537f7e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f535800c6a0 con 0x561059f647c0 2024-04-25T02:00:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f5379400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f44500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f5379400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f44500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47496/0 (socket says 172.21.15.102:47496) 2024-04-25T02:00:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f5379400640 1 -- 172.21.15.102:0/2589094496 learned_addr learned my addr 172.21.15.102:0/2589094496 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f5379e00640 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 0x561059f58640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.737+0000 7f537a800640 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f647c0 0x561059f64c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.738+0000 7f537a800640 1 -- 172.21.15.102:0/2589094496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 msgr2=0x561059f58640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.738+0000 7f537a800640 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 0x561059f58640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.738+0000 7f537a800640 1 -- 172.21.15.102:0/2589094496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 msgr2=0x561059f44500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.739+0000 7f537a800640 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f44500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.739+0000 7f537a800640 1 -- 172.21.15.102:0/2589094496 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561059f424b0 con 0x561059f647c0 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.739+0000 7f5379400640 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f44500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.739+0000 7f537a800640 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f647c0 0x561059f64c00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f53700ed6f0 tx=0x7f5370009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5891b35a2920bf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.739+0000 7f5362600640 1 -- 172.21.15.102:0/2589094496 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53700bc290 con 0x561059f647c0 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.739+0000 7f5362600640 1 -- 172.21.15.102:0/2589094496 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5370004850 con 0x561059f647c0 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.739+0000 7f5362600640 1 -- 172.21.15.102:0/2589094496 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53700bc4c0 con 0x561059f647c0 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.740+0000 7f537f7e2740 1 -- 172.21.15.102:0/2589094496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f647c0 msgr2=0x561059f64c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.740+0000 7f537f7e2740 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f647c0 0x561059f64c00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f53700ed6f0 tx=0x7f5370009650 comp rx=0 tx=0).stop 2024-04-25T02:00:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.740+0000 7f537f7e2740 1 -- 172.21.15.102:0/2589094496 shutdown_connections 2024-04-25T02:00:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.740+0000 7f537f7e2740 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f647c0 0x561059f64c00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.740+0000 7f537f7e2740 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f4cd50 0x561059f44500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.740+0000 7f537f7e2740 1 --2- 172.21.15.102:0/2589094496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f41d60 0x561059f58640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.740+0000 7f537f7e2740 1 -- 172.21.15.102:0/2589094496 >> 172.21.15.102:0/2589094496 conn(0x561059f3e080 msgr2=0x561059f3d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.741+0000 7f537f7e2740 1 -- 172.21.15.102:0/2589094496 shutdown_connections 2024-04-25T02:00:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.741+0000 7f537f7e2740 1 -- 172.21.15.102:0/2589094496 wait complete. 2024-04-25T02:00:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.742+0000 7f537f7e2740 1 -- start start 2024-04-25T02:00:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.743+0000 7f537f7e2740 1 Processor -- start 2024-04-25T02:00:30.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f41d60 0x561059f54fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f4cd50 0x561059f513b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f537f7e2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f647c0 0x561059f519c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f537f7e2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f535800cd10 con 0x561059f41d60 2024-04-25T02:00:30.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f537f7e2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f535800e180 con 0x561059f647c0 2024-04-25T02:00:30.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f537f7e2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f535800d3e0 con 0x561059f4cd50 2024-04-25T02:00:30.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f537f7e2740 10 client.0 _ll_register_callbacks cb 0x7f5364014050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:30.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f5379e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f41d60 0x561059f54fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f5379e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f41d60 0x561059f54fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47508/0 (socket says 172.21.15.102:47508) 2024-04-25T02:00:30.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f5379e00640 1 -- 172.21.15.102:0/617216753 learned_addr learned my addr 172.21.15.102:0/617216753 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:30.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.745+0000 7f5379400640 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f4cd50 0x561059f513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.746+0000 7f537a800640 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f647c0 0x561059f519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.746+0000 7f5379e00640 1 -- 172.21.15.102:0/617216753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f4cd50 msgr2=0x561059f513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:30.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.746+0000 7f5379e00640 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f4cd50 0x561059f513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.746+0000 7f5379e00640 1 -- 172.21.15.102:0/617216753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f647c0 msgr2=0x561059f519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:30.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.746+0000 7f5379e00640 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f647c0 0x561059f519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.746+0000 7f5379e00640 1 -- 172.21.15.102:0/617216753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561059f521c0 con 0x561059f41d60 2024-04-25T02:00:30.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.746+0000 7f5379e00640 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f41d60 0x561059f54fe0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x561059f5f760 tx=0x7f5368005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd2a49a311eb671d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:30.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.746+0000 7f5357400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:00:30.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.747+0000 7f5357400640 1 -- 172.21.15.102:0/617216753 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368004220 con 0x561059f41d60 2024-04-25T02:00:30.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.747+0000 7f5357400640 1 -- 172.21.15.102:0/617216753 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f53680043c0 con 0x561059f41d60 2024-04-25T02:00:30.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.747+0000 7f537f7e2740 1 -- 172.21.15.102:0/617216753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5370004850 con 0x561059f41d60 2024-04-25T02:00:30.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.748+0000 7f5357400640 1 -- 172.21.15.102:0/617216753 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f53680035f0 con 0x561059f41d60 2024-04-25T02:00:30.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.748+0000 7f537f7e2740 10 client.16212 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:30.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.748+0000 7f537f7e2740 1 -- 172.21.15.102:0/617216753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x561059fc41b0 con 0x561059f41d60 2024-04-25T02:00:30.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.748+0000 7f5357400640 1 -- 172.21.15.102:0/617216753 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5370004850 con 0x561059f41d60 2024-04-25T02:00:30.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.749+0000 7f5357400640 1 -- 172.21.15.102:0/617216753 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f5368022070 con 0x561059f41d60 2024-04-25T02:00:30.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.749+0000 7f537f7e2740 10 client.16212 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:30.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.749+0000 7f537f7e2740 10 client.16212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:30.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.749+0000 7f537f7e2740 20 client.16212 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:30.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.749+0000 7f537f7e2740 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561059fc4910 0x561059fc6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:30.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.749+0000 7f537f7e2740 4 client.16212 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T02:00:30.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.749+0000 7f537f7e2740 1 -- 172.21.15.102:0/617216753 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561059eada20 con 0x561059fc4910 2024-04-25T02:00:30.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.749+0000 7f5379400640 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561059fc4910 0x561059fc6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:30.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.750+0000 7f5379400640 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561059fc4910 0x561059fc6d30 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12391cb27e31f771 server_cookie=cc7d028f94758e39 in_seq=0 out_seq=0 2024-04-25T02:00:30.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.750+0000 7f5357400640 10 client.16212 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:30.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.751+0000 7f5357400640 1 -- 172.21.15.102:0/617216753 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561059eada20 con 0x561059fc4910 2024-04-25T02:00:30.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.751+0000 7f5357400640 10 client.16212 handle_command_reply: tid=0 2024-04-25T02:00:30.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.751+0000 7f537f7e2740 1 client.16212 shutdown 2024-04-25T02:00:30.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 -- 172.21.15.102:0/617216753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f41d60 msgr2=0x561059f54fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:30.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f41d60 0x561059f54fe0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x561059f5f760 tx=0x7f5368005c60 comp rx=0 tx=0).stop 2024-04-25T02:00:30.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 -- 172.21.15.102:0/617216753 shutdown_connections 2024-04-25T02:00:30.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561059fc4910 0x561059fc6d30 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561059f647c0 0x561059f519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561059f4cd50 0x561059f513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 --2- 172.21.15.102:0/617216753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561059f41d60 0x561059f54fe0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 -- 172.21.15.102:0/617216753 >> 172.21.15.102:0/617216753 conn(0x561059f3e080 msgr2=0x561059f637b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:30.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f5357400640 0 client.16212 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:30.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 -- 172.21.15.102:0/617216753 shutdown_connections 2024-04-25T02:00:30.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 1 -- 172.21.15.102:0/617216753 wait complete. 2024-04-25T02:00:30.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.752+0000 7f537f7e2740 20 client.16212 trim_cache size 0 max 16384 2024-04-25T02:00:30.760 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:00:30.760 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T02:00:30.760 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "fffdaca4-3b70-40e3-9c62-c8d14f2b23ce", 2024-04-25T02:00:30.760 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T02:00:30.760 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.754+0000 7f537c800640 1 -- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f535c0a2d80 msgr2=0x7f535c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.754+0000 7f537c800640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f535c0a2d80 0x7f535c0a5240 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x561059f43320 tx=0x7f536400c3c0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.754+0000 7f537c800640 1 -- 172.21.15.102:0/151670750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f537410bd00 msgr2=0x7f537410a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.754+0000 7f537c800640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f537410bd00 0x7f537410a930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5370005e30 tx=0x7f5370003f10 comp rx=0 tx=0).stop 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.754+0000 7f537c800640 1 -- 172.21.15.102:0/151670750 shutdown_connections 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.755+0000 7f537c800640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f535c0a2d80 0x7f535c0a5240 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.755+0000 7f537c800640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f537410bd00 0x7f537410a930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.755+0000 7f537c800640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53741032e0 0x7f537419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.755+0000 7f537c800640 1 --2- 172.21.15.102:0/151670750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53741029c0 0x7f537419eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.755+0000 7f537c800640 1 -- 172.21.15.102:0/151670750 >> 172.21.15.102:0/151670750 conn(0x7f5374077740 msgr2=0x7f53740fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.755+0000 7f537c800640 1 -- 172.21.15.102:0/151670750 shutdown_connections 2024-04-25T02:00:30.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:30.755+0000 7f537c800640 1 -- 172.21.15.102:0/151670750 wait complete. 2024-04-25T02:00:30.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T02:00:30.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T02:00:31.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.063+0000 7f0daf800640 1 -- 172.21.15.102:0/3367886324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 msgr2=0x7f0da8103e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.063+0000 7f0daf800640 1 --2- 172.21.15.102:0/3367886324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 0x7f0da8103e00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009930 tx=0x7f0d9c02a440 comp rx=0 tx=0).stop 2024-04-25T02:00:31.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.064+0000 7f0daf800640 1 -- 172.21.15.102:0/3367886324 shutdown_connections 2024-04-25T02:00:31.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.064+0000 7f0daf800640 1 --2- 172.21.15.102:0/3367886324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0da8104340 0x7f0da8110850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.064+0000 7f0daf800640 1 --2- 172.21.15.102:0/3367886324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 0x7f0da8103e00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.064+0000 7f0daf800640 1 --2- 172.21.15.102:0/3367886324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0da8109990 0x7f0da8109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.064+0000 7f0daf800640 1 -- 172.21.15.102:0/3367886324 >> 172.21.15.102:0/3367886324 conn(0x7f0da80ff4d0 msgr2=0x7f0da81018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.064+0000 7f0daf800640 1 -- 172.21.15.102:0/3367886324 shutdown_connections 2024-04-25T02:00:31.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.064+0000 7f0daf800640 1 -- 172.21.15.102:0/3367886324 wait complete. 2024-04-25T02:00:31.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.068+0000 7f0daf800640 1 Processor -- start 2024-04-25T02:00:31.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.068+0000 7f0daf800640 1 -- start start 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0daf800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 0x7f0da8073290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0daf800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0da8104340 0x7f0da806b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0daf800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0da8109990 0x7f0da806b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0daf800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0da8113dc0 con 0x7f0da8104340 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0daf800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0da8113c40 con 0x7f0da8109990 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0daf800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0da8113f40 con 0x7f0da8103980 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0dace00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 0x7f0da8073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0dad800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0da8109990 0x7f0da806b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0dad800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0da8109990 0x7f0da806b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59746/0 (socket says 172.21.15.102:59746) 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0dace00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 0x7f0da8073290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60666/0 (socket says 172.21.15.102:60666) 2024-04-25T02:00:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.069+0000 7f0dad800640 1 -- 172.21.15.102:0/3544652665 learned_addr learned my addr 172.21.15.102:0/3544652665 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.070+0000 7f0dad800640 1 -- 172.21.15.102:0/3544652665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 msgr2=0x7f0da8073290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.072+0000 7f0dad800640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 0x7f0da8073290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.072+0000 7f0dad800640 1 -- 172.21.15.102:0/3544652665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0da8104340 msgr2=0x7f0da806b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.072+0000 7f0dad800640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0da8104340 0x7f0da806b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.072+0000 7f0dad800640 1 -- 172.21.15.102:0/3544652665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da806c050 con 0x7f0da8109990 2024-04-25T02:00:31.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.072+0000 7f0dace00640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 0x7f0da8073290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:31.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.073+0000 7f0dad800640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0da8109990 0x7f0da806b790 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0da000eb10 tx=0x7f0da000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.074+0000 7f0da5600640 1 -- 172.21.15.102:0/3544652665 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0004b20 con 0x7f0da8109990 2024-04-25T02:00:31.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.074+0000 7f0da5600640 1 -- 172.21.15.102:0/3544652665 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da0004cc0 con 0x7f0da8109990 2024-04-25T02:00:31.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.074+0000 7f0daf800640 1 -- 172.21.15.102:0/3544652665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0da81af140 con 0x7f0da8109990 2024-04-25T02:00:31.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.074+0000 7f0daf800640 1 -- 172.21.15.102:0/3544652665 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0da81af6b0 con 0x7f0da8109990 2024-04-25T02:00:31.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.074+0000 7f0da5600640 1 -- 172.21.15.102:0/3544652665 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0010760 con 0x7f0da8109990 2024-04-25T02:00:31.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.081+0000 7f0da5600640 1 -- 172.21.15.102:0/3544652665 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0da0005020 con 0x7f0da8109990 2024-04-25T02:00:31.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0db2673740 1 Processor -- start 2024-04-25T02:00:31.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0da5600640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d900a2d80 0x7f0d900a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0db2673740 1 -- start start 2024-04-25T02:00:31.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0da5600640 1 -- 172.21.15.102:0/3544652665 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0da00cfa50 con 0x7f0da8109990 2024-04-25T02:00:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0db2673740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525de2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0db2673740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 0x555e525f12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0db2673740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0db2673740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555e52542560 con 0x555e525f1800 2024-04-25T02:00:31.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0db2673740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555e524ea930 con 0x555e525ddf00 2024-04-25T02:00:31.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.086+0000 7f0db2673740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555e5253e520 con 0x555e525de820 2024-04-25T02:00:31.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.087+0000 7f0da7e00640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d900a2d80 0x7f0d900a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.087+0000 7f0dad800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.087+0000 7f0da7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 0x555e525f12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.087+0000 7f0dad800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47544/0 (socket says 172.21.15.102:47544) 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.087+0000 7f0dad800640 1 -- 172.21.15.102:0/4092490977 learned_addr learned my addr 172.21.15.102:0/4092490977 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.088+0000 7f0dace00640 1 --2- 172.21.15.102:0/4092490977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525de2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.088+0000 7f0dad800640 1 -- 172.21.15.102:0/4092490977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 msgr2=0x555e525f12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.088+0000 7f0dad800640 1 --2- 172.21.15.102:0/4092490977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 0x555e525f12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.088+0000 7f0dad800640 1 -- 172.21.15.102:0/4092490977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 msgr2=0x555e525de2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.088+0000 7f0dad800640 1 --2- 172.21.15.102:0/4092490977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525de2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.088+0000 7f0dad800640 1 -- 172.21.15.102:0/4092490977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e523207d0 con 0x555e525f1800 2024-04-25T02:00:31.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.088+0000 7f0dad800640 1 --2- 172.21.15.102:0/4092490977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525f3bf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0da0098970 tx=0x7f0da009bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e5b83937af21cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.089+0000 7f0d9a000640 1 -- 172.21.15.102:0/4092490977 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da009bfa0 con 0x555e525f1800 2024-04-25T02:00:31.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.089+0000 7f0d9a000640 1 -- 172.21.15.102:0/4092490977 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da00a0350 con 0x555e525f1800 2024-04-25T02:00:31.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.089+0000 7f0d9a000640 1 -- 172.21.15.102:0/4092490977 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da00babe0 con 0x555e525f1800 2024-04-25T02:00:31.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.090+0000 7f0db2673740 1 -- 172.21.15.102:0/4092490977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 msgr2=0x555e525f3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.090+0000 7f0db2673740 1 --2- 172.21.15.102:0/4092490977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525f3bf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0da0098970 tx=0x7f0da009bba0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.090+0000 7f0db2673740 1 -- 172.21.15.102:0/4092490977 shutdown_connections 2024-04-25T02:00:31.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.090+0000 7f0db2673740 1 --2- 172.21.15.102:0/4092490977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525f3bf0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.090+0000 7f0db2673740 1 --2- 172.21.15.102:0/4092490977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 0x555e525f12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.090+0000 7f0db2673740 1 --2- 172.21.15.102:0/4092490977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525de2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.090+0000 7f0db2673740 1 -- 172.21.15.102:0/4092490977 >> 172.21.15.102:0/4092490977 conn(0x555e525da320 msgr2=0x555e525da700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.090+0000 7f0db2673740 1 -- 172.21.15.102:0/4092490977 shutdown_connections 2024-04-25T02:00:31.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.091+0000 7f0da7e00640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d900a2d80 0x7f0d900a5240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x555e525e0a00 tx=0x7f0d9c005a20 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.091+0000 7f0db2673740 1 -- 172.21.15.102:0/4092490977 wait complete. 2024-04-25T02:00:31.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.092+0000 7f0db2673740 1 -- start start 2024-04-25T02:00:31.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.092+0000 7f0db2673740 1 Processor -- start 2024-04-25T02:00:31.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.095+0000 7f0db2673740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525ddf00 0x555e525ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.095+0000 7f0db2673740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525de820 0x555e525eec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.095+0000 7f0db2673740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525ef150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.095+0000 7f0db2673740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555e525f45c0 con 0x555e525ddf00 2024-04-25T02:00:31.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.095+0000 7f0db2673740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555e524fdd90 con 0x555e525de820 2024-04-25T02:00:31.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.095+0000 7f0db2673740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555e525f4740 con 0x555e525f1800 2024-04-25T02:00:31.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.095+0000 7f0da7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525de820 0x555e525eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.095+0000 7f0db2673740 10 client.0 _ll_register_callbacks cb 0x555e524fd780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:31.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0dad800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0dad800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525ef150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60686/0 (socket says 172.21.15.102:60686) 2024-04-25T02:00:31.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0dad800640 1 -- 172.21.15.102:0/4128046592 learned_addr learned my addr 172.21.15.102:0/4128046592 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0dace00640 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525ddf00 0x555e525ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0dad800640 1 -- 172.21.15.102:0/4128046592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525de820 msgr2=0x555e525eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0dad800640 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525de820 0x555e525eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0dad800640 1 -- 172.21.15.102:0/4128046592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525ddf00 msgr2=0x555e525ee6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0dad800640 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525ddf00 0x555e525ee6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0dad800640 1 -- 172.21.15.102:0/4128046592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e522563f0 con 0x555e525f1800 2024-04-25T02:00:31.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.096+0000 7f0da7e00640 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525de820 0x555e525eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:31.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.097+0000 7f0dad800640 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525ef150 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0da00983b0 tx=0x7f0da00cd170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13e4f6926a5c93e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.097+0000 7f0d8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:00:31.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.097+0000 7f0d8ea00640 1 -- 172.21.15.102:0/4128046592 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da00a4c10 con 0x555e525f1800 2024-04-25T02:00:31.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.097+0000 7f0d8ea00640 1 -- 172.21.15.102:0/4128046592 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0da00badf0 con 0x555e525f1800 2024-04-25T02:00:31.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.097+0000 7f0db2673740 1 -- 172.21.15.102:0/4128046592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0da00a0350 con 0x555e525f1800 2024-04-25T02:00:31.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.097+0000 7f0d8ea00640 1 -- 172.21.15.102:0/4128046592 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0da009b3f0 con 0x555e525f1800 2024-04-25T02:00:31.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.097+0000 7f0db2673740 10 client.26659 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:31.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.097+0000 7f0db2673740 1 -- 172.21.15.102:0/4128046592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555e522160f0 con 0x555e525f1800 2024-04-25T02:00:31.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.098+0000 7f0d8ea00640 1 -- 172.21.15.102:0/4128046592 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da00a0350 con 0x555e525f1800 2024-04-25T02:00:31.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.098+0000 7f0d8ea00640 1 -- 172.21.15.102:0/4128046592 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f0da0085130 con 0x555e525f1800 2024-04-25T02:00:31.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.098+0000 7f0db2673740 10 client.26659 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:31.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.098+0000 7f0db2673740 10 client.26659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:31.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.098+0000 7f0db2673740 20 client.26659 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:31.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.098+0000 7f0db2673740 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555e526019b0 0x555e52603da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.098+0000 7f0db2673740 4 client.26659 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:31.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.098+0000 7f0db2673740 1 -- 172.21.15.102:0/4128046592 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e52555d90 con 0x555e526019b0 2024-04-25T02:00:31.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.099+0000 7f0dace00640 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555e526019b0 0x555e52603da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.100+0000 7f0dace00640 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555e526019b0 0x555e52603da0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=279126d536d4e474 server_cookie=b65da6a8235cecd0 in_seq=0 out_seq=0 2024-04-25T02:00:31.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.100+0000 7f0d8ea00640 10 client.26659 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:31.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.101+0000 7f0d8ea00640 1 -- 172.21.15.102:0/4128046592 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e52555d90 con 0x555e526019b0 2024-04-25T02:00:31.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.101+0000 7f0d8ea00640 10 client.26659 handle_command_reply: tid=0 2024-04-25T02:00:31.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.101+0000 7f0db2673740 1 client.26659 shutdown 2024-04-25T02:00:31.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.101+0000 7f0db2673740 1 -- 172.21.15.102:0/4128046592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 msgr2=0x555e525ef150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.101+0000 7f0db2673740 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525ef150 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0da00983b0 tx=0x7f0da00cd170 comp rx=0 tx=0).stop 2024-04-25T02:00:31.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.102+0000 7f0db2673740 1 -- 172.21.15.102:0/4128046592 shutdown_connections 2024-04-25T02:00:31.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.102+0000 7f0db2673740 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555e526019b0 0x555e52603da0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.102+0000 7f0db2673740 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525ef150 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.102+0000 7f0db2673740 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525de820 0x555e525eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.102+0000 7f0db2673740 1 --2- 172.21.15.102:0/4128046592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525ddf00 0x555e525ee6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.102+0000 7f0d8ea00640 0 client.26659 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:31.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.102+0000 7f0db2673740 1 -- 172.21.15.102:0/4128046592 >> 172.21.15.102:0/4128046592 conn(0x555e525da320 msgr2=0x555e525e0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.102+0000 7f0db2673740 1 -- 172.21.15.102:0/4128046592 shutdown_connections 2024-04-25T02:00:31.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.103+0000 7f0db2673740 1 -- 172.21.15.102:0/4128046592 wait complete. 2024-04-25T02:00:31.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.104+0000 7f0db2673740 20 client.26659 trim_cache size 0 max 16384 2024-04-25T02:00:31.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.135+0000 7f0db2673740 1 Processor -- start 2024-04-25T02:00:31.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.135+0000 7f0db2673740 1 -- start start 2024-04-25T02:00:31.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.136+0000 7f0db2673740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525fd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.136+0000 7f0db2673740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 0x555e525da700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.136+0000 7f0db2673740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.136+0000 7f0db2673740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d88010ff0 con 0x555e525f1800 2024-04-25T02:00:31.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.136+0000 7f0db2673740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8800e860 con 0x555e525ddf00 2024-04-25T02:00:31.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.136+0000 7f0db2673740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8800c6b0 con 0x555e525de820 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.136+0000 7f0da7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 0x555e525da700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.136+0000 7f0da7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 0x555e525da700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60692/0 (socket says 172.21.15.102:60692) 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.136+0000 7f0da7e00640 1 -- 172.21.15.102:0/203117000 learned_addr learned my addr 172.21.15.102:0/203117000 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.137+0000 7f0dace00640 1 --2- 172.21.15.102:0/203117000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.137+0000 7f0dace00640 1 -- 172.21.15.102:0/203117000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 msgr2=0x555e525da700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.137+0000 7f0dace00640 1 --2- 172.21.15.102:0/203117000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 0x555e525da700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.137+0000 7f0dace00640 1 -- 172.21.15.102:0/203117000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 msgr2=0x555e525eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.137+0000 7f0dad800640 1 --2- 172.21.15.102:0/203117000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.137+0000 7f0dace00640 1 --2- 172.21.15.102:0/203117000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525eeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.137+0000 7f0dace00640 1 -- 172.21.15.102:0/203117000 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e525de650 con 0x555e525ddf00 2024-04-25T02:00:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.138+0000 7f0dace00640 1 --2- 172.21.15.102:0/203117000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525fd520 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x555e52604310 tx=0x7f0d94005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8ac612b5790c7d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.138+0000 7f0d99600640 1 -- 172.21.15.102:0/203117000 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d940042f0 con 0x555e525ddf00 2024-04-25T02:00:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.138+0000 7f0d99600640 1 -- 172.21.15.102:0/203117000 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d94004490 con 0x555e525ddf00 2024-04-25T02:00:31.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.138+0000 7f0d99600640 1 -- 172.21.15.102:0/203117000 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d94004a00 con 0x555e525ddf00 2024-04-25T02:00:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.139+0000 7f0db2673740 1 -- 172.21.15.102:0/203117000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 msgr2=0x555e525fd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.139+0000 7f0db2673740 1 --2- 172.21.15.102:0/203117000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525fd520 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x555e52604310 tx=0x7f0d94005f90 comp rx=0 tx=0).stop 2024-04-25T02:00:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.139+0000 7f0db2673740 1 -- 172.21.15.102:0/203117000 shutdown_connections 2024-04-25T02:00:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.139+0000 7f0db2673740 1 --2- 172.21.15.102:0/203117000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525f1800 0x555e525eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.139+0000 7f0db2673740 1 --2- 172.21.15.102:0/203117000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525de820 0x555e525da700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.139+0000 7f0db2673740 1 --2- 172.21.15.102:0/203117000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525fd520 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.139+0000 7f0db2673740 1 -- 172.21.15.102:0/203117000 >> 172.21.15.102:0/203117000 conn(0x555e525da320 msgr2=0x555e525ee690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.139+0000 7f0db2673740 1 -- 172.21.15.102:0/203117000 shutdown_connections 2024-04-25T02:00:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.139+0000 7f0db2673740 1 -- 172.21.15.102:0/203117000 wait complete. 2024-04-25T02:00:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.140+0000 7f0db2673740 1 -- start start 2024-04-25T02:00:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.140+0000 7f0db2673740 1 Processor -- start 2024-04-25T02:00:31.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.141+0000 7f0db2673740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525f9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.141+0000 7f0db2673740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525de820 0x555e525eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.141+0000 7f0db2673740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525eb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.141+0000 7f0db2673740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8800cd20 con 0x555e525de820 2024-04-25T02:00:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.141+0000 7f0db2673740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8800e190 con 0x555e525ddf00 2024-04-25T02:00:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.141+0000 7f0db2673740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8800d3f0 con 0x555e525f1800 2024-04-25T02:00:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.141+0000 7f0db2673740 10 client.0 _ll_register_callbacks cb 0x7f0da009bd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.142+0000 7f0dace00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525f9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.142+0000 7f0dace00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525f9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59788/0 (socket says 172.21.15.102:59788) 2024-04-25T02:00:31.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.142+0000 7f0dace00640 1 -- 172.21.15.102:0/2223662392 learned_addr learned my addr 172.21.15.102:0/2223662392 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.142+0000 7f0dad800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525eb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.143+0000 7f0da7e00640 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525de820 0x555e525eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.143+0000 7f0dad800640 1 -- 172.21.15.102:0/2223662392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 msgr2=0x555e525f9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.143+0000 7f0dad800640 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525f9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.143+0000 7f0dad800640 1 -- 172.21.15.102:0/2223662392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525de820 msgr2=0x555e525eafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.143+0000 7f0dad800640 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525de820 0x555e525eafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.143+0000 7f0dad800640 1 -- 172.21.15.102:0/2223662392 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e525ebbc0 con 0x555e525f1800 2024-04-25T02:00:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.144+0000 7f0dad800640 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525eb500 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0da00cd280 tx=0x7f0da009be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64a6f78188ec7f6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.144+0000 7f0d8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.144+0000 7f0d8ea00640 1 -- 172.21.15.102:0/2223662392 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da00a4c10 con 0x555e525f1800 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.144+0000 7f0d8ea00640 1 -- 172.21.15.102:0/2223662392 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f0da009b590 con 0x555e525f1800 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.144+0000 7f0db2673740 1 -- 172.21.15.102:0/2223662392 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d94004490 con 0x555e525f1800 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.144+0000 7f0d8ea00640 1 -- 172.21.15.102:0/2223662392 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0da00a4e60 con 0x555e525f1800 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.144+0000 7f0db2673740 10 client.26665 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.144+0000 7f0db2673740 1 -- 172.21.15.102:0/2223662392 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555e525ebbc0 con 0x555e525f1800 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.145+0000 7f0d8ea00640 1 -- 172.21.15.102:0/2223662392 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d94004490 con 0x555e525f1800 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.145+0000 7f0d8ea00640 1 -- 172.21.15.102:0/2223662392 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7f0da00a16f0 con 0x555e525f1800 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.145+0000 7f0db2673740 10 client.26665 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.145+0000 7f0db2673740 10 client.26665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:31.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.145+0000 7f0db2673740 20 client.26665 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:31.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.145+0000 7f0db2673740 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555e52667e30 0x555e5266a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.145+0000 7f0db2673740 4 client.26665 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T02:00:31.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.145+0000 7f0db2673740 1 -- 172.21.15.102:0/2223662392 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555e5254faa0 con 0x555e52667e30 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.145+0000 7f0dace00640 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555e52667e30 0x555e5266a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.146+0000 7f0dace00640 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555e52667e30 0x555e5266a220 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6038a8cacef9fc6 server_cookie=701dc4f224647f33 in_seq=0 out_seq=0 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.146+0000 7f0d8ea00640 10 client.26665 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.146+0000 7f0d8ea00640 1 -- 172.21.15.102:0/2223662392 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555e5254faa0 con 0x555e52667e30 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.146+0000 7f0d8ea00640 10 client.26665 handle_command_reply: tid=0 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.146+0000 7f0db2673740 1 client.26665 shutdown 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.147+0000 7f0db2673740 1 -- 172.21.15.102:0/2223662392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 msgr2=0x555e525eb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.147+0000 7f0db2673740 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525eb500 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0da00cd280 tx=0x7f0da009be20 comp rx=0 tx=0).stop 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.147+0000 7f0db2673740 1 -- 172.21.15.102:0/2223662392 shutdown_connections 2024-04-25T02:00:31.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.147+0000 7f0db2673740 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x555e52667e30 0x555e5266a220 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.147+0000 7f0db2673740 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555e525f1800 0x555e525eb500 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.147+0000 7f0db2673740 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555e525de820 0x555e525eafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.147+0000 7f0db2673740 1 --2- 172.21.15.102:0/2223662392 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555e525ddf00 0x555e525f9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.147+0000 7f0db2673740 1 -- 172.21.15.102:0/2223662392 >> 172.21.15.102:0/2223662392 conn(0x555e525da320 msgr2=0x555e525eef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.147+0000 7f0d8ea00640 0 client.26665 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:31.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.148+0000 7f0db2673740 1 -- 172.21.15.102:0/2223662392 shutdown_connections 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.148+0000 7f0db2673740 1 -- 172.21.15.102:0/2223662392 wait complete. 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.148+0000 7f0db2673740 20 client.26665 trim_cache size 0 max 16384 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: "fffdaca4-3b70-40e3-9c62-c8d14f2b23ce": { 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "fffdaca4-3b70-40e3-9c62-c8d14f2b23ce", 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:00:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:00:31.154 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:00:31.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 -- 172.21.15.102:0/3544652665 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d900a2d80 msgr2=0x7f0d900a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d900a2d80 0x7f0d900a5240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x555e525e0a00 tx=0x7f0d9c005a20 comp rx=0 tx=0).stop 2024-04-25T02:00:31.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 -- 172.21.15.102:0/3544652665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0da8109990 msgr2=0x7f0da806b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0da8109990 0x7f0da806b790 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0da000eb10 tx=0x7f0da000c2d0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 -- 172.21.15.102:0/3544652665 shutdown_connections 2024-04-25T02:00:31.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0d900a2d80 0x7f0d900a5240 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0da8109990 0x7f0da806b790 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0da8104340 0x7f0da806b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 --2- 172.21.15.102:0/3544652665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0da8103980 0x7f0da8073290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 -- 172.21.15.102:0/3544652665 >> 172.21.15.102:0/3544652665 conn(0x7f0da80ff4d0 msgr2=0x7f0da80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 -- 172.21.15.102:0/3544652665 shutdown_connections 2024-04-25T02:00:31.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:00:31.149+0000 7f0daf800640 1 -- 172.21.15.102:0/3544652665 wait complete. 2024-04-25T02:00:31.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:30 smithi102 ceph-mon[27796]: pgmap v160: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.5 MiB/s wr, 674 op/s 2024-04-25T02:00:31.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:30 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T02:00:31.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:30 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:31.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:30 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:00:31.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:30 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:00:31.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:30 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:31.174 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:30 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:00:31.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:30 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:31.179 INFO:tasks.cephfs.filesystem:scrub status for tag:fffdaca4-3b70-40e3-9c62-c8d14f2b23ce - {'path': '/', 'tag': 'fffdaca4-3b70-40e3-9c62-c8d14f2b23ce', 'options': 'recursive,force'} 2024-04-25T02:00:31.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:30 smithi110 ceph-mon[32848]: pgmap v160: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.5 MiB/s wr, 674 op/s 2024-04-25T02:00:31.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:30 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T02:00:31.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:30 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:31.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:30 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:00:31.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:30 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:00:31.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:30 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:31.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:30 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:00:31.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:30 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:31.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:30 smithi193 ceph-mon[32981]: pgmap v160: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 7.5 MiB/s wr, 674 op/s 2024-04-25T02:00:31.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:30 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T02:00:31.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:30 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:31.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:30 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:00:31.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:30 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:00:31.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:30 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:31.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:30 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:00:31.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:30 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.508+0000 7fac85c00640 1 -- 172.21.15.102:0/1706610184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 msgr2=0x7fac80073df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.508+0000 7fac85c00640 1 --2- 172.21.15.102:0/1706610184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 0x7fac80073df0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fac78009ca0 tx=0x7fac7802a950 comp rx=0 tx=0).stop 2024-04-25T02:00:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.508+0000 7fac85c00640 1 -- 172.21.15.102:0/1706610184 shutdown_connections 2024-04-25T02:00:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.508+0000 7fac85c00640 1 --2- 172.21.15.102:0/1706610184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac8010e030 0x7fac80110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.508+0000 7fac85c00640 1 --2- 172.21.15.102:0/1706610184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac80074330 0x7fac8010daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.508+0000 7fac85c00640 1 --2- 172.21.15.102:0/1706610184 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 0x7fac80073df0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.508+0000 7fac85c00640 1 -- 172.21.15.102:0/1706610184 >> 172.21.15.102:0/1706610184 conn(0x7fac80078bd0 msgr2=0x7fac80078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.508+0000 7fac85c00640 1 -- 172.21.15.102:0/1706610184 shutdown_connections 2024-04-25T02:00:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.510+0000 7fac85c00640 1 -- 172.21.15.102:0/1706610184 wait complete. 2024-04-25T02:00:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.511+0000 7fac85c00640 1 Processor -- start 2024-04-25T02:00:31.515 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.511+0000 7fac85c00640 1 -- start start 2024-04-25T02:00:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.511+0000 7fac85c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 0x7fac80072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac85c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac80074330 0x7fac800733e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac85c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac8010e030 0x7fac8006c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac85c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fac80113de0 con 0x7fac80074330 2024-04-25T02:00:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac85c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fac80113c60 con 0x7fac80073a10 2024-04-25T02:00:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac85c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fac80113f60 con 0x7fac8010e030 2024-04-25T02:00:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac7ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 0x7fac80072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac7ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 0x7fac80072ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59806/0 (socket says 172.21.15.102:59806) 2024-04-25T02:00:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac7ea00640 1 -- 172.21.15.102:0/3073981752 learned_addr learned my addr 172.21.15.102:0/3073981752 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac7f400640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac8010e030 0x7fac8006c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.512+0000 7fac7e000640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac80074330 0x7fac800733e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.513+0000 7fac7ea00640 1 -- 172.21.15.102:0/3073981752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac8010e030 msgr2=0x7fac8006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.513+0000 7fac7ea00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac8010e030 0x7fac8006c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.513+0000 7fac7ea00640 1 -- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac80074330 msgr2=0x7fac800733e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.513+0000 7fac7ea00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac80074330 0x7fac800733e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.513+0000 7fac7ea00640 1 -- 172.21.15.102:0/3073981752 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8006c9c0 con 0x7fac80073a10 2024-04-25T02:00:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.513+0000 7fac7f400640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac8010e030 0x7fac8006c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.513+0000 7fac7ea00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 0x7fac80072ea0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fac78005e30 tx=0x7fac78003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.514+0000 7fac77400640 1 -- 172.21.15.102:0/3073981752 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac78034040 con 0x7fac80073a10 2024-04-25T02:00:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.514+0000 7fac85c00640 1 -- 172.21.15.102:0/3073981752 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac8006cc50 con 0x7fac80073a10 2024-04-25T02:00:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.514+0000 7fac77400640 1 -- 172.21.15.102:0/3073981752 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac7802fe20 con 0x7fac80073a10 2024-04-25T02:00:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.514+0000 7fac77400640 1 -- 172.21.15.102:0/3073981752 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac78004800 con 0x7fac80073a10 2024-04-25T02:00:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.514+0000 7fac85c00640 1 -- 172.21.15.102:0/3073981752 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac80070ba0 con 0x7fac80073a10 2024-04-25T02:00:31.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.519+0000 7fac77400640 1 -- 172.21.15.102:0/3073981752 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fac78037020 con 0x7fac80073a10 2024-04-25T02:00:31.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.520+0000 7fac88979740 1 Processor -- start 2024-04-25T02:00:31.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.520+0000 7fac88979740 1 -- start start 2024-04-25T02:00:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.521+0000 7fac88979740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ecbe00 0x55efb2ecc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.521+0000 7fac88979740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 0x55efb2ed6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.521+0000 7fac88979740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ed6fe0 0x55efb2ed93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.521+0000 7fac88979740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55efb2e2fab0 con 0x55efb2ed6fe0 2024-04-25T02:00:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.521+0000 7fac88979740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55efb2dd8640 con 0x55efb2ecbe00 2024-04-25T02:00:31.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.521+0000 7fac88979740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55efb2e2ba70 con 0x55efb2ecc720 2024-04-25T02:00:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.521+0000 7fac7f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ecbe00 0x55efb2ecc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.521+0000 7fac7f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ecbe00 0x55efb2ecc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59820/0 (socket says 172.21.15.102:59820) 2024-04-25T02:00:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.521+0000 7fac7f400640 1 -- 172.21.15.102:0/2733057360 learned_addr learned my addr 172.21.15.102:0/2733057360 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.522+0000 7fac7ea00640 1 --2- 172.21.15.102:0/2733057360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 0x55efb2ed6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.522+0000 7fac7f400640 1 -- 172.21.15.102:0/2733057360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 msgr2=0x55efb2ed6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.522+0000 7fac7f400640 1 --2- 172.21.15.102:0/2733057360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 0x55efb2ed6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.522+0000 7fac7f400640 1 -- 172.21.15.102:0/2733057360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ed6fe0 msgr2=0x55efb2ed93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.522+0000 7fac7f400640 1 --2- 172.21.15.102:0/2733057360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ed6fe0 0x55efb2ed93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.522+0000 7fac7f400640 1 -- 172.21.15.102:0/2733057360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb2b44210 con 0x55efb2ecbe00 2024-04-25T02:00:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.522+0000 7fac77400640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fac640a2f80 0x7fac640a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.523+0000 7fac77400640 1 -- 172.21.15.102:0/3073981752 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fac780f02f0 con 0x7fac80073a10 2024-04-25T02:00:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.523+0000 7fac7f400640 1 --2- 172.21.15.102:0/2733057360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ecbe00 0x55efb2ecc1e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009c60 tx=0x7fac6c0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1e30ea9dc26a2ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.523+0000 7fac7ea00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fac640a2f80 0x7fac640a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.523+0000 7fac6be00640 1 -- 172.21.15.102:0/2733057360 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c012450 con 0x55efb2ecbe00 2024-04-25T02:00:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.523+0000 7fac6be00640 1 -- 172.21.15.102:0/2733057360 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac6c012ad0 con 0x55efb2ecbe00 2024-04-25T02:00:31.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.523+0000 7fac6be00640 1 -- 172.21.15.102:0/2733057360 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c00fb80 con 0x55efb2ecbe00 2024-04-25T02:00:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac88979740 1 -- 172.21.15.102:0/2733057360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ecbe00 msgr2=0x55efb2ecc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac88979740 1 --2- 172.21.15.102:0/2733057360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ecbe00 0x55efb2ecc1e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009c60 tx=0x7fac6c0117b0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac7ea00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fac640a2f80 0x7fac640a5440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55efb2ecd3a0 tx=0x7fac7802a950 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac88979740 1 -- 172.21.15.102:0/2733057360 shutdown_connections 2024-04-25T02:00:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac88979740 1 --2- 172.21.15.102:0/2733057360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ed6fe0 0x55efb2ed93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac88979740 1 --2- 172.21.15.102:0/2733057360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 0x55efb2ed6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac88979740 1 --2- 172.21.15.102:0/2733057360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ecbe00 0x55efb2ecc1e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac88979740 1 -- 172.21.15.102:0/2733057360 >> 172.21.15.102:0/2733057360 conn(0x55efb2ec8310 msgr2=0x55efb2ec86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac88979740 1 -- 172.21.15.102:0/2733057360 shutdown_connections 2024-04-25T02:00:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.524+0000 7fac88979740 1 -- 172.21.15.102:0/2733057360 wait complete. 2024-04-25T02:00:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.525+0000 7fac88979740 1 -- start start 2024-04-25T02:00:31.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.525+0000 7fac88979740 1 Processor -- start 2024-04-25T02:00:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.526+0000 7fac88979740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ed4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.526+0000 7fac88979740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 0x55efb2ed4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.526+0000 7fac88979740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2ed4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.527+0000 7fac88979740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55efb2ed9c10 con 0x55efb2ecc720 2024-04-25T02:00:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.527+0000 7fac88979740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55efb2deb990 con 0x55efb2ed6fe0 2024-04-25T02:00:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.527+0000 7fac88979740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55efb2ed9d90 con 0x55efb2ecbe00 2024-04-25T02:00:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.527+0000 7fac88979740 10 client.0 _ll_register_callbacks cb 0x55efb2ded950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:31.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.527+0000 7fac7f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ed4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.527+0000 7fac7f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ed4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60734/0 (socket says 172.21.15.102:60734) 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.527+0000 7fac7f400640 1 -- 172.21.15.102:0/1173975598 learned_addr learned my addr 172.21.15.102:0/1173975598 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.527+0000 7fac7e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 0x55efb2ed4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.528+0000 7fac7f400640 1 -- 172.21.15.102:0/1173975598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 msgr2=0x55efb2ed4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.528+0000 7fac7f400640 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2ed4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.528+0000 7fac7f400640 1 -- 172.21.15.102:0/1173975598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 msgr2=0x55efb2ed4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.528+0000 7fac7f400640 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 0x55efb2ed4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.528+0000 7fac7f400640 1 -- 172.21.15.102:0/1173975598 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb2c10620 con 0x55efb2ecbe00 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.528+0000 7fac7e000640 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 0x55efb2ed4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.528+0000 7fac7f400640 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ed4000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009c30 tx=0x7fac6c009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdf3674c763ad1a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.528+0000 7fac68c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:00:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.529+0000 7fac68c00640 1 -- 172.21.15.102:0/1173975598 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c028070 con 0x55efb2ecbe00 2024-04-25T02:00:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.529+0000 7fac68c00640 1 -- 172.21.15.102:0/1173975598 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fac6c00f8c0 con 0x55efb2ecbe00 2024-04-25T02:00:31.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.529+0000 7fac88979740 1 -- 172.21.15.102:0/1173975598 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac6c012ad0 con 0x55efb2ecbe00 2024-04-25T02:00:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.529+0000 7fac68c00640 1 -- 172.21.15.102:0/1173975598 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fac6c032050 con 0x55efb2ecbe00 2024-04-25T02:00:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.529+0000 7fac88979740 10 client.26677 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.529+0000 7fac88979740 1 -- 172.21.15.102:0/1173975598 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55efb2ee5b70 con 0x55efb2ecbe00 2024-04-25T02:00:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.530+0000 7fac68c00640 1 -- 172.21.15.102:0/1173975598 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac6c012ad0 con 0x55efb2ecbe00 2024-04-25T02:00:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.531+0000 7fac68c00640 1 -- 172.21.15.102:0/1173975598 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7fac6c004750 con 0x55efb2ecbe00 2024-04-25T02:00:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.531+0000 7fac88979740 10 client.26677 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.531+0000 7fac88979740 10 client.26677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.531+0000 7fac88979740 20 client.26677 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.531+0000 7fac88979740 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55efb2ee6fd0 0x55efb2ee93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.531+0000 7fac88979740 4 client.26677 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.531+0000 7fac88979740 1 -- 172.21.15.102:0/1173975598 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efb2e3cff0 con 0x55efb2ee6fd0 2024-04-25T02:00:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.531+0000 7fac7e000640 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55efb2ee6fd0 0x55efb2ee93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.532+0000 7fac7e000640 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55efb2ee6fd0 0x55efb2ee93c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bc35183992d914b server_cookie=190eeeed6294f482 in_seq=0 out_seq=0 2024-04-25T02:00:31.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.532+0000 7fac68c00640 10 client.26677 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.533+0000 7fac68c00640 1 -- 172.21.15.102:0/1173975598 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55efb2e3cff0 con 0x55efb2ee6fd0 2024-04-25T02:00:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.533+0000 7fac68c00640 10 client.26677 handle_command_reply: tid=0 2024-04-25T02:00:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 client.26677 shutdown 2024-04-25T02:00:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 -- 172.21.15.102:0/1173975598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 msgr2=0x55efb2ed4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ed4000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009c30 tx=0x7fac6c009f60 comp rx=0 tx=0).stop 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 -- 172.21.15.102:0/1173975598 shutdown_connections 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55efb2ee6fd0 0x55efb2ee93c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2ed4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac68c00640 0 client.26677 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 0x55efb2ed4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 --2- 172.21.15.102:0/1173975598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ed4000 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 -- 172.21.15.102:0/1173975598 >> 172.21.15.102:0/1173975598 conn(0x55efb2ec8310 msgr2=0x55efb2ece040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 -- 172.21.15.102:0/1173975598 shutdown_connections 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.534+0000 7fac88979740 1 -- 172.21.15.102:0/1173975598 wait complete. 2024-04-25T02:00:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.535+0000 7fac88979740 20 client.26677 trim_cache size 0 max 16384 2024-04-25T02:00:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.559+0000 7fac88979740 1 Processor -- start 2024-04-25T02:00:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.559+0000 7fac88979740 1 -- start start 2024-04-25T02:00:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.559+0000 7fac88979740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ece7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.559+0000 7fac88979740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 0x55efb2ed48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.559+0000 7fac88979740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2eee990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.559+0000 7fac88979740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fac801ba6a0 con 0x55efb2ecc720 2024-04-25T02:00:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.559+0000 7fac88979740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fac801b7f10 con 0x55efb2ed6fe0 2024-04-25T02:00:31.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.559+0000 7fac88979740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fac801b5d60 con 0x55efb2ecbe00 2024-04-25T02:00:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ece7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ece7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60736/0 (socket says 172.21.15.102:60736) 2024-04-25T02:00:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7f400640 1 -- 172.21.15.102:0/901607239 learned_addr learned my addr 172.21.15.102:0/901607239 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7f400640 1 --2- 172.21.15.102:0/901607239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2eee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7e000640 1 --2- 172.21.15.102:0/901607239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 0x55efb2ed48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7f400640 1 -- 172.21.15.102:0/901607239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 msgr2=0x55efb2eee990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7f400640 1 --2- 172.21.15.102:0/901607239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2eee990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7f400640 1 -- 172.21.15.102:0/901607239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 msgr2=0x55efb2ed48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7f400640 1 --2- 172.21.15.102:0/901607239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 0x55efb2ed48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.560+0000 7fac7f400640 1 -- 172.21.15.102:0/901607239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb2ee9730 con 0x55efb2ecbe00 2024-04-25T02:00:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.561+0000 7fac7f400640 1 --2- 172.21.15.102:0/901607239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ece7c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009870 tx=0x7fac6c014070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=397658f32bbb0f21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.561+0000 7fac6b400640 1 -- 172.21.15.102:0/901607239 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c028070 con 0x55efb2ecbe00 2024-04-25T02:00:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.561+0000 7fac6b400640 1 -- 172.21.15.102:0/901607239 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac6c00fd00 con 0x55efb2ecbe00 2024-04-25T02:00:31.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.561+0000 7fac6b400640 1 -- 172.21.15.102:0/901607239 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c005000 con 0x55efb2ecbe00 2024-04-25T02:00:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 -- 172.21.15.102:0/901607239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 msgr2=0x55efb2ece7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 --2- 172.21.15.102:0/901607239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ece7c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fac6c009870 tx=0x7fac6c014070 comp rx=0 tx=0).stop 2024-04-25T02:00:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 -- 172.21.15.102:0/901607239 shutdown_connections 2024-04-25T02:00:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 --2- 172.21.15.102:0/901607239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2eee990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 --2- 172.21.15.102:0/901607239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecc720 0x55efb2ed48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 --2- 172.21.15.102:0/901607239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecbe00 0x55efb2ece7c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 -- 172.21.15.102:0/901607239 >> 172.21.15.102:0/901607239 conn(0x55efb2ec8310 msgr2=0x55efb2ec7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 -- 172.21.15.102:0/901607239 shutdown_connections 2024-04-25T02:00:31.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 -- 172.21.15.102:0/901607239 wait complete. 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.562+0000 7fac88979740 1 -- start start 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.563+0000 7fac88979740 1 Processor -- start 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.563+0000 7fac88979740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecbe00 0x55efb2edf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.563+0000 7fac88979740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 0x55efb2eda6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.563+0000 7fac88979740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2edac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.563+0000 7fac88979740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fac801b63d0 con 0x55efb2ecbe00 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.563+0000 7fac88979740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fac801b7840 con 0x55efb2ed6fe0 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.563+0000 7fac88979740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fac801b6aa0 con 0x55efb2ecc720 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.564+0000 7fac88979740 10 client.0 _ll_register_callbacks cb 0x7fac6c009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:31.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.564+0000 7fac7f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecbe00 0x55efb2edf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.564+0000 7fac7f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecbe00 0x55efb2edf350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47646/0 (socket says 172.21.15.102:47646) 2024-04-25T02:00:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.564+0000 7fac7f400640 1 -- 172.21.15.102:0/1111368814 learned_addr learned my addr 172.21.15.102:0/1111368814 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:31.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.564+0000 7fac7e000640 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 0x55efb2eda6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.565+0000 7fac7f400640 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2edac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.565+0000 7fac7f400640 1 -- 172.21.15.102:0/1111368814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 msgr2=0x55efb2eda6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.565+0000 7fac7f400640 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 0x55efb2eda6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.565+0000 7fac7f400640 1 -- 172.21.15.102:0/1111368814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecbe00 msgr2=0x55efb2edf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.565+0000 7fac7f400640 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecbe00 0x55efb2edf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.565+0000 7fac7f400640 1 -- 172.21.15.102:0/1111368814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb2edb430 con 0x55efb2ed6fe0 2024-04-25T02:00:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.565+0000 7fac7e000640 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 0x55efb2eda6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.566+0000 7fac7f400640 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2edac00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac6c006010 tx=0x7fac6c023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e6c5b24fdf70b06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:31.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.566+0000 7fac68c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:00:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.566+0000 7fac68c00640 1 -- 172.21.15.102:0/1111368814 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac6c028040 con 0x55efb2ed6fe0 2024-04-25T02:00:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.566+0000 7fac68c00640 1 -- 172.21.15.102:0/1111368814 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fac6c00fae0 con 0x55efb2ed6fe0 2024-04-25T02:00:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.566+0000 7fac88979740 1 -- 172.21.15.102:0/1111368814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac6c00fd00 con 0x55efb2ed6fe0 2024-04-25T02:00:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.567+0000 7fac68c00640 1 -- 172.21.15.102:0/1111368814 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fac6c059450 con 0x55efb2ed6fe0 2024-04-25T02:00:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.567+0000 7fac88979740 10 client.26717 fetch_fsmap learned FSMap version 33 2024-04-25T02:00:31.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.567+0000 7fac88979740 1 -- 172.21.15.102:0/1111368814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55efb2f4d1e0 con 0x55efb2ed6fe0 2024-04-25T02:00:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.567+0000 7fac68c00640 1 -- 172.21.15.102:0/1111368814 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac6c00fd00 con 0x55efb2ed6fe0 2024-04-25T02:00:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.567+0000 7fac68c00640 1 -- 172.21.15.102:0/1111368814 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 33) ==== 8350+0+0 (secure 0 0 0) 0x7fac6c066070 con 0x55efb2ed6fe0 2024-04-25T02:00:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.568+0000 7fac88979740 10 client.26717 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T02:00:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.568+0000 7fac88979740 10 client.26717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.568+0000 7fac88979740 20 client.26717 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.568+0000 7fac88979740 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55efb2f4d820 0x55efb2f4fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.568+0000 7fac88979740 4 client.26717 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:00:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.568+0000 7fac88979740 1 -- 172.21.15.102:0/1111368814 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55efb2e37340 con 0x55efb2f4d820 2024-04-25T02:00:31.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.568+0000 7fac7e000640 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55efb2f4d820 0x55efb2f4fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:31.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.568+0000 7fac68c00640 10 client.26717 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:31.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.569+0000 7fac7e000640 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55efb2f4d820 0x55efb2f4fc40 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b8d427d5fdc20c0 server_cookie=6541981dd5cec122 in_seq=0 out_seq=0 2024-04-25T02:00:31.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.582+0000 7fac68c00640 1 -- 172.21.15.102:0/1111368814 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55efb2e37340 con 0x55efb2f4d820 2024-04-25T02:00:31.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.582+0000 7fac68c00640 10 client.26717 handle_command_reply: tid=0 2024-04-25T02:00:31.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.582+0000 7fac88979740 1 client.26717 shutdown 2024-04-25T02:00:31.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.582+0000 7fac88979740 1 -- 172.21.15.102:0/1111368814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 msgr2=0x55efb2edac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.582+0000 7fac88979740 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2edac00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac6c006010 tx=0x7fac6c023000 comp rx=0 tx=0).stop 2024-04-25T02:00:31.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.582+0000 7fac88979740 1 -- 172.21.15.102:0/1111368814 shutdown_connections 2024-04-25T02:00:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.583+0000 7fac88979740 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55efb2f4d820 0x55efb2f4fc40 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.583+0000 7fac88979740 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55efb2ed6fe0 0x55efb2edac00 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.583+0000 7fac88979740 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55efb2ecc720 0x55efb2eda6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.583+0000 7fac68c00640 0 client.26717 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.583+0000 7fac88979740 1 --2- 172.21.15.102:0/1111368814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efb2ecbe00 0x55efb2edf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.583+0000 7fac88979740 1 -- 172.21.15.102:0/1111368814 >> 172.21.15.102:0/1111368814 conn(0x55efb2ec8310 msgr2=0x55efb2ed5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.583+0000 7fac88979740 1 -- 172.21.15.102:0/1111368814 shutdown_connections 2024-04-25T02:00:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.583+0000 7fac88979740 1 -- 172.21.15.102:0/1111368814 wait complete. 2024-04-25T02:00:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.583+0000 7fac88979740 20 client.26717 trim_cache size 0 max 16384 2024-04-25T02:00:31.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:00:31.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:00:31.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:00:31.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 -- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fac640a2f80 msgr2=0x7fac640a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fac640a2f80 0x7fac640a5440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55efb2ecd3a0 tx=0x7fac7802a950 comp rx=0 tx=0).stop 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 -- 172.21.15.102:0/3073981752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 msgr2=0x7fac80072ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 0x7fac80072ea0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fac78005e30 tx=0x7fac78003ce0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 -- 172.21.15.102:0/3073981752 shutdown_connections 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fac640a2f80 0x7fac640a5440 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fac8010e030 0x7fac8006c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac80074330 0x7fac800733e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 --2- 172.21.15.102:0/3073981752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80073a10 0x7fac80072ea0 secure :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fac78005e30 tx=0x7fac78003ce0 comp rx=0 tx=0).stop 2024-04-25T02:00:31.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.584+0000 7fac85c00640 1 -- 172.21.15.102:0/3073981752 >> 172.21.15.102:0/3073981752 conn(0x7fac80078bd0 msgr2=0x7fac800fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:31.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.585+0000 7fac85c00640 1 -- 172.21.15.102:0/3073981752 shutdown_connections 2024-04-25T02:00:31.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:31.585+0000 7fac85c00640 1 -- 172.21.15.102:0/3073981752 wait complete. 2024-04-25T02:00:32.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:31 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:00:32.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:31 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:32.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:31 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:32.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:31 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:00:32.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:31 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:32.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:31 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:32.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:31 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:00:32.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:31 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:32.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:31 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:00:33.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 280.0 s, 2338.4 tps, lat 42.552 ms stddev 49.740 2024-04-25T02:00:33.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:32 smithi102 ceph-mon[27796]: pgmap v161: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 9.7 MiB/s wr, 956 op/s 2024-04-25T02:00:33.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:32 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:00:33.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:32 smithi110 ceph-mon[32848]: pgmap v161: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 9.7 MiB/s wr, 956 op/s 2024-04-25T02:00:33.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:32 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:00:33.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:32 smithi193 ceph-mon[32981]: pgmap v161: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 9.7 MiB/s wr, 956 op/s 2024-04-25T02:00:33.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:32 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:00:34.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:33 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:00:34.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:33 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:00:34.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:33 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:00:35.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:34 smithi110 ceph-mon[32848]: pgmap v162: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 8.3 MiB/s wr, 857 op/s 2024-04-25T02:00:35.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:34 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:00:35.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:34 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:00:35.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:34 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:00:35.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:34 smithi193 ceph-mon[32981]: pgmap v162: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 8.3 MiB/s wr, 857 op/s 2024-04-25T02:00:35.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:34 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:00:35.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:34 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:00:35.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:34 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:00:35.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:34 smithi102 ceph-mon[27796]: pgmap v162: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 8.3 MiB/s wr, 857 op/s 2024-04-25T02:00:35.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:34 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:00:35.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:34 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:00:35.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:34 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:00:37.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:36 smithi110 ceph-mon[32848]: pgmap v163: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 7.4 MiB/s wr, 760 op/s 2024-04-25T02:00:37.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:36 smithi193 ceph-mon[32981]: pgmap v163: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 7.4 MiB/s wr, 760 op/s 2024-04-25T02:00:37.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:36 smithi102 ceph-mon[27796]: pgmap v163: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 7.4 MiB/s wr, 760 op/s 2024-04-25T02:00:38.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 285.0 s, 1953.4 tps, lat 50.896 ms stddev 55.620 2024-04-25T02:00:39.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:38 smithi110 ceph-mon[32848]: pgmap v164: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 10 MiB/s wr, 1.04k op/s 2024-04-25T02:00:39.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:38 smithi193 ceph-mon[32981]: pgmap v164: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 10 MiB/s wr, 1.04k op/s 2024-04-25T02:00:39.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:38 smithi102 ceph-mon[27796]: pgmap v164: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 10 MiB/s wr, 1.04k op/s 2024-04-25T02:00:41.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:40 smithi110 ceph-mon[32848]: pgmap v165: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 6.8 MiB/s wr, 714 op/s 2024-04-25T02:00:41.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:40 smithi193 ceph-mon[32981]: pgmap v165: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 6.8 MiB/s wr, 714 op/s 2024-04-25T02:00:41.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:40 smithi102 ceph-mon[27796]: pgmap v165: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 6.8 MiB/s wr, 714 op/s 2024-04-25T02:00:42.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:41 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:00:42.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:41 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:00:42.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:41 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:00:42.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:41 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:00:42.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:41 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:00:42.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:41 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:00:43.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 290.0 s, 2657.8 tps, lat 37.825 ms stddev 52.029 2024-04-25T02:00:43.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:42 smithi110 ceph-mon[32848]: pgmap v166: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 8.4 MiB/s wr, 887 op/s 2024-04-25T02:00:43.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:42 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:00:43.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:42 smithi193 ceph-mon[32981]: pgmap v166: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 8.4 MiB/s wr, 887 op/s 2024-04-25T02:00:43.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:42 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:00:43.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:42 smithi102 ceph-mon[27796]: pgmap v166: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 8.4 MiB/s wr, 887 op/s 2024-04-25T02:00:43.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:42 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:00:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.776+0000 7fae9d000640 1 -- 172.21.15.102:0/3041141487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae9810e100 msgr2=0x7fae981104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.776+0000 7fae9d000640 1 --2- 172.21.15.102:0/3041141487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae9810e100 0x7fae981104f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fae90009ca0 tx=0x7fae9002a8d0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.776+0000 7fae9d000640 1 -- 172.21.15.102:0/3041141487 shutdown_connections 2024-04-25T02:00:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.776+0000 7fae9d000640 1 --2- 172.21.15.102:0/3041141487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae9810e100 0x7fae981104f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.776+0000 7fae9d000640 1 --2- 172.21.15.102:0/3041141487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae98100e70 0x7fae9810dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.776+0000 7fae9d000640 1 --2- 172.21.15.102:0/3041141487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae98100550 0x7fae98100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.776+0000 7fae9d000640 1 -- 172.21.15.102:0/3041141487 >> 172.21.15.102:0/3041141487 conn(0x7fae980777c0 msgr2=0x7fae98077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.777+0000 7fae9d000640 1 -- 172.21.15.102:0/3041141487 shutdown_connections 2024-04-25T02:00:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.777+0000 7fae9d000640 1 -- 172.21.15.102:0/3041141487 wait complete. 2024-04-25T02:00:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.778+0000 7fae9d000640 1 Processor -- start 2024-04-25T02:00:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.778+0000 7fae9d000640 1 -- start start 2024-04-25T02:00:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.780+0000 7fae9d000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae98100550 0x7fae9810d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.780+0000 7fae9d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae98100e70 0x7fae9810aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.780+0000 7fae9d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9810e100 0x7fae9810afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.780+0000 7fae9d000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fae98113de0 con 0x7fae9810e100 2024-04-25T02:00:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.780+0000 7fae9d000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae98113c60 con 0x7fae98100550 2024-04-25T02:00:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.780+0000 7fae9d000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fae98113f60 con 0x7fae98100e70 2024-04-25T02:00:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.780+0000 7fae95e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae98100550 0x7fae9810d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.781+0000 7fae95e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae98100550 0x7fae9810d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46482/0 (socket says 172.21.15.102:46482) 2024-04-25T02:00:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.781+0000 7fae95e00640 1 -- 172.21.15.102:0/1795468553 learned_addr learned my addr 172.21.15.102:0/1795468553 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.781+0000 7fae96800640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9810e100 0x7fae9810afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.781+0000 7fae95e00640 1 -- 172.21.15.102:0/1795468553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae98100e70 msgr2=0x7fae9810aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.781+0000 7fae95e00640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae98100e70 0x7fae9810aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.781+0000 7fae95e00640 1 -- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9810e100 msgr2=0x7fae9810afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.781+0000 7fae95e00640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9810e100 0x7fae9810afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.781+0000 7fae95e00640 1 -- 172.21.15.102:0/1795468553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9810b870 con 0x7fae98100550 2024-04-25T02:00:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.782+0000 7fae95e00640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae98100550 0x7fae9810d960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fae8800b450 tx=0x7fae8800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:44.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.783+0000 7fae8ea00640 1 -- 172.21.15.102:0/1795468553 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae88015020 con 0x7fae98100550 2024-04-25T02:00:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.783+0000 7fae8ea00640 1 -- 172.21.15.102:0/1795468553 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae88011d70 con 0x7fae98100550 2024-04-25T02:00:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.783+0000 7fae8ea00640 1 -- 172.21.15.102:0/1795468553 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8800f340 con 0x7fae98100550 2024-04-25T02:00:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.783+0000 7fae9d000640 1 -- 172.21.15.102:0/1795468553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae981aab20 con 0x7fae98100550 2024-04-25T02:00:44.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.784+0000 7fae9d000640 1 -- 172.21.15.102:0/1795468553 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae981ab020 con 0x7fae98100550 2024-04-25T02:00:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.790+0000 7fae8ea00640 1 -- 172.21.15.102:0/1795468553 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fae88018020 con 0x7fae98100550 2024-04-25T02:00:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.792+0000 7fae9ff9f740 1 Processor -- start 2024-04-25T02:00:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.793+0000 7fae9ff9f740 1 -- start start 2024-04-25T02:00:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.793+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73ce00 0x55aebe73d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.793+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 0x55aebe747aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.793+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe747fe0 0x55aebe74a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.793+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55aebe6a0ab0 con 0x55aebe73d720 2024-04-25T02:00:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.793+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aebe649640 con 0x55aebe747fe0 2024-04-25T02:00:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.793+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55aebe69ca70 con 0x55aebe73ce00 2024-04-25T02:00:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae95400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe747fe0 0x55aebe74a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae95400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe747fe0 0x55aebe74a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46486/0 (socket says 172.21.15.102:46486) 2024-04-25T02:00:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae95400640 1 -- 172.21.15.102:0/2663988575 learned_addr learned my addr 172.21.15.102:0/2663988575 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae96800640 1 --2- 172.21.15.102:0/2663988575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73ce00 0x55aebe73d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae95e00640 1 --2- 172.21.15.102:0/2663988575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 0x55aebe747aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae95400640 1 -- 172.21.15.102:0/2663988575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73ce00 msgr2=0x55aebe73d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae95400640 1 --2- 172.21.15.102:0/2663988575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73ce00 0x55aebe73d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae95400640 1 -- 172.21.15.102:0/2663988575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 msgr2=0x55aebe747aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae95400640 1 --2- 172.21.15.102:0/2663988575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 0x55aebe747aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.794+0000 7fae95400640 1 -- 172.21.15.102:0/2663988575 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aebe3b5210 con 0x55aebe747fe0 2024-04-25T02:00:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.795+0000 7fae95400640 1 --2- 172.21.15.102:0/2663988575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe747fe0 0x55aebe74a3d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fae8000aa30 tx=0x7fae8000aef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=475662ccd3888462 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.795+0000 7fae87400640 1 -- 172.21.15.102:0/2663988575 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8000ce40 con 0x55aebe747fe0 2024-04-25T02:00:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.795+0000 7fae87400640 1 -- 172.21.15.102:0/2663988575 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae8000f930 con 0x55aebe747fe0 2024-04-25T02:00:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.795+0000 7fae87400640 1 -- 172.21.15.102:0/2663988575 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80018b80 con 0x55aebe747fe0 2024-04-25T02:00:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.795+0000 7fae8ea00640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fae7c0a2e20 0x7fae7c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.796+0000 7fae8ea00640 1 -- 172.21.15.102:0/1795468553 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fae880cee20 con 0x7fae98100550 2024-04-25T02:00:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.796+0000 7fae96800640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fae7c0a2e20 0x7fae7c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.796+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2663988575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe747fe0 msgr2=0x55aebe74a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.796+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2663988575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe747fe0 0x55aebe74a3d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fae8000aa30 tx=0x7fae8000aef0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.796+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2663988575 shutdown_connections 2024-04-25T02:00:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.796+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2663988575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe747fe0 0x55aebe74a3d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.796+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2663988575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 0x55aebe747aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.796+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2663988575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73ce00 0x55aebe73d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.796+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2663988575 >> 172.21.15.102:0/2663988575 conn(0x55aebe739310 msgr2=0x55aebe7396f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.798+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2663988575 shutdown_connections 2024-04-25T02:00:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.798+0000 7fae96800640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fae7c0a2e20 0x7fae7c0a52e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae90009430 tx=0x7fae90009360 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.798+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2663988575 wait complete. 2024-04-25T02:00:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.800+0000 7fae9ff9f740 1 -- start start 2024-04-25T02:00:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.801+0000 7fae9ff9f740 1 Processor -- start 2024-04-25T02:00:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.802+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe744e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.802+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 0x55aebe7453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.802+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 0x55aebe745900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.802+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55aebe74ac10 con 0x55aebe747fe0 2024-04-25T02:00:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.802+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aebe65c990 con 0x55aebe73ce00 2024-04-25T02:00:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.802+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55aebe74ad90 con 0x55aebe73d720 2024-04-25T02:00:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.802+0000 7fae9ff9f740 10 client.0 _ll_register_callbacks cb 0x55aebe65e950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae95e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe744e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae95e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe744e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46502/0 (socket says 172.21.15.102:46502) 2024-04-25T02:00:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae95e00640 1 -- 172.21.15.102:0/2582741240 learned_addr learned my addr 172.21.15.102:0/2582741240 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae95400640 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 0x55aebe7453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae96800640 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 0x55aebe745900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae95e00640 1 -- 172.21.15.102:0/2582741240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 msgr2=0x55aebe7453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae95e00640 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 0x55aebe7453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae95e00640 1 -- 172.21.15.102:0/2582741240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 msgr2=0x55aebe745900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae95e00640 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 0x55aebe745900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.803+0000 7fae95e00640 1 -- 172.21.15.102:0/2582741240 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aebe481620 con 0x55aebe73ce00 2024-04-25T02:00:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.804+0000 7fae95e00640 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe744e80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fae88010b40 tx=0x7fae880108c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c064f7a347fbbd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.804+0000 7fae7be00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:00:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.805+0000 7fae7be00640 1 -- 172.21.15.102:0/2582741240 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8809ade0 con 0x55aebe73ce00 2024-04-25T02:00:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.805+0000 7fae7be00640 1 -- 172.21.15.102:0/2582741240 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fae8809a8d0 con 0x55aebe73ce00 2024-04-25T02:00:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.805+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2582741240 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae8000f930 con 0x55aebe73ce00 2024-04-25T02:00:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.805+0000 7fae7be00640 1 -- 172.21.15.102:0/2582741240 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fae8809bb80 con 0x55aebe73ce00 2024-04-25T02:00:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.805+0000 7fae9ff9f740 10 client.26726 fetch_fsmap learned FSMap version 34 2024-04-25T02:00:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.805+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2582741240 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55aebe756a70 con 0x55aebe73ce00 2024-04-25T02:00:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.806+0000 7fae7be00640 1 -- 172.21.15.102:0/2582741240 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae8000f930 con 0x55aebe73ce00 2024-04-25T02:00:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.806+0000 7fae7be00640 1 -- 172.21.15.102:0/2582741240 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fae880b4230 con 0x55aebe73ce00 2024-04-25T02:00:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.806+0000 7fae9ff9f740 10 client.26726 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:00:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.806+0000 7fae9ff9f740 10 client.26726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.806+0000 7fae9ff9f740 20 client.26726 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.806+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aebe757ee0 0x55aebe75a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.806+0000 7fae9ff9f740 4 client.26726 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.806+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2582741240 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aebe6adff0 con 0x55aebe757ee0 2024-04-25T02:00:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.807+0000 7fae95400640 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aebe757ee0 0x55aebe75a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.809+0000 7fae95400640 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aebe757ee0 0x55aebe75a360 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8216c1e83fd0160b server_cookie=f231f1b49123063e in_seq=0 out_seq=0 2024-04-25T02:00:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.809+0000 7fae7be00640 10 client.26726 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.810+0000 7fae7be00640 1 -- 172.21.15.102:0/2582741240 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aebe6adff0 con 0x55aebe757ee0 2024-04-25T02:00:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.810+0000 7fae7be00640 10 client.26726 handle_command_reply: tid=0 2024-04-25T02:00:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.810+0000 7fae9ff9f740 1 client.26726 shutdown 2024-04-25T02:00:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.811+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2582741240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 msgr2=0x55aebe744e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.811+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe744e80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fae88010b40 tx=0x7fae880108c0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2582741240 shutdown_connections 2024-04-25T02:00:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aebe757ee0 0x55aebe75a360 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 0x55aebe745900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae7be00640 0 client.26726 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 0x55aebe7453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/2582741240 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe744e80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2582741240 >> 172.21.15.102:0/2582741240 conn(0x55aebe739310 msgr2=0x55aebe73f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2582741240 shutdown_connections 2024-04-25T02:00:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae9ff9f740 1 -- 172.21.15.102:0/2582741240 wait complete. 2024-04-25T02:00:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.812+0000 7fae9ff9f740 20 client.26726 trim_cache size 0 max 16384 2024-04-25T02:00:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.841+0000 7fae9ff9f740 1 Processor -- start 2024-04-25T02:00:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.841+0000 7fae9ff9f740 1 -- start start 2024-04-25T02:00:44.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.841+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe753b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.841+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 0x55aebe745720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.841+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 0x55aebe75fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.841+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fae74011130 con 0x55aebe747fe0 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.841+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae7400e9a0 con 0x55aebe73ce00 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.841+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fae7400c7f0 con 0x55aebe73d720 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.841+0000 7fae95e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe753b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae95e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe753b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46516/0 (socket says 172.21.15.102:46516) 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae95e00640 1 -- 172.21.15.102:0/1041403084 learned_addr learned my addr 172.21.15.102:0/1041403084 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae95400640 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 0x55aebe745720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae96800640 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 0x55aebe75fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae95e00640 1 -- 172.21.15.102:0/1041403084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 msgr2=0x55aebe745720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae95e00640 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 0x55aebe745720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae95e00640 1 -- 172.21.15.102:0/1041403084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 msgr2=0x55aebe75fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae95e00640 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 0x55aebe75fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae95e00640 1 -- 172.21.15.102:0/1041403084 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aebe74a740 con 0x55aebe73ce00 2024-04-25T02:00:44.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.842+0000 7fae96800640 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 0x55aebe75fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.843+0000 7fae95e00640 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe753b00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fae88009fd0 tx=0x7fae88099990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c967fa446d47f37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.844+0000 7fae86a00640 1 -- 172.21.15.102:0/1041403084 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8809f830 con 0x55aebe73ce00 2024-04-25T02:00:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.844+0000 7fae86a00640 1 -- 172.21.15.102:0/1041403084 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae880d2480 con 0x55aebe73ce00 2024-04-25T02:00:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.844+0000 7fae86a00640 1 -- 172.21.15.102:0/1041403084 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8809a8d0 con 0x55aebe73ce00 2024-04-25T02:00:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.845+0000 7fae9ff9f740 1 -- 172.21.15.102:0/1041403084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 msgr2=0x55aebe753b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:44.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.845+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe753b00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fae88009fd0 tx=0x7fae88099990 comp rx=0 tx=0).stop 2024-04-25T02:00:44.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.845+0000 7fae9ff9f740 1 -- 172.21.15.102:0/1041403084 shutdown_connections 2024-04-25T02:00:44.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.845+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe747fe0 0x55aebe75fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.845+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe73d720 0x55aebe745720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.845+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/1041403084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe753b00 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.845+0000 7fae9ff9f740 1 -- 172.21.15.102:0/1041403084 >> 172.21.15.102:0/1041403084 conn(0x55aebe739310 msgr2=0x55aebe737bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.845+0000 7fae9ff9f740 1 -- 172.21.15.102:0/1041403084 shutdown_connections 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.845+0000 7fae9ff9f740 1 -- 172.21.15.102:0/1041403084 wait complete. 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.846+0000 7fae9ff9f740 1 -- start start 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.846+0000 7fae9ff9f740 1 Processor -- start 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.847+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe741d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.847+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 0x55aebe742290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.847+0000 7fae9ff9f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe747fe0 0x55aebe7427d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.847+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fae7400ce60 con 0x55aebe73d720 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.847+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fae7400e2d0 con 0x55aebe73ce00 2024-04-25T02:00:44.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.847+0000 7fae9ff9f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fae7400d530 con 0x55aebe747fe0 2024-04-25T02:00:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.847+0000 7fae9ff9f740 10 client.0 _ll_register_callbacks cb 0x7fae880103e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.847+0000 7fae96800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe747fe0 0x55aebe7427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.848+0000 7fae96800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe747fe0 0x55aebe7427d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:57592/0 (socket says 172.21.15.102:57592) 2024-04-25T02:00:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.848+0000 7fae96800640 1 -- 172.21.15.102:0/4290354652 learned_addr learned my addr 172.21.15.102:0/4290354652 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.848+0000 7fae95e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe741d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.848+0000 7fae95400640 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 0x55aebe742290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.849+0000 7fae96800640 1 -- 172.21.15.102:0/4290354652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 msgr2=0x55aebe741d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.849+0000 7fae96800640 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe741d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.849+0000 7fae96800640 1 -- 172.21.15.102:0/4290354652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 msgr2=0x55aebe742290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.849+0000 7fae96800640 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 0x55aebe742290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.849+0000 7fae96800640 1 -- 172.21.15.102:0/4290354652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aebe7ce6f0 con 0x55aebe747fe0 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.849+0000 7fae95e00640 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe741d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.849+0000 7fae95400640 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 0x55aebe742290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.849+0000 7fae96800640 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe747fe0 0x55aebe7427d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fae9806af20 tx=0x7fae900095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e809735de8c83352 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.849+0000 7fae7be00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:00:44.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.850+0000 7fae7be00640 1 -- 172.21.15.102:0/4290354652 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90056060 con 0x55aebe747fe0 2024-04-25T02:00:44.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.850+0000 7fae9ff9f740 1 -- 172.21.15.102:0/4290354652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae880d2480 con 0x55aebe747fe0 2024-04-25T02:00:44.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.850+0000 7fae7be00640 1 -- 172.21.15.102:0/4290354652 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fae90004240 con 0x55aebe747fe0 2024-04-25T02:00:44.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.850+0000 7fae7be00640 1 -- 172.21.15.102:0/4290354652 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fae900319b0 con 0x55aebe747fe0 2024-04-25T02:00:44.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.850+0000 7fae9ff9f740 10 client.26698 fetch_fsmap learned FSMap version 34 2024-04-25T02:00:44.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.850+0000 7fae9ff9f740 1 -- 172.21.15.102:0/4290354652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55aebe7cfa10 con 0x55aebe747fe0 2024-04-25T02:00:44.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.850+0000 7fae7be00640 1 -- 172.21.15.102:0/4290354652 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae880d2480 con 0x55aebe747fe0 2024-04-25T02:00:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.851+0000 7fae7be00640 1 -- 172.21.15.102:0/4290354652 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fae90050070 con 0x55aebe747fe0 2024-04-25T02:00:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.851+0000 7fae9ff9f740 10 client.26698 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:00:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.851+0000 7fae9ff9f740 10 client.26698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.851+0000 7fae9ff9f740 20 client.26698 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.851+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aebe7d01d0 0x55aebe7d25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:44.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.851+0000 7fae9ff9f740 4 client.26698 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:00:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.851+0000 7fae9ff9f740 1 -- 172.21.15.102:0/4290354652 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aebe6a8340 con 0x55aebe7d01d0 2024-04-25T02:00:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.852+0000 7fae95e00640 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aebe7d01d0 0x55aebe7d25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:44.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.854+0000 7fae95e00640 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aebe7d01d0 0x55aebe7d25f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7364dfebf9c03cd4 server_cookie=9db5de672763adbc in_seq=0 out_seq=0 2024-04-25T02:00:44.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.854+0000 7fae7be00640 10 client.26698 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae7be00640 1 -- 172.21.15.102:0/4290354652 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aebe6a8340 con 0x55aebe7d01d0 2024-04-25T02:00:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae7be00640 10 client.26698 handle_command_reply: tid=0 2024-04-25T02:00:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae9ff9f740 1 client.26698 shutdown 2024-04-25T02:00:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae9ff9f740 1 -- 172.21.15.102:0/4290354652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe747fe0 msgr2=0x55aebe7427d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe747fe0 0x55aebe7427d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fae9806af20 tx=0x7fae900095c0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae9ff9f740 1 -- 172.21.15.102:0/4290354652 shutdown_connections 2024-04-25T02:00:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aebe7d01d0 0x55aebe7d25f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aebe747fe0 0x55aebe7427d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae7be00640 0 client.26698 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aebe73d720 0x55aebe742290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae9ff9f740 1 --2- 172.21.15.102:0/4290354652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aebe73ce00 0x55aebe741d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.866+0000 7fae9ff9f740 1 -- 172.21.15.102:0/4290354652 >> 172.21.15.102:0/4290354652 conn(0x55aebe739310 msgr2=0x55aebe753380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.867+0000 7fae9ff9f740 1 -- 172.21.15.102:0/4290354652 shutdown_connections 2024-04-25T02:00:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.867+0000 7fae9ff9f740 1 -- 172.21.15.102:0/4290354652 wait complete. 2024-04-25T02:00:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.867+0000 7fae9ff9f740 20 client.26698 trim_cache size 0 max 16384 2024-04-25T02:00:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:00:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:00:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:00:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:00:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 -- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fae7c0a2e20 msgr2=0x7fae7c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fae7c0a2e20 0x7fae7c0a52e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae90009430 tx=0x7fae90009360 comp rx=0 tx=0).stop 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 -- 172.21.15.102:0/1795468553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae98100550 msgr2=0x7fae9810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae98100550 0x7fae9810d960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fae8800b450 tx=0x7fae8800b910 comp rx=0 tx=0).stop 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 -- 172.21.15.102:0/1795468553 shutdown_connections 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fae7c0a2e20 0x7fae7c0a52e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae9810e100 0x7fae9810afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae98100e70 0x7fae9810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 --2- 172.21.15.102:0/1795468553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae98100550 0x7fae9810d960 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.869+0000 7fae9d000640 1 -- 172.21.15.102:0/1795468553 >> 172.21.15.102:0/1795468553 conn(0x7fae980777c0 msgr2=0x7fae9810f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.870+0000 7fae9d000640 1 -- 172.21.15.102:0/1795468553 shutdown_connections 2024-04-25T02:00:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:44.870+0000 7fae9d000640 1 -- 172.21.15.102:0/1795468553 wait complete. 2024-04-25T02:00:45.241 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:44 smithi110 ceph-mon[32848]: pgmap v167: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.7 MiB/s wr, 723 op/s 2024-04-25T02:00:45.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:44 smithi193 ceph-mon[32981]: pgmap v167: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.7 MiB/s wr, 723 op/s 2024-04-25T02:00:45.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:44 smithi102 ceph-mon[27796]: pgmap v167: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.7 MiB/s wr, 723 op/s 2024-04-25T02:00:47.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:46 smithi193 ceph-mon[32981]: pgmap v168: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.1 MiB/s wr, 566 op/s 2024-04-25T02:00:47.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:46 smithi102 ceph-mon[27796]: pgmap v168: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.1 MiB/s wr, 566 op/s 2024-04-25T02:00:47.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:46 smithi110 ceph-mon[32848]: pgmap v168: 97 pgs: 97 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.1 MiB/s wr, 566 op/s 2024-04-25T02:00:48.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 295.0 s, 2924.3 tps, lat 33.955 ms stddev 41.614 2024-04-25T02:00:49.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:48 smithi193 ceph-mon[32981]: pgmap v169: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.7 MiB/s wr, 860 op/s 2024-04-25T02:00:49.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:48 smithi102 ceph-mon[27796]: pgmap v169: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.7 MiB/s wr, 860 op/s 2024-04-25T02:00:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:48 smithi110 ceph-mon[32848]: pgmap v169: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.7 MiB/s wr, 860 op/s 2024-04-25T02:00:51.176 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:51 smithi102 ceph-mon[27796]: pgmap v170: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.7 MiB/s wr, 584 op/s 2024-04-25T02:00:51.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:51 smithi193 ceph-mon[32981]: pgmap v170: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.7 MiB/s wr, 584 op/s 2024-04-25T02:00:51.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:51 smithi110 ceph-mon[32848]: pgmap v170: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.7 MiB/s wr, 584 op/s 2024-04-25T02:00:53.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 300.0 s, 2599.8 tps, lat 38.727 ms stddev 45.179 2024-04-25T02:00:53.258 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:53 smithi193 ceph-mon[32981]: pgmap v171: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.1 MiB/s wr, 784 op/s 2024-04-25T02:00:53.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:53 smithi102 ceph-mon[27796]: pgmap v171: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.1 MiB/s wr, 784 op/s 2024-04-25T02:00:53.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:53 smithi110 ceph-mon[32848]: pgmap v171: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.1 MiB/s wr, 784 op/s 2024-04-25T02:00:55.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:55 smithi102 ceph-mon[27796]: pgmap v172: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.3 MiB/s wr, 730 op/s 2024-04-25T02:00:55.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:55 smithi110 ceph-mon[32848]: pgmap v172: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.3 MiB/s wr, 730 op/s 2024-04-25T02:00:55.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:55 smithi193 ceph-mon[32981]: pgmap v172: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.3 MiB/s wr, 730 op/s 2024-04-25T02:00:57.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:57 smithi102 ceph-mon[27796]: pgmap v173: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.9 MiB/s wr, 612 op/s 2024-04-25T02:00:57.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:57 smithi110 ceph-mon[32848]: pgmap v173: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.9 MiB/s wr, 612 op/s 2024-04-25T02:00:57.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:57 smithi193 ceph-mon[32981]: pgmap v173: 97 pgs: 97 active+clean; 3.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.9 MiB/s wr, 612 op/s 2024-04-25T02:00:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.061+0000 7f7861400640 1 -- 172.21.15.102:0/2984792313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f785c106630 msgr2=0x7f785c106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.061+0000 7f7861400640 1 --2- 172.21.15.102:0/2984792313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f785c106630 0x7f785c106a10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7854009900 tx=0x7f785402a420 comp rx=0 tx=0).stop 2024-04-25T02:00:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.062+0000 7f7861400640 1 -- 172.21.15.102:0/2984792313 shutdown_connections 2024-04-25T02:00:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.062+0000 7f7861400640 1 --2- 172.21.15.102:0/2984792313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785c101030 0x7f785c10d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.062+0000 7f7861400640 1 --2- 172.21.15.102:0/2984792313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f785c100670 0x7f785c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.062+0000 7f7861400640 1 --2- 172.21.15.102:0/2984792313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f785c106630 0x7f785c106a10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.062+0000 7f7861400640 1 -- 172.21.15.102:0/2984792313 >> 172.21.15.102:0/2984792313 conn(0x7f785c0777c0 msgr2=0x7f785c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.062+0000 7f7861400640 1 -- 172.21.15.102:0/2984792313 shutdown_connections 2024-04-25T02:00:58.068 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.062+0000 7f7861400640 1 -- 172.21.15.102:0/2984792313 wait complete. 2024-04-25T02:00:58.068 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.063+0000 7f7861400640 1 Processor -- start 2024-04-25T02:00:58.070 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.064+0000 7f7861400640 1 -- start start 2024-04-25T02:00:58.070 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.064+0000 7f7861400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f785c100670 0x7f785c19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.070 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.064+0000 7f7861400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785c101030 0x7f785c19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.064+0000 7f7861400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f785c106630 0x7f785c1a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.064+0000 7f7861400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f785c10f980 con 0x7f785c100670 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.064+0000 7f7861400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f785c10f800 con 0x7f785c106630 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.064+0000 7f7861400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f785c10fb00 con 0x7f785c101030 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f7859e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785c101030 0x7f785c19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f7859e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785c101030 0x7f785c19ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52402/0 (socket says 172.21.15.102:52402) 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f7859e00640 1 -- 172.21.15.102:0/2229012697 learned_addr learned my addr 172.21.15.102:0/2229012697 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f785b200640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f785c106630 0x7f785c1a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f7859e00640 1 -- 172.21.15.102:0/2229012697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f785c106630 msgr2=0x7f785c1a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f7859e00640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f785c106630 0x7f785c1a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f7859e00640 1 -- 172.21.15.102:0/2229012697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f785c100670 msgr2=0x7f785c19e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f7859e00640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f785c100670 0x7f785c19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.071 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f7859e00640 1 -- 172.21.15.102:0/2229012697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c1a34f0 con 0x7f785c101030 2024-04-25T02:00:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.065+0000 7f785b200640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f785c106630 0x7f785c1a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.066+0000 7f7859e00640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785c101030 0x7f785c19ea70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f784400ea90 tx=0x7f784400ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.066+0000 7f7853400640 1 -- 172.21.15.102:0/2229012697 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784400ce40 con 0x7f785c101030 2024-04-25T02:00:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.066+0000 7f7853400640 1 -- 172.21.15.102:0/2229012697 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78440035f0 con 0x7f785c101030 2024-04-25T02:00:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.066+0000 7f7853400640 1 -- 172.21.15.102:0/2229012697 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7844010690 con 0x7f785c101030 2024-04-25T02:00:58.072 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.066+0000 7f7861400640 1 -- 172.21.15.102:0/2229012697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785c1a37e0 con 0x7f785c101030 2024-04-25T02:00:58.073 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.066+0000 7f7861400640 1 -- 172.21.15.102:0/2229012697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f785c06b160 con 0x7f785c101030 2024-04-25T02:00:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 305.0 s, 3212.0 tps, lat 31.113 ms stddev 37.606 2024-04-25T02:00:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.073+0000 7f7853400640 1 -- 172.21.15.102:0/2229012697 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f78440037a0 con 0x7f785c101030 2024-04-25T02:00:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.074+0000 7f7864266740 1 Processor -- start 2024-04-25T02:00:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.075+0000 7f7864266740 1 -- start start 2024-04-25T02:00:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.075+0000 7f7864266740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d5871e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.075+0000 7f7864266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d587720 0x562a3d591aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.075+0000 7f7864266740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d591fe0 0x562a3d5943d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.075+0000 7f7864266740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x562a3d4eaab0 con 0x562a3d586e00 2024-04-25T02:00:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.075+0000 7f7864266740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562a3d493640 con 0x562a3d591fe0 2024-04-25T02:00:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.075+0000 7f7864266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562a3d4e6a70 con 0x562a3d587720 2024-04-25T02:00:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.075+0000 7f785a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d587720 0x562a3d591aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.076+0000 7f785a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d587720 0x562a3d591aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52418/0 (socket says 172.21.15.102:52418) 2024-04-25T02:00:58.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.076+0000 7f785a800640 1 -- 172.21.15.102:0/1733192900 learned_addr learned my addr 172.21.15.102:0/1733192900 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.076+0000 7f7859e00640 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d591fe0 0x562a3d5943d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.076+0000 7f785b200640 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d5871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.076+0000 7f785a800640 1 -- 172.21.15.102:0/1733192900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d591fe0 msgr2=0x562a3d5943d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:58.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.077+0000 7f785a800640 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d591fe0 0x562a3d5943d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.077+0000 7f785a800640 1 -- 172.21.15.102:0/1733192900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 msgr2=0x562a3d5871e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.077+0000 7f785a800640 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d5871e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.077+0000 7f785a800640 1 -- 172.21.15.102:0/1733192900 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3d1ff210 con 0x562a3d587720 2024-04-25T02:00:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.077+0000 7f785b200640 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d5871e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:58.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.077+0000 7f7859e00640 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d591fe0 0x562a3d5943d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.078+0000 7f7853400640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78400a2ea0 0x7f78400a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.078+0000 7f785a800640 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d587720 0x562a3d591aa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7854009230 tx=0x7f78540035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=818ca368ea480622 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.078+0000 7f7853400640 1 -- 172.21.15.102:0/2229012697 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f78440cf990 con 0x7f785c101030 2024-04-25T02:00:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.079+0000 7f785b200640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78400a2ea0 0x7f78400a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.079+0000 7f784b400640 1 -- 172.21.15.102:0/1733192900 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7854040030 con 0x562a3d587720 2024-04-25T02:00:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.079+0000 7f784b400640 1 -- 172.21.15.102:0/1733192900 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78540037e0 con 0x562a3d587720 2024-04-25T02:00:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.080+0000 7f785b200640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78400a2ea0 0x7f78400a5360 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f784c005fd0 tx=0x7f784c00b2c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.080+0000 7f7864266740 1 -- 172.21.15.102:0/1733192900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d587720 msgr2=0x562a3d591aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.080+0000 7f7864266740 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d587720 0x562a3d591aa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7854009230 tx=0x7f78540035f0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.080+0000 7f784b400640 1 -- 172.21.15.102:0/1733192900 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7854040030 con 0x562a3d587720 2024-04-25T02:00:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.080+0000 7f7864266740 1 -- 172.21.15.102:0/1733192900 shutdown_connections 2024-04-25T02:00:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.080+0000 7f7864266740 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d591fe0 0x562a3d5943d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.080+0000 7f7864266740 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d587720 0x562a3d591aa0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.080+0000 7f7864266740 1 --2- 172.21.15.102:0/1733192900 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d5871e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.080+0000 7f7864266740 1 -- 172.21.15.102:0/1733192900 >> 172.21.15.102:0/1733192900 conn(0x562a3d583310 msgr2=0x562a3d5836f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.081+0000 7f7864266740 1 -- 172.21.15.102:0/1733192900 shutdown_connections 2024-04-25T02:00:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.081+0000 7f7864266740 1 -- 172.21.15.102:0/1733192900 wait complete. 2024-04-25T02:00:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.081+0000 7f7864266740 1 -- start start 2024-04-25T02:00:58.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.082+0000 7f7864266740 1 Processor -- start 2024-04-25T02:00:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.083+0000 7f7864266740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d58ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.083+0000 7f7864266740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d587720 0x562a3d58f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.083+0000 7f7864266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.083+0000 7f7864266740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x562a3d594bb0 con 0x562a3d586e00 2024-04-25T02:00:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.083+0000 7f7864266740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562a3d4a6990 con 0x562a3d587720 2024-04-25T02:00:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.083+0000 7f7864266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562a3d594d30 con 0x562a3d591fe0 2024-04-25T02:00:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.083+0000 7f7864266740 10 client.0 _ll_register_callbacks cb 0x562a3d4a8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.084+0000 7f785b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.084+0000 7f785b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52424/0 (socket says 172.21.15.102:52424) 2024-04-25T02:00:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.084+0000 7f785b200640 1 -- 172.21.15.102:0/2832499709 learned_addr learned my addr 172.21.15.102:0/2832499709 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.084+0000 7f7859e00640 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d587720 0x562a3d58f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.084+0000 7f7859e00640 1 -- 172.21.15.102:0/2832499709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 msgr2=0x562a3d58f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.084+0000 7f7859e00640 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.084+0000 7f7859e00640 1 -- 172.21.15.102:0/2832499709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 msgr2=0x562a3d58ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.084+0000 7f785a800640 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d58ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.085+0000 7f7859e00640 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d58ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.085+0000 7f7859e00640 1 -- 172.21.15.102:0/2832499709 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3d2cb620 con 0x562a3d587720 2024-04-25T02:00:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.085+0000 7f785a800640 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d58ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:00:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.085+0000 7f7859e00640 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d587720 0x562a3d58f360 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f78440cd0b0 tx=0x7f784409c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=861f12b3ab8f9781 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.085+0000 7f783fe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:00:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.085+0000 7f783fe00640 1 -- 172.21.15.102:0/2832499709 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78440d1e60 con 0x562a3d587720 2024-04-25T02:00:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.085+0000 7f7864266740 1 -- 172.21.15.102:0/2832499709 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78540037e0 con 0x562a3d587720 2024-04-25T02:00:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.086+0000 7f783fe00640 1 -- 172.21.15.102:0/2832499709 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f78440984f0 con 0x562a3d587720 2024-04-25T02:00:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.086+0000 7f783fe00640 1 -- 172.21.15.102:0/2832499709 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f78440b8050 con 0x562a3d587720 2024-04-25T02:00:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.086+0000 7f7864266740 10 client.26744 fetch_fsmap learned FSMap version 34 2024-04-25T02:00:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.086+0000 7f7864266740 1 -- 172.21.15.102:0/2832499709 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562a3d5a0980 con 0x562a3d587720 2024-04-25T02:00:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.086+0000 7f783fe00640 1 -- 172.21.15.102:0/2832499709 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78540037e0 con 0x562a3d587720 2024-04-25T02:00:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.088+0000 7f783fe00640 1 -- 172.21.15.102:0/2832499709 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f7844097bd0 con 0x562a3d587720 2024-04-25T02:00:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.088+0000 7f7864266740 10 client.26744 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:00:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.088+0000 7f7864266740 10 client.26744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.088+0000 7f7864266740 20 client.26744 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.088+0000 7f7864266740 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562a3d5a1e90 0x562a3d5a4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.088+0000 7f7864266740 4 client.26744 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:00:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.088+0000 7f7864266740 1 -- 172.21.15.102:0/2832499709 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a3d4f7ff0 con 0x562a3d5a1e90 2024-04-25T02:00:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.089+0000 7f785a800640 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562a3d5a1e90 0x562a3d5a4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.090+0000 7f785a800640 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562a3d5a1e90 0x562a3d5a4320 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4eee3938b825cb64 server_cookie=b641227a837b6f0c in_seq=0 out_seq=0 2024-04-25T02:00:58.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.090+0000 7f783fe00640 10 client.26744 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.091+0000 7f783fe00640 1 -- 172.21.15.102:0/2832499709 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a3d4f7ff0 con 0x562a3d5a1e90 2024-04-25T02:00:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.091+0000 7f783fe00640 10 client.26744 handle_command_reply: tid=0 2024-04-25T02:00:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.091+0000 7f7864266740 1 client.26744 shutdown 2024-04-25T02:00:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.091+0000 7f7864266740 1 -- 172.21.15.102:0/2832499709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d587720 msgr2=0x562a3d58f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.091+0000 7f7864266740 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d587720 0x562a3d58f360 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f78440cd0b0 tx=0x7f784409c580 comp rx=0 tx=0).stop 2024-04-25T02:00:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f7864266740 1 -- 172.21.15.102:0/2832499709 shutdown_connections 2024-04-25T02:00:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f7864266740 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562a3d5a1e90 0x562a3d5a4320 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f7864266740 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f7864266740 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d587720 0x562a3d58f360 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f7864266740 1 --2- 172.21.15.102:0/2832499709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d586e00 0x562a3d58ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f7864266740 1 -- 172.21.15.102:0/2832499709 >> 172.21.15.102:0/2832499709 conn(0x562a3d583310 msgr2=0x562a3d589090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f783fe00640 0 client.26744 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f7864266740 1 -- 172.21.15.102:0/2832499709 shutdown_connections 2024-04-25T02:00:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f7864266740 1 -- 172.21.15.102:0/2832499709 wait complete. 2024-04-25T02:00:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.092+0000 7f7864266740 20 client.26744 trim_cache size 0 max 16384 2024-04-25T02:00:58.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.120+0000 7f7864266740 1 Processor -- start 2024-04-25T02:00:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.121+0000 7f7864266740 1 -- start start 2024-04-25T02:00:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f7864266740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d59dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f7864266740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 0x562a3d58df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f7864266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d582c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f7864266740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f785c1bea00 con 0x562a3d587720 2024-04-25T02:00:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f7864266740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f785c1bc270 con 0x562a3d586e00 2024-04-25T02:00:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f7864266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f785c071980 con 0x562a3d591fe0 2024-04-25T02:00:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f7859e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 0x562a3d58df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f785a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d59dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f785a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d59dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40504/0 (socket says 172.21.15.102:40504) 2024-04-25T02:00:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f785a800640 1 -- 172.21.15.102:0/1866525691 learned_addr learned my addr 172.21.15.102:0/1866525691 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:58.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.122+0000 7f785b200640 1 --2- 172.21.15.102:0/1866525691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d582c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.123+0000 7f785a800640 1 -- 172.21.15.102:0/1866525691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 msgr2=0x562a3d582c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.123+0000 7f785a800640 1 --2- 172.21.15.102:0/1866525691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d582c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.123+0000 7f785a800640 1 -- 172.21.15.102:0/1866525691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 msgr2=0x562a3d58df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.123+0000 7f785a800640 1 --2- 172.21.15.102:0/1866525691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 0x562a3d58df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.123+0000 7f785a800640 1 -- 172.21.15.102:0/1866525691 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a3d583140 con 0x562a3d586e00 2024-04-25T02:00:58.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.123+0000 7f785b200640 1 --2- 172.21.15.102:0/1866525691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d582c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:00:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.123+0000 7f785a800640 1 --2- 172.21.15.102:0/1866525691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d59dac0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7854004180 tx=0x7f7854005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=157fdd4c253433ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.124+0000 7f784aa00640 1 -- 172.21.15.102:0/1866525691 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7854040040 con 0x562a3d586e00 2024-04-25T02:00:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.124+0000 7f784aa00640 1 -- 172.21.15.102:0/1866525691 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78540037e0 con 0x562a3d586e00 2024-04-25T02:00:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.124+0000 7f784aa00640 1 -- 172.21.15.102:0/1866525691 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78540504b0 con 0x562a3d586e00 2024-04-25T02:00:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.126+0000 7f7864266740 1 -- 172.21.15.102:0/1866525691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 msgr2=0x562a3d59dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:58.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.126+0000 7f7864266740 1 --2- 172.21.15.102:0/1866525691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d59dac0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7854004180 tx=0x7f7854005fd0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.126+0000 7f7864266740 1 -- 172.21.15.102:0/1866525691 shutdown_connections 2024-04-25T02:00:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.126+0000 7f7864266740 1 --2- 172.21.15.102:0/1866525691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d582c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.126+0000 7f7864266740 1 --2- 172.21.15.102:0/1866525691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 0x562a3d58df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.126+0000 7f7864266740 1 --2- 172.21.15.102:0/1866525691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d59dac0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.126+0000 7f7864266740 1 -- 172.21.15.102:0/1866525691 >> 172.21.15.102:0/1866525691 conn(0x562a3d583310 msgr2=0x562a3d5887b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.126+0000 7f7864266740 1 -- 172.21.15.102:0/1866525691 shutdown_connections 2024-04-25T02:00:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.126+0000 7f7864266740 1 -- 172.21.15.102:0/1866525691 wait complete. 2024-04-25T02:00:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.127+0000 7f7864266740 1 -- start start 2024-04-25T02:00:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.127+0000 7f7864266740 1 Processor -- start 2024-04-25T02:00:58.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f7864266740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d58bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f7864266740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 0x562a3d58c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f7864266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f785b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f785b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52438/0 (socket says 172.21.15.102:52438) 2024-04-25T02:00:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f785b200640 1 -- 172.21.15.102:0/1336059935 learned_addr learned my addr 172.21.15.102:0/1336059935 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:00:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f785c071ff0 con 0x562a3d587720 2024-04-25T02:00:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f785c073460 con 0x562a3d586e00 2024-04-25T02:00:58.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f785c0726c0 con 0x562a3d591fe0 2024-04-25T02:00:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f785a800640 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d58bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f7864266740 10 client.0 _ll_register_callbacks cb 0x7f784409b9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:00:58.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.128+0000 7f7859e00640 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 0x562a3d58c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.129+0000 7f785b200640 1 -- 172.21.15.102:0/1336059935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 msgr2=0x562a3d58bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.129+0000 7f785b200640 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d58bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.129+0000 7f785b200640 1 -- 172.21.15.102:0/1336059935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 msgr2=0x562a3d58c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:00:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.129+0000 7f785b200640 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 0x562a3d58c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.129+0000 7f785b200640 1 -- 172.21.15.102:0/1336059935 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a3d6075d0 con 0x562a3d591fe0 2024-04-25T02:00:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.129+0000 7f785a800640 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d58bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:00:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.129+0000 7f785b200640 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58c600 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f784c018e40 tx=0x7f784c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20c81c65e060cac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:00:58.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.129+0000 7f783fe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:00:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.130+0000 7f783fe00640 1 -- 172.21.15.102:0/1336059935 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c004b20 con 0x562a3d591fe0 2024-04-25T02:00:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.130+0000 7f783fe00640 1 -- 172.21.15.102:0/1336059935 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f784c004cc0 con 0x562a3d591fe0 2024-04-25T02:00:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.130+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78540037e0 con 0x562a3d591fe0 2024-04-25T02:00:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.130+0000 7f783fe00640 1 -- 172.21.15.102:0/1336059935 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f784c026480 con 0x562a3d591fe0 2024-04-25T02:00:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.130+0000 7f7864266740 10 client.26716 fetch_fsmap learned FSMap version 34 2024-04-25T02:00:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.130+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562a3d608900 con 0x562a3d591fe0 2024-04-25T02:00:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.131+0000 7f783fe00640 1 -- 172.21.15.102:0/1336059935 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78540037e0 con 0x562a3d591fe0 2024-04-25T02:00:58.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.131+0000 7f783fe00640 1 -- 172.21.15.102:0/1336059935 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f784c005380 con 0x562a3d591fe0 2024-04-25T02:00:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.131+0000 7f7864266740 10 client.26716 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:00:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.131+0000 7f7864266740 10 client.26716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:00:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.131+0000 7f7864266740 20 client.26716 populate_metadata read hostname 'smithi102' 2024-04-25T02:00:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.131+0000 7f7864266740 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562a3d609110 0x562a3d60b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:00:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.131+0000 7f7864266740 4 client.26716 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:00:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.131+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a3d4f2340 con 0x562a3d609110 2024-04-25T02:00:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.132+0000 7f785a800640 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562a3d609110 0x562a3d60b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:00:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.132+0000 7f785a800640 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562a3d609110 0x562a3d60b530 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caac237fabefe1bc server_cookie=e2871c4f1e8e5655 in_seq=0 out_seq=0 2024-04-25T02:00:58.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.132+0000 7f783fe00640 10 client.26716 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.164+0000 7f783fe00640 1 -- 172.21.15.102:0/1336059935 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a3d4f2340 con 0x562a3d609110 2024-04-25T02:00:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f783fe00640 10 client.26716 handle_command_reply: tid=0 2024-04-25T02:00:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f7864266740 1 client.26716 shutdown 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 msgr2=0x562a3d58c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f7864266740 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58c600 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f784c018e40 tx=0x7f784c004a40 comp rx=0 tx=0).stop 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 shutdown_connections 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f7864266740 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562a3d609110 0x562a3d60b530 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f7864266740 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a3d591fe0 0x562a3d58c600 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f7864266740 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562a3d587720 0x562a3d58c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f7864266740 1 --2- 172.21.15.102:0/1336059935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562a3d586e00 0x562a3d58bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 >> 172.21.15.102:0/1336059935 conn(0x562a3d583310 msgr2=0x562a3d58f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.165+0000 7f783fe00640 0 client.26716 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.166+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 shutdown_connections 2024-04-25T02:00:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.166+0000 7f7864266740 1 -- 172.21.15.102:0/1336059935 wait complete. 2024-04-25T02:00:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.166+0000 7f7864266740 20 client.26716 trim_cache size 0 max 16384 2024-04-25T02:00:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:00:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:00:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:00:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:00:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 -- 172.21.15.102:0/2229012697 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78400a2ea0 msgr2=0x7f78400a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78400a2ea0 0x7f78400a5360 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f784c005fd0 tx=0x7f784c00b2c0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 -- 172.21.15.102:0/2229012697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785c101030 msgr2=0x7f785c19ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:00:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785c101030 0x7f785c19ea70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f784400ea90 tx=0x7f784400ef50 comp rx=0 tx=0).stop 2024-04-25T02:00:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 -- 172.21.15.102:0/2229012697 shutdown_connections 2024-04-25T02:00:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f78400a2ea0 0x7f78400a5360 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f785c106630 0x7f785c1a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785c101030 0x7f785c19ea70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 --2- 172.21.15.102:0/2229012697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f785c100670 0x7f785c19e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:00:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 -- 172.21.15.102:0/2229012697 >> 172.21.15.102:0/2229012697 conn(0x7f785c0777c0 msgr2=0x7f785c103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:00:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 -- 172.21.15.102:0/2229012697 shutdown_connections 2024-04-25T02:00:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:00:58.167+0000 7f7861400640 1 -- 172.21.15.102:0/2229012697 wait complete. 2024-04-25T02:00:59.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:00:59 smithi102 ceph-mon[27796]: pgmap v174: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-04-25T02:00:59.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:00:59 smithi110 ceph-mon[32848]: pgmap v174: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-04-25T02:00:59.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:00:59 smithi193 ceph-mon[32981]: pgmap v174: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 924 op/s 2024-04-25T02:01:01.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T02:01:01.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T02:01:01.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:01 smithi102 ceph-mon[27796]: pgmap v175: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.4 MiB/s wr, 630 op/s 2024-04-25T02:01:01.310 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:01 smithi193 ceph-mon[32981]: pgmap v175: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.4 MiB/s wr, 630 op/s 2024-04-25T02:01:01.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.390+0000 7f7ba9a00640 1 -- 172.21.15.102:0/652824283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ba41020b0 msgr2=0x7f7ba410e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.390+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/652824283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ba41020b0 0x7f7ba410e5c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009930 tx=0x7f7b9c02a420 comp rx=0 tx=0).stop 2024-04-25T02:01:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.390+0000 7f7ba9a00640 1 -- 172.21.15.102:0/652824283 shutdown_connections 2024-04-25T02:01:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.390+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/652824283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ba41020b0 0x7f7ba410e5c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.390+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/652824283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ba41016f0 0x7f7ba4101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.390+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/652824283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba4107700 0x7f7ba4107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.390+0000 7f7ba9a00640 1 -- 172.21.15.102:0/652824283 >> 172.21.15.102:0/652824283 conn(0x7f7ba40fd1c0 msgr2=0x7f7ba40ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.390+0000 7f7ba9a00640 1 -- 172.21.15.102:0/652824283 shutdown_connections 2024-04-25T02:01:01.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.390+0000 7f7ba9a00640 1 -- 172.21.15.102:0/652824283 wait complete. 2024-04-25T02:01:01.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.391+0000 7f7ba9a00640 1 Processor -- start 2024-04-25T02:01:01.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba9a00640 1 -- start start 2024-04-25T02:01:01.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba9a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba41016f0 0x7f7ba4073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ba41020b0 0x7f7ba4073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba9a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ba4107700 0x7f7ba40744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba9a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba4111b60 con 0x7f7ba41020b0 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba9a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41119e0 con 0x7f7ba41016f0 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba9a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba4111ce0 con 0x7f7ba4107700 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba41016f0 0x7f7ba4073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba41016f0 0x7f7ba4073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40532/0 (socket says 172.21.15.102:40532) 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3958920609 learned_addr learned my addr 172.21.15.102:0/3958920609 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba3400640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ba4107700 0x7f7ba40744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.392+0000 7f7ba2000640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ba41020b0 0x7f7ba4073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3958920609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ba4107700 msgr2=0x7f7ba40744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ba4107700 0x7f7ba40744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ba41020b0 msgr2=0x7f7ba4073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ba41020b0 0x7f7ba4073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3958920609 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba41a6500 con 0x7f7ba41016f0 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7ba3400640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ba4107700 0x7f7ba40744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba41016f0 0x7f7ba4073a40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009d50 tx=0x7f7b9400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7b9b400640 1 -- 172.21.15.102:0/3958920609 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94003b70 con 0x7f7ba41016f0 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7b9b400640 1 -- 172.21.15.102:0/3958920609 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b94003d10 con 0x7f7ba41016f0 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7ba9a00640 1 -- 172.21.15.102:0/3958920609 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba41a67f0 con 0x7f7ba41016f0 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7b9b400640 1 -- 172.21.15.102:0/3958920609 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94010a20 con 0x7f7ba41016f0 2024-04-25T02:01:01.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.393+0000 7f7ba9a00640 1 -- 172.21.15.102:0/3958920609 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba41a6d90 con 0x7f7ba41016f0 2024-04-25T02:01:01.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.396+0000 7f7b9b400640 1 -- 172.21.15.102:0/3958920609 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7b94003ec0 con 0x7f7ba41016f0 2024-04-25T02:01:01.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.397+0000 7f7bac865740 1 Processor -- start 2024-04-25T02:01:01.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.397+0000 7f7bac865740 1 -- start start 2024-04-25T02:01:01.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.397+0000 7f7bac865740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e35268d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.397+0000 7f7bac865740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352697ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.397+0000 7f7bac865740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 0x55e35269a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.397+0000 7f7bac865740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55e3525f1560 con 0x55e35268d920 2024-04-25T02:01:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.397+0000 7f7bac865740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e352599930 con 0x55e352697ff0 2024-04-25T02:01:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.397+0000 7f7bac865740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e3525ed520 con 0x55e35268d000 2024-04-25T02:01:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.397+0000 7f7ba3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e35268d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e35268d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52462/0 (socket says 172.21.15.102:52462) 2024-04-25T02:01:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba3400640 1 -- 172.21.15.102:0/1167847754 learned_addr learned my addr 172.21.15.102:0/1167847754 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba2000640 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 0x55e35269a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352697ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba3400640 1 -- 172.21.15.102:0/1167847754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 msgr2=0x55e35269a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba3400640 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 0x55e35269a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba3400640 1 -- 172.21.15.102:0/1167847754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 msgr2=0x55e352697ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba3400640 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352697ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba3400640 1 -- 172.21.15.102:0/1167847754 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3523cf7d0 con 0x55e35268d000 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.398+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352697ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.399+0000 7f7ba3400640 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e35268d3e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009900 tx=0x7f7b9c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75b1ba19779c98d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.399+0000 7f7b9b400640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7b880a2d80 0x7f7b880a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.399+0000 7f7b93e00640 1 -- 172.21.15.102:0/1167847754 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c035030 con 0x55e35268d000 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.399+0000 7f7b93e00640 1 -- 172.21.15.102:0/1167847754 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b9c005520 con 0x55e35268d000 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.399+0000 7f7b93e00640 1 -- 172.21.15.102:0/1167847754 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c0035f0 con 0x55e35268d000 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.399+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7b880a2d80 0x7f7b880a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.399+0000 7f7b9b400640 1 -- 172.21.15.102:0/3958920609 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7b940cfad0 con 0x7f7ba41016f0 2024-04-25T02:01:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7bac865740 1 -- 172.21.15.102:0/1167847754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 msgr2=0x55e35268d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7bac865740 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e35268d3e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009900 tx=0x7f7b9c004040 comp rx=0 tx=0).stop 2024-04-25T02:01:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7bac865740 1 -- 172.21.15.102:0/1167847754 shutdown_connections 2024-04-25T02:01:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7bac865740 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 0x55e35269a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7bac865740 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352697ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7bac865740 1 --2- 172.21.15.102:0/1167847754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e35268d3e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7bac865740 1 -- 172.21.15.102:0/1167847754 >> 172.21.15.102:0/1167847754 conn(0x55e352689320 msgr2=0x55e352689700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7b880a2d80 0x7f7b880a5240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55e35268e5c0 tx=0x7f7b940bc040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7bac865740 1 -- 172.21.15.102:0/1167847754 shutdown_connections 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.400+0000 7f7bac865740 1 -- 172.21.15.102:0/1167847754 wait complete. 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.401+0000 7f7bac865740 1 -- start start 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.401+0000 7f7bac865740 1 Processor -- start 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7bac865740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e352694fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7bac865740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d920 0x55e3526954f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7bac865740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 0x55e352695a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7bac865740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55e35269ad80 con 0x55e352697ff0 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7bac865740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e3525acd90 con 0x55e35268d920 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7bac865740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e35269af00 con 0x55e35268d000 2024-04-25T02:01:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7bac865740 10 client.0 _ll_register_callbacks cb 0x55e3525ac780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7ba3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e352694fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7ba3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e352694fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52468/0 (socket says 172.21.15.102:52468) 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7ba3400640 1 -- 172.21.15.102:0/2767922217 learned_addr learned my addr 172.21.15.102:0/2767922217 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7ba2000640 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d920 0x55e3526954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.402+0000 7f7ba3400640 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 0x55e352695a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7ba2000640 1 -- 172.21.15.102:0/2767922217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 msgr2=0x55e352694fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7ba2000640 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e352694fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7ba2000640 1 -- 172.21.15.102:0/2767922217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 msgr2=0x55e352695a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7ba2000640 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 0x55e352695a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7ba2000640 1 -- 172.21.15.102:0/2767922217 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3523053f0 con 0x55e35268d920 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7ba2000640 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d920 0x55e3526954f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c0126d0 tx=0x7f7b8c012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8123679af242522a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:01.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7b90c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7b90c00640 1 -- 172.21.15.102:0/2767922217 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c004050 con 0x55e35268d920 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7b90c00640 1 -- 172.21.15.102:0/2767922217 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7b8c0041f0 con 0x55e35268d920 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.403+0000 7f7bac865740 1 -- 172.21.15.102:0/2767922217 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b9c005520 con 0x55e35268d920 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7b90c00640 1 -- 172.21.15.102:0/2767922217 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7b8c0043f0 con 0x55e35268d920 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7bac865740 10 client.26762 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7bac865740 1 -- 172.21.15.102:0/2767922217 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e3522c50f0 con 0x55e35268d920 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7b90c00640 1 -- 172.21.15.102:0/2767922217 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b9c005520 con 0x55e35268d920 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7b90c00640 1 -- 172.21.15.102:0/2767922217 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f7b8c003bf0 con 0x55e35268d920 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7bac865740 10 client.26762 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7bac865740 10 client.26762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7bac865740 20 client.26762 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7bac865740 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e3526a8470 0x55e3526aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7bac865740 4 client.26762 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.404+0000 7f7bac865740 1 -- 172.21.15.102:0/2767922217 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e352604d90 con 0x55e3526a8470 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.405+0000 7f7ba3400640 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e3526a8470 0x55e3526aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.405+0000 7f7ba3400640 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e3526a8470 0x55e3526aa890 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7240819b88f9c6e3 server_cookie=e721adea0bbec2dc in_seq=0 out_seq=0 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.405+0000 7f7b90c00640 10 client.26762 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7b90c00640 1 -- 172.21.15.102:0/2767922217 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e352604d90 con 0x55e3526a8470 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7b90c00640 10 client.26762 handle_command_reply: tid=0 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 client.26762 shutdown 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 -- 172.21.15.102:0/2767922217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d920 msgr2=0x55e3526954f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d920 0x55e3526954f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c0126d0 tx=0x7f7b8c012b90 comp rx=0 tx=0).stop 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 -- 172.21.15.102:0/2767922217 shutdown_connections 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e3526a8470 0x55e3526aa890 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 0x55e352695a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d920 0x55e3526954f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 --2- 172.21.15.102:0/2767922217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e352694fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7b90c00640 0 client.26762 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 -- 172.21.15.102:0/2767922217 >> 172.21.15.102:0/2767922217 conn(0x55e352689320 msgr2=0x55e35268f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 -- 172.21.15.102:0/2767922217 shutdown_connections 2024-04-25T02:01:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 1 -- 172.21.15.102:0/2767922217 wait complete. 2024-04-25T02:01:01.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.406+0000 7f7bac865740 20 client.26762 trim_cache size 0 max 16384 2024-04-25T02:01:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.430+0000 7f7bac865740 1 Processor -- start 2024-04-25T02:01:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.430+0000 7f7bac865740 1 -- start start 2024-04-25T02:01:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7bac865740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e3526a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7bac865740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352695850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7bac865740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 0x55e3526b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7bac865740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41b1f70 con 0x55e35268d920 2024-04-25T02:01:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7bac865740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41af7e0 con 0x55e352697ff0 2024-04-25T02:01:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7bac865740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41ad630 con 0x55e35268d000 2024-04-25T02:01:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7ba2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352695850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7ba3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e3526a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7ba2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352695850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44254/0 (socket says 172.21.15.102:44254) 2024-04-25T02:01:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7ba2000640 1 -- 172.21.15.102:0/1841085620 learned_addr learned my addr 172.21.15.102:0/1841085620 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.431+0000 7f7ba3400640 1 --2- 172.21.15.102:0/1841085620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 0x55e3526b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.432+0000 7f7ba2000640 1 -- 172.21.15.102:0/1841085620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 msgr2=0x55e3526a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.432+0000 7f7ba2000640 1 --2- 172.21.15.102:0/1841085620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e3526a3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.432+0000 7f7ba2000640 1 -- 172.21.15.102:0/1841085620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 msgr2=0x55e3526b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.432+0000 7f7ba2000640 1 --2- 172.21.15.102:0/1841085620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 0x55e3526b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.432+0000 7f7ba2000640 1 -- 172.21.15.102:0/1841085620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e352684c80 con 0x55e35268d920 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.432+0000 7f7ba2000640 1 --2- 172.21.15.102:0/1841085620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352695850 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c010620 tx=0x7f7b8c010930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc639cb36e19e88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.432+0000 7f7b93400640 1 -- 172.21.15.102:0/1841085620 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c032070 con 0x55e35268d920 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.432+0000 7f7b93400640 1 -- 172.21.15.102:0/1841085620 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b8c003fe0 con 0x55e35268d920 2024-04-25T02:01:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.432+0000 7f7b93400640 1 -- 172.21.15.102:0/1841085620 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c029040 con 0x55e35268d920 2024-04-25T02:01:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.433+0000 7f7bac865740 1 -- 172.21.15.102:0/1841085620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 msgr2=0x55e352695850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.433+0000 7f7bac865740 1 --2- 172.21.15.102:0/1841085620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352695850 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c010620 tx=0x7f7b8c010930 comp rx=0 tx=0).stop 2024-04-25T02:01:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.433+0000 7f7bac865740 1 -- 172.21.15.102:0/1841085620 shutdown_connections 2024-04-25T02:01:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.433+0000 7f7bac865740 1 --2- 172.21.15.102:0/1841085620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e352697ff0 0x55e3526b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.433+0000 7f7bac865740 1 --2- 172.21.15.102:0/1841085620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e35268d920 0x55e352695850 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.433+0000 7f7bac865740 1 --2- 172.21.15.102:0/1841085620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d000 0x55e3526a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.433+0000 7f7bac865740 1 -- 172.21.15.102:0/1841085620 >> 172.21.15.102:0/1841085620 conn(0x55e352689320 msgr2=0x55e352689700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.433+0000 7f7bac865740 1 -- 172.21.15.102:0/1841085620 shutdown_connections 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.433+0000 7f7bac865740 1 -- 172.21.15.102:0/1841085620 wait complete. 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.434+0000 7f7bac865740 1 -- start start 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.434+0000 7f7bac865740 1 Processor -- start 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.434+0000 7f7bac865740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d000 0x55e3526a0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.434+0000 7f7bac865740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d920 0x55e352691840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7bac865740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 0x55e352691d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7bac865740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41adca0 con 0x55e352697ff0 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7bac865740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41af110 con 0x55e35268d000 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7bac865740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41ae370 con 0x55e35268d920 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7bac865740 10 client.0 _ll_register_callbacks cb 0x7f7b8c009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d920 0x55e352691840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d920 0x55e352691840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52486/0 (socket says 172.21.15.102:52486) 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba2000640 1 -- 172.21.15.102:0/3864408115 learned_addr learned my addr 172.21.15.102:0/3864408115 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba3400640 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d000 0x55e3526a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba3400640 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 0x55e352691d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba2000640 1 -- 172.21.15.102:0/3864408115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d000 msgr2=0x55e3526a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba2000640 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d000 0x55e3526a0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba2000640 1 -- 172.21.15.102:0/3864408115 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 msgr2=0x55e352691d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba2000640 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 0x55e352691d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba2000640 1 -- 172.21.15.102:0/3864408115 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3526923c0 con 0x55e35268d920 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba3400640 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d000 0x55e3526a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7ba2000640 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d920 0x55e352691840 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c010430 tx=0x7f7b8c00ffd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e04926b44411f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.435+0000 7f7b90c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.436+0000 7f7b90c00640 1 -- 172.21.15.102:0/3864408115 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c032070 con 0x55e35268d920 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.436+0000 7f7bac865740 1 -- 172.21.15.102:0/3864408115 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b8c003fe0 con 0x55e35268d920 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.436+0000 7f7b90c00640 1 -- 172.21.15.102:0/3864408115 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f7b8c004190 con 0x55e35268d920 2024-04-25T02:01:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.436+0000 7f7b90c00640 1 -- 172.21.15.102:0/3864408115 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7b8c029070 con 0x55e35268d920 2024-04-25T02:01:01.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.436+0000 7f7bac865740 10 client.26734 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:01.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.436+0000 7f7bac865740 1 -- 172.21.15.102:0/3864408115 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e35270e850 con 0x55e35268d920 2024-04-25T02:01:01.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.436+0000 7f7b90c00640 1 -- 172.21.15.102:0/3864408115 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b8c003fe0 con 0x55e35268d920 2024-04-25T02:01:01.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7b90c00640 1 -- 172.21.15.102:0/3864408115 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f7b8c005280 con 0x55e35268d920 2024-04-25T02:01:01.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7bac865740 10 client.26734 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:01.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7bac865740 10 client.26734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7bac865740 20 client.26734 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7bac865740 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e35270efd0 0x55e3527113f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7bac865740 4 client.26734 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7bac865740 1 -- 172.21.15.102:0/3864408115 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e3525feaa0 con 0x55e35270efd0 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7ba3400640 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e35270efd0 0x55e3527113f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7ba3400640 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e35270efd0 0x55e3527113f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30a7a2b7d8b9e665 server_cookie=d66bb2c2179e5e82 in_seq=0 out_seq=0 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.437+0000 7f7b90c00640 10 client.26734 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7b90c00640 1 -- 172.21.15.102:0/3864408115 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e3525feaa0 con 0x55e35270efd0 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7b90c00640 10 client.26734 handle_command_reply: tid=0 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 client.26734 shutdown 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 -- 172.21.15.102:0/3864408115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d920 msgr2=0x55e352691840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d920 0x55e352691840 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c010430 tx=0x7f7b8c00ffd0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 -- 172.21.15.102:0/3864408115 shutdown_connections 2024-04-25T02:01:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e35270efd0 0x55e3527113f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e352697ff0 0x55e352691d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7b90c00640 0 client.26734 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e35268d920 0x55e352691840 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 --2- 172.21.15.102:0/3864408115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e35268d000 0x55e3526a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 -- 172.21.15.102:0/3864408115 >> 172.21.15.102:0/3864408115 conn(0x55e352689320 msgr2=0x55e352687bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 -- 172.21.15.102:0/3864408115 shutdown_connections 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 1 -- 172.21.15.102:0/3864408115 wait complete. 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.438+0000 7f7bac865740 20 client.26734 trim_cache size 0 max 16384 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:01:01.447 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T02:01:01.448 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T02:01:01.448 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:01:01.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 -- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7b880a2d80 msgr2=0x7f7b880a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7b880a2d80 0x7f7b880a5240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55e35268e5c0 tx=0x7f7b940bc040 comp rx=0 tx=0).stop 2024-04-25T02:01:01.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 -- 172.21.15.102:0/3958920609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba41016f0 msgr2=0x7f7ba4073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba41016f0 0x7f7ba4073a40 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009d50 tx=0x7f7b9400c6a0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 -- 172.21.15.102:0/3958920609 shutdown_connections 2024-04-25T02:01:01.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7b880a2d80 0x7f7b880a5240 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ba4107700 0x7f7ba40744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ba41020b0 0x7f7ba4073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 --2- 172.21.15.102:0/3958920609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba41016f0 0x7f7ba4073a40 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 -- 172.21.15.102:0/3958920609 >> 172.21.15.102:0/3958920609 conn(0x7f7ba40fd1c0 msgr2=0x7f7ba40fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:01.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 -- 172.21.15.102:0/3958920609 shutdown_connections 2024-04-25T02:01:01.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.439+0000 7f7ba9a00640 1 -- 172.21.15.102:0/3958920609 wait complete. 2024-04-25T02:01:01.470 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T02:01:01.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T02:01:01.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:01 smithi110 ceph-mon[32848]: pgmap v175: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.4 MiB/s wr, 630 op/s 2024-04-25T02:01:01.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.676+0000 7fcc33c00640 1 -- 172.21.15.102:0/322871740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c075c70 msgr2=0x7fcc2c076050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.676+0000 7fcc33c00640 1 --2- 172.21.15.102:0/322871740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c075c70 0x7fcc2c076050 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009930 tx=0x7fcc2002a420 comp rx=0 tx=0).stop 2024-04-25T02:01:01.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.676+0000 7fcc33c00640 1 -- 172.21.15.102:0/322871740 shutdown_connections 2024-04-25T02:01:01.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.676+0000 7fcc33c00640 1 --2- 172.21.15.102:0/322871740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c1125b0 0x7fcc2c1149a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.676+0000 7fcc33c00640 1 --2- 172.21.15.102:0/322871740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc2c076590 0x7fcc2c112070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.676+0000 7fcc33c00640 1 --2- 172.21.15.102:0/322871740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c075c70 0x7fcc2c076050 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.676+0000 7fcc33c00640 1 -- 172.21.15.102:0/322871740 >> 172.21.15.102:0/322871740 conn(0x7fcc2c0ff400 msgr2=0x7fcc2c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:01.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.676+0000 7fcc33c00640 1 -- 172.21.15.102:0/322871740 shutdown_connections 2024-04-25T02:01:01.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.676+0000 7fcc33c00640 1 -- 172.21.15.102:0/322871740 wait complete. 2024-04-25T02:01:01.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.678+0000 7fcc33c00640 1 Processor -- start 2024-04-25T02:01:01.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.678+0000 7fcc33c00640 1 -- start start 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.678+0000 7fcc33c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc2c075c70 0x7fcc2c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.678+0000 7fcc33c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c076590 0x7fcc2c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.678+0000 7fcc33c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1125b0 0x7fcc2c1a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.678+0000 7fcc33c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c118150 con 0x7fcc2c075c70 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.678+0000 7fcc33c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c117fd0 con 0x7fcc2c076590 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.678+0000 7fcc33c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c1182d0 con 0x7fcc2c1125b0 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc31200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc2c075c70 0x7fcc2c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc31c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1125b0 0x7fcc2c1a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc31200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc2c075c70 0x7fcc2c19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44280/0 (socket says 172.21.15.102:44280) 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc31200640 1 -- 172.21.15.102:0/3332672334 learned_addr learned my addr 172.21.15.102:0/3332672334 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc2be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c076590 0x7fcc2c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc31c00640 1 -- 172.21.15.102:0/3332672334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c076590 msgr2=0x7fcc2c19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc31c00640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c076590 0x7fcc2c19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc31c00640 1 -- 172.21.15.102:0/3332672334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc2c075c70 msgr2=0x7fcc2c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc31c00640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc2c075c70 0x7fcc2c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.679+0000 7fcc31c00640 1 -- 172.21.15.102:0/3332672334 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2c1a3680 con 0x7fcc2c1125b0 2024-04-25T02:01:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.680+0000 7fcc31c00640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1125b0 0x7fcc2c1a2f50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcc24005fd0 tx=0x7fcc2400edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.680+0000 7fcc29600640 1 -- 172.21.15.102:0/3332672334 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24004d20 con 0x7fcc2c1125b0 2024-04-25T02:01:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.680+0000 7fcc29600640 1 -- 172.21.15.102:0/3332672334 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc24005020 con 0x7fcc2c1125b0 2024-04-25T02:01:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.680+0000 7fcc29600640 1 -- 172.21.15.102:0/3332672334 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc240116d0 con 0x7fcc2c1125b0 2024-04-25T02:01:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.680+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2c1a3970 con 0x7fcc2c1125b0 2024-04-25T02:01:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.680+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2c1ab200 con 0x7fcc2c1125b0 2024-04-25T02:01:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.682+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc2c0689b0 con 0x7fcc2c1125b0 2024-04-25T02:01:01.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.683+0000 7fcc29600640 1 -- 172.21.15.102:0/3332672334 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcc24005540 con 0x7fcc2c1125b0 2024-04-25T02:01:01.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.685+0000 7fcc29600640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcc140a2d80 0x7fcc140a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:01.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.685+0000 7fcc29600640 1 -- 172.21.15.102:0/3332672334 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fcc240cf3f0 con 0x7fcc2c1125b0 2024-04-25T02:01:01.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.686+0000 7fcc31200640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcc140a2d80 0x7fcc140a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:01.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.687+0000 7fcc31200640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcc140a2d80 0x7fcc140a5240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcc200096c0 tx=0x7fcc20005b50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:01.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.689+0000 7fcc29600640 1 -- 172.21.15.102:0/3332672334 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcc24016030 con 0x7fcc2c1125b0 2024-04-25T02:01:01.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.988+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc2c10d7a0 con 0x7fcc2c1125b0 2024-04-25T02:01:01.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.989+0000 7fcc29600640 1 -- 172.21.15.102:0/3332672334 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11902 (secure 0 0 0) 0x7fcc2409b990 con 0x7fcc2c1125b0 2024-04-25T02:01:01.996 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:01:01.996 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T02:00:31.796906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T02:01:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 34 2024-04-25T02:01:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.992+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcc140a2d80 msgr2=0x7fcc140a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.992+0000 7fcc33c00640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcc140a2d80 0x7fcc140a5240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcc200096c0 tx=0x7fcc20005b50 comp rx=0 tx=0).stop 2024-04-25T02:01:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.992+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1125b0 msgr2=0x7fcc2c1a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.992+0000 7fcc33c00640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1125b0 0x7fcc2c1a2f50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcc24005fd0 tx=0x7fcc2400edd0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.993+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 shutdown_connections 2024-04-25T02:01:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.993+0000 7fcc33c00640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fcc140a2d80 0x7fcc140a5240 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.993+0000 7fcc33c00640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1125b0 0x7fcc2c1a2f50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.993+0000 7fcc33c00640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c076590 0x7fcc2c19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.993+0000 7fcc33c00640 1 --2- 172.21.15.102:0/3332672334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc2c075c70 0x7fcc2c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.993+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 >> 172.21.15.102:0/3332672334 conn(0x7fcc2c0ff400 msgr2=0x7fcc2c113c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.993+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 shutdown_connections 2024-04-25T02:01:02.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:01.993+0000 7fcc33c00640 1 -- 172.21.15.102:0/3332672334 wait complete. 2024-04-25T02:01:02.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T02:01:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.223+0000 7f522d600640 1 -- 172.21.15.102:0/1460830569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52281007b0 msgr2=0x7f5228100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.223+0000 7f522d600640 1 --2- 172.21.15.102:0/1460830569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52281007b0 0x7f5228100b90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5220009be0 tx=0x7f522002a7e0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.224+0000 7f522d600640 1 -- 172.21.15.102:0/1460830569 shutdown_connections 2024-04-25T02:01:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.224+0000 7f522d600640 1 --2- 172.21.15.102:0/1460830569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5228105ad0 0x7f5228105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.224+0000 7f522d600640 1 --2- 172.21.15.102:0/1460830569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52281010d0 0x7f5228105590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.224+0000 7f522d600640 1 --2- 172.21.15.102:0/1460830569 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52281007b0 0x7f5228100b90 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.224+0000 7f522d600640 1 -- 172.21.15.102:0/1460830569 >> 172.21.15.102:0/1460830569 conn(0x7f5228077740 msgr2=0x7f5228077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.224+0000 7f522d600640 1 -- 172.21.15.102:0/1460830569 shutdown_connections 2024-04-25T02:01:02.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.224+0000 7f522d600640 1 -- 172.21.15.102:0/1460830569 wait complete. 2024-04-25T02:01:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.225+0000 7f522d600640 1 Processor -- start 2024-04-25T02:01:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.225+0000 7f522d600640 1 -- start start 2024-04-25T02:01:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.225+0000 7f522d600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52281007b0 0x7f522819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.225+0000 7f522d600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52281010d0 0x7f522819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f522d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5228105ad0 0x7f52281a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f522d600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f52281180d0 con 0x7f5228105ad0 2024-04-25T02:01:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f522d600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5228117f50 con 0x7f52281010d0 2024-04-25T02:01:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f522d600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5228118250 con 0x7f52281007b0 2024-04-25T02:01:02.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5226a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52281007b0 0x7f522819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5226a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52281007b0 0x7f522819e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52502/0 (socket says 172.21.15.102:52502) 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5226a00640 1 -- 172.21.15.102:0/489554945 learned_addr learned my addr 172.21.15.102:0/489554945 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5226000640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52281010d0 0x7f522819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5227400640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5228105ad0 0x7f52281a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5226a00640 1 -- 172.21.15.102:0/489554945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52281010d0 msgr2=0x7f522819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5226a00640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52281010d0 0x7f522819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5226a00640 1 -- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5228105ad0 msgr2=0x7f52281a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5226a00640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5228105ad0 0x7f52281a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5226a00640 1 -- 172.21.15.102:0/489554945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52281a3600 con 0x7f52281007b0 2024-04-25T02:01:02.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.226+0000 7f5227400640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5228105ad0 0x7f52281a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.227+0000 7f5226a00640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52281007b0 0x7f522819e610 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f522002add0 tx=0x7f52200305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.227+0000 7f521f400640 1 -- 172.21.15.102:0/489554945 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220041020 con 0x7f52281007b0 2024-04-25T02:01:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.227+0000 7f521f400640 1 -- 172.21.15.102:0/489554945 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5220003e50 con 0x7f52281007b0 2024-04-25T02:01:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.227+0000 7f521f400640 1 -- 172.21.15.102:0/489554945 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220036040 con 0x7f52281007b0 2024-04-25T02:01:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.227+0000 7f522d600640 1 -- 172.21.15.102:0/489554945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52281a3890 con 0x7f52281007b0 2024-04-25T02:01:02.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.227+0000 7f522d600640 1 -- 172.21.15.102:0/489554945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52281ab180 con 0x7f52281007b0 2024-04-25T02:01:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.230+0000 7f52303cf740 1 Processor -- start 2024-04-25T02:01:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f52303cf740 1 -- start start 2024-04-25T02:01:02.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f521f400640 1 -- 172.21.15.102:0/489554945 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5220044020 con 0x7f52281007b0 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f52303cf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f0406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f52303cf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 0x559f04076070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f52303cf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f040789a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f52303cf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559f03fd04e0 con 0x559f0406bee0 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f52303cf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559f03f78a70 con 0x559f0406b5c0 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f52303cf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f03fcc4a0 con 0x559f040765b0 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f5227400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f0406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f5227400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f0406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40614/0 (socket says 172.21.15.102:40614) 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f5227400640 1 -- 172.21.15.102:0/1606540819 learned_addr learned my addr 172.21.15.102:0/1606540819 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f5226a00640 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 0x559f04076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.231+0000 7f5226000640 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f040789a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5227400640 1 -- 172.21.15.102:0/1606540819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 msgr2=0x559f040789a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5227400640 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f040789a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5227400640 1 -- 172.21.15.102:0/1606540819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 msgr2=0x559f04076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5227400640 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 0x559f04076070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5227400640 1 -- 172.21.15.102:0/1606540819 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f03ce43f0 con 0x559f0406b5c0 2024-04-25T02:01:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5226000640 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f040789a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5226a00640 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 0x559f04076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:02.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5227400640 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f0406b9a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5218009c60 tx=0x7f52180107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=818a390f2d2c4df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5213400640 1 -- 172.21.15.102:0/1606540819 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521801c480 con 0x559f0406b5c0 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5213400640 1 -- 172.21.15.102:0/1606540819 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521801cb00 con 0x559f0406b5c0 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.232+0000 7f5213400640 1 -- 172.21.15.102:0/1606540819 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218011bb0 con 0x559f0406b5c0 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.233+0000 7f521f400640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f520c0a2d80 0x7f520c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.233+0000 7f521f400640 1 -- 172.21.15.102:0/489554945 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f52200f0110 con 0x7f52281007b0 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.233+0000 7f5226a00640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f520c0a2d80 0x7f520c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.233+0000 7f52303cf740 1 -- 172.21.15.102:0/1606540819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 msgr2=0x559f0406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.233+0000 7f52303cf740 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f0406b9a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5218009c60 tx=0x7f52180107a0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f52303cf740 1 -- 172.21.15.102:0/1606540819 shutdown_connections 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f52303cf740 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f040789a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f52303cf740 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 0x559f04076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f52303cf740 1 --2- 172.21.15.102:0/1606540819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f0406b9a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f52303cf740 1 -- 172.21.15.102:0/1606540819 >> 172.21.15.102:0/1606540819 conn(0x559f040678e0 msgr2=0x559f04067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f52303cf740 1 -- 172.21.15.102:0/1606540819 shutdown_connections 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f5226a00640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f520c0a2d80 0x7f520c0a5240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x559f0406cb80 tx=0x7f5220000f40 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f52303cf740 1 -- 172.21.15.102:0/1606540819 wait complete. 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f52303cf740 1 -- start start 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.234+0000 7f52303cf740 1 Processor -- start 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f52303cf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f04073570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f52303cf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f04073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f52303cf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f04073ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f52303cf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559f040793a0 con 0x559f0406b5c0 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f52303cf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559f03f8bec0 con 0x559f0406bee0 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f52303cf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f04079520 con 0x559f040765b0 2024-04-25T02:01:02.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f52303cf740 10 client.0 _ll_register_callbacks cb 0x559f03fc83f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f5226000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f04073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f5226000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f04073ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40616/0 (socket says 172.21.15.102:40616) 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f5226000640 1 -- 172.21.15.102:0/1248746825 learned_addr learned my addr 172.21.15.102:0/1248746825 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f5227400640 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f04073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.235+0000 7f5227400640 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f04073ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5226000640 1 -- 172.21.15.102:0/1248746825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 msgr2=0x559f04073ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5226000640 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f04073ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5226000640 1 -- 172.21.15.102:0/1248746825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 msgr2=0x559f04073570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5226000640 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f04073570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5226000640 1 -- 172.21.15.102:0/1248746825 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f03dae7d0 con 0x559f0406bee0 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5227400640 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f04073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5226000640 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f04073ab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5214012730 tx=0x7f5214012bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83bbf100c0a93e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5207e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:02.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5207e00640 1 -- 172.21.15.102:0/1248746825 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214004050 con 0x559f0406bee0 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f5207e00640 1 -- 172.21.15.102:0/1248746825 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f52140041f0 con 0x559f0406bee0 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.236+0000 7f52303cf740 1 -- 172.21.15.102:0/1248746825 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f521801cb00 con 0x559f0406bee0 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f5207e00640 1 -- 172.21.15.102:0/1248746825 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f5214004430 con 0x559f0406bee0 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f52303cf740 10 client.26786 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f52303cf740 1 -- 172.21.15.102:0/1248746825 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559f04085700 con 0x559f0406bee0 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f5207e00640 1 -- 172.21.15.102:0/1248746825 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f521801cb00 con 0x559f0406bee0 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f5207e00640 1 -- 172.21.15.102:0/1248746825 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f5214003bf0 con 0x559f0406bee0 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f52303cf740 10 client.26786 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f52303cf740 10 client.26786 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f52303cf740 20 client.26786 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f52303cf740 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x559f04086bc0 0x559f04088fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f52303cf740 4 client.26786 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:02.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.237+0000 7f52303cf740 1 -- 172.21.15.102:0/1248746825 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f03fdda20 con 0x559f04086bc0 2024-04-25T02:01:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.238+0000 7f5227400640 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x559f04086bc0 0x559f04088fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.238+0000 7f5227400640 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x559f04086bc0 0x559f04088fd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2de9012a00f38688 server_cookie=ca45c0d3a9d9579b in_seq=0 out_seq=0 2024-04-25T02:01:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.238+0000 7f5207e00640 10 client.26786 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T02:01:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f5207e00640 1 -- 172.21.15.102:0/1248746825 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f03fdda20 con 0x559f04086bc0 2024-04-25T02:01:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f5207e00640 10 client.26786 handle_command_reply: tid=0 2024-04-25T02:01:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f52303cf740 1 client.26786 shutdown 2024-04-25T02:01:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f52303cf740 1 -- 172.21.15.102:0/1248746825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 msgr2=0x559f04073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f52303cf740 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f04073ab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5214012730 tx=0x7f5214012bf0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f52303cf740 1 -- 172.21.15.102:0/1248746825 shutdown_connections 2024-04-25T02:01:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f52303cf740 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f04073ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f52303cf740 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x559f04086bc0 0x559f04088fd0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f52303cf740 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f04073ab0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f52303cf740 1 --2- 172.21.15.102:0/1248746825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f04073570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f5207e00640 0 client.26786 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T02:01:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.239+0000 7f52303cf740 1 -- 172.21.15.102:0/1248746825 >> 172.21.15.102:0/1248746825 conn(0x559f040678e0 msgr2=0x559f04066b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.240+0000 7f52303cf740 1 -- 172.21.15.102:0/1248746825 shutdown_connections 2024-04-25T02:01:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.240+0000 7f52303cf740 1 -- 172.21.15.102:0/1248746825 wait complete. 2024-04-25T02:01:02.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.240+0000 7f52303cf740 20 client.26786 trim_cache size 0 max 16384 2024-04-25T02:01:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f52303cf740 1 Processor -- start 2024-04-25T02:01:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f52303cf740 1 -- start start 2024-04-25T02:01:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f52303cf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f04082290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f52303cf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 0x559f0408e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f52303cf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f0408ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f52303cf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f52281b6350 con 0x559f0406bee0 2024-04-25T02:01:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f52303cf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f52281b3bc0 con 0x559f0406b5c0 2024-04-25T02:01:02.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f52303cf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f52281b1a10 con 0x559f040765b0 2024-04-25T02:01:02.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f5227400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f04082290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f5226000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 0x559f0408e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f5227400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f04082290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40628/0 (socket says 172.21.15.102:40628) 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f5227400640 1 -- 172.21.15.102:0/1000030026 learned_addr learned my addr 172.21.15.102:0/1000030026 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.256+0000 7f5227400640 1 --2- 172.21.15.102:0/1000030026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f0408ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.257+0000 7f5227400640 1 -- 172.21.15.102:0/1000030026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 msgr2=0x559f0408ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.257+0000 7f5227400640 1 --2- 172.21.15.102:0/1000030026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f0408ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.257+0000 7f5227400640 1 -- 172.21.15.102:0/1000030026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 msgr2=0x559f0408e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.257+0000 7f5227400640 1 --2- 172.21.15.102:0/1000030026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 0x559f0408e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.257+0000 7f5227400640 1 -- 172.21.15.102:0/1000030026 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f04071ac0 con 0x559f0406b5c0 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.257+0000 7f5227400640 1 --2- 172.21.15.102:0/1000030026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f04082290 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5218009860 tx=0x7f5218013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b832f95054530ce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.257+0000 7f5212a00640 1 -- 172.21.15.102:0/1000030026 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218024040 con 0x559f0406b5c0 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.257+0000 7f5212a00640 1 -- 172.21.15.102:0/1000030026 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5218025040 con 0x559f0406b5c0 2024-04-25T02:01:02.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.257+0000 7f5212a00640 1 -- 172.21.15.102:0/1000030026 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521801c480 con 0x559f0406b5c0 2024-04-25T02:01:02.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.258+0000 7f52303cf740 1 -- 172.21.15.102:0/1000030026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 msgr2=0x559f04082290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.258+0000 7f52303cf740 1 --2- 172.21.15.102:0/1000030026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f04082290 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5218009860 tx=0x7f5218013040 comp rx=0 tx=0).stop 2024-04-25T02:01:02.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.258+0000 7f52303cf740 1 -- 172.21.15.102:0/1000030026 shutdown_connections 2024-04-25T02:01:02.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.258+0000 7f52303cf740 1 --2- 172.21.15.102:0/1000030026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f0408ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.258+0000 7f52303cf740 1 --2- 172.21.15.102:0/1000030026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406bee0 0x559f0408e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.258+0000 7f52303cf740 1 --2- 172.21.15.102:0/1000030026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406b5c0 0x559f04082290 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.258+0000 7f52303cf740 1 -- 172.21.15.102:0/1000030026 >> 172.21.15.102:0/1000030026 conn(0x559f040678e0 msgr2=0x559f04067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.258+0000 7f52303cf740 1 -- 172.21.15.102:0/1000030026 shutdown_connections 2024-04-25T02:01:02.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.258+0000 7f52303cf740 1 -- 172.21.15.102:0/1000030026 wait complete. 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.259+0000 7f52303cf740 1 -- start start 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.259+0000 7f52303cf740 1 Processor -- start 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.259+0000 7f52303cf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f0407ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.259+0000 7f52303cf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f0407a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.259+0000 7f52303cf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f0407a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.259+0000 7f52303cf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f52281b2080 con 0x559f0406b5c0 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.259+0000 7f52303cf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f52281b34f0 con 0x559f0406bee0 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.259+0000 7f52303cf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f52281b2750 con 0x559f040765b0 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.259+0000 7f52303cf740 10 client.0 _ll_register_callbacks cb 0x7f5214009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5226000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f0407a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5226000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f0407a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40644/0 (socket says 172.21.15.102:40644) 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5226000640 1 -- 172.21.15.102:0/3578475748 learned_addr learned my addr 172.21.15.102:0/3578475748 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5227400640 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f0407ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5227400640 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f0407a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5226000640 1 -- 172.21.15.102:0/3578475748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 msgr2=0x559f0407a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5226000640 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f0407a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5226000640 1 -- 172.21.15.102:0/3578475748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 msgr2=0x559f0407ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5226000640 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f0407ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5226000640 1 -- 172.21.15.102:0/3578475748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f040ec150 con 0x559f0406bee0 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5227400640 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f0407ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5226000640 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f0407a050 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x559f04074b40 tx=0x7f521400ff50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f18f3d7b9fe1c21d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.260+0000 7f5207e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f5207e00640 1 -- 172.21.15.102:0/3578475748 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214029070 con 0x559f0406bee0 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f52303cf740 1 -- 172.21.15.102:0/3578475748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5218025040 con 0x559f0406bee0 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f5207e00640 1 -- 172.21.15.102:0/3578475748 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f5214005040 con 0x559f0406bee0 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f5207e00640 1 -- 172.21.15.102:0/3578475748 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f521403c050 con 0x559f0406bee0 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f52303cf740 10 client.26792 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f52303cf740 1 -- 172.21.15.102:0/3578475748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559f040ed320 con 0x559f0406bee0 2024-04-25T02:01:02.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f5207e00640 1 -- 172.21.15.102:0/3578475748 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5218025040 con 0x559f0406bee0 2024-04-25T02:01:02.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f5207e00640 1 -- 172.21.15.102:0/3578475748 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f52140045d0 con 0x559f0406bee0 2024-04-25T02:01:02.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f52303cf740 10 client.26792 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:02.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f52303cf740 10 client.26792 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T02:01:02.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.261+0000 7f52303cf740 20 client.26792 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:02.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.262+0000 7f52303cf740 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x559f040edac0 0x559f040efee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.262+0000 7f52303cf740 4 client.26792 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:01:02.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.262+0000 7f52303cf740 1 -- 172.21.15.102:0/3578475748 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f03fd7d70 con 0x559f040edac0 2024-04-25T02:01:02.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.262+0000 7f5227400640 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x559f040edac0 0x559f040efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.262+0000 7f5207e00640 10 client.26792 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T02:01:02.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.262+0000 7f5227400640 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x559f040edac0 0x559f040efee0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19502b3d574d53e5 server_cookie=764bff0315780174 in_seq=0 out_seq=0 2024-04-25T02:01:02.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.263+0000 7f5207e00640 1 -- 172.21.15.102:0/3578475748 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f03fd7d70 con 0x559f040edac0 2024-04-25T02:01:02.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.263+0000 7f5207e00640 10 client.26792 handle_command_reply: tid=0 2024-04-25T02:01:02.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.263+0000 7f52303cf740 1 client.26792 shutdown 2024-04-25T02:01:02.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.263+0000 7f52303cf740 1 -- 172.21.15.102:0/3578475748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 msgr2=0x559f0407a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.263+0000 7f52303cf740 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f0407a050 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x559f04074b40 tx=0x7f521400ff50 comp rx=0 tx=0).stop 2024-04-25T02:01:02.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.265+0000 7f52303cf740 1 -- 172.21.15.102:0/3578475748 shutdown_connections 2024-04-25T02:01:02.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.265+0000 7f52303cf740 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f040765b0 0x559f0407a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.265+0000 7f52303cf740 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x559f040edac0 0x559f040efee0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.265+0000 7f52303cf740 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f0406bee0 0x559f0407a050 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.265+0000 7f52303cf740 1 --2- 172.21.15.102:0/3578475748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f0406b5c0 0x559f0407ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.265+0000 7f52303cf740 1 -- 172.21.15.102:0/3578475748 >> 172.21.15.102:0/3578475748 conn(0x559f040678e0 msgr2=0x559f04073530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.265+0000 7f5207e00640 0 client.26792 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T02:01:02.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.267+0000 7f52303cf740 1 -- 172.21.15.102:0/3578475748 shutdown_connections 2024-04-25T02:01:02.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.267+0000 7f52303cf740 1 -- 172.21.15.102:0/3578475748 wait complete. 2024-04-25T02:01:02.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.268+0000 7f52303cf740 20 client.26792 trim_cache size 0 max 16384 2024-04-25T02:01:02.273 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:01:02.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.268+0000 7f522d600640 1 -- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f520c0a2d80 msgr2=0x7f520c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.268+0000 7f522d600640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f520c0a2d80 0x7f520c0a5240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x559f0406cb80 tx=0x7f5220000f40 comp rx=0 tx=0).stop 2024-04-25T02:01:02.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.268+0000 7f522d600640 1 -- 172.21.15.102:0/489554945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52281007b0 msgr2=0x7f522819e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.268+0000 7f522d600640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52281007b0 0x7f522819e610 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f522002add0 tx=0x7f52200305d0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.269+0000 7f522d600640 1 -- 172.21.15.102:0/489554945 shutdown_connections 2024-04-25T02:01:02.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.269+0000 7f522d600640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f520c0a2d80 0x7f520c0a5240 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.269+0000 7f522d600640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5228105ad0 0x7f52281a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.269+0000 7f522d600640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52281010d0 0x7f522819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.269+0000 7f522d600640 1 --2- 172.21.15.102:0/489554945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f52281007b0 0x7f522819e610 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.269+0000 7f522d600640 1 -- 172.21.15.102:0/489554945 >> 172.21.15.102:0/489554945 conn(0x7f5228077740 msgr2=0x7f5228103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.269+0000 7f522d600640 1 -- 172.21.15.102:0/489554945 shutdown_connections 2024-04-25T02:01:02.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.269+0000 7f522d600640 1 -- 172.21.15.102:0/489554945 wait complete. 2024-04-25T02:01:02.295 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:02 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3332672334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:01:02.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T02:01:02.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:02 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3332672334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:01:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.491+0000 7ff19aa00640 1 -- 172.21.15.102:0/1908649973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194108680 msgr2=0x7ff194114aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.491+0000 7ff19aa00640 1 --2- 172.21.15.102:0/1908649973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194108680 0x7ff194114aa0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009ca0 tx=0x7ff18c02a950 comp rx=0 tx=0).stop 2024-04-25T02:01:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.491+0000 7ff19aa00640 1 -- 172.21.15.102:0/1908649973 shutdown_connections 2024-04-25T02:01:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.491+0000 7ff19aa00640 1 --2- 172.21.15.102:0/1908649973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194108680 0x7ff194114aa0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.491+0000 7ff19aa00640 1 --2- 172.21.15.102:0/1908649973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff194107ce0 0x7ff194108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.491+0000 7ff19aa00640 1 --2- 172.21.15.102:0/1908649973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff19410dd00 0x7ff19410e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.491+0000 7ff19aa00640 1 -- 172.21.15.102:0/1908649973 >> 172.21.15.102:0/1908649973 conn(0x7ff194077560 msgr2=0x7ff194106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.491+0000 7ff19aa00640 1 -- 172.21.15.102:0/1908649973 shutdown_connections 2024-04-25T02:01:02.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.491+0000 7ff19aa00640 1 -- 172.21.15.102:0/1908649973 wait complete. 2024-04-25T02:01:02.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.492+0000 7ff19aa00640 1 Processor -- start 2024-04-25T02:01:02.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.492+0000 7ff19aa00640 1 -- start start 2024-04-25T02:01:02.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.492+0000 7ff19aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194107ce0 0x7ff19419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.492+0000 7ff19aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff194108680 0x7ff19419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff19aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff19410dd00 0x7ff1941a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff19aa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff194118030 con 0x7ff19410dd00 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff19aa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff194117eb0 con 0x7ff194108680 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff19aa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1941181b0 con 0x7ff194107ce0 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff199600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194107ce0 0x7ff19419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff199600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194107ce0 0x7ff19419e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52566/0 (socket says 172.21.15.102:52566) 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff199600640 1 -- 172.21.15.102:0/2655335407 learned_addr learned my addr 172.21.15.102:0/2655335407 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff198c00640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff194108680 0x7ff19419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff19a000640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff19410dd00 0x7ff1941a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff199600640 1 -- 172.21.15.102:0/2655335407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff194108680 msgr2=0x7ff19419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff199600640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff194108680 0x7ff19419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff199600640 1 -- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff19410dd00 msgr2=0x7ff1941a2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff199600640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff19410dd00 0x7ff1941a2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff199600640 1 -- 172.21.15.102:0/2655335407 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1941a3400 con 0x7ff194107ce0 2024-04-25T02:01:02.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff198c00640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff194108680 0x7ff19419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff19a000640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff19410dd00 0x7ff1941a2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.493+0000 7ff199600640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194107ce0 0x7ff19419e410 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff18800e9f0 tx=0x7ff18800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.494+0000 7ff192000640 1 -- 172.21.15.102:0/2655335407 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18800cd80 con 0x7ff194107ce0 2024-04-25T02:01:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.494+0000 7ff19aa00640 1 -- 172.21.15.102:0/2655335407 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1941a36f0 con 0x7ff194107ce0 2024-04-25T02:01:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.494+0000 7ff192000640 1 -- 172.21.15.102:0/2655335407 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff188005500 con 0x7ff194107ce0 2024-04-25T02:01:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.494+0000 7ff192000640 1 -- 172.21.15.102:0/2655335407 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188010690 con 0x7ff194107ce0 2024-04-25T02:01:02.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.494+0000 7ff19aa00640 1 -- 172.21.15.102:0/2655335407 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1941aaf80 con 0x7ff194107ce0 2024-04-25T02:01:02.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.497+0000 7ff19ee39740 1 Processor -- start 2024-04-25T02:01:02.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.497+0000 7ff19ee39740 1 -- start start 2024-04-25T02:01:02.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff192000640 1 -- 172.21.15.102:0/2655335407 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff1880046c0 con 0x7ff194107ce0 2024-04-25T02:01:02.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 0x5624b7fcfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fd00c0 0x5624b7fda250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fda790 0x5624b7fdcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff19ee39740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5624b7f344e0 con 0x5624b7fcf7a0 2024-04-25T02:01:02.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff19ee39740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5624b7edca70 con 0x5624b7fd00c0 2024-04-25T02:01:02.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff19ee39740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5624b7f304a0 con 0x5624b7fda790 2024-04-25T02:01:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff199600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fd00c0 0x5624b7fda250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff198c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fda790 0x5624b7fdcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff199600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fd00c0 0x5624b7fda250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40670/0 (socket says 172.21.15.102:40670) 2024-04-25T02:01:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff199600640 1 -- 172.21.15.102:0/598068505 learned_addr learned my addr 172.21.15.102:0/598068505 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.498+0000 7ff19a000640 1 --2- 172.21.15.102:0/598068505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 0x5624b7fcfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff199600640 1 -- 172.21.15.102:0/598068505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fda790 msgr2=0x5624b7fdcb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff199600640 1 --2- 172.21.15.102:0/598068505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fda790 0x5624b7fdcb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff199600640 1 -- 172.21.15.102:0/598068505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 msgr2=0x5624b7fcfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff199600640 1 --2- 172.21.15.102:0/598068505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 0x5624b7fcfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff199600640 1 -- 172.21.15.102:0/598068505 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624b7c483f0 con 0x5624b7fd00c0 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff198c00640 1 --2- 172.21.15.102:0/598068505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fda790 0x5624b7fdcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff199600640 1 --2- 172.21.15.102:0/598068505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fd00c0 0x5624b7fda250 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff1880ccfe0 tx=0x7ff1880d5a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6abe83d2d888c9bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff192000640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff17c0a2d00 0x7ff17c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff186600640 1 -- 172.21.15.102:0/598068505 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1880de070 con 0x5624b7fd00c0 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.499+0000 7ff186600640 1 -- 172.21.15.102:0/598068505 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff188013190 con 0x5624b7fd00c0 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff186600640 1 -- 172.21.15.102:0/598068505 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188015e70 con 0x5624b7fd00c0 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff192000640 1 -- 172.21.15.102:0/2655335407 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff1880cf8f0 con 0x7ff194107ce0 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff19a000640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff17c0a2d00 0x7ff17c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff19ee39740 1 -- 172.21.15.102:0/598068505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fd00c0 msgr2=0x5624b7fda250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff19ee39740 1 --2- 172.21.15.102:0/598068505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fd00c0 0x5624b7fda250 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff1880ccfe0 tx=0x7ff1880d5a90 comp rx=0 tx=0).stop 2024-04-25T02:01:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff19a000640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff17c0a2d00 0x7ff17c0a51c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff18c005fa0 tx=0x7ff18c004a40 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff19ee39740 1 -- 172.21.15.102:0/598068505 shutdown_connections 2024-04-25T02:01:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff19ee39740 1 --2- 172.21.15.102:0/598068505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fda790 0x5624b7fdcb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff19ee39740 1 --2- 172.21.15.102:0/598068505 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fd00c0 0x5624b7fda250 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff19ee39740 1 --2- 172.21.15.102:0/598068505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 0x5624b7fcfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.500+0000 7ff19ee39740 1 -- 172.21.15.102:0/598068505 >> 172.21.15.102:0/598068505 conn(0x5624b7fcbac0 msgr2=0x5624b7fcbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.501+0000 7ff19ee39740 1 -- 172.21.15.102:0/598068505 shutdown_connections 2024-04-25T02:01:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.501+0000 7ff19ee39740 1 -- 172.21.15.102:0/598068505 wait complete. 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.501+0000 7ff19ee39740 1 -- start start 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.501+0000 7ff19ee39740 1 Processor -- start 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fcf7a0 0x5624b7fd7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fda790 0x5624b7fd8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff19ee39740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5624b7fdd580 con 0x5624b7fda790 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff19ee39740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5624b7eefec0 con 0x5624b7fcf7a0 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff19ee39740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5624b7fdd700 con 0x5624b7fd00c0 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff19ee39740 10 client.0 _ll_register_callbacks cb 0x5624b7f2c3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff199600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fcf7a0 0x5624b7fd7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff198c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff199600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fcf7a0 0x5624b7fd7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:40678/0 (socket says 172.21.15.102:40678) 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.502+0000 7ff199600640 1 -- 172.21.15.102:0/222411224 learned_addr learned my addr 172.21.15.102:0/222411224 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff19a000640 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fda790 0x5624b7fd8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff199600640 1 -- 172.21.15.102:0/222411224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 msgr2=0x5624b7fd7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff199600640 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff199600640 1 -- 172.21.15.102:0/222411224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fda790 msgr2=0x5624b7fd8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff199600640 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fda790 0x5624b7fd8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff199600640 1 -- 172.21.15.102:0/222411224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624b7d127d0 con 0x5624b7fcf7a0 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff198c00640 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff199600640 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fcf7a0 0x5624b7fd7710 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff18809bdd0 tx=0x7ff18801eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=697b32010834abbb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff17aa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff17aa00640 1 -- 172.21.15.102:0/222411224 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18801e9b0 con 0x5624b7fcf7a0 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff17aa00640 1 -- 172.21.15.102:0/222411224 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff18809a140 con 0x5624b7fcf7a0 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff19ee39740 1 -- 172.21.15.102:0/222411224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5624b7f41a20 con 0x5624b7fcf7a0 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff17aa00640 1 -- 172.21.15.102:0/222411224 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff18809e020 con 0x5624b7fcf7a0 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff19ee39740 10 client.26804 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.503+0000 7ff19ee39740 1 -- 172.21.15.102:0/222411224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5624b7fe9800 con 0x5624b7fcf7a0 2024-04-25T02:01:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.504+0000 7ff17aa00640 1 -- 172.21.15.102:0/222411224 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5624b7f41a20 con 0x5624b7fcf7a0 2024-04-25T02:01:02.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:02 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3332672334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:01:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.505+0000 7ff17aa00640 1 -- 172.21.15.102:0/222411224 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7ff1880984b0 con 0x5624b7fcf7a0 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.505+0000 7ff19ee39740 10 client.26804 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.505+0000 7ff19ee39740 10 client.26804 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.505+0000 7ff19ee39740 20 client.26804 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.505+0000 7ff19ee39740 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624b7feab50 0x5624b7fecf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.505+0000 7ff19ee39740 4 client.26804 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.505+0000 7ff19ee39740 1 -- 172.21.15.102:0/222411224 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624b7f3bd70 con 0x5624b7feab50 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.505+0000 7ff198c00640 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624b7feab50 0x5624b7fecf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.506+0000 7ff198c00640 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624b7feab50 0x5624b7fecf40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61965793a447777b server_cookie=d61c66296c80309c in_seq=0 out_seq=0 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.506+0000 7ff17aa00640 10 client.26804 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.506+0000 7ff17aa00640 1 -- 172.21.15.102:0/222411224 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5624b7f3bd70 con 0x5624b7feab50 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.506+0000 7ff17aa00640 10 client.26804 handle_command_reply: tid=0 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 client.26804 shutdown 2024-04-25T02:01:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 -- 172.21.15.102:0/222411224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fcf7a0 msgr2=0x5624b7fd7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fcf7a0 0x5624b7fd7710 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff18809bdd0 tx=0x7ff18801eb50 comp rx=0 tx=0).stop 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 -- 172.21.15.102:0/222411224 shutdown_connections 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624b7feab50 0x5624b7fecf40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fda790 0x5624b7fd8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 --2- 172.21.15.102:0/222411224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fcf7a0 0x5624b7fd7710 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff17aa00640 0 client.26804 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 -- 172.21.15.102:0/222411224 >> 172.21.15.102:0/222411224 conn(0x5624b7fcbac0 msgr2=0x5624b7fd17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 -- 172.21.15.102:0/222411224 shutdown_connections 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 1 -- 172.21.15.102:0/222411224 wait complete. 2024-04-25T02:01:02.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.507+0000 7ff19ee39740 20 client.26804 trim_cache size 0 max 16384 2024-04-25T02:01:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.524+0000 7ff19ee39740 1 Processor -- start 2024-04-25T02:01:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.524+0000 7ff19ee39740 1 -- start start 2024-04-25T02:01:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.524+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.524+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.524+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 0x5624b7fca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.524+0000 7ff19ee39740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff1941b60d0 con 0x5624b7fd00c0 2024-04-25T02:01:02.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.524+0000 7ff19ee39740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff1941b3940 con 0x5624b7fda790 2024-04-25T02:01:02.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.524+0000 7ff19ee39740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1941b1790 con 0x5624b7fcf7a0 2024-04-25T02:01:02.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.524+0000 7ff199600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff198c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff199600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52600/0 (socket says 172.21.15.102:52600) 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff199600640 1 -- 172.21.15.102:0/3897052288 learned_addr learned my addr 172.21.15.102:0/3897052288 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff19a000640 1 --2- 172.21.15.102:0/3897052288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 0x5624b7fca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff199600640 1 -- 172.21.15.102:0/3897052288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 msgr2=0x5624b7fca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff199600640 1 --2- 172.21.15.102:0/3897052288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 0x5624b7fca9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff199600640 1 -- 172.21.15.102:0/3897052288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fd00c0 msgr2=0x5624b7fd7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff199600640 1 --2- 172.21.15.102:0/3897052288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff199600640 1 -- 172.21.15.102:0/3897052288 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624b7fcfef0 con 0x5624b7fcf7a0 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff198c00640 1 --2- 172.21.15.102:0/3897052288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff199600640 1 --2- 172.21.15.102:0/3897052288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe6440 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff18809bb40 tx=0x7ff188011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b603a6b87b6162b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.525+0000 7ff185c00640 1 -- 172.21.15.102:0/3897052288 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1880de070 con 0x5624b7fcf7a0 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.526+0000 7ff185c00640 1 -- 172.21.15.102:0/3897052288 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff18809c540 con 0x5624b7fcf7a0 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.526+0000 7ff185c00640 1 -- 172.21.15.102:0/3897052288 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18801ec00 con 0x5624b7fcf7a0 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.526+0000 7ff19ee39740 1 -- 172.21.15.102:0/3897052288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fcf7a0 msgr2=0x5624b7fe6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.526+0000 7ff19ee39740 1 --2- 172.21.15.102:0/3897052288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe6440 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff18809bb40 tx=0x7ff188011000 comp rx=0 tx=0).stop 2024-04-25T02:01:02.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.526+0000 7ff19ee39740 1 -- 172.21.15.102:0/3897052288 shutdown_connections 2024-04-25T02:01:02.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.526+0000 7ff19ee39740 1 --2- 172.21.15.102:0/3897052288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 0x5624b7fca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.526+0000 7ff19ee39740 1 --2- 172.21.15.102:0/3897052288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fd00c0 0x5624b7fd7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.526+0000 7ff19ee39740 1 --2- 172.21.15.102:0/3897052288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe6440 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.526+0000 7ff19ee39740 1 -- 172.21.15.102:0/3897052288 >> 172.21.15.102:0/3897052288 conn(0x5624b7fcbac0 msgr2=0x5624b7fca2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.527+0000 7ff19ee39740 1 -- 172.21.15.102:0/3897052288 shutdown_connections 2024-04-25T02:01:02.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.527+0000 7ff19ee39740 1 -- 172.21.15.102:0/3897052288 wait complete. 2024-04-25T02:01:02.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.527+0000 7ff19ee39740 1 -- start start 2024-04-25T02:01:02.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.527+0000 7ff19ee39740 1 Processor -- start 2024-04-25T02:01:02.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 0x5624b7fddf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19ee39740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 0x5624b7fde450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19ee39740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff1941b1e00 con 0x5624b7fcf7a0 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19ee39740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff1941b3270 con 0x5624b7fda790 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19ee39740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1941b24d0 con 0x5624b7fd00c0 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19ee39740 10 client.0 _ll_register_callbacks cb 0x7ff19406ae40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff199600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff198c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 0x5624b7fddf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 0x5624b7fde450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff199600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44388/0 (socket says 172.21.15.102:44388) 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff199600640 1 -- 172.21.15.102:0/1997657048 learned_addr learned my addr 172.21.15.102:0/1997657048 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19a000640 1 -- 172.21.15.102:0/1997657048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 msgr2=0x5624b7fddf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19a000640 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 0x5624b7fddf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19a000640 1 -- 172.21.15.102:0/1997657048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 msgr2=0x5624b7fe2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19a000640 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.528+0000 7ff19a000640 1 -- 172.21.15.102:0/1997657048 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624b7fde990 con 0x5624b7fda790 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.529+0000 7ff19a000640 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 0x5624b7fde450 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff18c006010 tx=0x7ff18c02a950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8902945d6cad139c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.529+0000 7ff17aa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.529+0000 7ff17aa00640 1 -- 172.21.15.102:0/1997657048 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c034040 con 0x5624b7fda790 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.529+0000 7ff17aa00640 1 -- 172.21.15.102:0/1997657048 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff18c041030 con 0x5624b7fda790 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.529+0000 7ff19ee39740 1 -- 172.21.15.102:0/1997657048 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff18809c540 con 0x5624b7fda790 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.529+0000 7ff17aa00640 1 -- 172.21.15.102:0/1997657048 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff18c02fed0 con 0x5624b7fda790 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.529+0000 7ff19ee39740 10 client.26810 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.529+0000 7ff19ee39740 1 -- 172.21.15.102:0/1997657048 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5624b8050be0 con 0x5624b7fda790 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.530+0000 7ff17aa00640 1 -- 172.21.15.102:0/1997657048 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff18809c540 con 0x5624b7fda790 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.530+0000 7ff17aa00640 1 -- 172.21.15.102:0/1997657048 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7ff18c030910 con 0x5624b7fda790 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.530+0000 7ff19ee39740 10 client.26810 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:02.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.530+0000 7ff19ee39740 10 client.26810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.530+0000 7ff19ee39740 20 client.26810 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.530+0000 7ff19ee39740 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624b8051330 0x5624b8053750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.530+0000 7ff19ee39740 4 client.26810 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.530+0000 7ff19ee39740 1 -- 172.21.15.102:0/1997657048 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5624b7eb96e0 con 0x5624b8051330 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.530+0000 7ff199600640 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624b8051330 0x5624b8053750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff199600640 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624b8051330 0x5624b8053750 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fd1c53fc292e5241 server_cookie=efbfe5c2800eb9c8 in_seq=0 out_seq=0 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff17aa00640 10 client.26810 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff17aa00640 1 -- 172.21.15.102:0/1997657048 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5624b7eb96e0 con 0x5624b8051330 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff17aa00640 10 client.26810 handle_command_reply: tid=0 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 client.26810 shutdown 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 -- 172.21.15.102:0/1997657048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 msgr2=0x5624b7fde450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 0x5624b7fde450 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff18c006010 tx=0x7ff18c02a950 comp rx=0 tx=0).stop 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 -- 172.21.15.102:0/1997657048 shutdown_connections 2024-04-25T02:01:02.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x5624b8051330 0x5624b8053750 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5624b7fda790 0x5624b7fde450 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624b7fd00c0 0x5624b7fddf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 --2- 172.21.15.102:0/1997657048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624b7fcf7a0 0x5624b7fe2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff17aa00640 0 client.26810 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T02:01:02.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 -- 172.21.15.102:0/1997657048 >> 172.21.15.102:0/1997657048 conn(0x5624b7fcbac0 msgr2=0x5624b7fd1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 -- 172.21.15.102:0/1997657048 shutdown_connections 2024-04-25T02:01:02.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.531+0000 7ff19ee39740 1 -- 172.21.15.102:0/1997657048 wait complete. 2024-04-25T02:01:02.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.532+0000 7ff19ee39740 20 client.26810 trim_cache size 0 max 16384 2024-04-25T02:01:02.544 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:01:02.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.532+0000 7ff19aa00640 1 -- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff17c0a2d00 msgr2=0x7ff17c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.532+0000 7ff19aa00640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff17c0a2d00 0x7ff17c0a51c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff18c005fa0 tx=0x7ff18c004a40 comp rx=0 tx=0).stop 2024-04-25T02:01:02.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.532+0000 7ff19aa00640 1 -- 172.21.15.102:0/2655335407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194107ce0 msgr2=0x7ff19419e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.532+0000 7ff19aa00640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194107ce0 0x7ff19419e410 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff18800e9f0 tx=0x7ff18800edf0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.532+0000 7ff19aa00640 1 -- 172.21.15.102:0/2655335407 shutdown_connections 2024-04-25T02:01:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.533+0000 7ff19aa00640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff17c0a2d00 0x7ff17c0a51c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.533+0000 7ff19aa00640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff19410dd00 0x7ff1941a2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.533+0000 7ff19aa00640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff194108680 0x7ff19419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.533+0000 7ff19aa00640 1 --2- 172.21.15.102:0/2655335407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff194107ce0 0x7ff19419e410 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.533+0000 7ff19aa00640 1 -- 172.21.15.102:0/2655335407 >> 172.21.15.102:0/2655335407 conn(0x7ff194077560 msgr2=0x7ff194078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.533+0000 7ff19aa00640 1 -- 172.21.15.102:0/2655335407 shutdown_connections 2024-04-25T02:01:02.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.533+0000 7ff19aa00640 1 -- 172.21.15.102:0/2655335407 wait complete. 2024-04-25T02:01:02.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T02:01:02.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.772+0000 7f831cc00640 1 -- 172.21.15.102:0/300944675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8318103920 msgr2=0x7f8318103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.772+0000 7f831cc00640 1 --2- 172.21.15.102:0/300944675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8318103920 0x7f8318103da0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8304009900 tx=0x7f830402a3f0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.772+0000 7f831cc00640 1 -- 172.21.15.102:0/300944675 shutdown_connections 2024-04-25T02:01:02.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.772+0000 7f831cc00640 1 --2- 172.21.15.102:0/300944675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83181042e0 0x7f83181107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.772+0000 7f831cc00640 1 --2- 172.21.15.102:0/300944675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8318103920 0x7f8318103da0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.772+0000 7f831cc00640 1 --2- 172.21.15.102:0/300944675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8318109930 0x7f8318109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.772+0000 7f831cc00640 1 -- 172.21.15.102:0/300944675 >> 172.21.15.102:0/300944675 conn(0x7f83180ff3b0 msgr2=0x7f83181017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.772+0000 7f831cc00640 1 -- 172.21.15.102:0/300944675 shutdown_connections 2024-04-25T02:01:02.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.772+0000 7f831cc00640 1 -- 172.21.15.102:0/300944675 wait complete. 2024-04-25T02:01:02.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.773+0000 7f831cc00640 1 Processor -- start 2024-04-25T02:01:02.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.773+0000 7f831cc00640 1 -- start start 2024-04-25T02:01:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.773+0000 7f831cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8318103920 0x7f831819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f831cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83181042e0 0x7f831819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f831cc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8318109930 0x7f831819eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f831cc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8318113d90 con 0x7f83181042e0 2024-04-25T02:01:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f831cc00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8318113c10 con 0x7f8318109930 2024-04-25T02:01:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f831cc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8318113f10 con 0x7f8318103920 2024-04-25T02:01:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8315e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8318103920 0x7f831819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8315e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8318103920 0x7f831819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60912/0 (socket says 172.21.15.102:60912) 2024-04-25T02:01:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8315e00640 1 -- 172.21.15.102:0/3350506869 learned_addr learned my addr 172.21.15.102:0/3350506869 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8316800640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8318109930 0x7f831819eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8315400640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83181042e0 0x7f831819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8316800640 1 -- 172.21.15.102:0/3350506869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8318103920 msgr2=0x7f831819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8316800640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8318103920 0x7f831819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8316800640 1 -- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83181042e0 msgr2=0x7f831819a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8316800640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83181042e0 0x7f831819a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8316800640 1 -- 172.21.15.102:0/3350506869 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831819f1f0 con 0x7f8318109930 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8315e00640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8318103920 0x7f831819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.774+0000 7f8315400640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83181042e0 0x7f831819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.775+0000 7f8316800640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8318109930 0x7f831819eac0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8310009fd0 tx=0x7f831000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.775+0000 7f830ea00640 1 -- 172.21.15.102:0/3350506869 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831000ce20 con 0x7f8318109930 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.775+0000 7f830ea00640 1 -- 172.21.15.102:0/3350506869 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8310005020 con 0x7f8318109930 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.775+0000 7f830ea00640 1 -- 172.21.15.102:0/3350506869 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310010720 con 0x7f8318109930 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.775+0000 7f831cc00640 1 -- 172.21.15.102:0/3350506869 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831819f4e0 con 0x7f8318109930 2024-04-25T02:01:02.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.775+0000 7f831cc00640 1 -- 172.21.15.102:0/3350506869 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83181a6d70 con 0x7f8318109930 2024-04-25T02:01:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.779+0000 7f830ea00640 1 -- 172.21.15.102:0/3350506869 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8310005540 con 0x7f8318109930 2024-04-25T02:01:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.779+0000 7f831fa72740 1 Processor -- start 2024-04-25T02:01:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.779+0000 7f831fa72740 1 -- start start 2024-04-25T02:01:02.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f831fa72740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968655c0 0x562e968659a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f831fa72740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96870070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f831fa72740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968705b0 0x562e968729a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f831fa72740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x562e967ca4e0 con 0x562e968705b0 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f831fa72740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562e96772a70 con 0x562e96865ee0 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f831fa72740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562e967c64a0 con 0x562e968655c0 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f8315e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96870070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f8315400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968655c0 0x562e968659a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f8315e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96870070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53002/0 (socket says 172.21.15.102:53002) 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f8315400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968705b0 0x562e968729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f8315e00640 1 -- 172.21.15.102:0/2758904492 learned_addr learned my addr 172.21.15.102:0/2758904492 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.780+0000 7f830ea00640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f82fc0a2d80 0x7f82fc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8315e00640 1 -- 172.21.15.102:0/2758904492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968655c0 msgr2=0x562e968659a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8315e00640 1 --2- 172.21.15.102:0/2758904492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968655c0 0x562e968659a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8315e00640 1 -- 172.21.15.102:0/2758904492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968705b0 msgr2=0x562e968729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8315e00640 1 --2- 172.21.15.102:0/2758904492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968705b0 0x562e968729a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8315e00640 1 -- 172.21.15.102:0/2758904492 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e964de3f0 con 0x562e96865ee0 2024-04-25T02:01:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f830ea00640 1 -- 172.21.15.102:0/3350506869 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f83100cfac0 con 0x7f8318109930 2024-04-25T02:01:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8316800640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f82fc0a2d80 0x7f82fc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8315400640 1 --2- 172.21.15.102:0/2758904492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968705b0 0x562e968729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8315e00640 1 --2- 172.21.15.102:0/2758904492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96870070 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f83080115c0 tx=0x7f8308011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a78692cc4a078bff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8303000640 1 -- 172.21.15.102:0/2758904492 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830800f660 con 0x562e96865ee0 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8303000640 1 -- 172.21.15.102:0/2758904492 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f830800fce0 con 0x562e96865ee0 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.781+0000 7f8303000640 1 -- 172.21.15.102:0/2758904492 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83080183b0 con 0x562e96865ee0 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.782+0000 7f831fa72740 1 -- 172.21.15.102:0/2758904492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 msgr2=0x562e96870070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.782+0000 7f8316800640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f82fc0a2d80 0x7f82fc0a5240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f83100cd240 tx=0x7f8310005b20 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.782+0000 7f831fa72740 1 --2- 172.21.15.102:0/2758904492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96870070 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f83080115c0 tx=0x7f8308011a80 comp rx=0 tx=0).stop 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.783+0000 7f831fa72740 1 -- 172.21.15.102:0/2758904492 shutdown_connections 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.783+0000 7f831fa72740 1 --2- 172.21.15.102:0/2758904492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968705b0 0x562e968729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.783+0000 7f831fa72740 1 --2- 172.21.15.102:0/2758904492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96870070 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.783+0000 7f831fa72740 1 --2- 172.21.15.102:0/2758904492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968655c0 0x562e968659a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.783+0000 7f831fa72740 1 -- 172.21.15.102:0/2758904492 >> 172.21.15.102:0/2758904492 conn(0x562e968618e0 msgr2=0x562e96861cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.783+0000 7f831fa72740 1 -- 172.21.15.102:0/2758904492 shutdown_connections 2024-04-25T02:01:02.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.783+0000 7f831fa72740 1 -- 172.21.15.102:0/2758904492 wait complete. 2024-04-25T02:01:02.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.783+0000 7f831fa72740 1 -- start start 2024-04-25T02:01:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.784+0000 7f831fa72740 1 Processor -- start 2024-04-25T02:01:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.784+0000 7f831fa72740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e9686d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.784+0000 7f831fa72740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e9686dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.784+0000 7f831fa72740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.784+0000 7f831fa72740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x562e968733a0 con 0x562e968655c0 2024-04-25T02:01:02.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.784+0000 7f831fa72740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x562e96785ec0 con 0x562e96865ee0 2024-04-25T02:01:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.784+0000 7f831fa72740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562e96873520 con 0x562e968705b0 2024-04-25T02:01:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.784+0000 7f831fa72740 10 client.0 _ll_register_callbacks cb 0x562e967c23f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e9686d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60928/0 (socket says 172.21.15.102:60928) 2024-04-25T02:01:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e9686dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 -- 172.21.15.102:0/534518652 learned_addr learned my addr 172.21.15.102:0/534518652 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 -- 172.21.15.102:0/534518652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 msgr2=0x562e9686dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e9686dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 -- 172.21.15.102:0/534518652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 msgr2=0x562e9686d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e9686d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 -- 172.21.15.102:0/534518652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e965a87d0 con 0x562e968705b0 2024-04-25T02:01:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f8315400640 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686dff0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8304005ab0 tx=0x7f8304005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0161b5c1666adfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f82f7400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f82f7400640 1 -- 172.21.15.102:0/534518652 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304035040 con 0x562e968705b0 2024-04-25T02:01:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.785+0000 7f82f7400640 1 -- 172.21.15.102:0/534518652 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8304003b70 con 0x562e968705b0 2024-04-25T02:01:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f831fa72740 1 -- 172.21.15.102:0/534518652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f830800fce0 con 0x562e968705b0 2024-04-25T02:01:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f82f7400640 1 -- 172.21.15.102:0/534518652 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8304053050 con 0x562e968705b0 2024-04-25T02:01:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f831fa72740 10 client.26788 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:02.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f831fa72740 1 -- 172.21.15.102:0/534518652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562e9687f660 con 0x562e968705b0 2024-04-25T02:01:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f82f7400640 1 -- 172.21.15.102:0/534518652 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f830800fce0 con 0x562e968705b0 2024-04-25T02:01:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f82f7400640 1 -- 172.21.15.102:0/534518652 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f830404a070 con 0x562e968705b0 2024-04-25T02:01:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f831fa72740 10 client.26788 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f831fa72740 10 client.26788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:02.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f831fa72740 20 client.26788 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:02.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f831fa72740 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562e96880ac0 0x562e96882eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f831fa72740 4 client.26788 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:02.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.786+0000 7f831fa72740 1 -- 172.21.15.102:0/534518652 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e967d7a20 con 0x562e96880ac0 2024-04-25T02:01:02.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.787+0000 7f8315e00640 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562e96880ac0 0x562e96882eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.787+0000 7f8315e00640 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562e96880ac0 0x562e96882eb0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=187928da92edf96a server_cookie=2badb44a0378ab7e in_seq=0 out_seq=0 2024-04-25T02:01:02.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.787+0000 7f82f7400640 10 client.26788 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:02.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f82f7400640 1 -- 172.21.15.102:0/534518652 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e967d7a20 con 0x562e96880ac0 2024-04-25T02:01:02.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f82f7400640 10 client.26788 handle_command_reply: tid=0 2024-04-25T02:01:02.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f831fa72740 1 client.26788 shutdown 2024-04-25T02:01:02.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f831fa72740 1 -- 172.21.15.102:0/534518652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 msgr2=0x562e9686dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f831fa72740 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686dff0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8304005ab0 tx=0x7f8304005a20 comp rx=0 tx=0).stop 2024-04-25T02:01:02.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f831fa72740 1 -- 172.21.15.102:0/534518652 shutdown_connections 2024-04-25T02:01:02.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f831fa72740 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562e96880ac0 0x562e96882eb0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f831fa72740 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686dff0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f82f7400640 0 client.26788 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:02.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f831fa72740 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e9686dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.788+0000 7f831fa72740 1 --2- 172.21.15.102:0/534518652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e9686d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.789+0000 7f831fa72740 1 -- 172.21.15.102:0/534518652 >> 172.21.15.102:0/534518652 conn(0x562e968618e0 msgr2=0x562e968676e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.789+0000 7f831fa72740 1 -- 172.21.15.102:0/534518652 shutdown_connections 2024-04-25T02:01:02.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.789+0000 7f831fa72740 1 -- 172.21.15.102:0/534518652 wait complete. 2024-04-25T02:01:02.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.789+0000 7f831fa72740 20 client.26788 trim_cache size 0 max 16384 2024-04-25T02:01:02.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.804+0000 7f831fa72740 1 Processor -- start 2024-04-25T02:01:02.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.804+0000 7f831fa72740 1 -- start start 2024-04-25T02:01:02.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f831fa72740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e9687c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f831fa72740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96867d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f831fa72740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e96888780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f831fa72740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f83181b1fa0 con 0x562e968655c0 2024-04-25T02:01:02.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f831fa72740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f83181af810 con 0x562e96865ee0 2024-04-25T02:01:02.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f831fa72740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f83181ad660 con 0x562e968705b0 2024-04-25T02:01:02.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f8315400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e9687c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f8315e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96867d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f8315400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e9687c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33178/0 (socket says 172.21.15.102:33178) 2024-04-25T02:01:02.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f8315400640 1 -- 172.21.15.102:0/3094192489 learned_addr learned my addr 172.21.15.102:0/3094192489 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.805+0000 7f8315400640 1 --2- 172.21.15.102:0/3094192489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e96888780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.806+0000 7f8315e00640 1 -- 172.21.15.102:0/3094192489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 msgr2=0x562e96888780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.806+0000 7f8315e00640 1 --2- 172.21.15.102:0/3094192489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e96888780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.806+0000 7f8315e00640 1 -- 172.21.15.102:0/3094192489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 msgr2=0x562e9687c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.806+0000 7f8315e00640 1 --2- 172.21.15.102:0/3094192489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e9687c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.806+0000 7f8315e00640 1 -- 172.21.15.102:0/3094192489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e968729a0 con 0x562e96865ee0 2024-04-25T02:01:02.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.806+0000 7f8315e00640 1 --2- 172.21.15.102:0/3094192489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96867d60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f830801b040 tx=0x7f83080198c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79f03986ace27b59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.806+0000 7f8302600640 1 -- 172.21.15.102:0/3094192489 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8308019490 con 0x562e96865ee0 2024-04-25T02:01:02.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.806+0000 7f8302600640 1 -- 172.21.15.102:0/3094192489 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8308019e10 con 0x562e96865ee0 2024-04-25T02:01:02.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.806+0000 7f8302600640 1 -- 172.21.15.102:0/3094192489 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830800f8c0 con 0x562e96865ee0 2024-04-25T02:01:02.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.807+0000 7f831fa72740 1 -- 172.21.15.102:0/3094192489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 msgr2=0x562e96867d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.807+0000 7f831fa72740 1 --2- 172.21.15.102:0/3094192489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96867d60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f830801b040 tx=0x7f83080198c0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.807+0000 7f831fa72740 1 -- 172.21.15.102:0/3094192489 shutdown_connections 2024-04-25T02:01:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.807+0000 7f831fa72740 1 --2- 172.21.15.102:0/3094192489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e96888780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.807+0000 7f831fa72740 1 --2- 172.21.15.102:0/3094192489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96867d60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.807+0000 7f831fa72740 1 --2- 172.21.15.102:0/3094192489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e9687c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.807+0000 7f831fa72740 1 -- 172.21.15.102:0/3094192489 >> 172.21.15.102:0/3094192489 conn(0x562e968618e0 msgr2=0x562e9686f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.807+0000 7f831fa72740 1 -- 172.21.15.102:0/3094192489 shutdown_connections 2024-04-25T02:01:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.807+0000 7f831fa72740 1 -- 172.21.15.102:0/3094192489 wait complete. 2024-04-25T02:01:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.808+0000 7f831fa72740 1 -- start start 2024-04-25T02:01:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.808+0000 7f831fa72740 1 Processor -- start 2024-04-25T02:01:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.808+0000 7f831fa72740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e96878b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f831fa72740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96869f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f831fa72740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f831fa72740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f83181adcd0 con 0x562e968655c0 2024-04-25T02:01:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f831fa72740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f83181af140 con 0x562e96865ee0 2024-04-25T02:01:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f831fa72740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f83181ae3a0 con 0x562e968705b0 2024-04-25T02:01:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f831fa72740 10 client.0 _ll_register_callbacks cb 0x7f8308005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96869f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96869f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53026/0 (socket says 172.21.15.102:53026) 2024-04-25T02:01:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315e00640 1 -- 172.21.15.102:0/2732969860 learned_addr learned my addr 172.21.15.102:0/2732969860 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315400640 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e96878b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315400640 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315e00640 1 -- 172.21.15.102:0/2732969860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 msgr2=0x562e9686a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315e00640 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315e00640 1 -- 172.21.15.102:0/2732969860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 msgr2=0x562e96878b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315e00640 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e96878b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315e00640 1 -- 172.21.15.102:0/2732969860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e9686ace0 con 0x562e96865ee0 2024-04-25T02:01:02.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.809+0000 7f8315400640 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:02.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.810+0000 7f8315e00640 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96869f70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8308011ac0 tx=0x7f8308005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a2603e0a2f9aeed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:02.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.810+0000 7f82f7400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:02.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.810+0000 7f82f7400640 1 -- 172.21.15.102:0/2732969860 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8308005130 con 0x562e96865ee0 2024-04-25T02:01:02.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.810+0000 7f831fa72740 1 -- 172.21.15.102:0/2732969860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8308019e10 con 0x562e96865ee0 2024-04-25T02:01:02.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.810+0000 7f82f7400640 1 -- 172.21.15.102:0/2732969860 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8308019490 con 0x562e96865ee0 2024-04-25T02:01:02.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.810+0000 7f82f7400640 1 -- 172.21.15.102:0/2732969860 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8308019690 con 0x562e96865ee0 2024-04-25T02:01:02.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.810+0000 7f831fa72740 10 client.26828 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:02.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.810+0000 7f831fa72740 1 -- 172.21.15.102:0/2732969860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562e968e7020 con 0x562e96865ee0 2024-04-25T02:01:02.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.810+0000 7f82f7400640 1 -- 172.21.15.102:0/2732969860 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8308019e10 con 0x562e96865ee0 2024-04-25T02:01:02.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.811+0000 7f82f7400640 1 -- 172.21.15.102:0/2732969860 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f83080035f0 con 0x562e96865ee0 2024-04-25T02:01:02.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.811+0000 7f831fa72740 10 client.26828 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:02.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.811+0000 7f831fa72740 10 client.26828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:02.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.811+0000 7f831fa72740 20 client.26828 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.811+0000 7f831fa72740 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562e968e74d0 0x562e968e98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.811+0000 7f831fa72740 4 client.26828 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:01:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.811+0000 7f831fa72740 1 -- 172.21.15.102:0/2732969860 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e967d1d70 con 0x562e968e74d0 2024-04-25T02:01:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.811+0000 7f8315400640 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562e968e74d0 0x562e968e98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.812+0000 7f8315400640 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562e968e74d0 0x562e968e98f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab685dd7f4336573 server_cookie=9dfe2c19bc0c7b1f in_seq=0 out_seq=0 2024-04-25T02:01:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.812+0000 7f82f7400640 10 client.26828 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f82f7400640 1 -- 172.21.15.102:0/2732969860 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e967d1d70 con 0x562e968e74d0 2024-04-25T02:01:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f82f7400640 10 client.26828 handle_command_reply: tid=0 2024-04-25T02:01:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f831fa72740 1 client.26828 shutdown 2024-04-25T02:01:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f831fa72740 1 -- 172.21.15.102:0/2732969860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 msgr2=0x562e96869f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f831fa72740 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96869f70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8308011ac0 tx=0x7f8308005750 comp rx=0 tx=0).stop 2024-04-25T02:01:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f831fa72740 1 -- 172.21.15.102:0/2732969860 shutdown_connections 2024-04-25T02:01:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f831fa72740 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x562e968e74d0 0x562e968e98f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f831fa72740 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562e968705b0 0x562e9686a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f82f7400640 0 client.26828 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f831fa72740 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562e96865ee0 0x562e96869f70 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f831fa72740 1 --2- 172.21.15.102:0/2732969860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562e968655c0 0x562e96878b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.813+0000 7f831fa72740 1 -- 172.21.15.102:0/2732969860 >> 172.21.15.102:0/2732969860 conn(0x562e968618e0 msgr2=0x562e9686d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.814+0000 7f831fa72740 1 -- 172.21.15.102:0/2732969860 shutdown_connections 2024-04-25T02:01:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.814+0000 7f831fa72740 1 -- 172.21.15.102:0/2732969860 wait complete. 2024-04-25T02:01:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.814+0000 7f831fa72740 20 client.26828 trim_cache size 0 max 16384 2024-04-25T02:01:02.832 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:01:02.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.814+0000 7f831cc00640 1 -- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f82fc0a2d80 msgr2=0x7f82fc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.814+0000 7f831cc00640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f82fc0a2d80 0x7f82fc0a5240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f83100cd240 tx=0x7f8310005b20 comp rx=0 tx=0).stop 2024-04-25T02:01:02.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.814+0000 7f831cc00640 1 -- 172.21.15.102:0/3350506869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8318109930 msgr2=0x7f831819eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:02.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.814+0000 7f831cc00640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8318109930 0x7f831819eac0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f8310009fd0 tx=0x7f831000ef10 comp rx=0 tx=0).stop 2024-04-25T02:01:02.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.815+0000 7f831cc00640 1 -- 172.21.15.102:0/3350506869 shutdown_connections 2024-04-25T02:01:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.815+0000 7f831cc00640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f82fc0a2d80 0x7f82fc0a5240 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.815+0000 7f831cc00640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8318109930 0x7f831819eac0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.815+0000 7f831cc00640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83181042e0 0x7f831819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.815+0000 7f831cc00640 1 --2- 172.21.15.102:0/3350506869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8318103920 0x7f831819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.815+0000 7f831cc00640 1 -- 172.21.15.102:0/3350506869 >> 172.21.15.102:0/3350506869 conn(0x7f83180ff3b0 msgr2=0x7f83180ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.815+0000 7f831cc00640 1 -- 172.21.15.102:0/3350506869 shutdown_connections 2024-04-25T02:01:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:02.815+0000 7f831cc00640 1 -- 172.21.15.102:0/3350506869 wait complete. 2024-04-25T02:01:03.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 310.0 s, 1143.1 tps, lat 80.630 ms stddev 170.076 2024-04-25T02:01:03.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:03 smithi102 ceph-mon[27796]: pgmap v176: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.5 MiB/s wr, 839 op/s 2024-04-25T02:01:03.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:03 smithi110 ceph-mon[32848]: pgmap v176: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.5 MiB/s wr, 839 op/s 2024-04-25T02:01:03.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:03 smithi193 ceph-mon[32981]: pgmap v176: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 9.5 MiB/s wr, 839 op/s 2024-04-25T02:01:03.840 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T02:01:03.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:01:03.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T02:01:04.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.070+0000 7fda1f200640 1 -- 172.21.15.102:0/1630407467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 msgr2=0x7fda20105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.070+0000 7fda1f200640 1 --2- 172.21.15.102:0/1630407467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 0x7fda20105e20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fda10009900 tx=0x7fda1002a4a0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.070+0000 7fda1f200640 1 -- 172.21.15.102:0/1630407467 shutdown_connections 2024-04-25T02:01:04.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.070+0000 7fda1f200640 1 --2- 172.21.15.102:0/1630407467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 0x7fda20105e20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.070+0000 7fda1f200640 1 --2- 172.21.15.102:0/1630407467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda20100eb0 0x7fda20105370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.070+0000 7fda1f200640 1 --2- 172.21.15.102:0/1630407467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda20100590 0x7fda20100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.070+0000 7fda1f200640 1 -- 172.21.15.102:0/1630407467 >> 172.21.15.102:0/1630407467 conn(0x7fda200fbfc0 msgr2=0x7fda200fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.070+0000 7fda1f200640 1 -- 172.21.15.102:0/1630407467 shutdown_connections 2024-04-25T02:01:04.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.070+0000 7fda1f200640 1 -- 172.21.15.102:0/1630407467 wait complete. 2024-04-25T02:01:04.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.072+0000 7fda1f200640 1 Processor -- start 2024-04-25T02:01:04.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.072+0000 7fda1f200640 1 -- start start 2024-04-25T02:01:04.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.072+0000 7fda1f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda20100590 0x7fda2019e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.072+0000 7fda1f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda20100eb0 0x7fda2019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.072+0000 7fda1f200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 0x7fda201a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.072+0000 7fda1f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fda20118020 con 0x7fda20100590 2024-04-25T02:01:04.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.072+0000 7fda1f200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fda20117ea0 con 0x7fda20105990 2024-04-25T02:01:04.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.072+0000 7fda1f200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fda201181a0 con 0x7fda20100eb0 2024-04-25T02:01:04.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 0x7fda201a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 0x7fda201a2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53042/0 (socket says 172.21.15.102:53042) 2024-04-25T02:01:04.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1e800640 1 -- 172.21.15.102:0/1264553941 learned_addr learned my addr 172.21.15.102:0/1264553941 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1d400640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda20100eb0 0x7fda2019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1de00640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda20100590 0x7fda2019e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1e800640 1 -- 172.21.15.102:0/1264553941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda20100eb0 msgr2=0x7fda2019ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1e800640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda20100eb0 0x7fda2019ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1e800640 1 -- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda20100590 msgr2=0x7fda2019e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1e800640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda20100590 0x7fda2019e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.073+0000 7fda1e800640 1 -- 172.21.15.102:0/1264553941 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda201a34f0 con 0x7fda20105990 2024-04-25T02:01:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.074+0000 7fda1de00640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda20100590 0x7fda2019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.074+0000 7fda1e800640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 0x7fda201a2dc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fda10006010 tx=0x7fda10032d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.074+0000 7fda16a00640 1 -- 172.21.15.102:0/1264553941 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1003f030 con 0x7fda20105990 2024-04-25T02:01:04.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.074+0000 7fda16a00640 1 -- 172.21.15.102:0/1264553941 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda10003ce0 con 0x7fda20105990 2024-04-25T02:01:04.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.074+0000 7fda16a00640 1 -- 172.21.15.102:0/1264553941 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10035070 con 0x7fda20105990 2024-04-25T02:01:04.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.075+0000 7fda1f200640 1 -- 172.21.15.102:0/1264553941 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda201a3780 con 0x7fda20105990 2024-04-25T02:01:04.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.075+0000 7fda1f200640 1 -- 172.21.15.102:0/1264553941 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda201ab070 con 0x7fda20105990 2024-04-25T02:01:04.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.080+0000 7fda16a00640 1 -- 172.21.15.102:0/1264553941 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fda10042030 con 0x7fda20105990 2024-04-25T02:01:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.083+0000 7fda16a00640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fda040a2ed0 0x7fda040a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.083+0000 7fda27bde740 1 Processor -- start 2024-04-25T02:01:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.083+0000 7fda16a00640 1 -- 172.21.15.102:0/1264553941 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fda100ef950 con 0x7fda20105990 2024-04-25T02:01:04.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.083+0000 7fda1de00640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fda040a2ed0 0x7fda040a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.083+0000 7fda27bde740 1 -- start start 2024-04-25T02:01:04.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda27bde740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fdf20 0x560fe23fe300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda27bde740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe24089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda27bde740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe240b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda27bde740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560fe235cf40 con 0x560fe2408f10 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda27bde740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560fe2362190 con 0x560fe23fe840 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda27bde740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560fe230aa10 con 0x560fe23fdf20 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda1d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe24089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda1d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe24089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53052/0 (socket says 172.21.15.102:53052) 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda1d400640 1 -- 172.21.15.102:0/2248269387 learned_addr learned my addr 172.21.15.102:0/2248269387 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.084+0000 7fda1e800640 1 --2- 172.21.15.102:0/2248269387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe240b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.085+0000 7fda1d400640 1 -- 172.21.15.102:0/2248269387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fdf20 msgr2=0x560fe23fe300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.085+0000 7fda1d400640 1 --2- 172.21.15.102:0/2248269387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fdf20 0x560fe23fe300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.085+0000 7fda1d400640 1 -- 172.21.15.102:0/2248269387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 msgr2=0x560fe240b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.085+0000 7fda1d400640 1 --2- 172.21.15.102:0/2248269387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe240b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.085+0000 7fda1d400640 1 -- 172.21.15.102:0/2248269387 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe213e7e0 con 0x560fe23fe840 2024-04-25T02:01:04.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.085+0000 7fda1e800640 1 --2- 172.21.15.102:0/2248269387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe240b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.085+0000 7fda1d400640 1 --2- 172.21.15.102:0/2248269387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe24089d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fda0800abe0 tx=0x7fda0800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=317e7b2f8799a04c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.085+0000 7fda1de00640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fda040a2ed0 0x7fda040a5390 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fda18007f00 tx=0x7fda1800c690 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.086+0000 7fda0f400640 1 -- 172.21.15.102:0/2248269387 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0801d070 con 0x560fe23fe840 2024-04-25T02:01:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.086+0000 7fda0f400640 1 -- 172.21.15.102:0/2248269387 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda08018b20 con 0x560fe23fe840 2024-04-25T02:01:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.086+0000 7fda0f400640 1 -- 172.21.15.102:0/2248269387 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08010b80 con 0x560fe23fe840 2024-04-25T02:01:04.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.086+0000 7fda27bde740 1 -- 172.21.15.102:0/2248269387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 msgr2=0x560fe24089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.086+0000 7fda27bde740 1 --2- 172.21.15.102:0/2248269387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe24089d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fda0800abe0 tx=0x7fda0800f790 comp rx=0 tx=0).stop 2024-04-25T02:01:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.087+0000 7fda27bde740 1 -- 172.21.15.102:0/2248269387 shutdown_connections 2024-04-25T02:01:04.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.087+0000 7fda27bde740 1 --2- 172.21.15.102:0/2248269387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe240b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.087+0000 7fda27bde740 1 --2- 172.21.15.102:0/2248269387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe24089d0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.087+0000 7fda27bde740 1 --2- 172.21.15.102:0/2248269387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fdf20 0x560fe23fe300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.087+0000 7fda27bde740 1 -- 172.21.15.102:0/2248269387 >> 172.21.15.102:0/2248269387 conn(0x560fe23fa240 msgr2=0x560fe23fa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.087+0000 7fda27bde740 1 -- 172.21.15.102:0/2248269387 shutdown_connections 2024-04-25T02:01:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.087+0000 7fda27bde740 1 -- 172.21.15.102:0/2248269387 wait complete. 2024-04-25T02:01:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.088+0000 7fda27bde740 1 -- start start 2024-04-25T02:01:04.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.088+0000 7fda27bde740 1 Processor -- start 2024-04-25T02:01:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda27bde740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2405f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda27bde740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2406450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda27bde740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe2406990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda27bde740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560fe231de70 con 0x560fe2408f10 2024-04-25T02:01:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda27bde740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560fe235e150 con 0x560fe23fdf20 2024-04-25T02:01:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda27bde740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560fe240bda0 con 0x560fe23fe840 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda27bde740 10 client.0 _ll_register_callbacks cb 0x560fe23591f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda1de00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2405f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda1d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2406450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.089+0000 7fda1de00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2405f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53066/0 (socket says 172.21.15.102:53066) 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1de00640 1 -- 172.21.15.102:0/25196248 learned_addr learned my addr 172.21.15.102:0/25196248 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2406450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:60976/0 (socket says 172.21.15.102:60976) 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1e800640 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe2406990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1d400640 1 -- 172.21.15.102:0/25196248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 msgr2=0x560fe2405f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1d400640 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2405f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1d400640 1 -- 172.21.15.102:0/25196248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 msgr2=0x560fe2406990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1d400640 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe2406990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1d400640 1 -- 172.21.15.102:0/25196248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fe219ef70 con 0x560fe23fe840 2024-04-25T02:01:04.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1e800640 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe2406990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda1d400640 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2406450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fda0800a2a0 tx=0x7fda08010b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bb0dc5881e36653 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.090+0000 7fda03e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.091+0000 7fda03e00640 1 -- 172.21.15.102:0/25196248 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda0801d050 con 0x560fe23fe840 2024-04-25T02:01:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.091+0000 7fda27bde740 1 -- 172.21.15.102:0/25196248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda08018b20 con 0x560fe23fe840 2024-04-25T02:01:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.091+0000 7fda03e00640 1 -- 172.21.15.102:0/25196248 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fda08024450 con 0x560fe23fe840 2024-04-25T02:01:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.091+0000 7fda03e00640 1 -- 172.21.15.102:0/25196248 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fda0800a7d0 con 0x560fe23fe840 2024-04-25T02:01:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.091+0000 7fda27bde740 10 client.26800 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:04.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.091+0000 7fda27bde740 1 -- 172.21.15.102:0/25196248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560fe2036170 con 0x560fe23fe840 2024-04-25T02:01:04.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.092+0000 7fda03e00640 1 -- 172.21.15.102:0/25196248 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda08018b20 con 0x560fe23fe840 2024-04-25T02:01:04.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.092+0000 7fda03e00640 1 -- 172.21.15.102:0/25196248 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fda08005020 con 0x560fe23fe840 2024-04-25T02:01:04.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.092+0000 7fda27bde740 10 client.26800 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:04.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.092+0000 7fda27bde740 10 client.26800 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:04.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.092+0000 7fda27bde740 20 client.26800 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.092+0000 7fda27bde740 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560fe2419150 0x560fe241b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.092+0000 7fda27bde740 4 client.26800 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.092+0000 7fda27bde740 1 -- 172.21.15.102:0/25196248 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fe236f6d0 con 0x560fe2419150 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.093+0000 7fda1de00640 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560fe2419150 0x560fe241b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.093+0000 7fda1de00640 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560fe2419150 0x560fe241b540 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc160d15c7152a8b server_cookie=828f0ed0f1f14b3b in_seq=0 out_seq=0 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.094+0000 7fda03e00640 10 client.26800 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.094+0000 7fda03e00640 1 -- 172.21.15.102:0/25196248 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560fe236f6d0 con 0x560fe2419150 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.094+0000 7fda03e00640 10 client.26800 handle_command_reply: tid=0 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.094+0000 7fda27bde740 1 client.26800 shutdown 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 -- 172.21.15.102:0/25196248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 msgr2=0x560fe2406450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2406450 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fda0800a2a0 tx=0x7fda08010b30 comp rx=0 tx=0).stop 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 -- 172.21.15.102:0/25196248 shutdown_connections 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560fe2419150 0x560fe241b540 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe2406990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2406450 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 --2- 172.21.15.102:0/25196248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2405f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 -- 172.21.15.102:0/25196248 >> 172.21.15.102:0/25196248 conn(0x560fe23fa240 msgr2=0x560fe2400040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda03e00640 0 client.26800 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:04.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 -- 172.21.15.102:0/25196248 shutdown_connections 2024-04-25T02:01:04.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 1 -- 172.21.15.102:0/25196248 wait complete. 2024-04-25T02:01:04.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.095+0000 7fda27bde740 20 client.26800 trim_cache size 0 max 16384 2024-04-25T02:01:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.119+0000 7fda27bde740 1 Processor -- start 2024-04-25T02:01:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.119+0000 7fda27bde740 1 -- start start 2024-04-25T02:01:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.119+0000 7fda27bde740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2414ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.119+0000 7fda27bde740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2400790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.119+0000 7fda27bde740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe24065e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.119+0000 7fda27bde740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fda201b60b0 con 0x560fe2408f10 2024-04-25T02:01:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.119+0000 7fda27bde740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fda201b3920 con 0x560fe23fdf20 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.119+0000 7fda27bde740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fda201b1770 con 0x560fe23fe840 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1de00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2414ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe24065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1de00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2414ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53068/0 (socket says 172.21.15.102:53068) 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1de00640 1 -- 172.21.15.102:0/665976235 learned_addr learned my addr 172.21.15.102:0/665976235 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1d400640 1 --2- 172.21.15.102:0/665976235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2400790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1de00640 1 -- 172.21.15.102:0/665976235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 msgr2=0x560fe2400790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1de00640 1 --2- 172.21.15.102:0/665976235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2400790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1de00640 1 -- 172.21.15.102:0/665976235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 msgr2=0x560fe24065e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1de00640 1 --2- 172.21.15.102:0/665976235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe24065e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1de00640 1 -- 172.21.15.102:0/665976235 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fe23f9b30 con 0x560fe23fdf20 2024-04-25T02:01:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1e800640 1 --2- 172.21.15.102:0/665976235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe24065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.120+0000 7fda1de00640 1 --2- 172.21.15.102:0/665976235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2414ae0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fda180077a0 tx=0x7fda18011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99b18e6a4936137b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.121+0000 7fda0ea00640 1 -- 172.21.15.102:0/665976235 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda18003a10 con 0x560fe23fdf20 2024-04-25T02:01:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.121+0000 7fda0ea00640 1 -- 172.21.15.102:0/665976235 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda18003bb0 con 0x560fe23fdf20 2024-04-25T02:01:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.121+0000 7fda0ea00640 1 -- 172.21.15.102:0/665976235 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1801f6b0 con 0x560fe23fdf20 2024-04-25T02:01:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 -- 172.21.15.102:0/665976235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 msgr2=0x560fe2414ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 --2- 172.21.15.102:0/665976235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2414ae0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fda180077a0 tx=0x7fda18011000 comp rx=0 tx=0).stop 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 -- 172.21.15.102:0/665976235 shutdown_connections 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 --2- 172.21.15.102:0/665976235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe2408f10 0x560fe24065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 --2- 172.21.15.102:0/665976235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe23fe840 0x560fe2400790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 --2- 172.21.15.102:0/665976235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fdf20 0x560fe2414ae0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 -- 172.21.15.102:0/665976235 >> 172.21.15.102:0/665976235 conn(0x560fe23fa240 msgr2=0x560fe2405ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 -- 172.21.15.102:0/665976235 shutdown_connections 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 -- 172.21.15.102:0/665976235 wait complete. 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 -- start start 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.122+0000 7fda27bde740 1 Processor -- start 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda27bde740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe23fdf20 0x560fe240cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda27bde740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe240d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda27bde740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe2408f10 0x560fe240d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda27bde740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fda201b1de0 con 0x560fe23fdf20 2024-04-25T02:01:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda27bde740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fda201b3250 con 0x560fe23fe840 2024-04-25T02:01:04.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda27bde740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fda201b24b0 con 0x560fe2408f10 2024-04-25T02:01:04.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda27bde740 10 client.0 _ll_register_callbacks cb 0x7fda0801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:04.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda1d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe240d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda1de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe23fdf20 0x560fe240cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda1e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe2408f10 0x560fe240d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda1d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe240d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53070/0 (socket says 172.21.15.102:53070) 2024-04-25T02:01:04.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.123+0000 7fda1d400640 1 -- 172.21.15.102:0/1859301741 learned_addr learned my addr 172.21.15.102:0/1859301741 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda1e800640 1 -- 172.21.15.102:0/1859301741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 msgr2=0x560fe240d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda1e800640 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe240d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda1e800640 1 -- 172.21.15.102:0/1859301741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe23fdf20 msgr2=0x560fe240cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda1e800640 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe23fdf20 0x560fe240cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda1e800640 1 -- 172.21.15.102:0/1859301741 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fe247e710 con 0x560fe2408f10 2024-04-25T02:01:04.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda1e800640 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe2408f10 0x560fe240d780 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fda100ed070 tx=0x7fda10009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdbe8e015bc2a0c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda03e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda03e00640 1 -- 172.21.15.102:0/1859301741 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda100be040 con 0x560fe2408f10 2024-04-25T02:01:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda27bde740 1 -- 172.21.15.102:0/1859301741 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda18003bb0 con 0x560fe2408f10 2024-04-25T02:01:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda03e00640 1 -- 172.21.15.102:0/1859301741 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fda100ed2f0 con 0x560fe2408f10 2024-04-25T02:01:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda03e00640 1 -- 172.21.15.102:0/1859301741 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fda10049040 con 0x560fe2408f10 2024-04-25T02:01:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda27bde740 10 client.26806 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:04.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.124+0000 7fda27bde740 1 -- 172.21.15.102:0/1859301741 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560fe247fa50 con 0x560fe2408f10 2024-04-25T02:01:04.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.125+0000 7fda03e00640 1 -- 172.21.15.102:0/1859301741 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda18003bb0 con 0x560fe2408f10 2024-04-25T02:01:04.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.125+0000 7fda03e00640 1 -- 172.21.15.102:0/1859301741 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fda100c4620 con 0x560fe2408f10 2024-04-25T02:01:04.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.125+0000 7fda27bde740 10 client.26806 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:04.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.125+0000 7fda27bde740 10 client.26806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:04.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.125+0000 7fda27bde740 20 client.26806 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:04.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.125+0000 7fda27bde740 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560fe24801d0 0x560fe24825f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.125+0000 7fda27bde740 4 client.26806 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T02:01:04.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.125+0000 7fda27bde740 1 -- 172.21.15.102:0/1859301741 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560fe2369a20 con 0x560fe24801d0 2024-04-25T02:01:04.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.125+0000 7fda1de00640 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560fe24801d0 0x560fe24825f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.126+0000 7fda1de00640 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560fe24801d0 0x560fe24825f0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63c041d03744f6e server_cookie=56ca5a82a2d21c7 in_seq=0 out_seq=0 2024-04-25T02:01:04.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.126+0000 7fda03e00640 10 client.26806 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:04.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda03e00640 1 -- 172.21.15.102:0/1859301741 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560fe2369a20 con 0x560fe24801d0 2024-04-25T02:01:04.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda03e00640 10 client.26806 handle_command_reply: tid=0 2024-04-25T02:01:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 client.26806 shutdown 2024-04-25T02:01:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 -- 172.21.15.102:0/1859301741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe2408f10 msgr2=0x560fe240d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe2408f10 0x560fe240d780 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fda100ed070 tx=0x7fda10009260 comp rx=0 tx=0).stop 2024-04-25T02:01:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 -- 172.21.15.102:0/1859301741 shutdown_connections 2024-04-25T02:01:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560fe24801d0 0x560fe24825f0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fe2408f10 0x560fe240d780 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560fe23fe840 0x560fe240d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 --2- 172.21.15.102:0/1859301741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560fe23fdf20 0x560fe240cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda03e00640 0 client.26806 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:04.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 -- 172.21.15.102:0/1859301741 >> 172.21.15.102:0/1859301741 conn(0x560fe23fa240 msgr2=0x560fe23f93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 -- 172.21.15.102:0/1859301741 shutdown_connections 2024-04-25T02:01:04.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.127+0000 7fda27bde740 1 -- 172.21.15.102:0/1859301741 wait complete. 2024-04-25T02:01:04.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda27bde740 20 client.26806 trim_cache size 0 max 16384 2024-04-25T02:01:04.143 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:01:04.143 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T02:01:04.143 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "6f9a9466-9b1b-426b-b8f0-8bb2a80b509a", 2024-04-25T02:01:04.143 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T02:01:04.143 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:01:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 -- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fda040a2ed0 msgr2=0x7fda040a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fda040a2ed0 0x7fda040a5390 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fda18007f00 tx=0x7fda1800c690 comp rx=0 tx=0).stop 2024-04-25T02:01:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 -- 172.21.15.102:0/1264553941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 msgr2=0x7fda201a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 0x7fda201a2dc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fda10006010 tx=0x7fda10032d10 comp rx=0 tx=0).stop 2024-04-25T02:01:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 -- 172.21.15.102:0/1264553941 shutdown_connections 2024-04-25T02:01:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fda040a2ed0 0x7fda040a5390 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fda20105990 0x7fda201a2dc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda20100eb0 0x7fda2019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 --2- 172.21.15.102:0/1264553941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda20100590 0x7fda2019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.128+0000 7fda1f200640 1 -- 172.21.15.102:0/1264553941 >> 172.21.15.102:0/1264553941 conn(0x7fda200fbfc0 msgr2=0x7fda200fcf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.129+0000 7fda1f200640 1 -- 172.21.15.102:0/1264553941 shutdown_connections 2024-04-25T02:01:04.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.129+0000 7fda1f200640 1 -- 172.21.15.102:0/1264553941 wait complete. 2024-04-25T02:01:04.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T02:01:04.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T02:01:04.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.364+0000 7f1ce6000640 1 -- 172.21.15.102:0/1791369047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ce01029c0 msgr2=0x7f1ce0102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.364+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1791369047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ce01029c0 0x7f1ce0102da0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009ca0 tx=0x7f1cd802a9e0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.364+0000 7f1ce6000640 1 -- 172.21.15.102:0/1791369047 shutdown_connections 2024-04-25T02:01:04.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.364+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1791369047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ce010bd00 0x7f1ce010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.364+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1791369047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce01032e0 0x7f1ce010b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.364+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1791369047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ce01029c0 0x7f1ce0102da0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.364+0000 7f1ce6000640 1 -- 172.21.15.102:0/1791369047 >> 172.21.15.102:0/1791369047 conn(0x7f1ce0077740 msgr2=0x7f1ce0077b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.364+0000 7f1ce6000640 1 -- 172.21.15.102:0/1791369047 shutdown_connections 2024-04-25T02:01:04.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.364+0000 7f1ce6000640 1 -- 172.21.15.102:0/1791369047 wait complete. 2024-04-25T02:01:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.365+0000 7f1ce6000640 1 Processor -- start 2024-04-25T02:01:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.365+0000 7f1ce6000640 1 -- start start 2024-04-25T02:01:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.365+0000 7f1ce6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ce01029c0 0x7f1ce0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1ce6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ce01032e0 0x7f1ce0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1ce6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce010bd00 0x7f1ce019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1ce6000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce0111b50 con 0x7f1ce010bd00 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1ce6000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce01119d0 con 0x7f1ce01032e0 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1ce6000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce0111cd0 con 0x7f1ce01029c0 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ce01029c0 0x7f1ce0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ce01029c0 0x7f1ce0198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:32778/0 (socket says 172.21.15.102:32778) 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdf400640 1 -- 172.21.15.102:0/1408749558 learned_addr learned my addr 172.21.15.102:0/1408749558 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce010bd00 0x7f1ce019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdea00640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ce01032e0 0x7f1ce0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdf400640 1 -- 172.21.15.102:0/1408749558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ce01032e0 msgr2=0x7f1ce0198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdf400640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ce01032e0 0x7f1ce0198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdf400640 1 -- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce010bd00 msgr2=0x7f1ce019c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdf400640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce010bd00 0x7f1ce019c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdf400640 1 -- 172.21.15.102:0/1408749558 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce019d040 con 0x7f1ce01029c0 2024-04-25T02:01:04.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.366+0000 7f1cdea00640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ce01032e0 0x7f1ce0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:04.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.367+0000 7f1cdf400640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ce01029c0 0x7f1ce0198050 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8004af0 tx=0x7f1cd8003f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.367+0000 7f1cd7e00640 1 -- 172.21.15.102:0/1408749558 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8047070 con 0x7f1ce01029c0 2024-04-25T02:01:04.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.367+0000 7f1ce6000640 1 -- 172.21.15.102:0/1408749558 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce019d2d0 con 0x7f1ce01029c0 2024-04-25T02:01:04.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.367+0000 7f1cd7e00640 1 -- 172.21.15.102:0/1408749558 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd802e870 con 0x7f1ce01029c0 2024-04-25T02:01:04.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.367+0000 7f1cd7e00640 1 -- 172.21.15.102:0/1408749558 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8033050 con 0x7f1ce01029c0 2024-04-25T02:01:04.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.367+0000 7f1ce6000640 1 -- 172.21.15.102:0/1408749558 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce01a4bc0 con 0x7f1ce01029c0 2024-04-25T02:01:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.370+0000 7f1ce8e55740 1 Processor -- start 2024-04-25T02:01:04.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1ce8e55740 1 -- start start 2024-04-25T02:01:04.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce25110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce33ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce367f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56438cd89560 con 0x56438ce256e0 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56438cd31930 con 0x56438ce34400 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56438cd85520 con 0x56438ce24d30 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1cd7e00640 1 -- 172.21.15.102:0/1408749558 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1cd804a070 con 0x7f1ce01029c0 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1cdfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce25110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1cdfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce25110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:32780/0 (socket says 172.21.15.102:32780) 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1cdfe00640 1 -- 172.21.15.102:0/1236434483 learned_addr learned my addr 172.21.15.102:0/1236434483 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.371+0000 7f1cdea00640 1 --2- 172.21.15.102:0/1236434483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.372+0000 7f1cdfe00640 1 -- 172.21.15.102:0/1236434483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 msgr2=0x56438ce367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.372+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/1236434483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce367f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.372+0000 7f1cdfe00640 1 -- 172.21.15.102:0/1236434483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 msgr2=0x56438ce33ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.372+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/1236434483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce33ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.372+0000 7f1cdfe00640 1 -- 172.21.15.102:0/1236434483 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56438cb677d0 con 0x56438ce24d30 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.372+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/1236434483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce25110 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009c50 tx=0x7f1cd0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e53c6c47aee3a071 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.372+0000 7f1ccfe00640 1 -- 172.21.15.102:0/1236434483 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd001e070 con 0x56438ce24d30 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.372+0000 7f1ccfe00640 1 -- 172.21.15.102:0/1236434483 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd0019b10 con 0x56438ce24d30 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.372+0000 7f1ccfe00640 1 -- 172.21.15.102:0/1236434483 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd000eae0 con 0x56438ce24d30 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1cd7e00640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f1cc40a2d00 0x7f1cc40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1ce8e55740 1 -- 172.21.15.102:0/1236434483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 msgr2=0x56438ce25110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/1236434483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce25110 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009c50 tx=0x7f1cd0011810 comp rx=0 tx=0).stop 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1cd7e00640 1 -- 172.21.15.102:0/1408749558 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f1cd80eff50 con 0x7f1ce01029c0 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1ce8e55740 1 -- 172.21.15.102:0/1236434483 shutdown_connections 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1cdf400640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f1cc40a2d00 0x7f1cc40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/1236434483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/1236434483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce33ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/1236434483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce25110 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1ce8e55740 1 -- 172.21.15.102:0/1236434483 >> 172.21.15.102:0/1236434483 conn(0x56438ce21320 msgr2=0x56438ce21700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1ce8e55740 1 -- 172.21.15.102:0/1236434483 shutdown_connections 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.373+0000 7f1ce8e55740 1 -- 172.21.15.102:0/1236434483 wait complete. 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.374+0000 7f1ce8e55740 1 -- start start 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.374+0000 7f1cdf400640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f1cc40a2d00 0x7f1cc40a51c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x56438ce27930 tx=0x7f1cd80a7050 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.374+0000 7f1ce8e55740 1 Processor -- start 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.374+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce313c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.374+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce31900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.374+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce31e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.374+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56438ce371c0 con 0x56438ce256e0 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.374+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56438cd44d90 con 0x56438ce34400 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.374+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56438ce37340 con 0x56438ce24d30 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1ce8e55740 10 client.0 _ll_register_callbacks cb 0x56438cd44780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce31900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce313c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:32786/0 (socket says 172.21.15.102:32786) 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdfe00640 1 -- 172.21.15.102:0/3333861609 learned_addr learned my addr 172.21.15.102:0/3333861609 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce31e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdea00640 1 -- 172.21.15.102:0/3333861609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 msgr2=0x56438ce313c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdea00640 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce313c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdea00640 1 -- 172.21.15.102:0/3333861609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 msgr2=0x56438ce31e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdea00640 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce31e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.375+0000 7f1cdea00640 1 -- 172.21.15.102:0/3333861609 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56438ca9d3f0 con 0x56438ce256e0 2024-04-25T02:01:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.376+0000 7f1cdea00640 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce31900 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0100030 tx=0x7f1cc8011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbb401169eced7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.376+0000 7f1cccc00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:01:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.376+0000 7f1cccc00640 1 -- 172.21.15.102:0/3333861609 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8010c30 con 0x56438ce256e0 2024-04-25T02:01:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.376+0000 7f1cccc00640 1 -- 172.21.15.102:0/3333861609 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f1cc8010dd0 con 0x56438ce256e0 2024-04-25T02:01:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.376+0000 7f1ce8e55740 1 -- 172.21.15.102:0/3333861609 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cd0019b10 con 0x56438ce256e0 2024-04-25T02:01:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.376+0000 7f1cccc00640 1 -- 172.21.15.102:0/3333861609 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f1cc8004020 con 0x56438ce256e0 2024-04-25T02:01:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.376+0000 7f1ce8e55740 10 client.16350 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.376+0000 7f1ce8e55740 1 -- 172.21.15.102:0/3333861609 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56438ca5d0f0 con 0x56438ce256e0 2024-04-25T02:01:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.377+0000 7f1cccc00640 1 -- 172.21.15.102:0/3333861609 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd0019b10 con 0x56438ce256e0 2024-04-25T02:01:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.377+0000 7f1cccc00640 1 -- 172.21.15.102:0/3333861609 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f1cc801a410 con 0x56438ce256e0 2024-04-25T02:01:04.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.377+0000 7f1ce8e55740 10 client.16350 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.377+0000 7f1ce8e55740 10 client.16350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.377+0000 7f1ce8e55740 20 client.16350 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.377+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56438ce44910 0x56438ce46d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.377+0000 7f1ce8e55740 4 client.16350 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.377+0000 7f1ce8e55740 1 -- 172.21.15.102:0/3333861609 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56438cd9cd90 con 0x56438ce44910 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.377+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56438ce44910 0x56438ce46d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.378+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56438ce44910 0x56438ce46d50 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfb59ed19ac8d4aa server_cookie=25bbe66f8f3c54bc in_seq=0 out_seq=0 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.378+0000 7f1cccc00640 10 client.16350 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.378+0000 7f1cccc00640 1 -- 172.21.15.102:0/3333861609 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56438cd9cd90 con 0x56438ce44910 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.378+0000 7f1cccc00640 10 client.16350 handle_command_reply: tid=0 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.378+0000 7f1ce8e55740 1 client.16350 shutdown 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 -- 172.21.15.102:0/3333861609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 msgr2=0x56438ce31900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce31900 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0100030 tx=0x7f1cc8011bc0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 -- 172.21.15.102:0/3333861609 shutdown_connections 2024-04-25T02:01:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56438ce44910 0x56438ce46d50 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce31e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce31900 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1cccc00640 0 client.16350 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:04.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/3333861609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce313c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 -- 172.21.15.102:0/3333861609 >> 172.21.15.102:0/3333861609 conn(0x56438ce21320 msgr2=0x56438ce27190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 -- 172.21.15.102:0/3333861609 shutdown_connections 2024-04-25T02:01:04.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 1 -- 172.21.15.102:0/3333861609 wait complete. 2024-04-25T02:01:04.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.379+0000 7f1ce8e55740 20 client.16350 trim_cache size 0 max 16384 2024-04-25T02:01:04.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.411+0000 7f1ce8e55740 1 Processor -- start 2024-04-25T02:01:04.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1ce8e55740 1 -- start start 2024-04-25T02:01:04.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce40120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce30560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce31c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce01adaf0 con 0x56438ce256e0 2024-04-25T02:01:04.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce01ab360 con 0x56438ce34400 2024-04-25T02:01:04.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce01a91d0 con 0x56438ce24d30 2024-04-25T02:01:04.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1cdfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce40120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1cdfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce40120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:32798/0 (socket says 172.21.15.102:32798) 2024-04-25T02:01:04.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1cdfe00640 1 -- 172.21.15.102:0/2036271116 learned_addr learned my addr 172.21.15.102:0/2036271116 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/2036271116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce31c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.412+0000 7f1cdea00640 1 --2- 172.21.15.102:0/2036271116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce30560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.413+0000 7f1cdfe00640 1 -- 172.21.15.102:0/2036271116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 msgr2=0x56438ce31c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.413+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/2036271116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce31c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.413+0000 7f1cdfe00640 1 -- 172.21.15.102:0/2036271116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 msgr2=0x56438ce30560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.413+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/2036271116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce30560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.413+0000 7f1cdfe00640 1 -- 172.21.15.102:0/2036271116 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56438ce367f0 con 0x56438ce24d30 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.413+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/2036271116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce40120 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000e570 tx=0x7f1cd002a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cbe176254eaeb94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.413+0000 7f1ccf400640 1 -- 172.21.15.102:0/2036271116 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd001e040 con 0x56438ce24d30 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.413+0000 7f1ccf400640 1 -- 172.21.15.102:0/2036271116 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd00050c0 con 0x56438ce24d30 2024-04-25T02:01:04.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.413+0000 7f1ccf400640 1 -- 172.21.15.102:0/2036271116 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd002f5e0 con 0x56438ce24d30 2024-04-25T02:01:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.414+0000 7f1ce8e55740 1 -- 172.21.15.102:0/2036271116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 msgr2=0x56438ce40120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.414+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/2036271116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce40120 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000e570 tx=0x7f1cd002a040 comp rx=0 tx=0).stop 2024-04-25T02:01:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.414+0000 7f1ce8e55740 1 -- 172.21.15.102:0/2036271116 shutdown_connections 2024-04-25T02:01:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.414+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/2036271116 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce34400 0x56438ce31c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.414+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/2036271116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce30560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.414+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/2036271116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce24d30 0x56438ce40120 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.414+0000 7f1ce8e55740 1 -- 172.21.15.102:0/2036271116 >> 172.21.15.102:0/2036271116 conn(0x56438ce21320 msgr2=0x56438ce1fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.414+0000 7f1ce8e55740 1 -- 172.21.15.102:0/2036271116 shutdown_connections 2024-04-25T02:01:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.414+0000 7f1ce8e55740 1 -- 172.21.15.102:0/2036271116 wait complete. 2024-04-25T02:01:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.415+0000 7f1ce8e55740 1 -- start start 2024-04-25T02:01:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.415+0000 7f1ce8e55740 1 Processor -- start 2024-04-25T02:01:04.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce24d30 0x56438ce3c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce3cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1ce8e55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce34400 0x56438ce2d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce01a9820 con 0x56438ce256e0 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce01aac90 con 0x56438ce24d30 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1ce8e55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce01a9ef0 con 0x56438ce34400 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1ce8e55740 10 client.0 _ll_register_callbacks cb 0x7f1cc800bef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1cdfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce24d30 0x56438ce3c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1cdfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce24d30 0x56438ce3c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53108/0 (socket says 172.21.15.102:53108) 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1cdfe00640 1 -- 172.21.15.102:0/269652357 learned_addr learned my addr 172.21.15.102:0/269652357 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.416+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce34400 0x56438ce2d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.417+0000 7f1cdfe00640 1 -- 172.21.15.102:0/269652357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce34400 msgr2=0x56438ce2d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.417+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce34400 0x56438ce2d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.417+0000 7f1cdfe00640 1 -- 172.21.15.102:0/269652357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 msgr2=0x56438ce3cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.417+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce3cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.417+0000 7f1cdfe00640 1 -- 172.21.15.102:0/269652357 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56438ce2dbb0 con 0x56438ce24d30 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.417+0000 7f1cdfe00640 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce24d30 0x56438ce3c630 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0011df0 tx=0x7f1cd002f1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4197517564a4958 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.417+0000 7f1cccc00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.418+0000 7f1cccc00640 1 -- 172.21.15.102:0/269652357 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd001e040 con 0x56438ce24d30 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.418+0000 7f1cccc00640 1 -- 172.21.15.102:0/269652357 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f1cd0005470 con 0x56438ce24d30 2024-04-25T02:01:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.418+0000 7f1ce8e55740 1 -- 172.21.15.102:0/269652357 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1cd00050c0 con 0x56438ce24d30 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.418+0000 7f1cccc00640 1 -- 172.21.15.102:0/269652357 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f1cd0019a60 con 0x56438ce24d30 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.418+0000 7f1ce8e55740 10 client.26864 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.418+0000 7f1ce8e55740 1 -- 172.21.15.102:0/269652357 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56438ce2b240 con 0x56438ce24d30 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.418+0000 7f1cccc00640 1 -- 172.21.15.102:0/269652357 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd00050c0 con 0x56438ce24d30 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.419+0000 7f1cccc00640 1 -- 172.21.15.102:0/269652357 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f1cd0030560 con 0x56438ce24d30 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.419+0000 7f1ce8e55740 10 client.26864 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.419+0000 7f1ce8e55740 10 client.26864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.419+0000 7f1ce8e55740 20 client.26864 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.419+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56438ce2b790 0x56438cea4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.419+0000 7f1ce8e55740 4 client.26864 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.419+0000 7f1ce8e55740 1 -- 172.21.15.102:0/269652357 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56438cd96aa0 con 0x56438ce2b790 2024-04-25T02:01:04.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.419+0000 7f1cdea00640 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56438ce2b790 0x56438cea4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1cdea00640 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56438ce2b790 0x56438cea4e60 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a688e44c306dc19 server_cookie=2087dc8eed23931 in_seq=0 out_seq=0 2024-04-25T02:01:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1cccc00640 10 client.26864 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:04.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1cccc00640 1 -- 172.21.15.102:0/269652357 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56438cd96aa0 con 0x56438ce2b790 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1cccc00640 10 client.26864 handle_command_reply: tid=0 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1ce8e55740 1 client.26864 shutdown 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1ce8e55740 1 -- 172.21.15.102:0/269652357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce24d30 msgr2=0x56438ce3c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce24d30 0x56438ce3c630 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0011df0 tx=0x7f1cd002f1c0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1ce8e55740 1 -- 172.21.15.102:0/269652357 shutdown_connections 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56438ce2b790 0x56438cea4e60 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56438ce34400 0x56438ce2d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56438ce256e0 0x56438ce3cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1ce8e55740 1 --2- 172.21.15.102:0/269652357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56438ce24d30 0x56438ce3c630 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1cccc00640 0 client.26864 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.420+0000 7f1ce8e55740 1 -- 172.21.15.102:0/269652357 >> 172.21.15.102:0/269652357 conn(0x56438ce21320 msgr2=0x56438ce303e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.421+0000 7f1ce8e55740 1 -- 172.21.15.102:0/269652357 shutdown_connections 2024-04-25T02:01:04.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.421+0000 7f1ce8e55740 1 -- 172.21.15.102:0/269652357 wait complete. 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.421+0000 7f1ce8e55740 20 client.26864 trim_cache size 0 max 16384 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout: "6f9a9466-9b1b-426b-b8f0-8bb2a80b509a": { 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "6f9a9466-9b1b-426b-b8f0-8bb2a80b509a", 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:01:04.428 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:01:04.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.421+0000 7f1ce6000640 1 -- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f1cc40a2d00 msgr2=0x7f1cc40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.421+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f1cc40a2d00 0x7f1cc40a51c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x56438ce27930 tx=0x7f1cd80a7050 comp rx=0 tx=0).stop 2024-04-25T02:01:04.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.421+0000 7f1ce6000640 1 -- 172.21.15.102:0/1408749558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ce01029c0 msgr2=0x7f1ce0198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:04.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.421+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ce01029c0 0x7f1ce0198050 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8004af0 tx=0x7f1cd8003f30 comp rx=0 tx=0).stop 2024-04-25T02:01:04.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.422+0000 7f1ce6000640 1 -- 172.21.15.102:0/1408749558 shutdown_connections 2024-04-25T02:01:04.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.422+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f1cc40a2d00 0x7f1cc40a51c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.422+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ce010bd00 0x7f1ce019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.422+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ce01032e0 0x7f1ce0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.422+0000 7f1ce6000640 1 --2- 172.21.15.102:0/1408749558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ce01029c0 0x7f1ce0198050 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:04.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.422+0000 7f1ce6000640 1 -- 172.21.15.102:0/1408749558 >> 172.21.15.102:0/1408749558 conn(0x7f1ce0077740 msgr2=0x7f1ce00fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:04.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.422+0000 7f1ce6000640 1 -- 172.21.15.102:0/1408749558 shutdown_connections 2024-04-25T02:01:04.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:04.422+0000 7f1ce6000640 1 -- 172.21.15.102:0/1408749558 wait complete. 2024-04-25T02:01:04.447 INFO:tasks.cephfs.filesystem:scrub status for tag:6f9a9466-9b1b-426b-b8f0-8bb2a80b509a - {'path': '/', 'tag': '6f9a9466-9b1b-426b-b8f0-8bb2a80b509a', 'options': 'recursive,force'} 2024-04-25T02:01:05.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: pgmap v177: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.5 MiB/s wr, 712 op/s 2024-04-25T02:01:05.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T02:01:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:01:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:01:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:01:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:01:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:05 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: pgmap v177: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.5 MiB/s wr, 712 op/s 2024-04-25T02:01:05.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T02:01:05.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:01:05.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:01:05.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:01:05.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:01:05.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:05 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: pgmap v177: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 7.5 MiB/s wr, 712 op/s 2024-04-25T02:01:05.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T02:01:05.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:01:05.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:01:05.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:01:05.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:01:05.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:05.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:05 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:07.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:07 smithi102 ceph-mon[27796]: pgmap v178: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 5.6 MiB/s wr, 593 op/s 2024-04-25T02:01:07.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:07 smithi110 ceph-mon[32848]: pgmap v178: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 5.6 MiB/s wr, 593 op/s 2024-04-25T02:01:07.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:07 smithi193 ceph-mon[32981]: pgmap v178: 97 pgs: 97 active+clean; 3.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 5.6 MiB/s wr, 593 op/s 2024-04-25T02:01:08.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 315.0 s, 138.6 tps, lat 732.376 ms stddev 453.185 2024-04-25T02:01:09.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:09 smithi102 ceph-mon[27796]: pgmap v179: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 6.5 MiB/s wr, 777 op/s 2024-04-25T02:01:09.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:09 smithi110 ceph-mon[32848]: pgmap v179: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 6.5 MiB/s wr, 777 op/s 2024-04-25T02:01:09.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:09 smithi193 ceph-mon[32981]: pgmap v179: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 6.5 MiB/s wr, 777 op/s 2024-04-25T02:01:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.326+0000 7f91a6a00640 1 -- 172.21.15.102:0/2267815936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91a0108600 msgr2=0x7f91a0114a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.326+0000 7f91a6a00640 1 --2- 172.21.15.102:0/2267815936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91a0108600 0x7f91a0114a40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9198009cd0 tx=0x7f919802a8e0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.326+0000 7f91a6a00640 1 -- 172.21.15.102:0/2267815936 shutdown_connections 2024-04-25T02:01:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.326+0000 7f91a6a00640 1 --2- 172.21.15.102:0/2267815936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91a0108600 0x7f91a0114a40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.326+0000 7f91a6a00640 1 --2- 172.21.15.102:0/2267815936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91a0107c80 0x7f91a01080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.326+0000 7f91a6a00640 1 --2- 172.21.15.102:0/2267815936 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91a010dc10 0x7f91a010dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.326+0000 7f91a6a00640 1 -- 172.21.15.102:0/2267815936 >> 172.21.15.102:0/2267815936 conn(0x7f91a0077520 msgr2=0x7f91a0106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.327+0000 7f91a6a00640 1 -- 172.21.15.102:0/2267815936 shutdown_connections 2024-04-25T02:01:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.327+0000 7f91a6a00640 1 -- 172.21.15.102:0/2267815936 wait complete. 2024-04-25T02:01:11.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.328+0000 7f91a6a00640 1 Processor -- start 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.328+0000 7f91a6a00640 1 -- start start 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.328+0000 7f91a6a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91a0107c80 0x7f91a019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.328+0000 7f91a6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91a0108600 0x7f91a019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.328+0000 7f91a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91a010dc10 0x7f91a01a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.328+0000 7f91a6a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f91a0117fd0 con 0x7f91a0108600 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.328+0000 7f91a6a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f91a0117e50 con 0x7f91a0107c80 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.328+0000 7f91a6a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91a0118150 con 0x7f91a010dc10 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.328+0000 7f91a5600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91a0107c80 0x7f91a019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91a010dc10 0x7f91a01a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a5600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91a0107c80 0x7f91a019e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53122/0 (socket says 172.21.15.102:53122) 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a5600640 1 -- 172.21.15.102:0/954036008 learned_addr learned my addr 172.21.15.102:0/954036008 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a4c00640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91a0108600 0x7f91a019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a5600640 1 -- 172.21.15.102:0/954036008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91a010dc10 msgr2=0x7f91a01a2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:11.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a5600640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91a010dc10 0x7f91a01a2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a5600640 1 -- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91a0108600 msgr2=0x7f91a019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a5600640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91a0108600 0x7f91a019e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a5600640 1 -- 172.21.15.102:0/954036008 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a01a33e0 con 0x7f91a0107c80 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a6000640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91a010dc10 0x7f91a01a2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.329+0000 7f91a5600640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91a0107c80 0x7f91a019e3f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f919000e9f0 tx=0x7f919000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.330+0000 7f919e000640 1 -- 172.21.15.102:0/954036008 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919000cd80 con 0x7f91a0107c80 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.330+0000 7f919e000640 1 -- 172.21.15.102:0/954036008 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9190005500 con 0x7f91a0107c80 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.330+0000 7f91a6a00640 1 -- 172.21.15.102:0/954036008 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a01a36d0 con 0x7f91a0107c80 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.330+0000 7f919e000640 1 -- 172.21.15.102:0/954036008 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190010690 con 0x7f91a0107c80 2024-04-25T02:01:11.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.330+0000 7f91a6a00640 1 -- 172.21.15.102:0/954036008 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a01aaf60 con 0x7f91a0107c80 2024-04-25T02:01:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.334+0000 7f91aafd3740 1 Processor -- start 2024-04-25T02:01:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.334+0000 7f91aafd3740 1 -- start start 2024-04-25T02:01:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c5ff390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c609c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c60a190 0x561f6c60c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91aafd3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561f6c562ab0 con 0x561f6c5fefb0 2024-04-25T02:01:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f919e000640 1 -- 172.21.15.102:0/954036008 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f91900046c0 con 0x7f91a0107c80 2024-04-25T02:01:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91aafd3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561f6c50b640 con 0x561f6c5ff8d0 2024-04-25T02:01:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91aafd3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561f6c55ea70 con 0x561f6c60a190 2024-04-25T02:01:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91a6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c5ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91a5600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c609c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91a6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c5ff390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33336/0 (socket says 172.21.15.102:33336) 2024-04-25T02:01:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91a5600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c609c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53126/0 (socket says 172.21.15.102:53126) 2024-04-25T02:01:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.335+0000 7f91a6000640 1 -- 172.21.15.102:0/3327695601 learned_addr learned my addr 172.21.15.102:0/3327695601 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:11.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f91a4c00640 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c60a190 0x561f6c60c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f91a5600640 1 -- 172.21.15.102:0/3327695601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c60a190 msgr2=0x561f6c60c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:11.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f91a5600640 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c60a190 0x561f6c60c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f91a5600640 1 -- 172.21.15.102:0/3327695601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 msgr2=0x561f6c5ff390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:11.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f91a5600640 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c5ff390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f91a5600640 1 -- 172.21.15.102:0/3327695601 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f6c277210 con 0x561f6c5ff8d0 2024-04-25T02:01:11.346 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f91a4c00640 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c60a190 0x561f6c60c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f91a6000640 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c5ff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f91a5600640 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c609c50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f91900cce10 tx=0x7f91900d5b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=999dc355a291e1fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f9196600640 1 -- 172.21.15.102:0/3327695601 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900de070 con 0x561f6c5ff8d0 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.336+0000 7f9196600640 1 -- 172.21.15.102:0/3327695601 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91900d7d50 con 0x561f6c5ff8d0 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.337+0000 7f9196600640 1 -- 172.21.15.102:0/3327695601 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900d1960 con 0x561f6c5ff8d0 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.337+0000 7f919e000640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f91840a2e40 0x7f91840a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.337+0000 7f919e000640 1 -- 172.21.15.102:0/954036008 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f91900cf690 con 0x7f91a0107c80 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.337+0000 7f91a6000640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f91840a2e40 0x7f91840a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.337+0000 7f91aafd3740 1 -- 172.21.15.102:0/3327695601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 msgr2=0x561f6c609c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.337+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c609c50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f91900cce10 tx=0x7f91900d5b40 comp rx=0 tx=0).stop 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.338+0000 7f91aafd3740 1 -- 172.21.15.102:0/3327695601 shutdown_connections 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.338+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c60a190 0x561f6c60c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.338+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c609c50 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.338+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3327695601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c5ff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.338+0000 7f91aafd3740 1 -- 172.21.15.102:0/3327695601 >> 172.21.15.102:0/3327695601 conn(0x561f6c5fb4c0 msgr2=0x561f6c5fb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.338+0000 7f91aafd3740 1 -- 172.21.15.102:0/3327695601 shutdown_connections 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.338+0000 7f91a6000640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f91840a2e40 0x7f91840a5300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9198006010 tx=0x7f919803d040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.338+0000 7f91aafd3740 1 -- 172.21.15.102:0/3327695601 wait complete. 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.338+0000 7f91aafd3740 1 -- start start 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91aafd3740 1 Processor -- start 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c6070d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5ff8d0 0x561f6c607610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c607b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91aafd3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561f6c60cdc0 con 0x561f6c5ff8d0 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91aafd3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561f6c51e990 con 0x561f6c60a190 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91aafd3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561f6c60cf40 con 0x561f6c5fefb0 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91aafd3740 10 client.0 _ll_register_callbacks cb 0x561f6c520950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91a5600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c6070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91a4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5ff8d0 0x561f6c607610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91a5600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c6070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:32850/0 (socket says 172.21.15.102:32850) 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.339+0000 7f91a5600640 1 -- 172.21.15.102:0/1585226150 learned_addr learned my addr 172.21.15.102:0/1585226150 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f91a6000640 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c607b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f91a5600640 1 -- 172.21.15.102:0/1585226150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 msgr2=0x561f6c607b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f91a5600640 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c607b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f91a5600640 1 -- 172.21.15.102:0/1585226150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5ff8d0 msgr2=0x561f6c607610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f91a5600640 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5ff8d0 0x561f6c607610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f91a5600640 1 -- 172.21.15.102:0/1585226150 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f6c343620 con 0x561f6c5fefb0 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f91a5600640 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c6070d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9190098040 tx=0x7f91900d1560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b56919cb389cc14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f918aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f918aa00640 1 -- 172.21.15.102:0/1585226150 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900de030 con 0x561f6c5fefb0 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f918aa00640 1 -- 172.21.15.102:0/1585226150 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f91900cd090 con 0x561f6c5fefb0 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.340+0000 7f91aafd3740 1 -- 172.21.15.102:0/1585226150 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91900d7d50 con 0x561f6c5fefb0 2024-04-25T02:01:11.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f918aa00640 1 -- 172.21.15.102:0/1585226150 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9190097e80 con 0x561f6c5fefb0 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f91aafd3740 10 client.26833 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f91aafd3740 1 -- 172.21.15.102:0/1585226150 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561f6c60cdc0 con 0x561f6c5fefb0 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f918aa00640 1 -- 172.21.15.102:0/1585226150 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91900d7d50 con 0x561f6c5fefb0 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f918aa00640 1 -- 172.21.15.102:0/1585226150 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f9190098250 con 0x561f6c5fefb0 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f91aafd3740 10 client.26833 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f91aafd3740 10 client.26833 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f91aafd3740 20 client.26833 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f91aafd3740 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561f6c619d30 0x561f6c61c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f91aafd3740 4 client.26833 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f91aafd3740 1 -- 172.21.15.102:0/1585226150 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f6c56fff0 con 0x561f6c619d30 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.341+0000 7f91a4c00640 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561f6c619d30 0x561f6c61c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.342+0000 7f91a4c00640 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561f6c619d30 0x561f6c61c120 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab2cae6ad82de8b4 server_cookie=39edf49902f3bdf2 in_seq=0 out_seq=0 2024-04-25T02:01:11.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.342+0000 7f918aa00640 10 client.26833 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:11.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f918aa00640 1 -- 172.21.15.102:0/1585226150 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f6c56fff0 con 0x561f6c619d30 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f918aa00640 10 client.26833 handle_command_reply: tid=0 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 client.26833 shutdown 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 -- 172.21.15.102:0/1585226150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 msgr2=0x561f6c6070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c6070d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9190098040 tx=0x7f91900d1560 comp rx=0 tx=0).stop 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 -- 172.21.15.102:0/1585226150 shutdown_connections 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561f6c619d30 0x561f6c61c120 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c607b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5ff8d0 0x561f6c607610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 --2- 172.21.15.102:0/1585226150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c6070d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f918aa00640 0 client.26833 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 -- 172.21.15.102:0/1585226150 >> 172.21.15.102:0/1585226150 conn(0x561f6c5fb4c0 msgr2=0x561f6c601240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 -- 172.21.15.102:0/1585226150 shutdown_connections 2024-04-25T02:01:11.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 1 -- 172.21.15.102:0/1585226150 wait complete. 2024-04-25T02:01:11.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.343+0000 7f91aafd3740 20 client.26833 trim_cache size 0 max 16384 2024-04-25T02:01:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.362+0000 7f91aafd3740 1 Processor -- start 2024-04-25T02:01:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.362+0000 7f91aafd3740 1 -- start start 2024-04-25T02:01:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.362+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c615c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.362+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5ff8d0 0x561f6c607970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c621860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91aafd3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f91a01b60c0 con 0x561f6c5fefb0 2024-04-25T02:01:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91aafd3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f91a01b3930 con 0x561f6c60a190 2024-04-25T02:01:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91aafd3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91a01b1780 con 0x561f6c5ff8d0 2024-04-25T02:01:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a5600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c615c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c621860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c621860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53152/0 (socket says 172.21.15.102:53152) 2024-04-25T02:01:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a6000640 1 -- 172.21.15.102:0/2720193593 learned_addr learned my addr 172.21.15.102:0/2720193593 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a5600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c615c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33366/0 (socket says 172.21.15.102:33366) 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a4c00640 1 --2- 172.21.15.102:0/2720193593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5ff8d0 0x561f6c607970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a6000640 1 -- 172.21.15.102:0/2720193593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5ff8d0 msgr2=0x561f6c607970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a6000640 1 --2- 172.21.15.102:0/2720193593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5ff8d0 0x561f6c607970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a6000640 1 -- 172.21.15.102:0/2720193593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 msgr2=0x561f6c615c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a6000640 1 --2- 172.21.15.102:0/2720193593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c615c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a6000640 1 -- 172.21.15.102:0/2720193593 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f6c601970 con 0x561f6c60a190 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.363+0000 7f91a5600640 1 --2- 172.21.15.102:0/2720193593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c615c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.364+0000 7f91a6000640 1 --2- 172.21.15.102:0/2720193593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c621860 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f919802aed0 tx=0x7f9198045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b94f5e972e75901 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.364+0000 7f9195c00640 1 -- 172.21.15.102:0/2720193593 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9198004710 con 0x561f6c60a190 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.364+0000 7f9195c00640 1 -- 172.21.15.102:0/2720193593 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9198056070 con 0x561f6c60a190 2024-04-25T02:01:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.364+0000 7f9195c00640 1 -- 172.21.15.102:0/2720193593 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9198046830 con 0x561f6c60a190 2024-04-25T02:01:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.365+0000 7f91aafd3740 1 -- 172.21.15.102:0/2720193593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 msgr2=0x561f6c621860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.365+0000 7f91aafd3740 1 --2- 172.21.15.102:0/2720193593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c621860 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f919802aed0 tx=0x7f9198045000 comp rx=0 tx=0).stop 2024-04-25T02:01:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.365+0000 7f91aafd3740 1 -- 172.21.15.102:0/2720193593 shutdown_connections 2024-04-25T02:01:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.365+0000 7f91aafd3740 1 --2- 172.21.15.102:0/2720193593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c60a190 0x561f6c621860 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.365+0000 7f91aafd3740 1 --2- 172.21.15.102:0/2720193593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5ff8d0 0x561f6c607970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.365+0000 7f91aafd3740 1 --2- 172.21.15.102:0/2720193593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c5fefb0 0x561f6c615c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.365+0000 7f91aafd3740 1 -- 172.21.15.102:0/2720193593 >> 172.21.15.102:0/2720193593 conn(0x561f6c5fb4c0 msgr2=0x561f6c5fa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.365+0000 7f91aafd3740 1 -- 172.21.15.102:0/2720193593 shutdown_connections 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.365+0000 7f91aafd3740 1 -- 172.21.15.102:0/2720193593 wait complete. 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91aafd3740 1 -- start start 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91aafd3740 1 Processor -- start 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c612190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c6126d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91aafd3740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c60a190 0x561f6c60f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91aafd3740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f91a01b1df0 con 0x561f6c60a190 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91aafd3740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f91a01b3260 con 0x561f6c5ff8d0 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91aafd3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91a01b24c0 con 0x561f6c5fefb0 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91aafd3740 10 client.0 _ll_register_callbacks cb 0x7f9190005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91a5600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c612190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.366+0000 7f91a4c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c6126d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a4c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c6126d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53156/0 (socket says 172.21.15.102:53156) 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a4c00640 1 -- 172.21.15.102:0/3780471955 learned_addr learned my addr 172.21.15.102:0/3780471955 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a5600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c612190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:32868/0 (socket says 172.21.15.102:32868) 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a6000640 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c60a190 0x561f6c60f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a5600640 1 -- 172.21.15.102:0/3780471955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 msgr2=0x561f6c6126d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a5600640 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c6126d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a5600640 1 -- 172.21.15.102:0/3780471955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c60a190 msgr2=0x561f6c60f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a5600640 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c60a190 0x561f6c60f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a5600640 1 -- 172.21.15.102:0/3780471955 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f6c60f7a0 con 0x561f6c5fefb0 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91a5600640 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c612190 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f919009bb70 tx=0x7f9190098500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c076a0f8d6ad073d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f918aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f918aa00640 1 -- 172.21.15.102:0/3780471955 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91900a8060 con 0x561f6c5fefb0 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f918aa00640 1 -- 172.21.15.102:0/3780471955 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f91900a43c0 con 0x561f6c5fefb0 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.367+0000 7f91aafd3740 1 -- 172.21.15.102:0/3780471955 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9198056070 con 0x561f6c5fefb0 2024-04-25T02:01:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f918aa00640 1 -- 172.21.15.102:0/3780471955 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f91900de030 con 0x561f6c5fefb0 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f91aafd3740 10 client.26839 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f91aafd3740 1 -- 172.21.15.102:0/3780471955 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x561f6c60f7a0 con 0x561f6c5fefb0 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f918aa00640 1 -- 172.21.15.102:0/3780471955 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9198056070 con 0x561f6c5fefb0 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f918aa00640 1 -- 172.21.15.102:0/3780471955 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f91900d10e0 con 0x561f6c5fefb0 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f91aafd3740 10 client.26839 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f91aafd3740 10 client.26839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f91aafd3740 20 client.26839 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561f6c680dd0 0x561f6c6831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f91aafd3740 4 client.26839 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:01:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.368+0000 7f91aafd3740 1 -- 172.21.15.102:0/3780471955 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f6c56a340 con 0x561f6c680dd0 2024-04-25T02:01:11.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.369+0000 7f91a4c00640 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561f6c680dd0 0x561f6c6831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:11.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.369+0000 7f91a4c00640 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561f6c680dd0 0x561f6c6831f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d25c434105191c0e server_cookie=40e766692c4c9e86 in_seq=0 out_seq=0 2024-04-25T02:01:11.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.369+0000 7f918aa00640 10 client.26839 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.381+0000 7f918aa00640 1 -- 172.21.15.102:0/3780471955 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f6c56a340 con 0x561f6c680dd0 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.381+0000 7f918aa00640 10 client.26839 handle_command_reply: tid=0 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.381+0000 7f91aafd3740 1 client.26839 shutdown 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.381+0000 7f91aafd3740 1 -- 172.21.15.102:0/3780471955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 msgr2=0x561f6c612190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.381+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c612190 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f919009bb70 tx=0x7f9190098500 comp rx=0 tx=0).stop 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.381+0000 7f91aafd3740 1 -- 172.21.15.102:0/3780471955 shutdown_connections 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.381+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561f6c680dd0 0x561f6c6831f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.382+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561f6c60a190 0x561f6c60f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.382+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561f6c5ff8d0 0x561f6c6126d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.382+0000 7f91aafd3740 1 --2- 172.21.15.102:0/3780471955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f6c5fefb0 0x561f6c612190 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.382+0000 7f91aafd3740 1 -- 172.21.15.102:0/3780471955 >> 172.21.15.102:0/3780471955 conn(0x561f6c5fb4c0 msgr2=0x561f6c600960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.382+0000 7f918aa00640 0 client.26839 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.382+0000 7f91aafd3740 1 -- 172.21.15.102:0/3780471955 shutdown_connections 2024-04-25T02:01:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.382+0000 7f91aafd3740 1 -- 172.21.15.102:0/3780471955 wait complete. 2024-04-25T02:01:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.382+0000 7f91aafd3740 20 client.26839 trim_cache size 0 max 16384 2024-04-25T02:01:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:01:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:01:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:01:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:01:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 -- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f91840a2e40 msgr2=0x7f91840a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f91840a2e40 0x7f91840a5300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9198006010 tx=0x7f919803d040 comp rx=0 tx=0).stop 2024-04-25T02:01:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 -- 172.21.15.102:0/954036008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91a0107c80 msgr2=0x7f91a019e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91a0107c80 0x7f91a019e3f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f919000e9f0 tx=0x7f919000edf0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 -- 172.21.15.102:0/954036008 shutdown_connections 2024-04-25T02:01:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f91840a2e40 0x7f91840a5300 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91a010dc10 0x7f91a01a2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91a0108600 0x7f91a019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 --2- 172.21.15.102:0/954036008 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91a0107c80 0x7f91a019e3f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 -- 172.21.15.102:0/954036008 >> 172.21.15.102:0/954036008 conn(0x7f91a0077520 msgr2=0x7f91a00784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 -- 172.21.15.102:0/954036008 shutdown_connections 2024-04-25T02:01:11.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:11.383+0000 7f91a6a00640 1 -- 172.21.15.102:0/954036008 wait complete. 2024-04-25T02:01:11.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:11 smithi102 ceph-mon[27796]: pgmap v180: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 3.4 MiB/s wr, 465 op/s 2024-04-25T02:01:11.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:11 smithi110 ceph-mon[32848]: pgmap v180: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 3.4 MiB/s wr, 465 op/s 2024-04-25T02:01:11.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:11 smithi193 ceph-mon[32981]: pgmap v180: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 3.4 MiB/s wr, 465 op/s 2024-04-25T02:01:13.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 320.0 s, 135.2 tps, lat 740.029 ms stddev 425.558 2024-04-25T02:01:13.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:13 smithi102 ceph-mon[27796]: pgmap v181: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 3.8 MiB/s wr, 542 op/s 2024-04-25T02:01:13.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:13 smithi110 ceph-mon[32848]: pgmap v181: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 3.8 MiB/s wr, 542 op/s 2024-04-25T02:01:13.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:13 smithi193 ceph-mon[32981]: pgmap v181: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 3.8 MiB/s wr, 542 op/s 2024-04-25T02:01:15.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:15 smithi102 ceph-mon[27796]: pgmap v182: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 1.8 MiB/s wr, 363 op/s 2024-04-25T02:01:15.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:15 smithi110 ceph-mon[32848]: pgmap v182: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 1.8 MiB/s wr, 363 op/s 2024-04-25T02:01:15.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:15 smithi193 ceph-mon[32981]: pgmap v182: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 1.8 MiB/s wr, 363 op/s 2024-04-25T02:01:17.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:17 smithi102 ceph-mon[27796]: pgmap v183: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.4 MiB/s wr, 290 op/s 2024-04-25T02:01:17.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:17 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:01:17.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:17 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:01:17.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:17 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:01:17.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:17 smithi110 ceph-mon[32848]: pgmap v183: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.4 MiB/s wr, 290 op/s 2024-04-25T02:01:17.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:17 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:01:17.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:17 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:01:17.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:17 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:01:17.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:17 smithi193 ceph-mon[32981]: pgmap v183: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.4 MiB/s wr, 290 op/s 2024-04-25T02:01:17.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:17 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:01:17.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:17 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:01:17.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:17 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:01:18.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 325.0 s, 123.8 tps, lat 764.995 ms stddev 471.350 2024-04-25T02:01:19.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:19 smithi102 ceph-mon[27796]: pgmap v184: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 2.3 MiB/s wr, 393 op/s 2024-04-25T02:01:19.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:19 smithi110 ceph-mon[32848]: pgmap v184: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 2.3 MiB/s wr, 393 op/s 2024-04-25T02:01:19.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:19 smithi193 ceph-mon[32981]: pgmap v184: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 2.3 MiB/s wr, 393 op/s 2024-04-25T02:01:21.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:21 smithi102 ceph-mon[27796]: pgmap v185: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.4 MiB/s wr, 209 op/s 2024-04-25T02:01:21.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:21 smithi110 ceph-mon[32848]: pgmap v185: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.4 MiB/s wr, 209 op/s 2024-04-25T02:01:21.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:21 smithi193 ceph-mon[32981]: pgmap v185: 97 pgs: 97 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.4 MiB/s wr, 209 op/s 2024-04-25T02:01:23.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 330.0 s, 1319.0 tps, lat 83.495 ms stddev 212.689 2024-04-25T02:01:23.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:23 smithi102 ceph-mon[27796]: pgmap v186: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 MiB/s wr, 985 op/s 2024-04-25T02:01:23.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:23 smithi110 ceph-mon[32848]: pgmap v186: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 MiB/s wr, 985 op/s 2024-04-25T02:01:23.508 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:23 smithi193 ceph-mon[32981]: pgmap v186: 97 pgs: 97 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 MiB/s wr, 985 op/s 2024-04-25T02:01:24.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.570+0000 7fb471a00640 1 -- 172.21.15.102:0/2229128397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb46c102110 msgr2=0x7fb46c10e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:24.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.570+0000 7fb471a00640 1 --2- 172.21.15.102:0/2229128397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb46c102110 0x7fb46c10e620 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb464009ca0 tx=0x7fb46402a8c0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.570+0000 7fb471a00640 1 -- 172.21.15.102:0/2229128397 shutdown_connections 2024-04-25T02:01:24.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.570+0000 7fb471a00640 1 --2- 172.21.15.102:0/2229128397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb46c102110 0x7fb46c10e620 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.570+0000 7fb471a00640 1 --2- 172.21.15.102:0/2229128397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb46c101750 0x7fb46c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.570+0000 7fb471a00640 1 --2- 172.21.15.102:0/2229128397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c107760 0x7fb46c107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.570+0000 7fb471a00640 1 -- 172.21.15.102:0/2229128397 >> 172.21.15.102:0/2229128397 conn(0x7fb46c078bd0 msgr2=0x7fb46c078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:24.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.571+0000 7fb471a00640 1 -- 172.21.15.102:0/2229128397 shutdown_connections 2024-04-25T02:01:24.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.571+0000 7fb471a00640 1 -- 172.21.15.102:0/2229128397 wait complete. 2024-04-25T02:01:24.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.572+0000 7fb471a00640 1 Processor -- start 2024-04-25T02:01:24.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.573+0000 7fb471a00640 1 -- start start 2024-04-25T02:01:24.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.573+0000 7fb471a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c101750 0x7fb46c10e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.573+0000 7fb471a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb46c102110 0x7fb46c10b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.573+0000 7fb471a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb46c107760 0x7fb46c10bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb471a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c075630 con 0x7fb46c101750 2024-04-25T02:01:24.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb471a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c0754b0 con 0x7fb46c107760 2024-04-25T02:01:24.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb471a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c0757b0 con 0x7fb46c102110 2024-04-25T02:01:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c101750 0x7fb46c10e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb46c102110 0x7fb46c10b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb46c102110 0x7fb46c10b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34694/0 (socket says 172.21.15.102:34694) 2024-04-25T02:01:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46a000640 1 -- 172.21.15.102:0/2877726194 learned_addr learned my addr 172.21.15.102:0/2877726194 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46b400640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb46c107760 0x7fb46c10bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46a000640 1 -- 172.21.15.102:0/2877726194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb46c107760 msgr2=0x7fb46c10bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46a000640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb46c107760 0x7fb46c10bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46a000640 1 -- 172.21.15.102:0/2877726194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c101750 msgr2=0x7fb46c10e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46a000640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c101750 0x7fb46c10e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.574+0000 7fb46a000640 1 -- 172.21.15.102:0/2877726194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46c1aa890 con 0x7fb46c102110 2024-04-25T02:01:24.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.575+0000 7fb46a000640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb46c102110 0x7fb46c10b640 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb45800e9f0 tx=0x7fb45800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:24.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.575+0000 7fb463400640 1 -- 172.21.15.102:0/2877726194 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45800cd80 con 0x7fb46c102110 2024-04-25T02:01:24.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.575+0000 7fb463400640 1 -- 172.21.15.102:0/2877726194 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb458005500 con 0x7fb46c102110 2024-04-25T02:01:24.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.575+0000 7fb463400640 1 -- 172.21.15.102:0/2877726194 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb458010690 con 0x7fb46c102110 2024-04-25T02:01:24.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.575+0000 7fb471a00640 1 -- 172.21.15.102:0/2877726194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb46c1aaac0 con 0x7fb46c102110 2024-04-25T02:01:24.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.576+0000 7fb471a00640 1 -- 172.21.15.102:0/2877726194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb46c1aaf90 con 0x7fb46c102110 2024-04-25T02:01:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.581+0000 7fb474522740 1 Processor -- start 2024-04-25T02:01:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.582+0000 7fb463400640 1 -- 172.21.15.102:0/2877726194 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb4580046c0 con 0x7fb46c102110 2024-04-25T02:01:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.582+0000 7fb474522740 1 -- start start 2024-04-25T02:01:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.582+0000 7fb474522740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x5592289931e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.582+0000 7fb474522740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.582+0000 7fb474522740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 0x5592289a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.582+0000 7fb474522740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5592288f6ab0 con 0x55922899dfe0 2024-04-25T02:01:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.582+0000 7fb46b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x5592289931e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb46b400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x5592289931e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33614/0 (socket says 172.21.15.102:33614) 2024-04-25T02:01:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb46b400640 1 -- 172.21.15.102:0/4071578123 learned_addr learned my addr 172.21.15.102:0/4071578123 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:24.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb474522740 1 -- 172.21.15.102:0/4071578123 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55922889f640 con 0x559228992e00 2024-04-25T02:01:24.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb474522740 1 -- 172.21.15.102:0/4071578123 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5592288f2a70 con 0x559228993720 2024-04-25T02:01:24.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb46aa00640 1 --2- 172.21.15.102:0/4071578123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb46a000640 1 --2- 172.21.15.102:0/4071578123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 0x5592289a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb46b400640 1 -- 172.21.15.102:0/4071578123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 msgr2=0x55922899daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb46b400640 1 --2- 172.21.15.102:0/4071578123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb46b400640 1 -- 172.21.15.102:0/4071578123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 msgr2=0x5592289a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb46b400640 1 --2- 172.21.15.102:0/4071578123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 0x5592289a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.583+0000 7fb46b400640 1 -- 172.21.15.102:0/4071578123 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55922860b210 con 0x559228992e00 2024-04-25T02:01:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.584+0000 7fb46b400640 1 --2- 172.21.15.102:0/4071578123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x5592289931e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb464009ca0 tx=0x7fb464005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e21f759c764a8d2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.584+0000 7fb457e00640 1 -- 172.21.15.102:0/4071578123 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464031da0 con 0x559228992e00 2024-04-25T02:01:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.584+0000 7fb457e00640 1 -- 172.21.15.102:0/4071578123 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb464036030 con 0x559228992e00 2024-04-25T02:01:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.584+0000 7fb457e00640 1 -- 172.21.15.102:0/4071578123 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464031740 con 0x559228992e00 2024-04-25T02:01:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.585+0000 7fb474522740 1 -- 172.21.15.102:0/4071578123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 msgr2=0x5592289931e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.585+0000 7fb474522740 1 --2- 172.21.15.102:0/4071578123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x5592289931e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb464009ca0 tx=0x7fb464005fa0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.585+0000 7fb474522740 1 -- 172.21.15.102:0/4071578123 shutdown_connections 2024-04-25T02:01:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.585+0000 7fb474522740 1 --2- 172.21.15.102:0/4071578123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 0x5592289a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.585+0000 7fb474522740 1 --2- 172.21.15.102:0/4071578123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.585+0000 7fb474522740 1 --2- 172.21.15.102:0/4071578123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x5592289931e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.585+0000 7fb474522740 1 -- 172.21.15.102:0/4071578123 >> 172.21.15.102:0/4071578123 conn(0x55922898f310 msgr2=0x55922898f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.585+0000 7fb474522740 1 -- 172.21.15.102:0/4071578123 shutdown_connections 2024-04-25T02:01:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.586+0000 7fb474522740 1 -- 172.21.15.102:0/4071578123 wait complete. 2024-04-25T02:01:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.586+0000 7fb474522740 1 -- start start 2024-04-25T02:01:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.586+0000 7fb474522740 1 Processor -- start 2024-04-25T02:01:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb474522740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x55922899af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb474522740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb474522740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 0x55922899b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb474522740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5592289a0a30 con 0x55922899dfe0 2024-04-25T02:01:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb474522740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5592288b2990 con 0x559228992e00 2024-04-25T02:01:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb474522740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5592289a0bb0 con 0x559228993720 2024-04-25T02:01:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb463400640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4500a2e90 0x7fb4500a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb474522740 10 client.0 _ll_register_callbacks cb 0x5592288b4950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb46aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb46aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34724/0 (socket says 172.21.15.102:34724) 2024-04-25T02:01:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb46aa00640 1 -- 172.21.15.102:0/2539241300 learned_addr learned my addr 172.21.15.102:0/2539241300 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb463400640 1 -- 172.21.15.102:0/2877726194 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb4580cf7e0 con 0x7fb46c102110 2024-04-25T02:01:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.587+0000 7fb46b400640 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x55922899af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb46b400640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4500a2e90 0x7fb4500a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb46a000640 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 0x55922899b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb46aa00640 1 -- 172.21.15.102:0/2539241300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 msgr2=0x55922899af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb46aa00640 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x55922899af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb46aa00640 1 -- 172.21.15.102:0/2539241300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 msgr2=0x55922899b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb46aa00640 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 0x55922899b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb46aa00640 1 -- 172.21.15.102:0/2539241300 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592286d7620 con 0x559228993720 2024-04-25T02:01:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb46a000640 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 0x55922899b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb46aa00640 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899b4a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00bde0 tx=0x7fb45c0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8f5b03e70385c9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb454c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.588+0000 7fb454c00640 1 -- 172.21.15.102:0/2539241300 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c012860 con 0x559228993720 2024-04-25T02:01:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.589+0000 7fb454c00640 1 -- 172.21.15.102:0/2539241300 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb45c012a00 con 0x559228993720 2024-04-25T02:01:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.589+0000 7fb474522740 1 -- 172.21.15.102:0/2539241300 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb464036030 con 0x559228993720 2024-04-25T02:01:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.589+0000 7fb46b400640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4500a2e90 0x7fb4500a5350 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb464009620 tx=0x7fb464005df0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.589+0000 7fb454c00640 1 -- 172.21.15.102:0/2539241300 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb45c012c00 con 0x559228993720 2024-04-25T02:01:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.589+0000 7fb474522740 10 client.26848 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.589+0000 7fb474522740 1 -- 172.21.15.102:0/2539241300 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5592289ac9d0 con 0x559228993720 2024-04-25T02:01:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.590+0000 7fb454c00640 1 -- 172.21.15.102:0/2539241300 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb464036030 con 0x559228993720 2024-04-25T02:01:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.590+0000 7fb454c00640 1 -- 172.21.15.102:0/2539241300 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fb45c0265b0 con 0x559228993720 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.590+0000 7fb474522740 10 client.26848 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.590+0000 7fb474522740 10 client.26848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.590+0000 7fb474522740 20 client.26848 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.590+0000 7fb474522740 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5592289ae340 0x5592289b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.590+0000 7fb474522740 4 client.26848 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.590+0000 7fb474522740 1 -- 172.21.15.102:0/2539241300 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559228903ff0 con 0x5592289ae340 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.591+0000 7fb46a000640 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5592289ae340 0x5592289b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.591+0000 7fb46a000640 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5592289ae340 0x5592289b0760 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f63967a48cd422 server_cookie=847d63f8766962c7 in_seq=0 out_seq=0 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.591+0000 7fb454c00640 10 client.26848 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.592+0000 7fb454c00640 1 -- 172.21.15.102:0/2539241300 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559228903ff0 con 0x5592289ae340 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.592+0000 7fb454c00640 10 client.26848 handle_command_reply: tid=0 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.592+0000 7fb474522740 1 client.26848 shutdown 2024-04-25T02:01:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 -- 172.21.15.102:0/2539241300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 msgr2=0x55922899b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899b4a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00bde0 tx=0x7fb45c0117b0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 -- 172.21.15.102:0/2539241300 shutdown_connections 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5592289ae340 0x5592289b0760 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55922899dfe0 0x55922899b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x55922899b4a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 --2- 172.21.15.102:0/2539241300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559228992e00 0x55922899af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 -- 172.21.15.102:0/2539241300 >> 172.21.15.102:0/2539241300 conn(0x55922898f310 msgr2=0x559228995040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb454c00640 0 client.26848 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 -- 172.21.15.102:0/2539241300 shutdown_connections 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.593+0000 7fb474522740 1 -- 172.21.15.102:0/2539241300 wait complete. 2024-04-25T02:01:24.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.594+0000 7fb474522740 20 client.26848 trim_cache size 0 max 16384 2024-04-25T02:01:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.617+0000 7fb474522740 1 Processor -- start 2024-04-25T02:01:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.617+0000 7fb474522740 1 -- start start 2024-04-25T02:01:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.617+0000 7fb474522740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228992e00 0x55922898ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.617+0000 7fb474522740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228993720 0x559228995040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb474522740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x55922899b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb46aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228992e00 0x55922898ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb474522740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c1b60e0 con 0x559228993720 2024-04-25T02:01:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb474522740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c1b3950 con 0x55922899dfe0 2024-04-25T02:01:24.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb474522740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c1b17a0 con 0x559228992e00 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb46aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228992e00 0x55922898ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34732/0 (socket says 172.21.15.102:34732) 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb46aa00640 1 -- 172.21.15.102:0/2900432346 learned_addr learned my addr 172.21.15.102:0/2900432346 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb46b400640 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x55922899b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb46a000640 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228993720 0x559228995040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb46aa00640 1 -- 172.21.15.102:0/2900432346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 msgr2=0x55922899b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb46aa00640 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x55922899b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb46aa00640 1 -- 172.21.15.102:0/2900432346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228993720 msgr2=0x559228995040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.618+0000 7fb46aa00640 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228993720 0x559228995040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.619+0000 7fb46aa00640 1 -- 172.21.15.102:0/2900432346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559228989530 con 0x559228992e00 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.619+0000 7fb46a000640 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228993720 0x559228995040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.619+0000 7fb46aa00640 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228992e00 0x55922898ec00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb45c000c00 tx=0x7fb45c00ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb218c9da2ab0159 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.619+0000 7fb457400640 1 -- 172.21.15.102:0/2900432346 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c035070 con 0x559228992e00 2024-04-25T02:01:24.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.619+0000 7fb457400640 1 -- 172.21.15.102:0/2900432346 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb45c0057f0 con 0x559228992e00 2024-04-25T02:01:24.625 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.619+0000 7fb457400640 1 -- 172.21.15.102:0/2900432346 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45c02c070 con 0x559228992e00 2024-04-25T02:01:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.620+0000 7fb474522740 1 -- 172.21.15.102:0/2900432346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228992e00 msgr2=0x55922898ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.620+0000 7fb474522740 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228992e00 0x55922898ec00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb45c000c00 tx=0x7fb45c00ba50 comp rx=0 tx=0).stop 2024-04-25T02:01:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.620+0000 7fb474522740 1 -- 172.21.15.102:0/2900432346 shutdown_connections 2024-04-25T02:01:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.620+0000 7fb474522740 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x55922899b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.620+0000 7fb474522740 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228993720 0x559228995040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.620+0000 7fb474522740 1 --2- 172.21.15.102:0/2900432346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228992e00 0x55922898ec00 secure :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb45c000c00 tx=0x7fb45c00ba50 comp rx=0 tx=0).stop 2024-04-25T02:01:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.620+0000 7fb474522740 1 -- 172.21.15.102:0/2900432346 >> 172.21.15.102:0/2900432346 conn(0x55922898f310 msgr2=0x55922899c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.621+0000 7fb474522740 1 -- 172.21.15.102:0/2900432346 shutdown_connections 2024-04-25T02:01:24.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.621+0000 7fb474522740 1 -- 172.21.15.102:0/2900432346 wait complete. 2024-04-25T02:01:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.622+0000 7fb474522740 1 -- start start 2024-04-25T02:01:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.622+0000 7fb474522740 1 Processor -- start 2024-04-25T02:01:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.623+0000 7fb474522740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x559228997fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.623+0000 7fb474522740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x5592289984e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.623+0000 7fb474522740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228998b30 0x559228a13880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.623+0000 7fb46aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x559228997fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.623+0000 7fb46aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x559228997fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:34740/0 (socket says 172.21.15.102:34740) 2024-04-25T02:01:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.623+0000 7fb46a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x5592289984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.623+0000 7fb46a000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x5592289984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33650/0 (socket says 172.21.15.102:33650) 2024-04-25T02:01:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c1b1e10 con 0x559228998b30 2024-04-25T02:01:24.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c1b3280 con 0x55922899dfe0 2024-04-25T02:01:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb46c1b24e0 con 0x559228993720 2024-04-25T02:01:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb46aa00640 1 -- 172.21.15.102:0/2975280979 learned_addr learned my addr 172.21.15.102:0/2975280979 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb474522740 10 client.0 _ll_register_callbacks cb 0x7fb45c00b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:24.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb46b400640 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228998b30 0x559228a13880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb46a000640 1 -- 172.21.15.102:0/2975280979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 msgr2=0x559228997fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb46a000640 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x559228997fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb46a000640 1 -- 172.21.15.102:0/2975280979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228998b30 msgr2=0x559228a13880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb46a000640 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228998b30 0x559228a13880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.624+0000 7fb46a000640 1 -- 172.21.15.102:0/2975280979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559228a14010 con 0x55922899dfe0 2024-04-25T02:01:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.625+0000 7fb46a000640 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x5592289984e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb4580ccf00 tx=0x7fb458004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be8d7c197b69bc85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.625+0000 7fb454c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.625+0000 7fb454c00640 1 -- 172.21.15.102:0/2975280979 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb45809c3d0 con 0x55922899dfe0 2024-04-25T02:01:24.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.625+0000 7fb454c00640 1 -- 172.21.15.102:0/2975280979 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb458098340 con 0x55922899dfe0 2024-04-25T02:01:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.625+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb45c0057f0 con 0x55922899dfe0 2024-04-25T02:01:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.625+0000 7fb46aa00640 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x559228997fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.625+0000 7fb46b400640 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228998b30 0x559228a13880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.625+0000 7fb454c00640 1 -- 172.21.15.102:0/2975280979 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb45809bcd0 con 0x55922899dfe0 2024-04-25T02:01:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.626+0000 7fb474522740 10 client.26900 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.626+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x559228a15320 con 0x55922899dfe0 2024-04-25T02:01:24.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.626+0000 7fb454c00640 1 -- 172.21.15.102:0/2975280979 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb45c0057f0 con 0x55922899dfe0 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.626+0000 7fb454c00640 1 -- 172.21.15.102:0/2975280979 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fb4580b3070 con 0x55922899dfe0 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.626+0000 7fb474522740 10 client.26900 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.626+0000 7fb474522740 10 client.26900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.626+0000 7fb474522740 20 client.26900 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.626+0000 7fb474522740 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559228a15b00 0x559228a17f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.627+0000 7fb474522740 4 client.26900 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.627+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5592288fe340 con 0x559228a15b00 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.627+0000 7fb46aa00640 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559228a15b00 0x559228a17f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.628+0000 7fb454c00640 10 client.26900 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:24.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.628+0000 7fb46aa00640 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559228a15b00 0x559228a17f20 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57469b905c054964 server_cookie=2ab739c69b87f5fe in_seq=0 out_seq=0 2024-04-25T02:01:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.640+0000 7fb454c00640 1 -- 172.21.15.102:0/2975280979 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5592288fe340 con 0x559228a15b00 2024-04-25T02:01:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb454c00640 10 client.26900 handle_command_reply: tid=0 2024-04-25T02:01:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 client.26900 shutdown 2024-04-25T02:01:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 msgr2=0x5592289984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x5592289984e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb4580ccf00 tx=0x7fb458004a40 comp rx=0 tx=0).stop 2024-04-25T02:01:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 shutdown_connections 2024-04-25T02:01:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559228a15b00 0x559228a17f20 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559228998b30 0x559228a13880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55922899dfe0 0x5592289984e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb454c00640 0 client.26900 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 --2- 172.21.15.102:0/2975280979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559228993720 0x559228997fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 >> 172.21.15.102:0/2975280979 conn(0x55922898f310 msgr2=0x559228994ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.641+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 shutdown_connections 2024-04-25T02:01:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.642+0000 7fb474522740 1 -- 172.21.15.102:0/2975280979 wait complete. 2024-04-25T02:01:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.642+0000 7fb474522740 20 client.26900 trim_cache size 0 max 16384 2024-04-25T02:01:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:01:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:01:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:01:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:01:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 -- 172.21.15.102:0/2877726194 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4500a2e90 msgr2=0x7fb4500a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4500a2e90 0x7fb4500a5350 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb464009620 tx=0x7fb464005df0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 -- 172.21.15.102:0/2877726194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb46c102110 msgr2=0x7fb46c10b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb46c102110 0x7fb46c10b640 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb45800e9f0 tx=0x7fb45800edf0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 -- 172.21.15.102:0/2877726194 shutdown_connections 2024-04-25T02:01:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4500a2e90 0x7fb4500a5350 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.648 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb46c107760 0x7fb46c10bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb46c102110 0x7fb46c10b640 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 --2- 172.21.15.102:0/2877726194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb46c101750 0x7fb46c10e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.643+0000 7fb471a00640 1 -- 172.21.15.102:0/2877726194 >> 172.21.15.102:0/2877726194 conn(0x7fb46c078bd0 msgr2=0x7fb46c0fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.644+0000 7fb471a00640 1 -- 172.21.15.102:0/2877726194 shutdown_connections 2024-04-25T02:01:24.649 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:24.644+0000 7fb471a00640 1 -- 172.21.15.102:0/2877726194 wait complete. 2024-04-25T02:01:25.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:25 smithi102 ceph-mon[27796]: pgmap v187: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 33 MiB/s wr, 1.27k op/s 2024-04-25T02:01:25.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:25 smithi110 ceph-mon[32848]: pgmap v187: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 33 MiB/s wr, 1.27k op/s 2024-04-25T02:01:25.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:25 smithi193 ceph-mon[32981]: pgmap v187: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 33 MiB/s wr, 1.27k op/s 2024-04-25T02:01:27.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:27 smithi102 ceph-mon[27796]: pgmap v188: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 33 MiB/s wr, 1.24k op/s 2024-04-25T02:01:27.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:27 smithi110 ceph-mon[32848]: pgmap v188: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 33 MiB/s wr, 1.24k op/s 2024-04-25T02:01:27.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:27 smithi193 ceph-mon[32981]: pgmap v188: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 33 MiB/s wr, 1.24k op/s 2024-04-25T02:01:28.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 335.0 s, 2796.5 tps, lat 36.260 ms stddev 44.848 2024-04-25T02:01:29.426 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:29 smithi102 ceph-mon[27796]: pgmap v189: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 MiB/s wr, 1.53k op/s 2024-04-25T02:01:29.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:29 smithi110 ceph-mon[32848]: pgmap v189: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 MiB/s wr, 1.53k op/s 2024-04-25T02:01:29.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:29 smithi193 ceph-mon[32981]: pgmap v189: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 MiB/s wr, 1.53k op/s 2024-04-25T02:01:31.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:31 smithi102 ceph-mon[27796]: pgmap v190: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 34 MiB/s wr, 1.43k op/s 2024-04-25T02:01:31.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:31 smithi110 ceph-mon[32848]: pgmap v190: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 34 MiB/s wr, 1.43k op/s 2024-04-25T02:01:31.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:31 smithi193 ceph-mon[32981]: pgmap v190: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 34 MiB/s wr, 1.43k op/s 2024-04-25T02:01:33.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 340.0 s, 2533.6 tps, lat 39.256 ms stddev 51.478 2024-04-25T02:01:33.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:33 smithi102 ceph-mon[27796]: pgmap v191: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 37 MiB/s wr, 1.68k op/s 2024-04-25T02:01:33.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:33 smithi110 ceph-mon[32848]: pgmap v191: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 37 MiB/s wr, 1.68k op/s 2024-04-25T02:01:33.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:33 smithi193 ceph-mon[32981]: pgmap v191: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 37 MiB/s wr, 1.68k op/s 2024-04-25T02:01:34.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T02:01:34.448 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T02:01:34.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.691+0000 7fb80e200640 1 -- 172.21.15.102:0/195450048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8081027a0 msgr2=0x7fb808102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:34.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.691+0000 7fb80e200640 1 --2- 172.21.15.102:0/195450048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8081027a0 0x7fb808102b80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009860 tx=0x7fb7fc02a3d0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.692+0000 7fb80e200640 1 -- 172.21.15.102:0/195450048 shutdown_connections 2024-04-25T02:01:34.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.692+0000 7fb80e200640 1 --2- 172.21.15.102:0/195450048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb80810bbc0 0x7fb80810e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.692+0000 7fb80e200640 1 --2- 172.21.15.102:0/195450048 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8081030c0 0x7fb80810b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.692+0000 7fb80e200640 1 --2- 172.21.15.102:0/195450048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8081027a0 0x7fb808102b80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.692+0000 7fb80e200640 1 -- 172.21.15.102:0/195450048 >> 172.21.15.102:0/195450048 conn(0x7fb8080fbfc0 msgr2=0x7fb8080fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:34.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.692+0000 7fb80e200640 1 -- 172.21.15.102:0/195450048 shutdown_connections 2024-04-25T02:01:34.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.692+0000 7fb80e200640 1 -- 172.21.15.102:0/195450048 wait complete. 2024-04-25T02:01:34.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.693+0000 7fb80e200640 1 Processor -- start 2024-04-25T02:01:34.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.693+0000 7fb80e200640 1 -- start start 2024-04-25T02:01:34.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.693+0000 7fb80e200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8081027a0 0x7fb80819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.693+0000 7fb80e200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8081030c0 0x7fb80819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.693+0000 7fb80e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb80810bbc0 0x7fb80819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.693+0000 7fb80e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb808111a10 con 0x7fb80810bbc0 2024-04-25T02:01:34.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.693+0000 7fb80e200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb808111890 con 0x7fb8081030c0 2024-04-25T02:01:34.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.693+0000 7fb80e200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb808111b90 con 0x7fb8081027a0 2024-04-25T02:01:34.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb807e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8081030c0 0x7fb80819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb807e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8081030c0 0x7fb80819a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32870/0 (socket says 172.21.15.102:32870) 2024-04-25T02:01:34.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb807e00640 1 -- 172.21.15.102:0/3632740925 learned_addr learned my addr 172.21.15.102:0/3632740925 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:34.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb80ce00640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8081027a0 0x7fb80819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb80d800640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb80810bbc0 0x7fb80819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb807e00640 1 -- 172.21.15.102:0/3632740925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8081027a0 msgr2=0x7fb80819a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:34.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb807e00640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8081027a0 0x7fb80819a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb807e00640 1 -- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb80810bbc0 msgr2=0x7fb80819e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:34.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb807e00640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb80810bbc0 0x7fb80819e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb807e00640 1 -- 172.21.15.102:0/3632740925 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80819f0d0 con 0x7fb8081030c0 2024-04-25T02:01:34.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.694+0000 7fb80d800640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb80810bbc0 0x7fb80819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:34.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.695+0000 7fb807e00640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8081030c0 0x7fb80819a620 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400e790 tx=0x7fb7f400ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:34.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.695+0000 7fb805600640 1 -- 172.21.15.102:0/3632740925 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f400eea0 con 0x7fb8081030c0 2024-04-25T02:01:34.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.695+0000 7fb805600640 1 -- 172.21.15.102:0/3632740925 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7f4005500 con 0x7fb8081030c0 2024-04-25T02:01:34.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.695+0000 7fb805600640 1 -- 172.21.15.102:0/3632740925 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4010430 con 0x7fb8081030c0 2024-04-25T02:01:34.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.695+0000 7fb80e200640 1 -- 172.21.15.102:0/3632740925 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80819f3c0 con 0x7fb8081030c0 2024-04-25T02:01:34.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.695+0000 7fb80e200640 1 -- 172.21.15.102:0/3632740925 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8081a6c50 con 0x7fb8081030c0 2024-04-25T02:01:34.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.700+0000 7fb805600640 1 -- 172.21.15.102:0/3632740925 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb7f4005020 con 0x7fb8081030c0 2024-04-25T02:01:34.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.701+0000 7fb812616740 1 Processor -- start 2024-04-25T02:01:34.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb812616740 1 -- start start 2024-04-25T02:01:34.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb805600640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb7f00a2d80 0x7fb7f00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb805600640 1 -- 172.21.15.102:0/3632740925 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb7f40cf630 con 0x7fb8081030c0 2024-04-25T02:01:34.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb812616740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c81fec5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb812616740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fecb00 0x560c81ff6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb812616740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81ff71d0 0x560c81ff95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb812616740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560c81f50560 con 0x560c81fecb00 2024-04-25T02:01:34.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb812616740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560c81ef8930 con 0x560c81ff71d0 2024-04-25T02:01:34.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb812616740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560c81f4c520 con 0x560c81fec1e0 2024-04-25T02:01:34.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb80d800640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb7f00a2d80 0x7fb7f00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb80d800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81ff71d0 0x560c81ff95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb80ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c81fec5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb807e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fecb00 0x560c81ff6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb80d800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81ff71d0 0x560c81ff95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32874/0 (socket says 172.21.15.102:32874) 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb80ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c81fec5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42396/0 (socket says 172.21.15.102:42396) 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.702+0000 7fb80d800640 1 -- 172.21.15.102:0/2909811815 learned_addr learned my addr 172.21.15.102:0/2909811815 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb80ce00640 1 -- 172.21.15.102:0/2909811815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81ff71d0 msgr2=0x560c81ff95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb80ce00640 1 --2- 172.21.15.102:0/2909811815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81ff71d0 0x560c81ff95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb80ce00640 1 -- 172.21.15.102:0/2909811815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fecb00 msgr2=0x560c81ff6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:34.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb80ce00640 1 --2- 172.21.15.102:0/2909811815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fecb00 0x560c81ff6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb80ce00640 1 -- 172.21.15.102:0/2909811815 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c81d2e7d0 con 0x560c81fec1e0 2024-04-25T02:01:34.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb80d800640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb7f00a2d80 0x7fb7f00a5240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb800009950 tx=0x7fb800009e40 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:34.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb80ce00640 1 --2- 172.21.15.102:0/2909811815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c81fec5c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009830 tx=0x7fb7fc030ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76a9fa30c82b2721 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:34.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb7fa000640 1 -- 172.21.15.102:0/2909811815 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc004030 con 0x560c81fec1e0 2024-04-25T02:01:34.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb7fa000640 1 -- 172.21.15.102:0/2909811815 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7fc041030 con 0x560c81fec1e0 2024-04-25T02:01:34.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.703+0000 7fb7fa000640 1 -- 172.21.15.102:0/2909811815 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc0053c0 con 0x560c81fec1e0 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.704+0000 7fb812616740 1 -- 172.21.15.102:0/2909811815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 msgr2=0x560c81fec5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.704+0000 7fb812616740 1 --2- 172.21.15.102:0/2909811815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c81fec5c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009830 tx=0x7fb7fc030ab0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.704+0000 7fb812616740 1 -- 172.21.15.102:0/2909811815 shutdown_connections 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.704+0000 7fb812616740 1 --2- 172.21.15.102:0/2909811815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81ff71d0 0x560c81ff95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.704+0000 7fb812616740 1 --2- 172.21.15.102:0/2909811815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fecb00 0x560c81ff6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.704+0000 7fb812616740 1 --2- 172.21.15.102:0/2909811815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c81fec5c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.704+0000 7fb812616740 1 -- 172.21.15.102:0/2909811815 >> 172.21.15.102:0/2909811815 conn(0x560c81fe8500 msgr2=0x560c81fe88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.705+0000 7fb812616740 1 -- 172.21.15.102:0/2909811815 shutdown_connections 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.705+0000 7fb812616740 1 -- 172.21.15.102:0/2909811815 wait complete. 2024-04-25T02:01:34.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.705+0000 7fb812616740 1 -- start start 2024-04-25T02:01:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.706+0000 7fb812616740 1 Processor -- start 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb812616740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fec1e0 0x560c81ff40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb812616740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb812616740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81ff71d0 0x560c81ff4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb812616740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560c81ff9f60 con 0x560c81fec1e0 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb812616740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560c81f0bd90 con 0x560c81fecb00 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb812616740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560c81ffa0e0 con 0x560c81ff71d0 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb812616740 10 client.0 _ll_register_callbacks cb 0x560c81f0b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb807e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb807e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32882/0 (socket says 172.21.15.102:32882) 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb807e00640 1 -- 172.21.15.102:0/1526328655 learned_addr learned my addr 172.21.15.102:0/1526328655 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb80d800640 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81ff71d0 0x560c81ff4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.707+0000 7fb80ce00640 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fec1e0 0x560c81ff40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.708+0000 7fb807e00640 1 -- 172.21.15.102:0/1526328655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81ff71d0 msgr2=0x560c81ff4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:34.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.708+0000 7fb807e00640 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81ff71d0 0x560c81ff4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.708+0000 7fb807e00640 1 -- 172.21.15.102:0/1526328655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fec1e0 msgr2=0x560c81ff40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:34.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.708+0000 7fb807e00640 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fec1e0 0x560c81ff40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.708+0000 7fb807e00640 1 -- 172.21.15.102:0/1526328655 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c81c643f0 con 0x560c81fecb00 2024-04-25T02:01:34.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.708+0000 7fb80d800640 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81ff71d0 0x560c81ff4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:34.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.708+0000 7fb80ce00640 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fec1e0 0x560c81ff40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:34.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.709+0000 7fb807e00640 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff4620 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb7f40cce10 tx=0x7fb7f409bb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d21cb2786ba2a858 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:34.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.709+0000 7fb7eea00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:34.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.710+0000 7fb7eea00640 1 -- 172.21.15.102:0/1526328655 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4005310 con 0x560c81fecb00 2024-04-25T02:01:34.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.710+0000 7fb7eea00640 1 -- 172.21.15.102:0/1526328655 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb7f40981f0 con 0x560c81fecb00 2024-04-25T02:01:34.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.710+0000 7fb812616740 1 -- 172.21.15.102:0/1526328655 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7fc041030 con 0x560c81fecb00 2024-04-25T02:01:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.711+0000 7fb7eea00640 1 -- 172.21.15.102:0/1526328655 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb7f40a4360 con 0x560c81fecb00 2024-04-25T02:01:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.711+0000 7fb812616740 10 client.26912 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.711+0000 7fb812616740 1 -- 172.21.15.102:0/1526328655 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560c81c240f0 con 0x560c81fecb00 2024-04-25T02:01:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.711+0000 7fb7eea00640 1 -- 172.21.15.102:0/1526328655 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7fc041030 con 0x560c81fecb00 2024-04-25T02:01:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.712+0000 7fb7eea00640 1 -- 172.21.15.102:0/1526328655 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fb7f409afd0 con 0x560c81fecb00 2024-04-25T02:01:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.712+0000 7fb812616740 10 client.26912 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:34.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.712+0000 7fb812616740 10 client.26912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.712+0000 7fb812616740 20 client.26912 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.712+0000 7fb812616740 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560c8200fcd0 0x560c82012140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.712+0000 7fb812616740 4 client.26912 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:34.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.712+0000 7fb812616740 1 -- 172.21.15.102:0/1526328655 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c81f63d90 con 0x560c8200fcd0 2024-04-25T02:01:34.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.712+0000 7fb80ce00640 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560c8200fcd0 0x560c82012140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.713+0000 7fb80ce00640 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560c8200fcd0 0x560c82012140 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d0bdcbf5765237c server_cookie=e053b143abe9fd0 in_seq=0 out_seq=0 2024-04-25T02:01:34.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.713+0000 7fb7eea00640 10 client.26912 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:34.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb7eea00640 1 -- 172.21.15.102:0/1526328655 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560c81f63d90 con 0x560c8200fcd0 2024-04-25T02:01:34.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb7eea00640 10 client.26912 handle_command_reply: tid=0 2024-04-25T02:01:34.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 client.26912 shutdown 2024-04-25T02:01:34.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 -- 172.21.15.102:0/1526328655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 msgr2=0x560c81ff4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:34.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff4620 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb7f40cce10 tx=0x7fb7f409bb80 comp rx=0 tx=0).stop 2024-04-25T02:01:34.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 -- 172.21.15.102:0/1526328655 shutdown_connections 2024-04-25T02:01:34.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560c8200fcd0 0x560c82012140 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81ff71d0 0x560c81ff4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb7eea00640 0 client.26912 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:34.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff4620 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 --2- 172.21.15.102:0/1526328655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81fec1e0 0x560c81ff40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 -- 172.21.15.102:0/1526328655 >> 172.21.15.102:0/1526328655 conn(0x560c81fe8500 msgr2=0x560c81fe7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:34.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 -- 172.21.15.102:0/1526328655 shutdown_connections 2024-04-25T02:01:34.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.714+0000 7fb812616740 1 -- 172.21.15.102:0/1526328655 wait complete. 2024-04-25T02:01:34.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.715+0000 7fb812616740 20 client.26912 trim_cache size 0 max 16384 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb812616740 1 Processor -- start 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb812616740 1 -- start start 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb812616740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fec1e0 0x560c81fe7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb812616740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fecb00 0x560c81ff47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb812616740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 0x560c82017a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb812616740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb808071d10 con 0x560c81ff71d0 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb812616740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb80806f580 con 0x560c81fec1e0 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb812616740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb80806d410 con 0x560c81fecb00 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb80ce00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fec1e0 0x560c81fe7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb80d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 0x560c82017a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb80ce00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fec1e0 0x560c81fe7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32888/0 (socket says 172.21.15.102:32888) 2024-04-25T02:01:34.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb80ce00640 1 -- 172.21.15.102:0/3967582259 learned_addr learned my addr 172.21.15.102:0/3967582259 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:34.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.743+0000 7fb807e00640 1 --2- 172.21.15.102:0/3967582259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fecb00 0x560c81ff47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.744+0000 7fb80ce00640 1 -- 172.21.15.102:0/3967582259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fecb00 msgr2=0x560c81ff47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:34.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.744+0000 7fb80ce00640 1 --2- 172.21.15.102:0/3967582259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fecb00 0x560c81ff47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.744+0000 7fb80ce00640 1 -- 172.21.15.102:0/3967582259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 msgr2=0x560c82017a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:34.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.744+0000 7fb80ce00640 1 --2- 172.21.15.102:0/3967582259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 0x560c82017a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.744+0000 7fb80ce00640 1 -- 172.21.15.102:0/3967582259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c81fe2ca0 con 0x560c81fec1e0 2024-04-25T02:01:34.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.744+0000 7fb80ce00640 1 --2- 172.21.15.102:0/3967582259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fec1e0 0x560c81fe7f80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc02a8f0 tx=0x7fb7fc02ab50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a8f6ae1b162357 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:34.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.744+0000 7fb7f9600640 1 -- 172.21.15.102:0/3967582259 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc004050 con 0x560c81fec1e0 2024-04-25T02:01:34.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.745+0000 7fb7f9600640 1 -- 172.21.15.102:0/3967582259 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7fc041070 con 0x560c81fec1e0 2024-04-25T02:01:34.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.745+0000 7fb7f9600640 1 -- 172.21.15.102:0/3967582259 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc055360 con 0x560c81fec1e0 2024-04-25T02:01:34.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.746+0000 7fb812616740 1 -- 172.21.15.102:0/3967582259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fec1e0 msgr2=0x560c81fe7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:34.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.746+0000 7fb812616740 1 --2- 172.21.15.102:0/3967582259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fec1e0 0x560c81fe7f80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc02a8f0 tx=0x7fb7fc02ab50 comp rx=0 tx=0).stop 2024-04-25T02:01:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.746+0000 7fb812616740 1 -- 172.21.15.102:0/3967582259 shutdown_connections 2024-04-25T02:01:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.746+0000 7fb812616740 1 --2- 172.21.15.102:0/3967582259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 0x560c82017a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.746+0000 7fb812616740 1 --2- 172.21.15.102:0/3967582259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fecb00 0x560c81ff47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.746+0000 7fb812616740 1 --2- 172.21.15.102:0/3967582259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fec1e0 0x560c81fe7f80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.746+0000 7fb812616740 1 -- 172.21.15.102:0/3967582259 >> 172.21.15.102:0/3967582259 conn(0x560c81fe8500 msgr2=0x560c81ff40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.746+0000 7fb812616740 1 -- 172.21.15.102:0/3967582259 shutdown_connections 2024-04-25T02:01:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.746+0000 7fb812616740 1 -- 172.21.15.102:0/3967582259 wait complete. 2024-04-25T02:01:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.747+0000 7fb812616740 1 -- start start 2024-04-25T02:01:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.747+0000 7fb812616740 1 Processor -- start 2024-04-25T02:01:34.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb812616740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c82007e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb812616740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb812616740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 0x560c81ff0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb807e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb807e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32900/0 (socket says 172.21.15.102:32900) 2024-04-25T02:01:34.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb80ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c82007e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb80ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c82007e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42418/0 (socket says 172.21.15.102:42418) 2024-04-25T02:01:34.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb807e00640 1 -- 172.21.15.102:0/1793866597 learned_addr learned my addr 172.21.15.102:0/1793866597 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:34.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb80806da60 con 0x560c81ff71d0 2024-04-25T02:01:34.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.748+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb80806eeb0 con 0x560c81fecb00 2024-04-25T02:01:34.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.749+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb80806e110 con 0x560c81fec1e0 2024-04-25T02:01:34.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.749+0000 7fb80d800640 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 0x560c81ff0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.749+0000 7fb812616740 10 client.0 _ll_register_callbacks cb 0x7fb7f4005720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.749+0000 7fb807e00640 1 -- 172.21.15.102:0/1793866597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 msgr2=0x560c82007e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.749+0000 7fb807e00640 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c82007e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.749+0000 7fb807e00640 1 -- 172.21.15.102:0/1793866597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 msgr2=0x560c81ff0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.749+0000 7fb807e00640 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 0x560c81ff0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.750+0000 7fb807e00640 1 -- 172.21.15.102:0/1793866597 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c81ff1590 con 0x560c81fecb00 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.750+0000 7fb807e00640 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff0990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb7f409b830 tx=0x7fb7f4004fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77e1923f950c8811 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.750+0000 7fb7eea00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.750+0000 7fb7eea00640 1 -- 172.21.15.102:0/1793866597 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4005310 con 0x560c81fecb00 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.750+0000 7fb7eea00640 1 -- 172.21.15.102:0/1793866597 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7fb7f40a4500 con 0x560c81fecb00 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.750+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7fc041070 con 0x560c81fecb00 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.750+0000 7fb7eea00640 1 -- 172.21.15.102:0/1793866597 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb7f409acd0 con 0x560c81fecb00 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.750+0000 7fb812616740 10 client.26918 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.750+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560c82076160 con 0x560c81fecb00 2024-04-25T02:01:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.751+0000 7fb7eea00640 1 -- 172.21.15.102:0/1793866597 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7fc041070 con 0x560c81fecb00 2024-04-25T02:01:34.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.751+0000 7fb7eea00640 1 -- 172.21.15.102:0/1793866597 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fb7f40ba310 con 0x560c81fecb00 2024-04-25T02:01:34.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.751+0000 7fb812616740 10 client.26918 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:34.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.751+0000 7fb812616740 10 client.26918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:34.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.751+0000 7fb812616740 20 client.26918 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:34.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.751+0000 7fb812616740 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560c82076780 0x560c82078ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:34.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.751+0000 7fb812616740 4 client.26918 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T02:01:34.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.751+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560c81f5daa0 con 0x560c82076780 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.751+0000 7fb80ce00640 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560c82076780 0x560c82078ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb80ce00640 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560c82076780 0x560c82078ba0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d56c8bdb3cf52b5c server_cookie=5c6065168799215b in_seq=0 out_seq=0 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb7eea00640 10 client.26918 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb7eea00640 1 -- 172.21.15.102:0/1793866597 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560c81f5daa0 con 0x560c82076780 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb7eea00640 10 client.26918 handle_command_reply: tid=0 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb812616740 1 client.26918 shutdown 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 msgr2=0x560c81ff0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb812616740 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff0990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb7f409b830 tx=0x7fb7f4004fb0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 shutdown_connections 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb812616740 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560c82076780 0x560c82078ba0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb812616740 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560c81ff71d0 0x560c81ff0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb812616740 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560c81fecb00 0x560c81ff0990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb812616740 1 --2- 172.21.15.102:0/1793866597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c81fec1e0 0x560c82007e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 >> 172.21.15.102:0/1793866597 conn(0x560c81fe8500 msgr2=0x560c81ff4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:34.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.752+0000 7fb7eea00640 0 client.26918 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:34.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.753+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 shutdown_connections 2024-04-25T02:01:34.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.753+0000 7fb812616740 1 -- 172.21.15.102:0/1793866597 wait complete. 2024-04-25T02:01:34.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.753+0000 7fb812616740 20 client.26918 trim_cache size 0 max 16384 2024-04-25T02:01:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:01:34.759 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T02:01:34.759 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T02:01:34.759 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:01:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.753+0000 7fb80e200640 1 -- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb7f00a2d80 msgr2=0x7fb7f00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.753+0000 7fb80e200640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb7f00a2d80 0x7fb7f00a5240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb800009950 tx=0x7fb800009e40 comp rx=0 tx=0).stop 2024-04-25T02:01:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.753+0000 7fb80e200640 1 -- 172.21.15.102:0/3632740925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8081030c0 msgr2=0x7fb80819a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.753+0000 7fb80e200640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8081030c0 0x7fb80819a620 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb7f400e790 tx=0x7fb7f400ec50 comp rx=0 tx=0).stop 2024-04-25T02:01:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.754+0000 7fb80e200640 1 -- 172.21.15.102:0/3632740925 shutdown_connections 2024-04-25T02:01:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.754+0000 7fb80e200640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb7f00a2d80 0x7fb7f00a5240 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.754+0000 7fb80e200640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb80810bbc0 0x7fb80819e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.754+0000 7fb80e200640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8081030c0 0x7fb80819a620 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.754+0000 7fb80e200640 1 --2- 172.21.15.102:0/3632740925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8081027a0 0x7fb80819a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:34.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.754+0000 7fb80e200640 1 -- 172.21.15.102:0/3632740925 >> 172.21.15.102:0/3632740925 conn(0x7fb8080fbfc0 msgr2=0x7fb808100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:34.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.754+0000 7fb80e200640 1 -- 172.21.15.102:0/3632740925 shutdown_connections 2024-04-25T02:01:34.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:34.754+0000 7fb80e200640 1 -- 172.21.15.102:0/3632740925 wait complete. 2024-04-25T02:01:34.779 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T02:01:34.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T02:01:35.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.021+0000 7ff213c00640 1 -- 172.21.15.102:0/3360296339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c109920 msgr2=0x7ff20c109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.021+0000 7ff213c00640 1 --2- 172.21.15.102:0/3360296339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c109920 0x7ff20c109d00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff200009930 tx=0x7ff20002a460 comp rx=0 tx=0).stop 2024-04-25T02:01:35.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.021+0000 7ff213c00640 1 -- 172.21.15.102:0/3360296339 shutdown_connections 2024-04-25T02:01:35.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.021+0000 7ff213c00640 1 --2- 172.21.15.102:0/3360296339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff20c1042d0 0x7ff20c1107e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.021+0000 7ff213c00640 1 --2- 172.21.15.102:0/3360296339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff20c103910 0x7ff20c103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.021+0000 7ff213c00640 1 --2- 172.21.15.102:0/3360296339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c109920 0x7ff20c109d00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.021+0000 7ff213c00640 1 -- 172.21.15.102:0/3360296339 >> 172.21.15.102:0/3360296339 conn(0x7ff20c0ff400 msgr2=0x7ff20c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:35.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.022+0000 7ff213c00640 1 -- 172.21.15.102:0/3360296339 shutdown_connections 2024-04-25T02:01:35.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.022+0000 7ff213c00640 1 -- 172.21.15.102:0/3360296339 wait complete. 2024-04-25T02:01:35.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.023+0000 7ff213c00640 1 Processor -- start 2024-04-25T02:01:35.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.023+0000 7ff213c00640 1 -- start start 2024-04-25T02:01:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.023+0000 7ff213c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c103910 0x7ff20c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff213c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff20c1042d0 0x7ff20c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff213c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff20c109920 0x7ff20c19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff213c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff20c113da0 con 0x7ff20c109920 2024-04-25T02:01:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff213c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff20c113c20 con 0x7ff20c1042d0 2024-04-25T02:01:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff213c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff20c113f20 con 0x7ff20c103910 2024-04-25T02:01:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff211200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c103910 0x7ff20c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff20be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff20c1042d0 0x7ff20c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff211200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c103910 0x7ff20c19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42438/0 (socket says 172.21.15.102:42438) 2024-04-25T02:01:35.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff211200640 1 -- 172.21.15.102:0/3412839119 learned_addr learned my addr 172.21.15.102:0/3412839119 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:35.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff211c00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff20c109920 0x7ff20c19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff211200640 1 -- 172.21.15.102:0/3412839119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff20c1042d0 msgr2=0x7ff20c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff211200640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff20c1042d0 0x7ff20c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff211200640 1 -- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff20c109920 msgr2=0x7ff20c19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff211200640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff20c109920 0x7ff20c19eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff211200640 1 -- 172.21.15.102:0/3412839119 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff20c19f230 con 0x7ff20c103910 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.024+0000 7ff20be00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff20c1042d0 0x7ff20c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.025+0000 7ff211200640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c103910 0x7ff20c19a240 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff20002a980 tx=0x7ff200003c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.025+0000 7ff209600640 1 -- 172.21.15.102:0/3412839119 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff20003f070 con 0x7ff20c103910 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.025+0000 7ff209600640 1 -- 172.21.15.102:0/3412839119 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff20003b460 con 0x7ff20c103910 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.025+0000 7ff209600640 1 -- 172.21.15.102:0/3412839119 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff200035030 con 0x7ff20c103910 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.025+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff20c19f4c0 con 0x7ff20c103910 2024-04-25T02:01:35.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.025+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff20c1a6db0 con 0x7ff20c103910 2024-04-25T02:01:35.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.026+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff20c105050 con 0x7ff20c103910 2024-04-25T02:01:35.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.030+0000 7ff209600640 1 -- 172.21.15.102:0/3412839119 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff200042070 con 0x7ff20c103910 2024-04-25T02:01:35.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.033+0000 7ff209600640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1f40a2d80 0x7ff1f40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.033+0000 7ff209600640 1 -- 172.21.15.102:0/3412839119 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7ff2000efdb0 con 0x7ff20c103910 2024-04-25T02:01:35.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.034+0000 7ff20be00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1f40a2d80 0x7ff1f40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.035+0000 7ff20be00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1f40a2d80 0x7ff1f40a5240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff20c1019d0 tx=0x7ff1f800a2d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:35.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.037+0000 7ff209600640 1 -- 172.21.15.102:0/3412839119 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff2000f4070 con 0x7ff20c103910 2024-04-25T02:01:35.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:35 smithi102 ceph-mon[27796]: pgmap v192: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 17 MiB/s wr, 1.03k op/s 2024-04-25T02:01:35.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:35 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:01:35.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.430+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff20c10be80 con 0x7ff20c103910 2024-04-25T02:01:35.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.431+0000 7ff209600640 1 -- 172.21.15.102:0/3412839119 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11902 (secure 0 0 0) 0x7ff2000bc9a0 con 0x7ff20c103910 2024-04-25T02:01:35.435 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:01:35.436 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T02:00:31.796906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24533,24541,24587]},"id":1}]} 2024-04-25T02:01:35.436 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 34 2024-04-25T02:01:35.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1f40a2d80 msgr2=0x7ff1f40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1f40a2d80 0x7ff1f40a5240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff20c1019d0 tx=0x7ff1f800a2d0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c103910 msgr2=0x7ff20c19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c103910 0x7ff20c19a240 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff20002a980 tx=0x7ff200003c80 comp rx=0 tx=0).stop 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 shutdown_connections 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff1f40a2d80 0x7ff1f40a5240 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff20c109920 0x7ff20c19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff20c1042d0 0x7ff20c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 --2- 172.21.15.102:0/3412839119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff20c103910 0x7ff20c19a240 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.434+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 >> 172.21.15.102:0/3412839119 conn(0x7ff20c0ff400 msgr2=0x7ff20c0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.435+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 shutdown_connections 2024-04-25T02:01:35.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.435+0000 7ff213c00640 1 -- 172.21.15.102:0/3412839119 wait complete. 2024-04-25T02:01:35.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T02:01:35.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:35 smithi110 ceph-mon[32848]: pgmap v192: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 17 MiB/s wr, 1.03k op/s 2024-04-25T02:01:35.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:35 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:01:35.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:35 smithi193 ceph-mon[32981]: pgmap v192: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 17 MiB/s wr, 1.03k op/s 2024-04-25T02:01:35.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:35 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:01:35.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.775+0000 7f27e6400640 1 -- 172.21.15.102:0/2923826351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e01125b0 msgr2=0x7f27e01149a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.775+0000 7f27e6400640 1 --2- 172.21.15.102:0/2923826351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e01125b0 0x7f27e01149a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009cd0 tx=0x7f27d802a8e0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.775+0000 7f27e6400640 1 -- 172.21.15.102:0/2923826351 shutdown_connections 2024-04-25T02:01:35.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.775+0000 7f27e6400640 1 --2- 172.21.15.102:0/2923826351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e01125b0 0x7f27e01149a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.775+0000 7f27e6400640 1 --2- 172.21.15.102:0/2923826351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e0076590 0x7f27e0112070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.775+0000 7f27e6400640 1 --2- 172.21.15.102:0/2923826351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27e0075c70 0x7f27e0076050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.775+0000 7f27e6400640 1 -- 172.21.15.102:0/2923826351 >> 172.21.15.102:0/2923826351 conn(0x7f27e00ff3a0 msgr2=0x7f27e01017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:35.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.775+0000 7f27e6400640 1 -- 172.21.15.102:0/2923826351 shutdown_connections 2024-04-25T02:01:35.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.775+0000 7f27e6400640 1 -- 172.21.15.102:0/2923826351 wait complete. 2024-04-25T02:01:35.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.776+0000 7f27e6400640 1 Processor -- start 2024-04-25T02:01:35.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.776+0000 7f27e6400640 1 -- start start 2024-04-25T02:01:35.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.777+0000 7f27e6400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e0075c70 0x7f27e019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.777+0000 7f27e6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e0076590 0x7f27e019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.777+0000 7f27e6400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27e01125b0 0x7f27e01a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.777+0000 7f27e6400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f27e0118140 con 0x7f27e0076590 2024-04-25T02:01:35.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.777+0000 7f27e6400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f27e0117fc0 con 0x7f27e0075c70 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.777+0000 7f27e6400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01182c0 con 0x7f27e01125b0 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.777+0000 7f27df400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e0075c70 0x7f27e019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.777+0000 7f27df400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e0075c70 0x7f27e019e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32936/0 (socket says 172.21.15.102:32936) 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.777+0000 7f27df400640 1 -- 172.21.15.102:0/3651680112 learned_addr learned my addr 172.21.15.102:0/3651680112 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27dfe00640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27e01125b0 0x7f27e01a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27df400640 1 -- 172.21.15.102:0/3651680112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27e01125b0 msgr2=0x7f27e01a2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27dea00640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e0076590 0x7f27e019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27df400640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27e01125b0 0x7f27e01a2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27df400640 1 -- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e0076590 msgr2=0x7f27e019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27df400640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e0076590 0x7f27e019eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27df400640 1 -- 172.21.15.102:0/3651680112 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e01a35b0 con 0x7f27e0075c70 2024-04-25T02:01:35.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27dea00640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e0076590 0x7f27e019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:35.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27dfe00640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27e01125b0 0x7f27e01a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:35.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.778+0000 7f27df400640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e0075c70 0x7f27e019e5c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f27d000e9f0 tx=0x7f27d000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:35.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.779+0000 7f27d7e00640 1 -- 172.21.15.102:0/3651680112 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d000cd80 con 0x7f27e0075c70 2024-04-25T02:01:35.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.779+0000 7f27d7e00640 1 -- 172.21.15.102:0/3651680112 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27d0005500 con 0x7f27e0075c70 2024-04-25T02:01:35.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.779+0000 7f27d7e00640 1 -- 172.21.15.102:0/3651680112 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d0010690 con 0x7f27e0075c70 2024-04-25T02:01:35.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.779+0000 7f27e6400640 1 -- 172.21.15.102:0/3651680112 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e01a38a0 con 0x7f27e0075c70 2024-04-25T02:01:35.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.779+0000 7f27e6400640 1 -- 172.21.15.102:0/3651680112 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e01ab130 con 0x7f27e0075c70 2024-04-25T02:01:35.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.783+0000 7f27d7e00640 1 -- 172.21.15.102:0/3651680112 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f27d00046c0 con 0x7f27e0075c70 2024-04-25T02:01:35.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.784+0000 7f27e922f740 1 Processor -- start 2024-04-25T02:01:35.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27e922f740 1 -- start start 2024-04-25T02:01:35.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27d7e00640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f27c40a2ed0 0x7f27c40a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27e922f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2889a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27d7e00640 1 -- 172.21.15.102:0/3651680112 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f27d00cf900 con 0x7f27e0075c70 2024-04-25T02:01:35.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27e922f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b288ee0 0x55a15b293070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27dfe00640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f27c40a2ed0 0x7f27c40a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27e922f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 0x55a15b2959a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27e922f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a15b1ed4e0 con 0x55a15b2935b0 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27e922f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a15b195a70 con 0x55a15b288ee0 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.785+0000 7f27e922f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a15b1e94a0 con 0x55a15b2885c0 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2889a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42454/0 (socket says 172.21.15.102:42454) 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27df400640 1 -- 172.21.15.102:0/3319253523 learned_addr learned my addr 172.21.15.102:0/3319253523 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27dea00640 1 --2- 172.21.15.102:0/3319253523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b288ee0 0x55a15b293070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27df400640 1 -- 172.21.15.102:0/3319253523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b288ee0 msgr2=0x55a15b293070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27df400640 1 --2- 172.21.15.102:0/3319253523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b288ee0 0x55a15b293070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27df400640 1 -- 172.21.15.102:0/3319253523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 msgr2=0x55a15b2959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27df400640 1 --2- 172.21.15.102:0/3319253523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 0x55a15b2959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27df400640 1 -- 172.21.15.102:0/3319253523 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a15af013f0 con 0x55a15b2885c0 2024-04-25T02:01:35.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27dea00640 1 --2- 172.21.15.102:0/3319253523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b288ee0 0x55a15b293070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:35.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.786+0000 7f27df400640 1 --2- 172.21.15.102:0/3319253523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2889a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f27d0098250 tx=0x7f27d00056b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a519476ba8fe72d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:35.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.787+0000 7f27d4c00640 1 -- 172.21.15.102:0/3319253523 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d009bdf0 con 0x55a15b2885c0 2024-04-25T02:01:35.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.787+0000 7f27d4c00640 1 -- 172.21.15.102:0/3319253523 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27d00a01a0 con 0x55a15b2885c0 2024-04-25T02:01:35.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.787+0000 7f27d4c00640 1 -- 172.21.15.102:0/3319253523 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d009b4a0 con 0x55a15b2885c0 2024-04-25T02:01:35.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.787+0000 7f27dfe00640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f27c40a2ed0 0x7f27c40a5390 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f27d8005e30 tx=0x7f27d8031590 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:35.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.788+0000 7f27e922f740 1 -- 172.21.15.102:0/3319253523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 msgr2=0x55a15b2889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.788+0000 7f27e922f740 1 --2- 172.21.15.102:0/3319253523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2889a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f27d0098250 tx=0x7f27d00056b0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.789+0000 7f27e922f740 1 -- 172.21.15.102:0/3319253523 shutdown_connections 2024-04-25T02:01:35.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.789+0000 7f27e922f740 1 --2- 172.21.15.102:0/3319253523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 0x55a15b2959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.789+0000 7f27e922f740 1 --2- 172.21.15.102:0/3319253523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b288ee0 0x55a15b293070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.789+0000 7f27e922f740 1 --2- 172.21.15.102:0/3319253523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2889a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.789+0000 7f27e922f740 1 -- 172.21.15.102:0/3319253523 >> 172.21.15.102:0/3319253523 conn(0x55a15b2848e0 msgr2=0x55a15b284cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.790+0000 7f27e922f740 1 -- 172.21.15.102:0/3319253523 shutdown_connections 2024-04-25T02:01:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.790+0000 7f27e922f740 1 -- 172.21.15.102:0/3319253523 wait complete. 2024-04-25T02:01:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.790+0000 7f27e922f740 1 -- start start 2024-04-25T02:01:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.791+0000 7f27e922f740 1 Processor -- start 2024-04-25T02:01:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.791+0000 7f27e922f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b2905d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.791+0000 7f27e922f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b288ee0 0x55a15b290b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27e922f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2935b0 0x55a15b291050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27e922f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a15b296340 con 0x55a15b288ee0 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27e922f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a15b1a8ec0 con 0x55a15b2885c0 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27e922f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a15b2964c0 con 0x55a15b2935b0 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27e922f740 10 client.0 _ll_register_callbacks cb 0x55a15b1e53f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27df400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b2905d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27df400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b2905d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:32958/0 (socket says 172.21.15.102:32958) 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27df400640 1 -- 172.21.15.102:0/916112590 learned_addr learned my addr 172.21.15.102:0/916112590 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27dea00640 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b288ee0 0x55a15b290b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27dfe00640 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2935b0 0x55a15b291050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27df400640 1 -- 172.21.15.102:0/916112590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2935b0 msgr2=0x55a15b291050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27df400640 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2935b0 0x55a15b291050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27df400640 1 -- 172.21.15.102:0/916112590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b288ee0 msgr2=0x55a15b290b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27df400640 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b288ee0 0x55a15b290b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.792+0000 7f27df400640 1 -- 172.21.15.102:0/916112590 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a15afcb7d0 con 0x55a15b2885c0 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.793+0000 7f27df400640 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b2905d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f27d0005820 tx=0x7f27d0009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f951240196e4fece server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.793+0000 7f27c9600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.793+0000 7f27c9600640 1 -- 172.21.15.102:0/916112590 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d009bdf0 con 0x55a15b2885c0 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.793+0000 7f27c9600640 1 -- 172.21.15.102:0/916112590 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f27d009a0f0 con 0x55a15b2885c0 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.793+0000 7f27e922f740 1 -- 172.21.15.102:0/916112590 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27d00a01a0 con 0x55a15b2885c0 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.794+0000 7f27c9600640 1 -- 172.21.15.102:0/916112590 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f27d0098460 con 0x55a15b2885c0 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.794+0000 7f27e922f740 10 client.26933 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.794+0000 7f27e922f740 1 -- 172.21.15.102:0/916112590 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a15b2a2670 con 0x55a15b2885c0 2024-04-25T02:01:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.794+0000 7f27c9600640 1 -- 172.21.15.102:0/916112590 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27d00a01a0 con 0x55a15b2885c0 2024-04-25T02:01:35.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.794+0000 7f27c9600640 1 -- 172.21.15.102:0/916112590 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f27d0086ab0 con 0x55a15b2885c0 2024-04-25T02:01:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.794+0000 7f27e922f740 10 client.26933 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.794+0000 7f27e922f740 10 client.26933 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T02:01:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.794+0000 7f27e922f740 20 client.26933 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.795+0000 7f27e922f740 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55a15b2a39c0 0x55a15b2a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.795+0000 7f27e922f740 4 client.26933 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.795+0000 7f27e922f740 1 -- 172.21.15.102:0/916112590 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a15b1faa20 con 0x55a15b2a39c0 2024-04-25T02:01:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.795+0000 7f27dea00640 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55a15b2a39c0 0x55a15b2a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.796+0000 7f27dea00640 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55a15b2a39c0 0x55a15b2a5db0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=efac70e285a050f4 server_cookie=cc141d4d48d24114 in_seq=0 out_seq=0 2024-04-25T02:01:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.796+0000 7f27c9600640 10 client.26933 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T02:01:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.797+0000 7f27c9600640 1 -- 172.21.15.102:0/916112590 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a15b1faa20 con 0x55a15b2a39c0 2024-04-25T02:01:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.797+0000 7f27c9600640 10 client.26933 handle_command_reply: tid=0 2024-04-25T02:01:35.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.797+0000 7f27e922f740 1 client.26933 shutdown 2024-04-25T02:01:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.798+0000 7f27e922f740 1 -- 172.21.15.102:0/916112590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 msgr2=0x55a15b2905d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.798+0000 7f27e922f740 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b2905d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f27d0005820 tx=0x7f27d0009310 comp rx=0 tx=0).stop 2024-04-25T02:01:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.798+0000 7f27e922f740 1 -- 172.21.15.102:0/916112590 shutdown_connections 2024-04-25T02:01:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.798+0000 7f27e922f740 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2935b0 0x55a15b291050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.798+0000 7f27e922f740 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b288ee0 0x55a15b290b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.798+0000 7f27e922f740 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55a15b2a39c0 0x55a15b2a5db0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.798+0000 7f27c9600640 0 client.26933 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T02:01:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.798+0000 7f27e922f740 1 --2- 172.21.15.102:0/916112590 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b2905d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.798+0000 7f27e922f740 1 -- 172.21.15.102:0/916112590 >> 172.21.15.102:0/916112590 conn(0x55a15b2848e0 msgr2=0x55a15b28a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.799+0000 7f27e922f740 1 -- 172.21.15.102:0/916112590 shutdown_connections 2024-04-25T02:01:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.799+0000 7f27e922f740 1 -- 172.21.15.102:0/916112590 wait complete. 2024-04-25T02:01:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.799+0000 7f27e922f740 20 client.26933 trim_cache size 0 max 16384 2024-04-25T02:01:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.824+0000 7f27e922f740 1 Processor -- start 2024-04-25T02:01:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.824+0000 7f27e922f740 1 -- start start 2024-04-25T02:01:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.824+0000 7f27e922f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b290e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.824+0000 7f27e922f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b288ee0 0x55a15b2a8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.824+0000 7f27e922f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 0x55a15b2a9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27e922f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01b6470 con 0x55a15b2935b0 2024-04-25T02:01:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27e922f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01b3ce0 con 0x55a15b2885c0 2024-04-25T02:01:35.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27e922f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01b1b30 con 0x55a15b288ee0 2024-04-25T02:01:35.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27dea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b288ee0 0x55a15b2a8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27dea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b288ee0 0x55a15b2a8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42470/0 (socket says 172.21.15.102:42470) 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27dea00640 1 -- 172.21.15.102:0/148114226 learned_addr learned my addr 172.21.15.102:0/148114226 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27df400640 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b290e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27dea00640 1 -- 172.21.15.102:0/148114226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 msgr2=0x55a15b290e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27dea00640 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b290e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27dea00640 1 -- 172.21.15.102:0/148114226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 msgr2=0x55a15b2a9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.825+0000 7f27dfe00640 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 0x55a15b2a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.826+0000 7f27dea00640 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 0x55a15b2a9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.826+0000 7f27dea00640 1 -- 172.21.15.102:0/148114226 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a15b289850 con 0x55a15b288ee0 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.826+0000 7f27dfe00640 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 0x55a15b2a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.826+0000 7f27dea00640 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b288ee0 0x55a15b2a8dd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005fd0 tx=0x7f27cc00d6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=954703bafd7229b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.826+0000 7f27cbe00640 1 -- 172.21.15.102:0/148114226 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0129a0 con 0x55a15b288ee0 2024-04-25T02:01:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.826+0000 7f27cbe00640 1 -- 172.21.15.102:0/148114226 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27cc01f070 con 0x55a15b288ee0 2024-04-25T02:01:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.826+0000 7f27cbe00640 1 -- 172.21.15.102:0/148114226 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27cc019750 con 0x55a15b288ee0 2024-04-25T02:01:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.827+0000 7f27e922f740 1 -- 172.21.15.102:0/148114226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b288ee0 msgr2=0x55a15b2a8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.827+0000 7f27e922f740 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b288ee0 0x55a15b2a8dd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f27cc005fd0 tx=0x7f27cc00d6d0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.828+0000 7f27e922f740 1 -- 172.21.15.102:0/148114226 shutdown_connections 2024-04-25T02:01:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.828+0000 7f27e922f740 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b2935b0 0x55a15b2a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.828+0000 7f27e922f740 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b288ee0 0x55a15b2a8dd0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.828+0000 7f27e922f740 1 --2- 172.21.15.102:0/148114226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2885c0 0x55a15b290e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.828+0000 7f27e922f740 1 -- 172.21.15.102:0/148114226 >> 172.21.15.102:0/148114226 conn(0x55a15b2848e0 msgr2=0x55a15b29efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.828+0000 7f27e922f740 1 -- 172.21.15.102:0/148114226 shutdown_connections 2024-04-25T02:01:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.828+0000 7f27e922f740 1 -- 172.21.15.102:0/148114226 wait complete. 2024-04-25T02:01:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.829+0000 7f27e922f740 1 -- start start 2024-04-25T02:01:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.829+0000 7f27e922f740 1 Processor -- start 2024-04-25T02:01:35.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27e922f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2a6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27e922f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b288ee0 0x55a15b2a6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27e922f740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2935b0 0x55a15b2a73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27e922f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01b21a0 con 0x55a15b288ee0 2024-04-25T02:01:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27e922f740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01b3610 con 0x55a15b2935b0 2024-04-25T02:01:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27e922f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f27e01b2870 con 0x55a15b2885c0 2024-04-25T02:01:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27e922f740 10 client.0 _ll_register_callbacks cb 0x7f27d009b9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2a6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42476/0 (socket says 172.21.15.102:42476) 2024-04-25T02:01:35.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27df400640 1 -- 172.21.15.102:0/3512271310 learned_addr learned my addr 172.21.15.102:0/3512271310 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.830+0000 7f27dfe00640 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2935b0 0x55a15b2a73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.831+0000 7f27df400640 1 -- 172.21.15.102:0/3512271310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2935b0 msgr2=0x55a15b2a73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.831+0000 7f27df400640 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2935b0 0x55a15b2a73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.831+0000 7f27df400640 1 -- 172.21.15.102:0/3512271310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b288ee0 msgr2=0x55a15b2a6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.831+0000 7f27df400640 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b288ee0 0x55a15b2a6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.831+0000 7f27df400640 1 -- 172.21.15.102:0/3512271310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a15b308d00 con 0x55a15b2885c0 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.831+0000 7f27dfe00640 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2935b0 0x55a15b2a73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.831+0000 7f27df400640 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2a6970 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f27d00987c0 tx=0x7f27d00cd090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2d33545c552fcdb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.832+0000 7f27c9600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.832+0000 7f27c9600640 1 -- 172.21.15.102:0/3512271310 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d009bdf0 con 0x55a15b2885c0 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.832+0000 7f27c9600640 1 -- 172.21.15.102:0/3512271310 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f27d009b240 con 0x55a15b2885c0 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.832+0000 7f27e922f740 1 -- 172.21.15.102:0/3512271310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27cc01f070 con 0x55a15b2885c0 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.832+0000 7f27c9600640 1 -- 172.21.15.102:0/3512271310 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f27d0098600 con 0x55a15b2885c0 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.832+0000 7f27e922f740 10 client.26893 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.832+0000 7f27e922f740 1 -- 172.21.15.102:0/3512271310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55a15b309f50 con 0x55a15b2885c0 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.832+0000 7f27c9600640 1 -- 172.21.15.102:0/3512271310 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27cc01f070 con 0x55a15b2885c0 2024-04-25T02:01:35.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.833+0000 7f27c9600640 1 -- 172.21.15.102:0/3512271310 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f27d00a1330 con 0x55a15b2885c0 2024-04-25T02:01:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.833+0000 7f27e922f740 10 client.26893 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.833+0000 7f27e922f740 10 client.26893 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T02:01:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.833+0000 7f27e922f740 20 client.26893 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.833+0000 7f27e922f740 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55a15b2a39c0 0x55a15b30c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.833+0000 7f27e922f740 4 client.26893 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:01:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.833+0000 7f27e922f740 1 -- 172.21.15.102:0/3512271310 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a15b1f4d70 con 0x55a15b2a39c0 2024-04-25T02:01:35.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.833+0000 7f27dea00640 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55a15b2a39c0 0x55a15b30c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27dea00640 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55a15b2a39c0 0x55a15b30c4a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b2bb496800b63496 server_cookie=d091e0d0f03955d8 in_seq=0 out_seq=0 2024-04-25T02:01:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27c9600640 10 client.26893 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T02:01:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27c9600640 1 -- 172.21.15.102:0/3512271310 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a15b1f4d70 con 0x55a15b2a39c0 2024-04-25T02:01:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27c9600640 10 client.26893 handle_command_reply: tid=0 2024-04-25T02:01:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27e922f740 1 client.26893 shutdown 2024-04-25T02:01:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27e922f740 1 -- 172.21.15.102:0/3512271310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 msgr2=0x55a15b2a6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27e922f740 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2a6970 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f27d00987c0 tx=0x7f27d00cd090 comp rx=0 tx=0).stop 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27e922f740 1 -- 172.21.15.102:0/3512271310 shutdown_connections 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27e922f740 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x55a15b2a39c0 0x55a15b30c4a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27e922f740 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a15b2935b0 0x55a15b2a73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27e922f740 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a15b288ee0 0x55a15b2a6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27e922f740 1 --2- 172.21.15.102:0/3512271310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a15b2885c0 0x55a15b2a6970 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.834+0000 7f27e922f740 1 -- 172.21.15.102:0/3512271310 >> 172.21.15.102:0/3512271310 conn(0x55a15b2848e0 msgr2=0x55a15b289d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.835+0000 7f27c9600640 0 client.26893 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.835+0000 7f27e922f740 1 -- 172.21.15.102:0/3512271310 shutdown_connections 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.835+0000 7f27e922f740 1 -- 172.21.15.102:0/3512271310 wait complete. 2024-04-25T02:01:35.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.836+0000 7f27e922f740 20 client.26893 trim_cache size 0 max 16384 2024-04-25T02:01:35.842 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.836+0000 7f27e6400640 1 -- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f27c40a2ed0 msgr2=0x7f27c40a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.836+0000 7f27e6400640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f27c40a2ed0 0x7f27c40a5390 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f27d8005e30 tx=0x7f27d8031590 comp rx=0 tx=0).stop 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.836+0000 7f27e6400640 1 -- 172.21.15.102:0/3651680112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e0075c70 msgr2=0x7f27e019e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.836+0000 7f27e6400640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e0075c70 0x7f27e019e5c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f27d000e9f0 tx=0x7f27d000edf0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.837+0000 7f27e6400640 1 -- 172.21.15.102:0/3651680112 shutdown_connections 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.837+0000 7f27e6400640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f27c40a2ed0 0x7f27c40a5390 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.837+0000 7f27e6400640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27e01125b0 0x7f27e01a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.837+0000 7f27e6400640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27e0076590 0x7f27e019eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.837+0000 7f27e6400640 1 --2- 172.21.15.102:0/3651680112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e0075c70 0x7f27e019e5c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.837+0000 7f27e6400640 1 -- 172.21.15.102:0/3651680112 >> 172.21.15.102:0/3651680112 conn(0x7f27e00ff3a0 msgr2=0x7f27e0101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:35.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.837+0000 7f27e6400640 1 -- 172.21.15.102:0/3651680112 shutdown_connections 2024-04-25T02:01:35.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:35.837+0000 7f27e6400640 1 -- 172.21.15.102:0/3651680112 wait complete. 2024-04-25T02:01:35.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T02:01:36.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.151+0000 7f40db200640 1 -- 172.21.15.102:0/65993173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d4108740 msgr2=0x7f40d4108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.151+0000 7f40db200640 1 --2- 172.21.15.102:0/65993173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d4108740 0x7f40d4108ba0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f40c0007380 tx=0x7f40c0029f80 comp rx=0 tx=0).stop 2024-04-25T02:01:36.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.151+0000 7f40db200640 1 -- 172.21.15.102:0/65993173 shutdown_connections 2024-04-25T02:01:36.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.151+0000 7f40db200640 1 --2- 172.21.15.102:0/65993173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d41090e0 0x7f40d4114f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.151+0000 7f40db200640 1 --2- 172.21.15.102:0/65993173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d4108740 0x7f40d4108ba0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.151+0000 7f40db200640 1 --2- 172.21.15.102:0/65993173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40d410d120 0x7f40d410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.151+0000 7f40db200640 1 -- 172.21.15.102:0/65993173 >> 172.21.15.102:0/65993173 conn(0x7f40d406ccb0 msgr2=0x7f40d406d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.152+0000 7f40db200640 1 -- 172.21.15.102:0/65993173 shutdown_connections 2024-04-25T02:01:36.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.152+0000 7f40db200640 1 -- 172.21.15.102:0/65993173 wait complete. 2024-04-25T02:01:36.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.154+0000 7f40db200640 1 Processor -- start 2024-04-25T02:01:36.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.154+0000 7f40db200640 1 -- start start 2024-04-25T02:01:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40db200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40d4108740 0x7f40d41a7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40db200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d41090e0 0x7f40d41a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40db200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d410d120 0x7f40d41a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40db200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41169a0 con 0x7f40d41090e0 2024-04-25T02:01:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40db200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40d4116820 con 0x7f40d410d120 2024-04-25T02:01:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40db200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40d4116b20 con 0x7f40d4108740 2024-04-25T02:01:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40d4108740 0x7f40d41a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40d3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d41090e0 0x7f40d41a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40d4108740 0x7f40d41a7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42498/0 (socket says 172.21.15.102:42498) 2024-04-25T02:01:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40d3e00640 1 -- 172.21.15.102:0/4180913731 learned_addr learned my addr 172.21.15.102:0/4180913731 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.155+0000 7f40d9200640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d410d120 0x7f40d41a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.157+0000 7f40d3e00640 1 -- 172.21.15.102:0/4180913731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d410d120 msgr2=0x7f40d41a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.157+0000 7f40d3e00640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d410d120 0x7f40d41a2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.157+0000 7f40d3e00640 1 -- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d41090e0 msgr2=0x7f40d41a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.157+0000 7f40d3e00640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d41090e0 0x7f40d41a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.157+0000 7f40d3e00640 1 -- 172.21.15.102:0/4180913731 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d41a32b0 con 0x7f40d4108740 2024-04-25T02:01:36.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.157+0000 7f40d3400640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d41090e0 0x7f40d41a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:36.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.160+0000 7f40d3e00640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40d4108740 0x7f40d41a7650 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f40c400ddc0 tx=0x7f40c400a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.161+0000 7f40d0c00640 1 -- 172.21.15.102:0/4180913731 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c401b070 con 0x7f40d4108740 2024-04-25T02:01:36.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.161+0000 7f40d0c00640 1 -- 172.21.15.102:0/4180913731 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c400be40 con 0x7f40d4108740 2024-04-25T02:01:36.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.162+0000 7f40d0c00640 1 -- 172.21.15.102:0/4180913731 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c401b070 con 0x7f40d4108740 2024-04-25T02:01:36.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.162+0000 7f40db200640 1 -- 172.21.15.102:0/4180913731 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d41a35a0 con 0x7f40d4108740 2024-04-25T02:01:36.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.162+0000 7f40db200640 1 -- 172.21.15.102:0/4180913731 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d41af5e0 con 0x7f40d4108740 2024-04-25T02:01:36.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.167+0000 7f40ddfaf740 1 Processor -- start 2024-04-25T02:01:36.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.168+0000 7f40ddfaf740 1 -- start start 2024-04-25T02:01:36.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.168+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 0x555aed4389a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.168+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed443070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.168+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed4459a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.168+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555aed39d4e0 con 0x555aed4385c0 2024-04-25T02:01:36.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.168+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555aed345a70 con 0x555aed438ee0 2024-04-25T02:01:36.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.168+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555aed3994a0 con 0x555aed4435b0 2024-04-25T02:01:36.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.168+0000 7f40d0c00640 1 -- 172.21.15.102:0/4180913731 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f40c401e070 con 0x7f40d4108740 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.168+0000 7f40d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed443070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.169+0000 7f40d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed443070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33006/0 (socket says 172.21.15.102:33006) 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.169+0000 7f40d3e00640 1 -- 172.21.15.102:0/3054363570 learned_addr learned my addr 172.21.15.102:0/3054363570 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.169+0000 7f40d3400640 1 --2- 172.21.15.102:0/3054363570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed4459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.169+0000 7f40d3e00640 1 -- 172.21.15.102:0/3054363570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 msgr2=0x555aed4459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.169+0000 7f40d3e00640 1 --2- 172.21.15.102:0/3054363570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed4459a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.169+0000 7f40d3e00640 1 -- 172.21.15.102:0/3054363570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 msgr2=0x555aed4389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.169+0000 7f40d3e00640 1 --2- 172.21.15.102:0/3054363570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 0x555aed4389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.169+0000 7f40d3e00640 1 -- 172.21.15.102:0/3054363570 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555aed0b13f0 con 0x555aed438ee0 2024-04-25T02:01:36.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.169+0000 7f40d3e00640 1 --2- 172.21.15.102:0/3054363570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed443070 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f40c40cc3b0 tx=0x7f40c40d2bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=757bcd10cfbabaab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.170+0000 7f40c9600640 1 -- 172.21.15.102:0/3054363570 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c40dd070 con 0x555aed438ee0 2024-04-25T02:01:36.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.170+0000 7f40c9600640 1 -- 172.21.15.102:0/3054363570 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c40d7dd0 con 0x555aed438ee0 2024-04-25T02:01:36.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.170+0000 7f40c9600640 1 -- 172.21.15.102:0/3054363570 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c40d8380 con 0x555aed438ee0 2024-04-25T02:01:36.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.171+0000 7f40d0c00640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40b80a3000 0x7f40b80a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.171+0000 7f40d0c00640 1 -- 172.21.15.102:0/4180913731 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f40c40cec40 con 0x7f40d4108740 2024-04-25T02:01:36.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.171+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3054363570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 msgr2=0x555aed443070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.171+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3054363570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed443070 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f40c40cc3b0 tx=0x7f40c40d2bb0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.172+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3054363570 shutdown_connections 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.172+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3054363570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed4459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.172+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3054363570 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed443070 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.172+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3054363570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 0x555aed4389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.172+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3054363570 >> 172.21.15.102:0/3054363570 conn(0x555aed4348e0 msgr2=0x555aed434cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.172+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3054363570 shutdown_connections 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.172+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3054363570 wait complete. 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.172+0000 7f40d9200640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40b80a3000 0x7f40b80a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.172+0000 7f40ddfaf740 1 -- start start 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.173+0000 7f40ddfaf740 1 Processor -- start 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.173+0000 7f40d9200640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40b80a3000 0x7f40b80a54c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f40cc005fd0 tx=0x7f40cc005f60 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.173+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 0x555aed4405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.173+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed440ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.173+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed441020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.173+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x555aed4463a0 con 0x555aed4385c0 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.173+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x555aed358ec0 con 0x555aed438ee0 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.173+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555aed446520 con 0x555aed4435b0 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.173+0000 7f40ddfaf740 10 client.0 _ll_register_callbacks cb 0x555aed3953f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed440ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed441020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed440ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33020/0 (socket says 172.21.15.102:33020) 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3400640 1 -- 172.21.15.102:0/2683843852 learned_addr learned my addr 172.21.15.102:0/2683843852 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3e00640 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 0x555aed4405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3400640 1 -- 172.21.15.102:0/2683843852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 msgr2=0x555aed441020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3400640 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed441020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3400640 1 -- 172.21.15.102:0/2683843852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 msgr2=0x555aed4405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3400640 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 0x555aed4405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3400640 1 -- 172.21.15.102:0/2683843852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555aed17b7d0 con 0x555aed438ee0 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3e00640 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 0x555aed4405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40d3400640 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed440ae0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f40c0007380 tx=0x7f40c0005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76a9c0b82ab1e0cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.174+0000 7f40be000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.175+0000 7f40be000640 1 -- 172.21.15.102:0/2683843852 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0041070 con 0x555aed438ee0 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.175+0000 7f40be000640 1 -- 172.21.15.102:0/2683843852 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f40c00092e0 con 0x555aed438ee0 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.175+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2683843852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40c40d7dd0 con 0x555aed438ee0 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.175+0000 7f40be000640 1 -- 172.21.15.102:0/2683843852 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f40c004a050 con 0x555aed438ee0 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.175+0000 7f40ddfaf740 10 client.26948 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:36.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.175+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2683843852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555aed452490 con 0x555aed438ee0 2024-04-25T02:01:36.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.175+0000 7f40be000640 1 -- 172.21.15.102:0/2683843852 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c40d7dd0 con 0x555aed438ee0 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.176+0000 7f40be000640 1 -- 172.21.15.102:0/2683843852 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f40c003d450 con 0x555aed438ee0 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.176+0000 7f40ddfaf740 10 client.26948 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.176+0000 7f40ddfaf740 10 client.26948 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.176+0000 7f40ddfaf740 20 client.26948 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.176+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x555aed4539a0 0x555aed455d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.176+0000 7f40ddfaf740 4 client.26948 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.176+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2683843852 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555aed3aaa20 con 0x555aed4539a0 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.176+0000 7f40d3e00640 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x555aed4539a0 0x555aed455d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.178+0000 7f40d3e00640 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x555aed4539a0 0x555aed455d90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=856fdef751712cd0 server_cookie=9f2c1af5148dda48 in_seq=0 out_seq=0 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.178+0000 7f40be000640 10 client.26948 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.179+0000 7f40be000640 1 -- 172.21.15.102:0/2683843852 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555aed3aaa20 con 0x555aed4539a0 2024-04-25T02:01:36.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.179+0000 7f40be000640 10 client.26948 handle_command_reply: tid=0 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.180+0000 7f40ddfaf740 1 client.26948 shutdown 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.180+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2683843852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 msgr2=0x555aed440ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.180+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed440ae0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f40c0007380 tx=0x7f40c0005e00 comp rx=0 tx=0).stop 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2683843852 shutdown_connections 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x555aed4539a0 0x555aed455d90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed441020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed438ee0 0x555aed440ae0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2683843852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed4385c0 0x555aed4405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2683843852 >> 172.21.15.102:0/2683843852 conn(0x555aed4348e0 msgr2=0x555aed433b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40be000640 0 client.26948 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T02:01:36.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2683843852 shutdown_connections 2024-04-25T02:01:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2683843852 wait complete. 2024-04-25T02:01:36.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.181+0000 7f40ddfaf740 20 client.26948 trim_cache size 0 max 16384 2024-04-25T02:01:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.208+0000 7f40ddfaf740 1 Processor -- start 2024-04-25T02:01:36.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.208+0000 7f40ddfaf740 1 -- start start 2024-04-25T02:01:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.208+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.208+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed438ee0 0x555aed45b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.208+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed45bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.208+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41c7340 con 0x555aed438ee0 2024-04-25T02:01:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.208+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41c4bb0 con 0x555aed4385c0 2024-04-25T02:01:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.208+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41c2a60 con 0x555aed4435b0 2024-04-25T02:01:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed45bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed45bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42522/0 (socket says 172.21.15.102:42522) 2024-04-25T02:01:36.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d9200640 1 -- 172.21.15.102:0/2100211394 learned_addr learned my addr 172.21.15.102:0/2100211394 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d3e00640 1 --2- 172.21.15.102:0/2100211394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d3400640 1 --2- 172.21.15.102:0/2100211394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed438ee0 0x555aed45b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d9200640 1 -- 172.21.15.102:0/2100211394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 msgr2=0x555aed44f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d9200640 1 --2- 172.21.15.102:0/2100211394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d9200640 1 -- 172.21.15.102:0/2100211394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed438ee0 msgr2=0x555aed45b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d9200640 1 --2- 172.21.15.102:0/2100211394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed438ee0 0x555aed45b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.209+0000 7f40d9200640 1 -- 172.21.15.102:0/2100211394 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555aed43eb10 con 0x555aed4435b0 2024-04-25T02:01:36.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.211+0000 7f40d9200640 1 --2- 172.21.15.102:0/2100211394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed45bbe0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f40cc01e760 tx=0x7f40cc012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcb4f48d2576eec1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.211+0000 7f40c8c00640 1 -- 172.21.15.102:0/2100211394 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc02b070 con 0x555aed4435b0 2024-04-25T02:01:36.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.212+0000 7f40c8c00640 1 -- 172.21.15.102:0/2100211394 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40cc010ce0 con 0x555aed4435b0 2024-04-25T02:01:36.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.212+0000 7f40c8c00640 1 -- 172.21.15.102:0/2100211394 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc02b070 con 0x555aed4435b0 2024-04-25T02:01:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.213+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2100211394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 msgr2=0x555aed45bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.213+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2100211394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed45bbe0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f40cc01e760 tx=0x7f40cc012000 comp rx=0 tx=0).stop 2024-04-25T02:01:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.213+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2100211394 shutdown_connections 2024-04-25T02:01:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.213+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2100211394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed45bbe0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.213+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2100211394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed438ee0 0x555aed45b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.213+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/2100211394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.213+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2100211394 >> 172.21.15.102:0/2100211394 conn(0x555aed4348e0 msgr2=0x555aed434cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.213+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2100211394 shutdown_connections 2024-04-25T02:01:36.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.213+0000 7f40ddfaf740 1 -- 172.21.15.102:0/2100211394 wait complete. 2024-04-25T02:01:36.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.214+0000 7f40ddfaf740 1 -- start start 2024-04-25T02:01:36.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.214+0000 7f40ddfaf740 1 Processor -- start 2024-04-25T02:01:36.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed438ee0 0x555aed43cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40ddfaf740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed43d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41c30b0 con 0x555aed438ee0 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41c44e0 con 0x555aed4385c0 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40ddfaf740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41c3740 con 0x555aed4435b0 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33044/0 (socket says 172.21.15.102:33044) 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40ddfaf740 10 client.0 _ll_register_callbacks cb 0x7f40c0005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.215+0000 7f40d3e00640 1 -- 172.21.15.102:0/3508065327 learned_addr learned my addr 172.21.15.102:0/3508065327 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.216+0000 7f40d3e00640 1 -- 172.21.15.102:0/3508065327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 msgr2=0x555aed43d4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.216+0000 7f40d3e00640 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed43d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.216+0000 7f40d3e00640 1 -- 172.21.15.102:0/3508065327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed438ee0 msgr2=0x555aed43cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.216+0000 7f40d3e00640 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed438ee0 0x555aed43cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.216+0000 7f40d3e00640 1 -- 172.21.15.102:0/3508065327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555aed43dcb0 con 0x555aed4385c0 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.216+0000 7f40d3e00640 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44baf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f40c40a3980 tx=0x7f40c40d9ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f604825e6c07d94e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.217+0000 7f40be000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.217+0000 7f40be000640 1 -- 172.21.15.102:0/3508065327 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c40dd050 con 0x555aed4385c0 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.217+0000 7f40be000640 1 -- 172.21.15.102:0/3508065327 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f40c40995f0 con 0x555aed4385c0 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.217+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3508065327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40cc010ce0 con 0x555aed4385c0 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.218+0000 7f40be000640 1 -- 172.21.15.102:0/3508065327 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f40c4095e60 con 0x555aed4385c0 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.218+0000 7f40ddfaf740 10 client.26954 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.218+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3508065327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=34}) -- 0x555aed4b9e20 con 0x555aed4385c0 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.218+0000 7f40be000640 1 -- 172.21.15.102:0/3508065327 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f40c400b040 con 0x555aed4385c0 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.218+0000 7f40ddfaf740 10 client.26954 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:36.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.219+0000 7f40ddfaf740 10 client.26954 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T02:01:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.219+0000 7f40ddfaf740 20 client.26954 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.219+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x555aed4ba040 0x555aed4bc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.219+0000 7f40ddfaf740 4 client.26954 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:01:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.219+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3508065327 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555aed3a4d70 con 0x555aed4ba040 2024-04-25T02:01:36.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.219+0000 7f40d3400640 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x555aed4ba040 0x555aed4bc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.222+0000 7f40be000640 10 client.26954 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.222+0000 7f40d3400640 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x555aed4ba040 0x555aed4bc430 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d51b4d02124fb3a9 server_cookie=2ec6a54b2c01dbf5 in_seq=0 out_seq=0 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.222+0000 7f40be000640 1 -- 172.21.15.102:0/3508065327 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555aed3a4d70 con 0x555aed4ba040 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.222+0000 7f40be000640 10 client.26954 handle_command_reply: tid=0 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.222+0000 7f40ddfaf740 1 client.26954 shutdown 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3508065327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 msgr2=0x555aed44baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44baf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f40c40a3980 tx=0x7f40c40d9ab0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3508065327 shutdown_connections 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x555aed4ba040 0x555aed4bc430 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555aed4435b0 0x555aed43d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555aed438ee0 0x555aed43cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 --2- 172.21.15.102:0/3508065327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555aed4385c0 0x555aed44baf0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3508065327 >> 172.21.15.102:0/3508065327 conn(0x555aed4348e0 msgr2=0x555aed43b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40be000640 0 client.26954 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T02:01:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3508065327 shutdown_connections 2024-04-25T02:01:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 1 -- 172.21.15.102:0/3508065327 wait complete. 2024-04-25T02:01:36.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.223+0000 7f40ddfaf740 20 client.26954 trim_cache size 0 max 16384 2024-04-25T02:01:36.232 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:01:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 -- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40b80a3000 msgr2=0x7f40b80a54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40b80a3000 0x7f40b80a54c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f40cc005fd0 tx=0x7f40cc005f60 comp rx=0 tx=0).stop 2024-04-25T02:01:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 -- 172.21.15.102:0/4180913731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40d4108740 msgr2=0x7f40d41a7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40d4108740 0x7f40d41a7650 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f40c400ddc0 tx=0x7f40c400a620 comp rx=0 tx=0).stop 2024-04-25T02:01:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 -- 172.21.15.102:0/4180913731 shutdown_connections 2024-04-25T02:01:36.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40b80a3000 0x7f40b80a54c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d410d120 0x7f40d41a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d41090e0 0x7f40d41a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 --2- 172.21.15.102:0/4180913731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40d4108740 0x7f40d41a7650 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.224+0000 7f40db200640 1 -- 172.21.15.102:0/4180913731 >> 172.21.15.102:0/4180913731 conn(0x7f40d406ccb0 msgr2=0x7f40d4071830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.225+0000 7f40db200640 1 -- 172.21.15.102:0/4180913731 shutdown_connections 2024-04-25T02:01:36.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.225+0000 7f40db200640 1 -- 172.21.15.102:0/4180913731 wait complete. 2024-04-25T02:01:36.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T02:01:36.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:36 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3412839119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:01:36.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:36 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3412839119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:01:36.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:36 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3412839119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:01:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.520+0000 7f4f11200640 1 -- 172.21.15.102:0/428548428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c107720 msgr2=0x7f4f0c107b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.520+0000 7f4f11200640 1 --2- 172.21.15.102:0/428548428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c107720 0x7f4f0c107b00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8009900 tx=0x7f4ef802a410 comp rx=0 tx=0).stop 2024-04-25T02:01:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.520+0000 7f4f11200640 1 -- 172.21.15.102:0/428548428 shutdown_connections 2024-04-25T02:01:36.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.520+0000 7f4f11200640 1 --2- 172.21.15.102:0/428548428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f0c06a700 0x7f4f0c10e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.520+0000 7f4f11200640 1 --2- 172.21.15.102:0/428548428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f0c069d40 0x7f4f0c06a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.520+0000 7f4f11200640 1 --2- 172.21.15.102:0/428548428 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c107720 0x7f4f0c107b00 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.520+0000 7f4f11200640 1 -- 172.21.15.102:0/428548428 >> 172.21.15.102:0/428548428 conn(0x7f4f0c076440 msgr2=0x7f4f0c076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.520+0000 7f4f11200640 1 -- 172.21.15.102:0/428548428 shutdown_connections 2024-04-25T02:01:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.521+0000 7f4f11200640 1 -- 172.21.15.102:0/428548428 wait complete. 2024-04-25T02:01:36.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.523+0000 7f4f11200640 1 Processor -- start 2024-04-25T02:01:36.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f11200640 1 -- start start 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f11200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c069d40 0x7f4f0c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f11200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f0c06a700 0x7f4f0c19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f11200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f0c107720 0x7f4f0c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f11200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c111b30 con 0x7f4f0c107720 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f11200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c1119b0 con 0x7f4f0c069d40 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f11200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c111cb0 con 0x7f4f0c06a700 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f09e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c069d40 0x7f4f0c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f09e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c069d40 0x7f4f0c19a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33060/0 (socket says 172.21.15.102:33060) 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.524+0000 7f4f09e00640 1 -- 172.21.15.102:0/1558810586 learned_addr learned my addr 172.21.15.102:0/1558810586 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f0a800640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f0c107720 0x7f4f0c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f09400640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f0c06a700 0x7f4f0c19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f09e00640 1 -- 172.21.15.102:0/1558810586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f0c06a700 msgr2=0x7f4f0c19a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f09e00640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f0c06a700 0x7f4f0c19a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f09e00640 1 -- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f0c107720 msgr2=0x7f4f0c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f09e00640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f0c107720 0x7f4f0c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f09e00640 1 -- 172.21.15.102:0/1558810586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f0c19f260 con 0x7f4f0c069d40 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f09400640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f0c06a700 0x7f4f0c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f0a800640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f0c107720 0x7f4f0c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.525+0000 7f4f09e00640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c069d40 0x7f4f0c19a270 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8006010 tx=0x7f4ef8003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.526+0000 7f4f02a00640 1 -- 172.21.15.102:0/1558810586 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef803f070 con 0x7f4f0c069d40 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.526+0000 7f4f02a00640 1 -- 172.21.15.102:0/1558810586 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ef8031c30 con 0x7f4f0c069d40 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.526+0000 7f4f02a00640 1 -- 172.21.15.102:0/1558810586 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8035030 con 0x7f4f0c069d40 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.526+0000 7f4f11200640 1 -- 172.21.15.102:0/1558810586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f0c19f4f0 con 0x7f4f0c069d40 2024-04-25T02:01:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.526+0000 7f4f11200640 1 -- 172.21.15.102:0/1558810586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f0c1a6de0 con 0x7f4f0c069d40 2024-04-25T02:01:36.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.531+0000 7f4f02a00640 1 -- 172.21.15.102:0/1558810586 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4ef8042070 con 0x7f4f0c069d40 2024-04-25T02:01:36.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.532+0000 7f4f13f96740 1 Processor -- start 2024-04-25T02:01:36.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.533+0000 7f4f13f96740 1 -- start start 2024-04-25T02:01:36.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.533+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0f310 0x55d252d0f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.533+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d22880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.533+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d251b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.533+0000 7f4f13f96740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d252c744e0 con 0x55d252d0fcc0 2024-04-25T02:01:36.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.533+0000 7f4f13f96740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d252c1ca70 con 0x55d252d0f310 2024-04-25T02:01:36.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.534+0000 7f4f13f96740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d252c704a0 con 0x55d252d22dc0 2024-04-25T02:01:36.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.534+0000 7f4f02a00640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4ef00a2d00 0x7f4ef00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.534+0000 7f4f02a00640 1 -- 172.21.15.102:0/1558810586 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f4ef80efa30 con 0x7f4f0c069d40 2024-04-25T02:01:36.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.535+0000 7f4f09e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d22880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.535+0000 7f4f09400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d251b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.535+0000 7f4f09e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d22880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37214/0 (socket says 172.21.15.102:37214) 2024-04-25T02:01:36.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.535+0000 7f4f09400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d251b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42544/0 (socket says 172.21.15.102:42544) 2024-04-25T02:01:36.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.535+0000 7f4f09400640 1 -- 172.21.15.102:0/850932076 learned_addr learned my addr 172.21.15.102:0/850932076 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.535+0000 7f4f0a800640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4ef00a2d00 0x7f4ef00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.535+0000 7f4f0a800640 1 --2- 172.21.15.102:0/850932076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0f310 0x55d252d0f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.536+0000 7f4f09e00640 1 -- 172.21.15.102:0/850932076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 msgr2=0x55d252d251b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.536+0000 7f4f09e00640 1 --2- 172.21.15.102:0/850932076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d251b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.536+0000 7f4f09e00640 1 -- 172.21.15.102:0/850932076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0f310 msgr2=0x55d252d0f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.536+0000 7f4f09e00640 1 --2- 172.21.15.102:0/850932076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0f310 0x55d252d0f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.536+0000 7f4f09e00640 1 -- 172.21.15.102:0/850932076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2529883f0 con 0x55d252d0fcc0 2024-04-25T02:01:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.536+0000 7f4f09e00640 1 --2- 172.21.15.102:0/850932076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d22880 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80ed150 tx=0x7f4ef80bbb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23d634f6f932780 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.536+0000 7f4ef7400640 1 -- 172.21.15.102:0/850932076 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8049040 con 0x55d252d0fcc0 2024-04-25T02:01:36.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.536+0000 7f4f0a800640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4ef00a2d00 0x7f4ef00a51c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f04009fd0 tx=0x7f4f04009510 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.536+0000 7f4ef7400640 1 -- 172.21.15.102:0/850932076 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ef80bc6e0 con 0x55d252d0fcc0 2024-04-25T02:01:36.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.538+0000 7f4f13f96740 1 -- 172.21.15.102:0/850932076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 msgr2=0x55d252d22880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.538+0000 7f4f13f96740 1 --2- 172.21.15.102:0/850932076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d22880 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80ed150 tx=0x7f4ef80bbb20 comp rx=0 tx=0).stop 2024-04-25T02:01:36.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.539+0000 7f4f13f96740 1 -- 172.21.15.102:0/850932076 shutdown_connections 2024-04-25T02:01:36.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.539+0000 7f4f13f96740 1 --2- 172.21.15.102:0/850932076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d251b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.539+0000 7f4f13f96740 1 --2- 172.21.15.102:0/850932076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d22880 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.539+0000 7f4f13f96740 1 --2- 172.21.15.102:0/850932076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0f310 0x55d252d0f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.539+0000 7f4f13f96740 1 -- 172.21.15.102:0/850932076 >> 172.21.15.102:0/850932076 conn(0x55d252d0b8e0 msgr2=0x55d252d0bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.539+0000 7f4f13f96740 1 -- 172.21.15.102:0/850932076 shutdown_connections 2024-04-25T02:01:36.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.539+0000 7f4f13f96740 1 -- 172.21.15.102:0/850932076 wait complete. 2024-04-25T02:01:36.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.539+0000 7f4f13f96740 1 -- start start 2024-04-25T02:01:36.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f13f96740 1 Processor -- start 2024-04-25T02:01:36.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d1c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d1cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252d22420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f13f96740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d252d121d0 con 0x55d252d0fcc0 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f13f96740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d252c2fec0 con 0x55d252d22dc0 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f13f96740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d252d12350 con 0x55d252d0f310 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f13f96740 10 client.0 _ll_register_callbacks cb 0x55d252c6c3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f09e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d1c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.540+0000 7f4f0a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252d22420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4f09e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d1c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42556/0 (socket says 172.21.15.102:42556) 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4f09e00640 1 -- 172.21.15.102:0/1447493946 learned_addr learned my addr 172.21.15.102:0/1447493946 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4f09400640 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d1cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4f09e00640 1 -- 172.21.15.102:0/1447493946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 msgr2=0x55d252d22420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4f09e00640 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252d22420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4f09e00640 1 -- 172.21.15.102:0/1447493946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 msgr2=0x55d252d1cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4f09e00640 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d1cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4f09e00640 1 -- 172.21.15.102:0/1447493946 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d252a527d0 con 0x55d252d0f310 2024-04-25T02:01:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4f09e00640 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d1c700 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80bbce0 tx=0x7f4ef80ba4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1182aceeee6bd9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.541+0000 7f4eebe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:36.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.542+0000 7f4eebe00640 1 -- 172.21.15.102:0/1447493946 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8049040 con 0x55d252d0f310 2024-04-25T02:01:36.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.542+0000 7f4eebe00640 1 -- 172.21.15.102:0/1447493946 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f4ef803b460 con 0x55d252d0f310 2024-04-25T02:01:36.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.542+0000 7f4f13f96740 1 -- 172.21.15.102:0/1447493946 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ef80bc6e0 con 0x55d252d0f310 2024-04-25T02:01:36.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.542+0000 7f4eebe00640 1 -- 172.21.15.102:0/1447493946 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4ef8036e10 con 0x55d252d0f310 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.542+0000 7f4f13f96740 10 client.26920 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.542+0000 7f4f13f96740 1 -- 172.21.15.102:0/1447493946 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d252d121d0 con 0x55d252d0f310 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.542+0000 7f4eebe00640 1 -- 172.21.15.102:0/1447493946 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ef80bc6e0 con 0x55d252d0f310 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.543+0000 7f4eebe00640 1 -- 172.21.15.102:0/1447493946 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f4ef804c060 con 0x55d252d0f310 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.543+0000 7f4f13f96740 10 client.26920 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.543+0000 7f4f13f96740 10 client.26920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.543+0000 7f4f13f96740 20 client.26920 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.543+0000 7f4f13f96740 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d252d4c4f0 0x55d252d4e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.543+0000 7f4f13f96740 4 client.26920 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.544+0000 7f4f13f96740 1 -- 172.21.15.102:0/1447493946 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d252c81a20 con 0x55d252d4c4f0 2024-04-25T02:01:36.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.544+0000 7f4f09400640 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d252d4c4f0 0x55d252d4e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.547+0000 7f4f09400640 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d252d4c4f0 0x55d252d4e8e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3127eb4622ef89f4 server_cookie=8bf282218cb31d47 in_seq=0 out_seq=0 2024-04-25T02:01:36.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.547+0000 7f4eebe00640 10 client.26920 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.547+0000 7f4eebe00640 1 -- 172.21.15.102:0/1447493946 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d252c81a20 con 0x55d252d4c4f0 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.547+0000 7f4eebe00640 10 client.26920 handle_command_reply: tid=0 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 client.26920 shutdown 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 -- 172.21.15.102:0/1447493946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 msgr2=0x55d252d1c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d1c700 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80bbce0 tx=0x7f4ef80ba4f0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 -- 172.21.15.102:0/1447493946 shutdown_connections 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d252d4c4f0 0x55d252d4e8e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252d22420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d1cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 --2- 172.21.15.102:0/1447493946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d1c700 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4eebe00640 0 client.26920 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:36.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 -- 172.21.15.102:0/1447493946 >> 172.21.15.102:0/1447493946 conn(0x55d252d0b8e0 msgr2=0x55d252d11750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 -- 172.21.15.102:0/1447493946 shutdown_connections 2024-04-25T02:01:36.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 1 -- 172.21.15.102:0/1447493946 wait complete. 2024-04-25T02:01:36.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.548+0000 7f4f13f96740 20 client.26920 trim_cache size 0 max 16384 2024-04-25T02:01:36.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.611+0000 7f4f13f96740 1 Processor -- start 2024-04-25T02:01:36.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.612+0000 7f4f13f96740 1 -- start start 2024-04-25T02:01:36.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.614+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0f310 0x55d252d22390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.614+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0fcc0 0x55d252d540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.614+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d54600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.614+0000 7f4f13f96740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c071c30 con 0x55d252d0f310 2024-04-25T02:01:36.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.614+0000 7f4f13f96740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c06f4a0 con 0x55d252d0fcc0 2024-04-25T02:01:36.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.614+0000 7f4f13f96740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c06d310 con 0x55d252d22dc0 2024-04-25T02:01:36.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.615+0000 7f4f0a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d54600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.615+0000 7f4f0a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d54600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42564/0 (socket says 172.21.15.102:42564) 2024-04-25T02:01:36.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.615+0000 7f4f0a800640 1 -- 172.21.15.102:0/4051521695 learned_addr learned my addr 172.21.15.102:0/4051521695 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.616+0000 7f4f0a800640 1 -- 172.21.15.102:0/4051521695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0fcc0 msgr2=0x55d252d540c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:01:36.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.616+0000 7f4f09400640 1 --2- 172.21.15.102:0/4051521695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0fcc0 0x55d252d540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.616+0000 7f4f0a800640 1 --2- 172.21.15.102:0/4051521695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0fcc0 0x55d252d540c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.616+0000 7f4f0a800640 1 -- 172.21.15.102:0/4051521695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0f310 msgr2=0x55d252d22390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.616+0000 7f4f0a800640 1 --2- 172.21.15.102:0/4051521695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0f310 0x55d252d22390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.616+0000 7f4f0a800640 1 -- 172.21.15.102:0/4051521695 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d252d17890 con 0x55d252d22dc0 2024-04-25T02:01:36.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.617+0000 7f4f0a800640 1 --2- 172.21.15.102:0/4051521695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d54600 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4f04012a70 tx=0x7f4f040094a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b284bc6786ca30b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.618+0000 7f4ef6a00640 1 -- 172.21.15.102:0/4051521695 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f04004b20 con 0x55d252d22dc0 2024-04-25T02:01:36.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.619+0000 7f4ef6a00640 1 -- 172.21.15.102:0/4051521695 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f04021cd0 con 0x55d252d22dc0 2024-04-25T02:01:36.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.619+0000 7f4ef6a00640 1 -- 172.21.15.102:0/4051521695 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f0402b830 con 0x55d252d22dc0 2024-04-25T02:01:36.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.620+0000 7f4f13f96740 1 -- 172.21.15.102:0/4051521695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 msgr2=0x55d252d54600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.621+0000 7f4f13f96740 1 --2- 172.21.15.102:0/4051521695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d54600 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4f04012a70 tx=0x7f4f040094a0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.621+0000 7f4f13f96740 1 -- 172.21.15.102:0/4051521695 shutdown_connections 2024-04-25T02:01:36.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.621+0000 7f4f13f96740 1 --2- 172.21.15.102:0/4051521695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d22dc0 0x55d252d54600 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.621+0000 7f4f13f96740 1 --2- 172.21.15.102:0/4051521695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d0fcc0 0x55d252d540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.621+0000 7f4f13f96740 1 --2- 172.21.15.102:0/4051521695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0f310 0x55d252d22390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.621+0000 7f4f13f96740 1 -- 172.21.15.102:0/4051521695 >> 172.21.15.102:0/4051521695 conn(0x55d252d0b8e0 msgr2=0x55d252d0a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.622+0000 7f4f13f96740 1 -- 172.21.15.102:0/4051521695 shutdown_connections 2024-04-25T02:01:36.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.622+0000 7f4f13f96740 1 -- 172.21.15.102:0/4051521695 wait complete. 2024-04-25T02:01:36.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.623+0000 7f4f13f96740 1 -- start start 2024-04-25T02:01:36.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.623+0000 7f4f13f96740 1 Processor -- start 2024-04-25T02:01:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.625+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d14e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.625+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d15340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.625+0000 7f4f13f96740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252db1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.625+0000 7f4f13f96740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c06d960 con 0x55d252d0fcc0 2024-04-25T02:01:36.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.625+0000 7f4f13f96740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c06edd0 con 0x55d252d22dc0 2024-04-25T02:01:36.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.625+0000 7f4f13f96740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0c06e030 con 0x55d252d0f310 2024-04-25T02:01:36.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.625+0000 7f4f13f96740 10 client.0 _ll_register_callbacks cb 0x7f4f04009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.626+0000 7f4f0a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252db1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.626+0000 7f4f0a800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252db1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33102/0 (socket says 172.21.15.102:33102) 2024-04-25T02:01:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.626+0000 7f4f0a800640 1 -- 172.21.15.102:0/67105843 learned_addr learned my addr 172.21.15.102:0/67105843 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.626+0000 7f4f09400640 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d15340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.626+0000 7f4f09e00640 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d14e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.627+0000 7f4f0a800640 1 -- 172.21.15.102:0/67105843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 msgr2=0x55d252d14e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.627+0000 7f4f0a800640 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d14e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.627+0000 7f4f0a800640 1 -- 172.21.15.102:0/67105843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 msgr2=0x55d252d15340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.627+0000 7f4f0a800640 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d15340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.628+0000 7f4f0a800640 1 -- 172.21.15.102:0/67105843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d252d0fa60 con 0x55d252d22dc0 2024-04-25T02:01:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.628+0000 7f4f09400640 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d15340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.628+0000 7f4f09e00640 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d14e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:01:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.629+0000 7f4eebe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.629+0000 7f4f0a800640 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252db1410 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4f04012a40 tx=0x7f4f04021aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13c2578d28facbad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:36.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.629+0000 7f4eebe00640 1 -- 172.21.15.102:0/67105843 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f04004c50 con 0x55d252d22dc0 2024-04-25T02:01:36.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.630+0000 7f4eebe00640 1 -- 172.21.15.102:0/67105843 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f4f04027040 con 0x55d252d22dc0 2024-04-25T02:01:36.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.630+0000 7f4f13f96740 1 -- 172.21.15.102:0/67105843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f04021cd0 con 0x55d252d22dc0 2024-04-25T02:01:36.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.631+0000 7f4eebe00640 1 -- 172.21.15.102:0/67105843 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4f0403a050 con 0x55d252d22dc0 2024-04-25T02:01:36.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.631+0000 7f4f13f96740 10 client.26969 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:36.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.631+0000 7f4f13f96740 1 -- 172.21.15.102:0/67105843 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55d252db2a20 con 0x55d252d22dc0 2024-04-25T02:01:36.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.632+0000 7f4eebe00640 1 -- 172.21.15.102:0/67105843 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f04021cd0 con 0x55d252d22dc0 2024-04-25T02:01:36.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.632+0000 7f4eebe00640 1 -- 172.21.15.102:0/67105843 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f4f0400c030 con 0x55d252d22dc0 2024-04-25T02:01:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.633+0000 7f4f13f96740 10 client.26969 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.633+0000 7f4f13f96740 10 client.26969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.633+0000 7f4f13f96740 20 client.26969 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.633+0000 7f4f13f96740 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d252db3120 0x55d252db5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.633+0000 7f4f13f96740 4 client.26969 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:01:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.633+0000 7f4f13f96740 1 -- 172.21.15.102:0/67105843 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d252c7bd70 con 0x55d252db3120 2024-04-25T02:01:36.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.633+0000 7f4f09e00640 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d252db3120 0x55d252db5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.634+0000 7f4eebe00640 10 client.26969 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:36.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.634+0000 7f4f09e00640 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d252db3120 0x55d252db5540 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=198d9f4c5b748d7a server_cookie=a232e0a5ebac53e7 in_seq=0 out_seq=0 2024-04-25T02:01:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.634+0000 7f4eebe00640 1 -- 172.21.15.102:0/67105843 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d252c7bd70 con 0x55d252db3120 2024-04-25T02:01:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.634+0000 7f4eebe00640 10 client.26969 handle_command_reply: tid=0 2024-04-25T02:01:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4f13f96740 1 client.26969 shutdown 2024-04-25T02:01:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4f13f96740 1 -- 172.21.15.102:0/67105843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 msgr2=0x55d252db1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4f13f96740 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252db1410 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4f04012a40 tx=0x7f4f04021aa0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4f13f96740 1 -- 172.21.15.102:0/67105843 shutdown_connections 2024-04-25T02:01:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4f13f96740 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d252db3120 0x55d252db5540 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4f13f96740 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d252d22dc0 0x55d252db1410 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4f13f96740 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d252d0fcc0 0x55d252d15340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4f13f96740 1 --2- 172.21.15.102:0/67105843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d252d0f310 0x55d252d14e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4f13f96740 1 -- 172.21.15.102:0/67105843 >> 172.21.15.102:0/67105843 conn(0x55d252d0b8e0 msgr2=0x55d252d0ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.635+0000 7f4eebe00640 0 client.26969 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:36.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.636+0000 7f4f13f96740 1 -- 172.21.15.102:0/67105843 shutdown_connections 2024-04-25T02:01:36.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.636+0000 7f4f13f96740 1 -- 172.21.15.102:0/67105843 wait complete. 2024-04-25T02:01:36.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.636+0000 7f4f13f96740 20 client.26969 trim_cache size 0 max 16384 2024-04-25T02:01:36.641 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:01:36.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 -- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4ef00a2d00 msgr2=0x7f4ef00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4ef00a2d00 0x7f4ef00a51c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f04009fd0 tx=0x7f4f04009510 comp rx=0 tx=0).stop 2024-04-25T02:01:36.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 -- 172.21.15.102:0/1558810586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c069d40 msgr2=0x7f4f0c19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:36.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c069d40 0x7f4f0c19a270 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8006010 tx=0x7f4ef8003c10 comp rx=0 tx=0).stop 2024-04-25T02:01:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 -- 172.21.15.102:0/1558810586 shutdown_connections 2024-04-25T02:01:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f4ef00a2d00 0x7f4ef00a51c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f0c107720 0x7f4f0c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f0c06a700 0x7f4f0c19a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 --2- 172.21.15.102:0/1558810586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f0c069d40 0x7f4f0c19a270 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:36.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.638+0000 7f4f11200640 1 -- 172.21.15.102:0/1558810586 >> 172.21.15.102:0/1558810586 conn(0x7f4f0c076440 msgr2=0x7f4f0c101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:36.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.639+0000 7f4f11200640 1 -- 172.21.15.102:0/1558810586 shutdown_connections 2024-04-25T02:01:36.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:36.639+0000 7f4f11200640 1 -- 172.21.15.102:0/1558810586 wait complete. 2024-04-25T02:01:37.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:37 smithi102 ceph-mon[27796]: pgmap v193: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.6 MiB/s wr, 675 op/s 2024-04-25T02:01:37.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:37 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:01:37.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:37 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:01:37.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:37 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:01:37.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:37 smithi110 ceph-mon[32848]: pgmap v193: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.6 MiB/s wr, 675 op/s 2024-04-25T02:01:37.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:37 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:01:37.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:37 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:01:37.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:37 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:01:37.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:37 smithi193 ceph-mon[32981]: pgmap v193: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 6.6 MiB/s wr, 675 op/s 2024-04-25T02:01:37.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:37 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:01:37.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:37 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:01:37.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:37 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:01:37.679 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T02:01:37.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:01:37.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T02:01:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.898+0000 7f8743200640 1 -- 172.21.15.102:0/2802821031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744103780 msgr2=0x7f8744103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.898+0000 7f8743200640 1 --2- 172.21.15.102:0/2802821031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744103780 0x7f8744103c00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8730009900 tx=0x7f873002a420 comp rx=0 tx=0).stop 2024-04-25T02:01:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.899+0000 7f8743200640 1 -- 172.21.15.102:0/2802821031 shutdown_connections 2024-04-25T02:01:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.899+0000 7f8743200640 1 --2- 172.21.15.102:0/2802821031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8744104140 0x7f8744110650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.899+0000 7f8743200640 1 --2- 172.21.15.102:0/2802821031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744103780 0x7f8744103c00 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.899+0000 7f8743200640 1 --2- 172.21.15.102:0/2802821031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8744109790 0x7f8744109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.899+0000 7f8743200640 1 -- 172.21.15.102:0/2802821031 >> 172.21.15.102:0/2802821031 conn(0x7f87440ff270 msgr2=0x7f8744101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.900+0000 7f8743200640 1 -- 172.21.15.102:0/2802821031 shutdown_connections 2024-04-25T02:01:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.900+0000 7f8743200640 1 -- 172.21.15.102:0/2802821031 wait complete. 2024-04-25T02:01:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.901+0000 7f8743200640 1 Processor -- start 2024-04-25T02:01:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.903+0000 7f8743200640 1 -- start start 2024-04-25T02:01:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.903+0000 7f8743200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8744103780 0x7f8744115dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.903+0000 7f8743200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8744104140 0x7f8744116310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.903+0000 7f8743200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744109790 0x7f874410f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.904+0000 7f8743200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8744076ee0 con 0x7f8744103780 2024-04-25T02:01:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.904+0000 7f8743200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8744076d60 con 0x7f8744109790 2024-04-25T02:01:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.904+0000 7f8743200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8744077060 con 0x7f8744104140 2024-04-25T02:01:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.904+0000 7f8742800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744109790 0x7f874410f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.904+0000 7f8742800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744109790 0x7f874410f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33122/0 (socket says 172.21.15.102:33122) 2024-04-25T02:01:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.904+0000 7f8742800640 1 -- 172.21.15.102:0/1969144263 learned_addr learned my addr 172.21.15.102:0/1969144263 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.904+0000 7f8741400640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8744104140 0x7f8744116310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.904+0000 7f8741e00640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8744103780 0x7f8744115dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.909+0000 7f8742800640 1 -- 172.21.15.102:0/1969144263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8744104140 msgr2=0x7f8744116310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.910+0000 7f8742800640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8744104140 0x7f8744116310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.910+0000 7f8742800640 1 -- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8744103780 msgr2=0x7f8744115dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.910+0000 7f8742800640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8744103780 0x7f8744115dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.910+0000 7f8742800640 1 -- 172.21.15.102:0/1969144263 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874410fa80 con 0x7f8744109790 2024-04-25T02:01:37.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.910+0000 7f8741e00640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8744103780 0x7f8744115dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.911+0000 7f8742800640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744109790 0x7f874410f1c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f873400b3f0 tx=0x7f873400b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:37.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.911+0000 7f873aa00640 1 -- 172.21.15.102:0/1969144263 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734015020 con 0x7f8744109790 2024-04-25T02:01:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.911+0000 7f873aa00640 1 -- 172.21.15.102:0/1969144263 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8734011d10 con 0x7f8744109790 2024-04-25T02:01:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.911+0000 7f8743200640 1 -- 172.21.15.102:0/1969144263 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f874410fd70 con 0x7f8744109790 2024-04-25T02:01:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.912+0000 7f8743200640 1 -- 172.21.15.102:0/1969144263 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8744119cb0 con 0x7f8744109790 2024-04-25T02:01:37.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.912+0000 7f873aa00640 1 -- 172.21.15.102:0/1969144263 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8734015020 con 0x7f8744109790 2024-04-25T02:01:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.925+0000 7f873aa00640 1 -- 172.21.15.102:0/1969144263 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8734018020 con 0x7f8744109790 2024-04-25T02:01:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.929+0000 7f874bdd2740 1 Processor -- start 2024-04-25T02:01:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.931+0000 7f874bdd2740 1 -- start start 2024-04-25T02:01:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.931+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da50e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.931+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 0x5634da518c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.931+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 0x5634da51b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.931+0000 7f874bdd2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5634da471ab0 con 0x5634da50e8d0 2024-04-25T02:01:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.931+0000 7f874bdd2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5634da41a640 con 0x5634da50dfb0 2024-04-25T02:01:37.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.931+0000 7f874bdd2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5634da46da70 con 0x5634da519190 2024-04-25T02:01:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.934+0000 7f8741400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 0x5634da51b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.934+0000 7f8741400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da50e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.934+0000 7f8741400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da50e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33132/0 (socket says 172.21.15.102:33132) 2024-04-25T02:01:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.934+0000 7f8741400640 1 -- 172.21.15.102:0/3677035306 learned_addr learned my addr 172.21.15.102:0/3677035306 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.935+0000 7f873aa00640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87280a30f0 0x7f87280a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.935+0000 7f873aa00640 1 -- 172.21.15.102:0/1969144263 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f8734011630 con 0x7f8744109790 2024-04-25T02:01:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.937+0000 7f8741e00640 1 --2- 172.21.15.102:0/3677035306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 0x5634da518c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.937+0000 7f8742800640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87280a30f0 0x7f87280a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.938+0000 7f8741400640 1 -- 172.21.15.102:0/3677035306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 msgr2=0x5634da50e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.938+0000 7f8741400640 1 --2- 172.21.15.102:0/3677035306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da50e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.938+0000 7f8741400640 1 -- 172.21.15.102:0/3677035306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 msgr2=0x5634da518c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.938+0000 7f8741400640 1 --2- 172.21.15.102:0/3677035306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 0x5634da518c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.938+0000 7f8741400640 1 -- 172.21.15.102:0/3677035306 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634da186210 con 0x5634da519190 2024-04-25T02:01:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.939+0000 7f8741400640 1 --2- 172.21.15.102:0/3677035306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 0x5634da51b580 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8730009230 tx=0x7f8730031470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baeba49d1d5cfaf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.940+0000 7f872f400640 1 -- 172.21.15.102:0/3677035306 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730035040 con 0x5634da519190 2024-04-25T02:01:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.940+0000 7f872f400640 1 -- 172.21.15.102:0/3677035306 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8730005020 con 0x5634da519190 2024-04-25T02:01:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.940+0000 7f872f400640 1 -- 172.21.15.102:0/3677035306 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730004a00 con 0x5634da519190 2024-04-25T02:01:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.943+0000 7f8742800640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87280a30f0 0x7f87280a55b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8734097670 tx=0x7f873400bf60 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.943+0000 7f874bdd2740 1 -- 172.21.15.102:0/3677035306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 msgr2=0x5634da51b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.943+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3677035306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 0x5634da51b580 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8730009230 tx=0x7f8730031470 comp rx=0 tx=0).stop 2024-04-25T02:01:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.944+0000 7f874bdd2740 1 -- 172.21.15.102:0/3677035306 shutdown_connections 2024-04-25T02:01:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.944+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3677035306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 0x5634da51b580 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.944+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3677035306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 0x5634da518c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.944+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3677035306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da50e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.944+0000 7f874bdd2740 1 -- 172.21.15.102:0/3677035306 >> 172.21.15.102:0/3677035306 conn(0x5634da50a4c0 msgr2=0x5634da50a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.944+0000 7f874bdd2740 1 -- 172.21.15.102:0/3677035306 shutdown_connections 2024-04-25T02:01:37.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.945+0000 7f874bdd2740 1 -- 172.21.15.102:0/3677035306 wait complete. 2024-04-25T02:01:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.946+0000 7f874bdd2740 1 -- start start 2024-04-25T02:01:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.946+0000 7f874bdd2740 1 Processor -- start 2024-04-25T02:01:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.948+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da516240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.948+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 0x5634da516780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.948+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 0x5634da516cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.948+0000 7f874bdd2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5634da52cc80 con 0x5634da50e8d0 2024-04-25T02:01:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.948+0000 7f874bdd2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5634da42d990 con 0x5634da50dfb0 2024-04-25T02:01:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.948+0000 7f874bdd2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5634da52ce00 con 0x5634da519190 2024-04-25T02:01:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.948+0000 7f874bdd2740 10 client.0 _ll_register_callbacks cb 0x5634da42f950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.949+0000 7f8741400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 0x5634da516cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.949+0000 7f8741400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da516240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.949+0000 7f8741400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da516240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33136/0 (socket says 172.21.15.102:33136) 2024-04-25T02:01:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.949+0000 7f8741400640 1 -- 172.21.15.102:0/193748783 learned_addr learned my addr 172.21.15.102:0/193748783 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.950+0000 7f8741e00640 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 0x5634da516780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.950+0000 7f8741400640 1 -- 172.21.15.102:0/193748783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 msgr2=0x5634da516cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.950+0000 7f8741400640 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 0x5634da516cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.950+0000 7f8741400640 1 -- 172.21.15.102:0/193748783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 msgr2=0x5634da516780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.952+0000 7f8741400640 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 0x5634da516780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.952+0000 7f8741400640 1 -- 172.21.15.102:0/193748783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634da252620 con 0x5634da50dfb0 2024-04-25T02:01:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.952+0000 7f8741e00640 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 0x5634da516780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.952+0000 7f8741400640 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da516240 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x5634da50a0f0 tx=0x7f8730000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=342bafbf6f9084ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.952+0000 7f8723e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.953+0000 7f8723e00640 1 -- 172.21.15.102:0/193748783 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730035040 con 0x5634da50dfb0 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.953+0000 7f874bdd2740 1 -- 172.21.15.102:0/193748783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8730005020 con 0x5634da50dfb0 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.953+0000 7f8723e00640 1 -- 172.21.15.102:0/193748783 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f87300051d0 con 0x5634da50dfb0 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.953+0000 7f8723e00640 1 -- 172.21.15.102:0/193748783 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f87300053d0 con 0x5634da50dfb0 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.954+0000 7f874bdd2740 10 client.26981 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.954+0000 7f874bdd2740 1 -- 172.21.15.102:0/193748783 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5634da538d60 con 0x5634da50dfb0 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.954+0000 7f8723e00640 1 -- 172.21.15.102:0/193748783 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8730005020 con 0x5634da50dfb0 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.954+0000 7f8723e00640 1 -- 172.21.15.102:0/193748783 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f87300317b0 con 0x5634da50dfb0 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.954+0000 7f874bdd2740 10 client.26981 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.954+0000 7f874bdd2740 10 client.26981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.954+0000 7f874bdd2740 20 client.26981 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.955+0000 7f874bdd2740 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634da53a0b0 0x5634da53c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.955+0000 7f874bdd2740 4 client.26981 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.955+0000 7f874bdd2740 1 -- 172.21.15.102:0/193748783 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634da47eff0 con 0x5634da53a0b0 2024-04-25T02:01:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.955+0000 7f8741e00640 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634da53a0b0 0x5634da53c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.957+0000 7f8723e00640 10 client.26981 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.957+0000 7f8741e00640 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634da53a0b0 0x5634da53c4a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=172dee1db420881a server_cookie=194ac7f093ee389d in_seq=0 out_seq=0 2024-04-25T02:01:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.958+0000 7f8723e00640 1 -- 172.21.15.102:0/193748783 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634da47eff0 con 0x5634da53a0b0 2024-04-25T02:01:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.958+0000 7f8723e00640 10 client.26981 handle_command_reply: tid=0 2024-04-25T02:01:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.959+0000 7f874bdd2740 1 client.26981 shutdown 2024-04-25T02:01:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.960+0000 7f874bdd2740 1 -- 172.21.15.102:0/193748783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 msgr2=0x5634da516240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.960+0000 7f874bdd2740 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da516240 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x5634da50a0f0 tx=0x7f8730000bc0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.960+0000 7f874bdd2740 1 -- 172.21.15.102:0/193748783 shutdown_connections 2024-04-25T02:01:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.960+0000 7f874bdd2740 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634da53a0b0 0x5634da53c4a0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.960+0000 7f874bdd2740 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da519190 0x5634da516cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.960+0000 7f874bdd2740 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50e8d0 0x5634da516780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.960+0000 7f874bdd2740 1 --2- 172.21.15.102:0/193748783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50dfb0 0x5634da516240 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.960+0000 7f874bdd2740 1 -- 172.21.15.102:0/193748783 >> 172.21.15.102:0/193748783 conn(0x5634da50a4c0 msgr2=0x5634da510240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.961+0000 7f8723e00640 0 client.26981 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.961+0000 7f874bdd2740 1 -- 172.21.15.102:0/193748783 shutdown_connections 2024-04-25T02:01:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.962+0000 7f874bdd2740 1 -- 172.21.15.102:0/193748783 wait complete. 2024-04-25T02:01:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:37.967+0000 7f874bdd2740 20 client.26981 trim_cache size 0 max 16384 2024-04-25T02:01:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.000+0000 7f874bdd2740 1 Processor -- start 2024-04-25T02:01:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.000+0000 7f874bdd2740 1 -- start start 2024-04-25T02:01:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.000+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50dfb0 0x5634da535c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.001+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50e8d0 0x5634da510980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.001+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da519190 0x5634da518070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.001+0000 7f874bdd2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f87441b6230 con 0x5634da50dfb0 2024-04-25T02:01:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.001+0000 7f874bdd2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8744072590 con 0x5634da519190 2024-04-25T02:01:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.001+0000 7f874bdd2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f87440703e0 con 0x5634da50e8d0 2024-04-25T02:01:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.001+0000 7f8741e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50e8d0 0x5634da510980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.001+0000 7f8741e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50e8d0 0x5634da510980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42622/0 (socket says 172.21.15.102:42622) 2024-04-25T02:01:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.001+0000 7f8741e00640 1 -- 172.21.15.102:0/3920411728 learned_addr learned my addr 172.21.15.102:0/3920411728 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.002+0000 7f8741e00640 1 -- 172.21.15.102:0/3920411728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da519190 msgr2=0x5634da518070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:01:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.002+0000 7f8741e00640 1 --2- 172.21.15.102:0/3920411728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da519190 0x5634da518070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.002+0000 7f8741e00640 1 -- 172.21.15.102:0/3920411728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50dfb0 msgr2=0x5634da535c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.002+0000 7f8741e00640 1 --2- 172.21.15.102:0/3920411728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50dfb0 0x5634da535c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.002+0000 7f8741e00640 1 -- 172.21.15.102:0/3920411728 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634da510ec0 con 0x5634da50e8d0 2024-04-25T02:01:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.002+0000 7f8741e00640 1 --2- 172.21.15.102:0/3920411728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50e8d0 0x5634da510980 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f873c009fa0 tx=0x7f873c00e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68724aad2d41d063 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.002+0000 7f872ea00640 1 -- 172.21.15.102:0/3920411728 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c0177b0 con 0x5634da50e8d0 2024-04-25T02:01:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.002+0000 7f872ea00640 1 -- 172.21.15.102:0/3920411728 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f873c01f030 con 0x5634da50e8d0 2024-04-25T02:01:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.002+0000 7f872ea00640 1 -- 172.21.15.102:0/3920411728 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c01b750 con 0x5634da50e8d0 2024-04-25T02:01:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.003+0000 7f874bdd2740 1 -- 172.21.15.102:0/3920411728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50e8d0 msgr2=0x5634da510980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.003+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3920411728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50e8d0 0x5634da510980 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f873c009fa0 tx=0x7f873c00e040 comp rx=0 tx=0).stop 2024-04-25T02:01:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.003+0000 7f874bdd2740 1 -- 172.21.15.102:0/3920411728 shutdown_connections 2024-04-25T02:01:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.003+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3920411728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da519190 0x5634da518070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.003+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3920411728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50e8d0 0x5634da510980 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.003+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3920411728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da50dfb0 0x5634da535c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.003+0000 7f874bdd2740 1 -- 172.21.15.102:0/3920411728 >> 172.21.15.102:0/3920411728 conn(0x5634da50a4c0 msgr2=0x5634da517960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.004+0000 7f874bdd2740 1 -- 172.21.15.102:0/3920411728 shutdown_connections 2024-04-25T02:01:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.004+0000 7f874bdd2740 1 -- 172.21.15.102:0/3920411728 wait complete. 2024-04-25T02:01:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.004+0000 7f874bdd2740 1 -- start start 2024-04-25T02:01:38.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.005+0000 7f874bdd2740 1 Processor -- start 2024-04-25T02:01:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.006+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50dfb0 0x5634da532190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.006+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50e8d0 0x5634da5326d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.006+0000 7f8741400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50dfb0 0x5634da532190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.006+0000 7f8741400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50dfb0 0x5634da532190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42624/0 (socket says 172.21.15.102:42624) 2024-04-25T02:01:38.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.006+0000 7f8741e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50e8d0 0x5634da5326d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.006+0000 7f8741e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50e8d0 0x5634da5326d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33150/0 (socket says 172.21.15.102:33150) 2024-04-25T02:01:38.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.007+0000 7f874bdd2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da519190 0x5634da52f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.007+0000 7f874bdd2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8744070a50 con 0x5634da519190 2024-04-25T02:01:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.007+0000 7f874bdd2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8744071ec0 con 0x5634da50e8d0 2024-04-25T02:01:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.007+0000 7f874bdd2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8744071120 con 0x5634da50dfb0 2024-04-25T02:01:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.007+0000 7f874bdd2740 10 client.0 _ll_register_callbacks cb 0x7f873c009770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.007+0000 7f8741400640 1 -- 172.21.15.102:0/3828838388 learned_addr learned my addr 172.21.15.102:0/3828838388 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.008+0000 7f8741e00640 1 -- 172.21.15.102:0/3828838388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50dfb0 msgr2=0x5634da532190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.008+0000 7f8741e00640 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50dfb0 0x5634da532190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.008+0000 7f8741e00640 1 -- 172.21.15.102:0/3828838388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da519190 msgr2=0x5634da52f250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:01:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.008+0000 7f8741e00640 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da519190 0x5634da52f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.008+0000 7f8741e00640 1 -- 172.21.15.102:0/3828838388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634da52fa50 con 0x5634da50e8d0 2024-04-25T02:01:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.009+0000 7f8741e00640 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50e8d0 0x5634da5326d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x5634da538cf0 tx=0x7f873c00c160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5df644e62e9f1f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.009+0000 7f8723e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.010+0000 7f8723e00640 1 -- 172.21.15.102:0/3828838388 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c017bc0 con 0x5634da50e8d0 2024-04-25T02:01:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.010+0000 7f874bdd2740 1 -- 172.21.15.102:0/3828838388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f873c01f030 con 0x5634da50e8d0 2024-04-25T02:01:38.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.011+0000 7f8723e00640 1 -- 172.21.15.102:0/3828838388 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f873c0179c0 con 0x5634da50e8d0 2024-04-25T02:01:38.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.012+0000 7f8723e00640 1 -- 172.21.15.102:0/3828838388 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f873c017bc0 con 0x5634da50e8d0 2024-04-25T02:01:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.012+0000 7f874bdd2740 10 client.26984 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.013+0000 7f874bdd2740 1 -- 172.21.15.102:0/3828838388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5634da5a0ea0 con 0x5634da50e8d0 2024-04-25T02:01:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.014+0000 7f8723e00640 1 -- 172.21.15.102:0/3828838388 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f873c01f030 con 0x5634da50e8d0 2024-04-25T02:01:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.015+0000 7f8723e00640 1 -- 172.21.15.102:0/3828838388 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f873c02b020 con 0x5634da50e8d0 2024-04-25T02:01:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.016+0000 7f874bdd2740 10 client.26984 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.016+0000 7f874bdd2740 10 client.26984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.016+0000 7f874bdd2740 20 client.26984 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.016+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634da5a1660 0x5634da5a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.016+0000 7f874bdd2740 4 client.26984 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:01:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.016+0000 7f874bdd2740 1 -- 172.21.15.102:0/3828838388 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5634da479340 con 0x5634da5a1660 2024-04-25T02:01:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.017+0000 7f8741400640 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634da5a1660 0x5634da5a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.019+0000 7f8741400640 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634da5a1660 0x5634da5a3a80 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a75d421719dd992 server_cookie=e3815c75f5b8d79c in_seq=0 out_seq=0 2024-04-25T02:01:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.019+0000 7f8723e00640 10 client.26984 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.027+0000 7f1797c00640 1 -- 172.21.15.102:0/839169343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900ffa10 msgr2=0x7f17900ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.027+0000 7f1797c00640 1 --2- 172.21.15.102:0/839169343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900ffa10 0x7f17900ffe90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f177c01a910 tx=0x7f177c03b430 comp rx=0 tx=0).stop 2024-04-25T02:01:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.028+0000 7f1797c00640 1 -- 172.21.15.102:0/839169343 shutdown_connections 2024-04-25T02:01:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.028+0000 7f1797c00640 1 --2- 172.21.15.102:0/839169343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17901003d0 0x7f179010c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.028+0000 7f1797c00640 1 --2- 172.21.15.102:0/839169343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17900ffa10 0x7f17900ffe90 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.028+0000 7f1797c00640 1 --2- 172.21.15.102:0/839169343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17901059d0 0x7f1790105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.028+0000 7f1797c00640 1 -- 172.21.15.102:0/839169343 >> 172.21.15.102:0/839169343 conn(0x7f17900767a0 msgr2=0x7f17900fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.029+0000 7f1797c00640 1 -- 172.21.15.102:0/839169343 shutdown_connections 2024-04-25T02:01:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.029+0000 7f1797c00640 1 -- 172.21.15.102:0/839169343 wait complete. 2024-04-25T02:01:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.030+0000 7f1797c00640 1 Processor -- start 2024-04-25T02:01:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.031+0000 7f1797c00640 1 -- start start 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f1797c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17900ffa10 0x7f1790108260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f1797c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17901003d0 0x7f17901087a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f1797c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17901059d0 0x7f179019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f1797c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f179006d6c0 con 0x7f17901059d0 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f1797c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f179006d540 con 0x7f17901003d0 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f1797c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f179006d840 con 0x7f17900ffa10 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f1795200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17900ffa10 0x7f1790108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f1795200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17900ffa10 0x7f1790108260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42650/0 (socket says 172.21.15.102:42650) 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f1795200640 1 -- 172.21.15.102:0/3134530524 learned_addr learned my addr 172.21.15.102:0/3134530524 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.032+0000 7f178fe00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17901003d0 0x7f17901087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.033+0000 7f1795c00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17901059d0 0x7f179019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.033+0000 7f178fe00640 1 -- 172.21.15.102:0/3134530524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17900ffa10 msgr2=0x7f1790108260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.033+0000 7f178fe00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17900ffa10 0x7f1790108260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.033+0000 7f178fe00640 1 -- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17901059d0 msgr2=0x7f179019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.033+0000 7f178fe00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17901059d0 0x7f179019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.033+0000 7f178fe00640 1 -- 172.21.15.102:0/3134530524 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17901931a0 con 0x7f17901003d0 2024-04-25T02:01:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.034+0000 7f8723e00640 1 -- 172.21.15.102:0/3828838388 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5634da479340 con 0x5634da5a1660 2024-04-25T02:01:38.040 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.034+0000 7f8723e00640 10 client.26984 handle_command_reply: tid=0 2024-04-25T02:01:38.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.034+0000 7f178fe00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17901003d0 0x7f17901087a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f177c01a860 tx=0x7f177c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.034+0000 7f178d600640 1 -- 172.21.15.102:0/3134530524 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c04f070 con 0x7f17901003d0 2024-04-25T02:01:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.034+0000 7f178d600640 1 -- 172.21.15.102:0/3134530524 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f177c004a40 con 0x7f17901003d0 2024-04-25T02:01:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.034+0000 7f1797c00640 1 -- 172.21.15.102:0/3134530524 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1790193430 con 0x7f17901003d0 2024-04-25T02:01:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.034+0000 7f874bdd2740 1 client.26984 shutdown 2024-04-25T02:01:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.034+0000 7f1797c00640 1 -- 172.21.15.102:0/3134530524 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1790193970 con 0x7f17901003d0 2024-04-25T02:01:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.035+0000 7f178d600640 1 -- 172.21.15.102:0/3134530524 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c046030 con 0x7f17901003d0 2024-04-25T02:01:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.036+0000 7f874bdd2740 1 -- 172.21.15.102:0/3828838388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50e8d0 msgr2=0x5634da5326d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.036+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50e8d0 0x5634da5326d0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x5634da538cf0 tx=0x7f873c00c160 comp rx=0 tx=0).stop 2024-04-25T02:01:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.039+0000 7f874bdd2740 1 -- 172.21.15.102:0/3828838388 shutdown_connections 2024-04-25T02:01:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.039+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634da5a1660 0x5634da5a3a80 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.039+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634da519190 0x5634da52f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.039+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634da50e8d0 0x5634da5326d0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.039+0000 7f874bdd2740 1 --2- 172.21.15.102:0/3828838388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634da50dfb0 0x5634da532190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.039+0000 7f874bdd2740 1 -- 172.21.15.102:0/3828838388 >> 172.21.15.102:0/3828838388 conn(0x5634da50a4c0 msgr2=0x5634da516a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.039+0000 7f8723e00640 0 client.26984 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.039+0000 7f874bdd2740 1 -- 172.21.15.102:0/3828838388 shutdown_connections 2024-04-25T02:01:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.039+0000 7f874bdd2740 1 -- 172.21.15.102:0/3828838388 wait complete. 2024-04-25T02:01:38.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.040+0000 7f874bdd2740 20 client.26984 trim_cache size 0 max 16384 2024-04-25T02:01:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:01:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:01:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:01:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:01:38.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.040+0000 7f178d600640 1 -- 172.21.15.102:0/3134530524 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f177c052070 con 0x7f17901003d0 2024-04-25T02:01:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.042+0000 7f8743200640 1 -- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87280a30f0 msgr2=0x7f87280a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.042+0000 7f178d600640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17780a2e30 0x7f17780a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.042+0000 7f8743200640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87280a30f0 0x7f87280a55b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8734097670 tx=0x7f873400bf60 comp rx=0 tx=0).stop 2024-04-25T02:01:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.042+0000 7f8743200640 1 -- 172.21.15.102:0/1969144263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744109790 msgr2=0x7f874410f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.042+0000 7f8743200640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744109790 0x7f874410f1c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f873400b3f0 tx=0x7f873400b8b0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.042+0000 7f178d600640 1 -- 172.21.15.102:0/3134530524 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f177c1013d0 con 0x7f17901003d0 2024-04-25T02:01:38.048 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.043+0000 7f8743200640 1 -- 172.21.15.102:0/1969144263 shutdown_connections 2024-04-25T02:01:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.043+0000 7f8743200640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f87280a30f0 0x7f87280a55b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.043+0000 7f8743200640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8744109790 0x7f874410f1c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.043+0000 7f8743200640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8744104140 0x7f8744116310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.043+0000 7f8743200640 1 --2- 172.21.15.102:0/1969144263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8744103780 0x7f8744115dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.043+0000 7f1795200640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17780a2e30 0x7f17780a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.043+0000 7f8743200640 1 -- 172.21.15.102:0/1969144263 >> 172.21.15.102:0/1969144263 conn(0x7f87440ff270 msgr2=0x7f87440ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.044+0000 7f8743200640 1 -- 172.21.15.102:0/1969144263 shutdown_connections 2024-04-25T02:01:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:38.044+0000 7f8743200640 1 -- 172.21.15.102:0/1969144263 wait complete. 2024-04-25T02:01:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.044+0000 7f179ab70740 1 Processor -- start 2024-04-25T02:01:38.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.045+0000 7f179ab70740 1 -- start start 2024-04-25T02:01:38.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.045+0000 7f1795200640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17780a2e30 0x7f17780a52f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f17840058c0 tx=0x7f178400a380 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.046+0000 7f179ab70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f35140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.046+0000 7f179ab70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f35680 0x5621a0f3f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.047+0000 7f179ab70740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f3fd50 0x5621a0f42140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.047+0000 7f179ab70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5621a0e93f40 con 0x5621a0f35680 2024-04-25T02:01:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.047+0000 7f179ab70740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5621a0e99190 con 0x5621a0f3fd50 2024-04-25T02:01:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.047+0000 7f179ab70740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5621a0e41a10 con 0x5621a0f34d60 2024-04-25T02:01:38.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.047+0000 7f1795200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f35140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.047+0000 7f1795200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f35140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42666/0 (socket says 172.21.15.102:42666) 2024-04-25T02:01:38.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.047+0000 7f1795200640 1 -- 172.21.15.102:0/1601273382 learned_addr learned my addr 172.21.15.102:0/1601273382 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.047+0000 7f1795c00640 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f3fd50 0x5621a0f42140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.047+0000 7f178fe00640 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f35680 0x5621a0f3f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.048+0000 7f1795c00640 1 -- 172.21.15.102:0/1601273382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 msgr2=0x5621a0f35140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.048+0000 7f1795c00640 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f35140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.048+0000 7f1795c00640 1 -- 172.21.15.102:0/1601273382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f35680 msgr2=0x5621a0f3f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.048+0000 7f1795c00640 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f35680 0x5621a0f3f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.048+0000 7f1795c00640 1 -- 172.21.15.102:0/1601273382 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a0c757e0 con 0x5621a0f3fd50 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.048+0000 7f1795200640 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f35140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.048+0000 7f1795c00640 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f3fd50 0x5621a0f42140 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1788009e80 tx=0x7f17880117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ef0cd358a9f0989 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.048+0000 7f1782000640 1 -- 172.21.15.102:0/1601273382 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17880099c0 con 0x5621a0f3fd50 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.049+0000 7f1782000640 1 -- 172.21.15.102:0/1601273382 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1788003b70 con 0x5621a0f3fd50 2024-04-25T02:01:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.049+0000 7f1782000640 1 -- 172.21.15.102:0/1601273382 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788012990 con 0x5621a0f3fd50 2024-04-25T02:01:38.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.050+0000 7f179ab70740 1 -- 172.21.15.102:0/1601273382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f3fd50 msgr2=0x5621a0f42140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.050+0000 7f179ab70740 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f3fd50 0x5621a0f42140 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1788009e80 tx=0x7f17880117b0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.051+0000 7f179ab70740 1 -- 172.21.15.102:0/1601273382 shutdown_connections 2024-04-25T02:01:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.051+0000 7f179ab70740 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f3fd50 0x5621a0f42140 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.051+0000 7f179ab70740 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f35680 0x5621a0f3f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.051+0000 7f179ab70740 1 --2- 172.21.15.102:0/1601273382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f35140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.051+0000 7f179ab70740 1 -- 172.21.15.102:0/1601273382 >> 172.21.15.102:0/1601273382 conn(0x5621a0f31080 msgr2=0x5621a0f31460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.051+0000 7f179ab70740 1 -- 172.21.15.102:0/1601273382 shutdown_connections 2024-04-25T02:01:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.051+0000 7f179ab70740 1 -- 172.21.15.102:0/1601273382 wait complete. 2024-04-25T02:01:38.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.052+0000 7f179ab70740 1 -- start start 2024-04-25T02:01:38.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.052+0000 7f179ab70740 1 Processor -- start 2024-04-25T02:01:38.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.053+0000 7f179ab70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f3ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.053+0000 7f179ab70740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 0x5621a0f3d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.053+0000 7f179ab70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 0x5621a0f3d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f178fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 0x5621a0f3d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f1795200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f3ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f178fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 0x5621a0f3d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33180/0 (socket says 172.21.15.102:33180) 2024-04-25T02:01:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f1795200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f3ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42680/0 (socket says 172.21.15.102:42680) 2024-04-25T02:01:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f179ab70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5621a0e54e70 con 0x5621a0f3fd50 2024-04-25T02:01:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f178fe00640 1 -- 172.21.15.102:0/3138814828 learned_addr learned my addr 172.21.15.102:0/3138814828 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5621a0e95150 con 0x5621a0f35680 2024-04-25T02:01:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5621a0f42be0 con 0x5621a0f34d60 2024-04-25T02:01:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f179ab70740 10 client.0 _ll_register_callbacks cb 0x5621a0e901f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:38.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f178fe00640 1 -- 172.21.15.102:0/3138814828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 msgr2=0x5621a0f3ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f178fe00640 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f3ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f178fe00640 1 -- 172.21.15.102:0/3138814828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 msgr2=0x5621a0f3d740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:01:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f1795c00640 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 0x5621a0f3d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f178fe00640 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 0x5621a0f3d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.054+0000 7f178fe00640 1 -- 172.21.15.102:0/3138814828 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621a0cd5f70 con 0x5621a0f35680 2024-04-25T02:01:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.055+0000 7f178fe00640 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 0x5621a0f3d200 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f177c0c9c90 tx=0x7f177c0cd2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3c1e1320c4b6a36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.055+0000 7f1776a00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:38.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.055+0000 7f1776a00640 1 -- 172.21.15.102:0/3138814828 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c0e6060 con 0x5621a0f35680 2024-04-25T02:01:38.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.055+0000 7f1776a00640 1 -- 172.21.15.102:0/3138814828 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f177c0c9ea0 con 0x5621a0f35680 2024-04-25T02:01:38.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.055+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1788003b70 con 0x5621a0f35680 2024-04-25T02:01:38.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.055+0000 7f1776a00640 1 -- 172.21.15.102:0/3138814828 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f177c0eb050 con 0x5621a0f35680 2024-04-25T02:01:38.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.055+0000 7f179ab70740 10 client.26996 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:38.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.055+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5621a0b6d170 con 0x5621a0f35680 2024-04-25T02:01:38.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.056+0000 7f1776a00640 1 -- 172.21.15.102:0/3138814828 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1788003b70 con 0x5621a0f35680 2024-04-25T02:01:38.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.057+0000 7f1776a00640 1 -- 172.21.15.102:0/3138814828 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f177c0fed70 con 0x5621a0f35680 2024-04-25T02:01:38.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.057+0000 7f179ab70740 10 client.26996 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:38.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.057+0000 7f179ab70740 10 client.26996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:38.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.057+0000 7f179ab70740 20 client.26996 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:38.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.057+0000 7f179ab70740 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621a0f50150 0x5621a0f525b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.057+0000 7f179ab70740 4 client.26996 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:38.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.057+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621a0ea66d0 con 0x5621a0f50150 2024-04-25T02:01:38.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.058+0000 7f1795200640 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621a0f50150 0x5621a0f525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.058+0000 7f1776a00640 10 client.26996 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.058+0000 7f1795200640 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621a0f50150 0x5621a0f525b0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31ee6265eb435a00 server_cookie=9afa5a326888549c in_seq=0 out_seq=0 2024-04-25T02:01:38.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.059+0000 7f1776a00640 1 -- 172.21.15.102:0/3138814828 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621a0ea66d0 con 0x5621a0f50150 2024-04-25T02:01:38.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.059+0000 7f1776a00640 10 client.26996 handle_command_reply: tid=0 2024-04-25T02:01:38.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.059+0000 7f179ab70740 1 client.26996 shutdown 2024-04-25T02:01:38.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.060+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 msgr2=0x5621a0f3d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.060+0000 7f179ab70740 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 0x5621a0f3d200 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f177c0c9c90 tx=0x7f177c0cd2a0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.061+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 shutdown_connections 2024-04-25T02:01:38.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.061+0000 7f179ab70740 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621a0f50150 0x5621a0f525b0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.061+0000 7f179ab70740 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 0x5621a0f3d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.061+0000 7f179ab70740 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 0x5621a0f3d200 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.061+0000 7f179ab70740 1 --2- 172.21.15.102:0/3138814828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f34d60 0x5621a0f3ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.061+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 >> 172.21.15.102:0/3138814828 conn(0x5621a0f31080 msgr2=0x5621a0f36d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.061+0000 7f1776a00640 0 client.26996 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.062+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 shutdown_connections 2024-04-25T02:01:38.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.062+0000 7f179ab70740 1 -- 172.21.15.102:0/3138814828 wait complete. 2024-04-25T02:01:38.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.062+0000 7f179ab70740 20 client.26996 trim_cache size 0 max 16384 2024-04-25T02:01:38.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 345.0 s, 2359.0 tps, lat 42.632 ms stddev 58.845 2024-04-25T02:01:38.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.104+0000 7f179ab70740 1 Processor -- start 2024-04-25T02:01:38.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.104+0000 7f179ab70740 1 -- start start 2024-04-25T02:01:38.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.105+0000 7f179ab70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f34d60 0x5621a0f4b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.105+0000 7f179ab70740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 0x5621a0f57ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.105+0000 7f179ab70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f3fd50 0x5621a0f58400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.105+0000 7f179ab70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1770010e60 con 0x5621a0f34d60 2024-04-25T02:01:38.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.105+0000 7f179ab70740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f177000e6d0 con 0x5621a0f35680 2024-04-25T02:01:38.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.105+0000 7f179ab70740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f177000c5d0 con 0x5621a0f3fd50 2024-04-25T02:01:38.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.106+0000 7f1795c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f3fd50 0x5621a0f58400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.106+0000 7f1795c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f3fd50 0x5621a0f58400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42686/0 (socket says 172.21.15.102:42686) 2024-04-25T02:01:38.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.106+0000 7f1795c00640 1 -- 172.21.15.102:0/332155245 learned_addr learned my addr 172.21.15.102:0/332155245 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.106+0000 7f1795200640 1 --2- 172.21.15.102:0/332155245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f34d60 0x5621a0f4b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.107+0000 7f1795c00640 1 -- 172.21.15.102:0/332155245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 msgr2=0x5621a0f57ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.107+0000 7f1795c00640 1 --2- 172.21.15.102:0/332155245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 0x5621a0f57ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.107+0000 7f1795c00640 1 -- 172.21.15.102:0/332155245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f34d60 msgr2=0x5621a0f4b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.107+0000 7f1795c00640 1 --2- 172.21.15.102:0/332155245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f34d60 0x5621a0f4b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.107+0000 7f1795c00640 1 -- 172.21.15.102:0/332155245 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a0f52c30 con 0x5621a0f3fd50 2024-04-25T02:01:38.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.108+0000 7f1795c00640 1 --2- 172.21.15.102:0/332155245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f3fd50 0x5621a0f58400 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1788006010 tx=0x7f1788005a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed777d7049be5817 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.108+0000 7f1781600640 1 -- 172.21.15.102:0/332155245 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1788028070 con 0x5621a0f3fd50 2024-04-25T02:01:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.109+0000 7f1781600640 1 -- 172.21.15.102:0/332155245 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1788003e50 con 0x5621a0f3fd50 2024-04-25T02:01:38.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.109+0000 7f1781600640 1 -- 172.21.15.102:0/332155245 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17880124b0 con 0x5621a0f3fd50 2024-04-25T02:01:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.112+0000 7f179ab70740 1 -- 172.21.15.102:0/332155245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f3fd50 msgr2=0x5621a0f58400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.112+0000 7f179ab70740 1 --2- 172.21.15.102:0/332155245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f3fd50 0x5621a0f58400 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1788006010 tx=0x7f1788005a40 comp rx=0 tx=0).stop 2024-04-25T02:01:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.114+0000 7f179ab70740 1 -- 172.21.15.102:0/332155245 shutdown_connections 2024-04-25T02:01:38.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.114+0000 7f179ab70740 1 --2- 172.21.15.102:0/332155245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f3fd50 0x5621a0f58400 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.114+0000 7f179ab70740 1 --2- 172.21.15.102:0/332155245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f35680 0x5621a0f57ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.114+0000 7f179ab70740 1 --2- 172.21.15.102:0/332155245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f34d60 0x5621a0f4b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.114+0000 7f179ab70740 1 -- 172.21.15.102:0/332155245 >> 172.21.15.102:0/332155245 conn(0x5621a0f31080 msgr2=0x5621a0f301e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.115+0000 7f179ab70740 1 -- 172.21.15.102:0/332155245 shutdown_connections 2024-04-25T02:01:38.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.115+0000 7f179ab70740 1 -- 172.21.15.102:0/332155245 wait complete. 2024-04-25T02:01:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.115+0000 7f179ab70740 1 -- start start 2024-04-25T02:01:38.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.116+0000 7f179ab70740 1 Processor -- start 2024-04-25T02:01:38.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.117+0000 7f179ab70740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f34d60 0x5621a0f44e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.118+0000 7f179ab70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f35680 0x5621a0f453d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.118+0000 7f179ab70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 0x5621a0f45910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.118+0000 7f179ab70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f177000cb90 con 0x5621a0f3fd50 2024-04-25T02:01:38.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.118+0000 7f179ab70740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f177000e000 con 0x5621a0f34d60 2024-04-25T02:01:38.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.118+0000 7f179ab70740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f177000d260 con 0x5621a0f35680 2024-04-25T02:01:38.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.118+0000 7f179ab70740 10 client.0 _ll_register_callbacks cb 0x7f177c0cd400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:38.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.119+0000 7f1795c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 0x5621a0f45910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.119+0000 7f1795200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f34d60 0x5621a0f44e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.120+0000 7f1795200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f34d60 0x5621a0f44e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33200/0 (socket says 172.21.15.102:33200) 2024-04-25T02:01:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.120+0000 7f178fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f35680 0x5621a0f453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.120+0000 7f178fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f35680 0x5621a0f453d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42694/0 (socket says 172.21.15.102:42694) 2024-04-25T02:01:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.120+0000 7f178fe00640 1 -- 172.21.15.102:0/2397767077 learned_addr learned my addr 172.21.15.102:0/2397767077 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.121+0000 7f178fe00640 1 -- 172.21.15.102:0/2397767077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f34d60 msgr2=0x5621a0f44e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.121+0000 7f178fe00640 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f34d60 0x5621a0f44e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.121+0000 7f178fe00640 1 -- 172.21.15.102:0/2397767077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 msgr2=0x5621a0f45910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.121+0000 7f178fe00640 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 0x5621a0f45910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.121+0000 7f178fe00640 1 -- 172.21.15.102:0/2397767077 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621a0fb6740 con 0x5621a0f35680 2024-04-25T02:01:38.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.121+0000 7f1795200640 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f34d60 0x5621a0f44e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:38.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.122+0000 7f178fe00640 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f35680 0x5621a0f453d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f177c0ec400 tx=0x7f177c004e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=546cd1fe96eb4c43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.122+0000 7f1776a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.123+0000 7f1776a00640 1 -- 172.21.15.102:0/2397767077 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177c059070 con 0x5621a0f35680 2024-04-25T02:01:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.123+0000 7f1776a00640 1 -- 172.21.15.102:0/2397767077 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f177c0b80b0 con 0x5621a0f35680 2024-04-25T02:01:38.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.123+0000 7f179ab70740 1 -- 172.21.15.102:0/2397767077 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1788003e50 con 0x5621a0f35680 2024-04-25T02:01:38.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.123+0000 7f1776a00640 1 -- 172.21.15.102:0/2397767077 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f177c0e6020 con 0x5621a0f35680 2024-04-25T02:01:38.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.124+0000 7f179ab70740 10 client.26959 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:38.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.124+0000 7f179ab70740 1 -- 172.21.15.102:0/2397767077 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5621a0fb79b0 con 0x5621a0f35680 2024-04-25T02:01:38.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.124+0000 7f1776a00640 1 -- 172.21.15.102:0/2397767077 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1788003e50 con 0x5621a0f35680 2024-04-25T02:01:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.125+0000 7f1776a00640 1 -- 172.21.15.102:0/2397767077 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f177c0d60f0 con 0x5621a0f35680 2024-04-25T02:01:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.125+0000 7f179ab70740 10 client.26959 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.125+0000 7f179ab70740 10 client.26959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.125+0000 7f179ab70740 20 client.26959 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:38.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.125+0000 7f179ab70740 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621a0fb7fd0 0x5621a0fba3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.125+0000 7f179ab70740 4 client.26959 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T02:01:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.125+0000 7f179ab70740 1 -- 172.21.15.102:0/2397767077 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5621a0ea0a20 con 0x5621a0fb7fd0 2024-04-25T02:01:38.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.125+0000 7f1795200640 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621a0fb7fd0 0x5621a0fba3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.127+0000 7f1795200640 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621a0fb7fd0 0x5621a0fba3f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c50a31622c3ac6ee server_cookie=ef9e887de3afb659 in_seq=0 out_seq=0 2024-04-25T02:01:38.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.127+0000 7f1776a00640 10 client.26959 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.128+0000 7f1776a00640 1 -- 172.21.15.102:0/2397767077 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5621a0ea0a20 con 0x5621a0fb7fd0 2024-04-25T02:01:38.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.128+0000 7f1776a00640 10 client.26959 handle_command_reply: tid=0 2024-04-25T02:01:38.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.129+0000 7f179ab70740 1 client.26959 shutdown 2024-04-25T02:01:38.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.129+0000 7f179ab70740 1 -- 172.21.15.102:0/2397767077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f35680 msgr2=0x5621a0f453d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.129+0000 7f179ab70740 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f35680 0x5621a0f453d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f177c0ec400 tx=0x7f177c004e00 comp rx=0 tx=0).stop 2024-04-25T02:01:38.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.131+0000 7f179ab70740 1 -- 172.21.15.102:0/2397767077 shutdown_connections 2024-04-25T02:01:38.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.131+0000 7f179ab70740 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621a0fb7fd0 0x5621a0fba3f0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.131+0000 7f179ab70740 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621a0f3fd50 0x5621a0f45910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.131+0000 7f179ab70740 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621a0f35680 0x5621a0f453d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.131+0000 7f179ab70740 1 --2- 172.21.15.102:0/2397767077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621a0f34d60 0x5621a0f44e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.131+0000 7f1776a00640 0 client.26959 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.131+0000 7f179ab70740 1 -- 172.21.15.102:0/2397767077 >> 172.21.15.102:0/2397767077 conn(0x5621a0f31080 msgr2=0x5621a0f364f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.131+0000 7f179ab70740 1 -- 172.21.15.102:0/2397767077 shutdown_connections 2024-04-25T02:01:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.132+0000 7f179ab70740 1 -- 172.21.15.102:0/2397767077 wait complete. 2024-04-25T02:01:38.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.132+0000 7f179ab70740 20 client.26959 trim_cache size 0 max 16384 2024-04-25T02:01:38.137 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:01:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "return_code": 0, 2024-04-25T02:01:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_tag": "1d67d1ea-ed6c-4f41-ae39-2db31f2f79db", 2024-04-25T02:01:38.138 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": "asynchronous" 2024-04-25T02:01:38.138 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.133+0000 7f1797c00640 1 -- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17780a2e30 msgr2=0x7f17780a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.133+0000 7f1797c00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17780a2e30 0x7f17780a52f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f17840058c0 tx=0x7f178400a380 comp rx=0 tx=0).stop 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.133+0000 7f1797c00640 1 -- 172.21.15.102:0/3134530524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17901003d0 msgr2=0x7f17901087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.133+0000 7f1797c00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17901003d0 0x7f17901087a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f177c01a860 tx=0x7f177c0046c0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.133+0000 7f1797c00640 1 -- 172.21.15.102:0/3134530524 shutdown_connections 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.134+0000 7f1797c00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17780a2e30 0x7f17780a52f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.134+0000 7f1797c00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17901059d0 0x7f179019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.134+0000 7f1797c00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17901003d0 0x7f17901087a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.134+0000 7f1797c00640 1 --2- 172.21.15.102:0/3134530524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17900ffa10 0x7f1790108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.134+0000 7f1797c00640 1 -- 172.21.15.102:0/3134530524 >> 172.21.15.102:0/3134530524 conn(0x7f17900767a0 msgr2=0x7f1790072250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.134+0000 7f1797c00640 1 -- 172.21.15.102:0/3134530524 shutdown_connections 2024-04-25T02:01:38.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.134+0000 7f1797c00640 1 -- 172.21.15.102:0/3134530524 wait complete. 2024-04-25T02:01:38.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T02:01:38.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T02:01:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.467+0000 7f2157e00640 1 -- 172.21.15.102:0/2854481865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 msgr2=0x7f2158109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.467+0000 7f2157e00640 1 --2- 172.21.15.102:0/2854481865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 0x7f2158109bd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2148009900 tx=0x7f214802a3f0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.467+0000 7f2157e00640 1 -- 172.21.15.102:0/2854481865 shutdown_connections 2024-04-25T02:01:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.467+0000 7f2157e00640 1 --2- 172.21.15.102:0/2854481865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21581041a0 0x7f21581106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.467+0000 7f2157e00640 1 --2- 172.21.15.102:0/2854481865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21581037e0 0x7f2158103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.467+0000 7f2157e00640 1 --2- 172.21.15.102:0/2854481865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 0x7f2158109bd0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.467+0000 7f2157e00640 1 -- 172.21.15.102:0/2854481865 >> 172.21.15.102:0/2854481865 conn(0x7f21580ff2b0 msgr2=0x7f21581016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.468+0000 7f2157e00640 1 -- 172.21.15.102:0/2854481865 shutdown_connections 2024-04-25T02:01:38.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.468+0000 7f2157e00640 1 -- 172.21.15.102:0/2854481865 wait complete. 2024-04-25T02:01:38.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.470+0000 7f2157e00640 1 Processor -- start 2024-04-25T02:01:38.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.470+0000 7f2157e00640 1 -- start start 2024-04-25T02:01:38.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.471+0000 7f2157e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21581037e0 0x7f215819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.471+0000 7f2157e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21581041a0 0x7f215819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.471+0000 7f2157e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 0x7f215819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.471+0000 7f2157e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2158113c10 con 0x7f21581037e0 2024-04-25T02:01:38.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.471+0000 7f2157e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2158113a90 con 0x7f21581097f0 2024-04-25T02:01:38.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.471+0000 7f2157e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2158113d90 con 0x7f21581041a0 2024-04-25T02:01:38.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.471+0000 7f2157400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 0x7f215819e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.471+0000 7f2157400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 0x7f215819e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33226/0 (socket says 172.21.15.102:33226) 2024-04-25T02:01:38.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.471+0000 7f2157400640 1 -- 172.21.15.102:0/2129328389 learned_addr learned my addr 172.21.15.102:0/2129328389 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.472+0000 7f2156a00640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21581037e0 0x7f215819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.472+0000 7f2157400640 1 -- 172.21.15.102:0/2129328389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21581041a0 msgr2=0x7f215819a550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T02:01:38.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.472+0000 7f2157400640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21581041a0 0x7f215819a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.472+0000 7f2157400640 1 -- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21581037e0 msgr2=0x7f215819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.472+0000 7f2157400640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21581037e0 0x7f215819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.472+0000 7f2157400640 1 -- 172.21.15.102:0/2129328389 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215819f000 con 0x7f21581097f0 2024-04-25T02:01:38.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.472+0000 7f2156a00640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21581037e0 0x7f215819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:38.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.473+0000 7f2157400640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 0x7f215819e8d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f215000eb10 tx=0x7f215000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.473+0000 7f214fe00640 1 -- 172.21.15.102:0/2129328389 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150004b20 con 0x7f21581097f0 2024-04-25T02:01:38.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.473+0000 7f214fe00640 1 -- 172.21.15.102:0/2129328389 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2150004cc0 con 0x7f21581097f0 2024-04-25T02:01:38.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.473+0000 7f2157e00640 1 -- 172.21.15.102:0/2129328389 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f215819f2f0 con 0x7f21581097f0 2024-04-25T02:01:38.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.473+0000 7f2157e00640 1 -- 172.21.15.102:0/2129328389 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21581a6b80 con 0x7f21581097f0 2024-04-25T02:01:38.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.473+0000 7f214fe00640 1 -- 172.21.15.102:0/2129328389 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150010760 con 0x7f21581097f0 2024-04-25T02:01:38.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.479+0000 7f214fe00640 1 -- 172.21.15.102:0/2129328389 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2150010900 con 0x7f21581097f0 2024-04-25T02:01:38.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.480+0000 7f2160122740 1 Processor -- start 2024-04-25T02:01:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.480+0000 7f2160122740 1 -- start start 2024-04-25T02:01:38.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.481+0000 7f2160122740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877dddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.481+0000 7f2160122740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877de310 0x5634877e84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.481+0000 7f2160122740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877e89e0 0x5634877eadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.481+0000 7f2160122740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x563487739560 con 0x5634877de310 2024-04-25T02:01:38.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.481+0000 7f2160122740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5634876e1930 con 0x5634877dd9f0 2024-04-25T02:01:38.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.481+0000 7f2160122740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563487735520 con 0x5634877e89e0 2024-04-25T02:01:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.482+0000 7f2156a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877de310 0x5634877e84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.482+0000 7f2156a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877de310 0x5634877e84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37392/0 (socket says 172.21.15.102:37392) 2024-04-25T02:01:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.482+0000 7f2156a00640 1 -- 172.21.15.102:0/3204988273 learned_addr learned my addr 172.21.15.102:0/3204988273 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.482+0000 7f214fe00640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f213c0a2dc0 0x7f213c0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.483+0000 7f214fe00640 1 -- 172.21.15.102:0/2129328389 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f21500cf950 con 0x7f21581097f0 2024-04-25T02:01:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.483+0000 7f2156a00640 1 -- 172.21.15.102:0/3204988273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877e89e0 msgr2=0x5634877eadd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:01:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.483+0000 7f2156a00640 1 --2- 172.21.15.102:0/3204988273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877e89e0 0x5634877eadd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.483+0000 7f2156a00640 1 -- 172.21.15.102:0/3204988273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 msgr2=0x5634877dddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.483+0000 7f2157400640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f213c0a2dc0 0x7f213c0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.483+0000 7f214e000640 1 --2- 172.21.15.102:0/3204988273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877dddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.484+0000 7f2156a00640 1 --2- 172.21.15.102:0/3204988273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877dddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.484+0000 7f2156a00640 1 -- 172.21.15.102:0/3204988273 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634875177d0 con 0x5634877de310 2024-04-25T02:01:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.484+0000 7f214e000640 1 --2- 172.21.15.102:0/3204988273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877dddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.485+0000 7f2157400640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f213c0a2dc0 0x7f213c0a5280 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2150009ae0 tx=0x7f2150009a70 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.485+0000 7f2156a00640 1 --2- 172.21.15.102:0/3204988273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877de310 0x5634877e84a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2148009850 tx=0x7f21480040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84549ce58cc09ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.486+0000 7f2147400640 1 -- 172.21.15.102:0/3204988273 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214803f070 con 0x5634877de310 2024-04-25T02:01:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.486+0000 7f2147400640 1 -- 172.21.15.102:0/3204988273 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21480044c0 con 0x5634877de310 2024-04-25T02:01:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.486+0000 7f2147400640 1 -- 172.21.15.102:0/3204988273 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148035030 con 0x5634877de310 2024-04-25T02:01:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.488+0000 7f2160122740 1 -- 172.21.15.102:0/3204988273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877de310 msgr2=0x5634877e84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.488+0000 7f2160122740 1 --2- 172.21.15.102:0/3204988273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877de310 0x5634877e84a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2148009850 tx=0x7f21480040d0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.488+0000 7f2160122740 1 -- 172.21.15.102:0/3204988273 shutdown_connections 2024-04-25T02:01:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.488+0000 7f2160122740 1 --2- 172.21.15.102:0/3204988273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877e89e0 0x5634877eadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.488+0000 7f2160122740 1 --2- 172.21.15.102:0/3204988273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877de310 0x5634877e84a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.488+0000 7f2160122740 1 --2- 172.21.15.102:0/3204988273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877dddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.488+0000 7f2160122740 1 -- 172.21.15.102:0/3204988273 >> 172.21.15.102:0/3204988273 conn(0x5634877d9d10 msgr2=0x5634877da0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.489+0000 7f2160122740 1 -- 172.21.15.102:0/3204988273 shutdown_connections 2024-04-25T02:01:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.489+0000 7f2160122740 1 -- 172.21.15.102:0/3204988273 wait complete. 2024-04-25T02:01:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.489+0000 7f2160122740 1 -- start start 2024-04-25T02:01:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.490+0000 7f2160122740 1 Processor -- start 2024-04-25T02:01:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2160122740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877e5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2160122740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2160122740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877e89e0 0x5634877e6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2160122740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5634877eb770 con 0x5634877e89e0 2024-04-25T02:01:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2160122740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5634876f4d90 con 0x5634877dd9f0 2024-04-25T02:01:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2160122740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5634877eb8f0 con 0x5634877de310 2024-04-25T02:01:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2160122740 10 client.0 _ll_register_callbacks cb 0x5634876f4780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2156a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2156a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42720/0 (socket says 172.21.15.102:42720) 2024-04-25T02:01:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.491+0000 7f2156a00640 1 -- 172.21.15.102:0/2829069800 learned_addr learned my addr 172.21.15.102:0/2829069800 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.492+0000 7f214e000640 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877e5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.492+0000 7f2156a00640 1 -- 172.21.15.102:0/2829069800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 msgr2=0x5634877e5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.492+0000 7f2156a00640 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877e5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.493+0000 7f2156a00640 1 -- 172.21.15.102:0/2829069800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877e89e0 msgr2=0x5634877e6480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:01:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.493+0000 7f2156a00640 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877e89e0 0x5634877e6480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.493+0000 7f2156a00640 1 -- 172.21.15.102:0/2829069800 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56348744d3f0 con 0x5634877de310 2024-04-25T02:01:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.493+0000 7f214e000640 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877e5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:01:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.494+0000 7f2156a00640 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e5f40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f21480092a0 tx=0x7f21480048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cead578e71ec9f2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.494+0000 7f213be00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:01:38.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.495+0000 7f213be00640 1 -- 172.21.15.102:0/2829069800 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214804c070 con 0x5634877de310 2024-04-25T02:01:38.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.495+0000 7f213be00640 1 -- 172.21.15.102:0/2829069800 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f2148003a00 con 0x5634877de310 2024-04-25T02:01:38.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.495+0000 7f2160122740 1 -- 172.21.15.102:0/2829069800 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56348774cd90 con 0x5634877de310 2024-04-25T02:01:38.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.496+0000 7f213be00640 1 -- 172.21.15.102:0/2829069800 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f214803f070 con 0x5634877de310 2024-04-25T02:01:38.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.496+0000 7f2160122740 10 client.26965 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.496+0000 7f2160122740 1 -- 172.21.15.102:0/2829069800 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56348740d0f0 con 0x5634877de310 2024-04-25T02:01:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.497+0000 7f213be00640 1 -- 172.21.15.102:0/2829069800 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2148003c00 con 0x5634877de310 2024-04-25T02:01:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.497+0000 7f213be00640 1 -- 172.21.15.102:0/2829069800 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f21480315d0 con 0x5634877de310 2024-04-25T02:01:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.497+0000 7f2160122740 10 client.26965 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.497+0000 7f2160122740 10 client.26965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.497+0000 7f2160122740 20 client.26965 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.497+0000 7f2160122740 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634877f8eb0 0x5634877fb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.497+0000 7f2160122740 4 client.26965 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.497+0000 7f2160122740 1 -- 172.21.15.102:0/2829069800 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56348774cd90 con 0x5634877f8eb0 2024-04-25T02:01:38.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.498+0000 7f214e000640 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634877f8eb0 0x5634877fb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.500+0000 7f214e000640 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634877f8eb0 0x5634877fb2d0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a08acc66eeba3afe server_cookie=29e11f3e4e10702d in_seq=0 out_seq=0 2024-04-25T02:01:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.500+0000 7f213be00640 10 client.26965 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.501+0000 7f213be00640 1 -- 172.21.15.102:0/2829069800 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56348774cd90 con 0x5634877f8eb0 2024-04-25T02:01:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.501+0000 7f213be00640 10 client.26965 handle_command_reply: tid=0 2024-04-25T02:01:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.502+0000 7f2160122740 1 client.26965 shutdown 2024-04-25T02:01:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.502+0000 7f2160122740 1 -- 172.21.15.102:0/2829069800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 msgr2=0x5634877e5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.502+0000 7f2160122740 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e5f40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f21480092a0 tx=0x7f21480048f0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.503+0000 7f2160122740 1 -- 172.21.15.102:0/2829069800 shutdown_connections 2024-04-25T02:01:38.510 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:01:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: 2024-04-25T02:01:38.192+0000 7f18fa88c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x200000001fd [...7,head] /volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004A auth{2=1} v1635 ap=1 snaprealm=0x56336f8be900 s=16777216 n(v0 rc2024-04-25T02:00:36.004263+0000 b16777216 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25235=0-37748736@7} caps={25235=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25235 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x56336fb54000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x200000001fd:[<0x10000000007/00000001000000000000004A v1621>,<0x10000000006/pg_wal v1002>,<0x10000000005/data v520>,<0x100000001fb/postgres v306>,<0x100000001fa/81669b28-eb2a-4861-893e-c7b62ed09a76 v327>,<0x10000000001/sv_1 v344>,<0x10000000000/qa v334>,<0x1/volumes v309>]//[]","memoryvalue":"(3)0x200000001fd:[<0x10000000007/00000001000000000000004A v1635>,<0x10000000006/pg_wal v1072>,<0x10000000005/data v593>,<0x100000001fb/postgres v352>,<0x100000001fa/81669b28-eb2a-4861-893e-c7b62ed09a76 v315>,<0x10000000001/sv_1 v388>,<0x10000000000/qa v330>,<0x1/volumes v356>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T02:01:38.510 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:01:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: 2024-04-25T02:01:38.194+0000 7f18fa88c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x20000000204 [...8,head] /volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004D auth{2=1} v1640 ap=1 snaprealm=0x56336f666fc0 s=16777216 n(v0 rc2024-04-25T02:00:55.512857+0000 b16777216 1=1+0) old_inodes=1 caps={25235=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x56336fb35180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x20000000204:[<0x10000000007/00000001000000000000004D v1629>,<0x10000000006/pg_wal v1033>,<0x10000000005/data v557>,<0x100000001fb/postgres v328>,<0x100000001fa/81669b28-eb2a-4861-893e-c7b62ed09a76 v352>,<0x10000000001/sv_1 v368>,<0x10000000000/qa v362>,<0x1/volumes v336>]//[]","memoryvalue":"(3)0x20000000204:[<0x10000000007/00000001000000000000004D v1640>,<0x10000000006/pg_wal v1072>,<0x10000000005/data v593>,<0x100000001fb/postgres v352>,<0x100000001fa/81669b28-eb2a-4861-893e-c7b62ed09a76 v315>,<0x10000000001/sv_1 v388>,<0x10000000000/qa v330>,<0x1/volumes v356>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T02:01:38.510 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:01:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: 2024-04-25T02:01:38.197+0000 7f18fa88c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x100000005e9 [8,head] /volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000C auth{0=1} v119 ap=1 s=188416 n(v0 rc2024-04-25T02:01:31.229115+0000 b188416 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25235=0-4194304@7} caps={25235=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25235 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x56336fdd0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x100000005e9:[<0x1000000000f/000C v109>,<0x10000000006/pg_subtrans v1035>,<0x10000000005/data v557>,<0x100000001fb/postgres v328>,<0x100000001fa/81669b28-eb2a-4861-893e-c7b62ed09a76 v352>,<0x10000000001/sv_1 v368>,<0x10000000000/qa v362>,<0x1/volumes v336>]//[]","memoryvalue":"(3)0x100000005e9:[<0x1000000000f/000C v119>,<0x10000000006/pg_subtrans v1078>,<0x10000000005/data v593>,<0x100000001fb/postgres v352>,<0x100000001fa/81669b28-eb2a-4861-893e-c7b62ed09a76 v315>,<0x10000000001/sv_1 v388>,<0x10000000000/qa v330>,<0x1/volumes v356>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T02:01:38.510 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:01:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: 2024-04-25T02:01:38.197+0000 7f18fa88c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x100000005e8 [8,head] /volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000B auth v115 ap=1 s=262144 n(v0 rc2024-04-25T02:00:54.728874+0000 b262144 1=1+0) caps={25235=pAsLsXsFsc/-@1} | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x56336fa7c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x100000005e8:[<0x1000000000f/000B v107>,<0x10000000006/pg_subtrans v1006>,<0x10000000005/data v520>,<0x100000001fb/postgres v306>,<0x100000001fa/81669b28-eb2a-4861-893e-c7b62ed09a76 v327>,<0x10000000001/sv_1 v344>,<0x10000000000/qa v334>,<0x1/volumes v309>]//[]","memoryvalue":"(3)0x100000005e8:[<0x1000000000f/000B v115>,<0x10000000006/pg_subtrans v1078>,<0x10000000005/data v593>,<0x100000001fb/postgres v352>,<0x100000001fa/81669b28-eb2a-4861-893e-c7b62ed09a76 v315>,<0x10000000001/sv_1 v388>,<0x10000000000/qa v330>,<0x1/volumes v356>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T02:01:38.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.503+0000 7f2160122740 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5634877f8eb0 0x5634877fb2d0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.503+0000 7f2160122740 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877e89e0 0x5634877e6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.503+0000 7f2160122740 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e5f40 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.503+0000 7f2160122740 1 --2- 172.21.15.102:0/2829069800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877dd9f0 0x5634877e5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.503+0000 7f2160122740 1 -- 172.21.15.102:0/2829069800 >> 172.21.15.102:0/2829069800 conn(0x5634877d9d10 msgr2=0x5634877df180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.503+0000 7f213be00640 0 client.26965 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.504+0000 7f2160122740 1 -- 172.21.15.102:0/2829069800 shutdown_connections 2024-04-25T02:01:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.504+0000 7f2160122740 1 -- 172.21.15.102:0/2829069800 wait complete. 2024-04-25T02:01:38.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.504+0000 7f2160122740 20 client.26965 trim_cache size 0 max 16384 2024-04-25T02:01:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.550+0000 7f2160122740 1 Processor -- start 2024-04-25T02:01:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.550+0000 7f2160122740 1 -- start start 2024-04-25T02:01:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.550+0000 7f2160122740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877dd9f0 0x5634877d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.550+0000 7f2160122740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.550+0000 7f2160122740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877e89e0 0x563487800980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.550+0000 7f2160122740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f21340110e0 con 0x5634877dd9f0 2024-04-25T02:01:38.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.550+0000 7f2160122740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f213400e950 con 0x5634877e89e0 2024-04-25T02:01:38.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.550+0000 7f2160122740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f213400c760 con 0x5634877de310 2024-04-25T02:01:38.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.551+0000 7f214e000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877e89e0 0x563487800980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.551+0000 7f214e000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877e89e0 0x563487800980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33252/0 (socket says 172.21.15.102:33252) 2024-04-25T02:01:38.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.551+0000 7f214e000640 1 -- 172.21.15.102:0/349532395 learned_addr learned my addr 172.21.15.102:0/349532395 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.553+0000 7f2156a00640 1 --2- 172.21.15.102:0/349532395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.553+0000 7f2156a00640 1 -- 172.21.15.102:0/349532395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877e89e0 msgr2=0x563487800980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.553+0000 7f2156a00640 1 --2- 172.21.15.102:0/349532395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877e89e0 0x563487800980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.553+0000 7f2156a00640 1 -- 172.21.15.102:0/349532395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877dd9f0 msgr2=0x5634877d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.553+0000 7f2156a00640 1 --2- 172.21.15.102:0/349532395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877dd9f0 0x5634877d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.553+0000 7f2156a00640 1 -- 172.21.15.102:0/349532395 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634877fb640 con 0x5634877de310 2024-04-25T02:01:38.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.554+0000 7f2156a00640 1 --2- 172.21.15.102:0/349532395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e0150 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2148009900 tx=0x7f21480040d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bada1cfba004f4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.554+0000 7f2146a00640 1 -- 172.21.15.102:0/349532395 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f214803f070 con 0x5634877de310 2024-04-25T02:01:38.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.554+0000 7f2146a00640 1 -- 172.21.15.102:0/349532395 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2148003ce0 con 0x5634877de310 2024-04-25T02:01:38.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.554+0000 7f2146a00640 1 -- 172.21.15.102:0/349532395 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148035040 con 0x5634877de310 2024-04-25T02:01:38.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.557+0000 7f2160122740 1 -- 172.21.15.102:0/349532395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 msgr2=0x5634877e0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.557+0000 7f2160122740 1 --2- 172.21.15.102:0/349532395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e0150 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2148009900 tx=0x7f21480040d0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.557+0000 7f2160122740 1 -- 172.21.15.102:0/349532395 shutdown_connections 2024-04-25T02:01:38.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.557+0000 7f2160122740 1 --2- 172.21.15.102:0/349532395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877e89e0 0x563487800980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.557+0000 7f2160122740 1 --2- 172.21.15.102:0/349532395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877de310 0x5634877e0150 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.557+0000 7f2160122740 1 --2- 172.21.15.102:0/349532395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877dd9f0 0x5634877d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.557+0000 7f2160122740 1 -- 172.21.15.102:0/349532395 >> 172.21.15.102:0/349532395 conn(0x5634877d9d10 msgr2=0x5634877cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.558+0000 7f2160122740 1 -- 172.21.15.102:0/349532395 shutdown_connections 2024-04-25T02:01:38.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.558+0000 7f2160122740 1 -- 172.21.15.102:0/349532395 wait complete. 2024-04-25T02:01:38.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.559+0000 7f2160122740 1 -- start start 2024-04-25T02:01:38.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.559+0000 7f2160122740 1 Processor -- start 2024-04-25T02:01:38.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f2160122740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877dd9f0 0x563487856620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f2160122740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877de310 0x563487856b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f2160122740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877e89e0 0x5634877fdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f2160122740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f213400ce10 con 0x5634877e89e0 2024-04-25T02:01:38.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f2160122740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f213400e280 con 0x5634877de310 2024-04-25T02:01:38.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f2160122740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f213400d4e0 con 0x5634877dd9f0 2024-04-25T02:01:38.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f2160122740 10 client.0 _ll_register_callbacks cb 0x7f21480042c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:38.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f2156a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877de310 0x563487856b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f214e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877dd9f0 0x563487856620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f214e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877dd9f0 0x563487856620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:42732/0 (socket says 172.21.15.102:42732) 2024-04-25T02:01:38.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f2156a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877de310 0x563487856b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33260/0 (socket says 172.21.15.102:33260) 2024-04-25T02:01:38.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.560+0000 7f214e000640 1 -- 172.21.15.102:0/3023538204 learned_addr learned my addr 172.21.15.102:0/3023538204 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:38.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.561+0000 7f2156a00640 1 -- 172.21.15.102:0/3023538204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877dd9f0 msgr2=0x563487856620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.561+0000 7f2156a00640 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877dd9f0 0x563487856620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.561+0000 7f2156a00640 1 -- 172.21.15.102:0/3023538204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877e89e0 msgr2=0x5634877fdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:38.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.561+0000 7f2156a00640 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877e89e0 0x5634877fdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.561+0000 7f2156a00640 1 -- 172.21.15.102:0/3023538204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634877fe770 con 0x5634877de310 2024-04-25T02:01:38.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.561+0000 7f2156a00640 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877de310 0x563487856b60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x56348777fba0 tx=0x7f2148031df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8fb911a33fb4c19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:38.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.561+0000 7f213be00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.562+0000 7f213be00640 1 -- 172.21.15.102:0/3023538204 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148035070 con 0x5634877de310 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.562+0000 7f213be00640 1 -- 172.21.15.102:0/3023538204 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f214803b460 con 0x5634877de310 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.562+0000 7f2160122740 1 -- 172.21.15.102:0/3023538204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2148003ce0 con 0x5634877de310 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.563+0000 7f213be00640 1 -- 172.21.15.102:0/3023538204 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f214803f040 con 0x5634877de310 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.563+0000 7f2160122740 10 client.27011 fetch_fsmap learned FSMap version 34 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.563+0000 7f2160122740 1 -- 172.21.15.102:0/3023538204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56348785eee0 con 0x5634877de310 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.564+0000 7f213be00640 1 -- 172.21.15.102:0/3023538204 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2148003ce0 con 0x5634877de310 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.564+0000 7f213be00640 1 -- 172.21.15.102:0/3023538204 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f2148044570 con 0x5634877de310 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.566+0000 7f2160122740 10 client.27011 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.566+0000 7f2160122740 10 client.27011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.566+0000 7f2160122740 20 client.27011 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:38.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.566+0000 7f2160122740 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56348785f3e0 0x563487861800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:38.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.566+0000 7f2160122740 4 client.27011 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T02:01:38.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.566+0000 7f2160122740 1 -- 172.21.15.102:0/3023538204 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563487746aa0 con 0x56348785f3e0 2024-04-25T02:01:38.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.567+0000 7f214e000640 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56348785f3e0 0x563487861800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:38.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.567+0000 7f214e000640 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56348785f3e0 0x563487861800 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a75bd5f15f08e385 server_cookie=167fc03de04da4a7 in_seq=0 out_seq=0 2024-04-25T02:01:38.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.568+0000 7f213be00640 10 client.27011 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.568+0000 7f213be00640 1 -- 172.21.15.102:0/3023538204 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563487746aa0 con 0x56348785f3e0 2024-04-25T02:01:38.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.568+0000 7f213be00640 10 client.27011 handle_command_reply: tid=0 2024-04-25T02:01:38.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.568+0000 7f2160122740 1 client.27011 shutdown 2024-04-25T02:01:38.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.569+0000 7f2160122740 1 -- 172.21.15.102:0/3023538204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877de310 msgr2=0x563487856b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.569+0000 7f2160122740 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877de310 0x563487856b60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x56348777fba0 tx=0x7f2148031df0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.569+0000 7f2160122740 1 -- 172.21.15.102:0/3023538204 shutdown_connections 2024-04-25T02:01:38.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.569+0000 7f2160122740 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56348785f3e0 0x563487861800 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.569+0000 7f2160122740 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634877e89e0 0x5634877fdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.569+0000 7f2160122740 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5634877de310 0x563487856b60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.569+0000 7f2160122740 1 --2- 172.21.15.102:0/3023538204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634877dd9f0 0x563487856620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.569+0000 7f2160122740 1 -- 172.21.15.102:0/3023538204 >> 172.21.15.102:0/3023538204 conn(0x5634877d9d10 msgr2=0x5634877f3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.569+0000 7f213be00640 0 client.27011 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:38.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.570+0000 7f2160122740 1 -- 172.21.15.102:0/3023538204 shutdown_connections 2024-04-25T02:01:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.571+0000 7f2160122740 1 -- 172.21.15.102:0/3023538204 wait complete. 2024-04-25T02:01:38.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.571+0000 7f2160122740 20 client.27011 trim_cache size 0 max 16384 2024-04-25T02:01:38.578 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:01:38.578 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T02:01:38.579 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": { 2024-04-25T02:01:38.579 INFO:teuthology.orchestra.run.smithi102.stdout: "1d67d1ea-ed6c-4f41-ae39-2db31f2f79db": { 2024-04-25T02:01:38.579 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/", 2024-04-25T02:01:38.579 INFO:teuthology.orchestra.run.smithi102.stdout: "tag": "1d67d1ea-ed6c-4f41-ae39-2db31f2f79db", 2024-04-25T02:01:38.579 INFO:teuthology.orchestra.run.smithi102.stdout: "options": "recursive,force" 2024-04-25T02:01:38.579 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:01:38.579 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:01:38.579 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:01:38.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.576+0000 7f2157e00640 1 -- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f213c0a2dc0 msgr2=0x7f213c0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.576+0000 7f2157e00640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f213c0a2dc0 0x7f213c0a5280 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2150009ae0 tx=0x7f2150009a70 comp rx=0 tx=0).stop 2024-04-25T02:01:38.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.576+0000 7f2157e00640 1 -- 172.21.15.102:0/2129328389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 msgr2=0x7f215819e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.576+0000 7f2157e00640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 0x7f215819e8d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f215000eb10 tx=0x7f215000c2d0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.576+0000 7f2157e00640 1 -- 172.21.15.102:0/2129328389 shutdown_connections 2024-04-25T02:01:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.577+0000 7f2157e00640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f213c0a2dc0 0x7f213c0a5280 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.577+0000 7f2157e00640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21581097f0 0x7f215819e8d0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.577+0000 7f2157e00640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21581041a0 0x7f215819a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.577+0000 7f2157e00640 1 --2- 172.21.15.102:0/2129328389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21581037e0 0x7f215819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:38.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.577+0000 7f2157e00640 1 -- 172.21.15.102:0/2129328389 >> 172.21.15.102:0/2129328389 conn(0x7f21580ff2b0 msgr2=0x7f21580ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:38.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.579+0000 7f2157e00640 1 -- 172.21.15.102:0/2129328389 shutdown_connections 2024-04-25T02:01:38.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:01:38.579+0000 7f2157e00640 1 -- 172.21.15.102:0/2129328389 wait complete. 2024-04-25T02:01:38.614 INFO:tasks.cephfs.filesystem:scrub status for tag:1d67d1ea-ed6c-4f41-ae39-2db31f2f79db - {'path': '/', 'tag': '1d67d1ea-ed6c-4f41-ae39-2db31f2f79db', 'options': 'recursive,force'} 2024-04-25T02:01:39.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: pgmap v194: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.3 MiB/s wr, 941 op/s 2024-04-25T02:01:39.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub queued for path: / 2024-04-25T02:01:39.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:01:39.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub summary: active paths [/] 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: Scrub error on inode 0x200000001fd (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004A) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: Scrub error on inode 0x20000000204 (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004D) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: Scrub error on inode 0x100000005e9 (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000C) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: Scrub error on inode 0x100000005e8 (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000B) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:39 smithi102 ceph-mon[27796]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: pgmap v194: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.3 MiB/s wr, 941 op/s 2024-04-25T02:01:39.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub queued for path: / 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub summary: active paths [/] 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: Scrub error on inode 0x200000001fd (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004A) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: Scrub error on inode 0x20000000204 (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004D) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: Scrub error on inode 0x100000005e9 (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000C) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: Scrub error on inode 0x100000005e8 (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000B) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:39 smithi110 ceph-mon[32848]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: pgmap v194: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.3 MiB/s wr, 941 op/s 2024-04-25T02:01:39.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub queued for path: / 2024-04-25T02:01:39.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:01:39.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:01:39.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:01:39.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub summary: active paths [/] 2024-04-25T02:01:39.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: Scrub error on inode 0x200000001fd (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004A) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: Scrub error on inode 0x20000000204 (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004D) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: Scrub error on inode 0x100000005e9 (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000C) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: Scrub error on inode 0x100000005e8 (/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000B) see mds.i log and `damage ls` output for details 2024-04-25T02:01:39.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:39.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:39 smithi193 ceph-mon[32981]: scrub summary: idle+waiting paths [/] 2024-04-25T02:01:41.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:41 smithi102 ceph-mon[27796]: pgmap v195: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.4 MiB/s wr, 650 op/s 2024-04-25T02:01:41.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:41 smithi110 ceph-mon[32848]: pgmap v195: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.4 MiB/s wr, 650 op/s 2024-04-25T02:01:41.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:41 smithi193 ceph-mon[32981]: pgmap v195: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.4 MiB/s wr, 650 op/s 2024-04-25T02:01:43.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 350.0 s, 3888.8 tps, lat 25.624 ms stddev 28.161 2024-04-25T02:01:43.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:43 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a[27792]: 2024-04-25T02:01:43.138+0000 7f4d19000700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-25T02:01:43.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:43 smithi102 ceph-mon[27796]: pgmap v196: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 9.2 MiB/s wr, 958 op/s 2024-04-25T02:01:43.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:43 smithi110 ceph-mon[32848]: pgmap v196: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 9.2 MiB/s wr, 958 op/s 2024-04-25T02:01:43.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:43 smithi193 ceph-mon[32981]: pgmap v196: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 9.2 MiB/s wr, 958 op/s 2024-04-25T02:01:44.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:44 smithi102 ceph-mon[27796]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-25T02:01:44.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:44 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] up:active 2024-04-25T02:01:44.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:44 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:01:44.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:44 smithi110 ceph-mon[32848]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-25T02:01:44.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:44 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] up:active 2024-04-25T02:01:44.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:44 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:01:44.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:44 smithi193 ceph-mon[32981]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-25T02:01:44.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:44 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] up:active 2024-04-25T02:01:44.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:44 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:01:45.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:45 smithi102 ceph-mon[27796]: pgmap v197: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.5 MiB/s wr, 812 op/s 2024-04-25T02:01:45.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:45 smithi110 ceph-mon[32848]: pgmap v197: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.5 MiB/s wr, 812 op/s 2024-04-25T02:01:45.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:45 smithi193 ceph-mon[32981]: pgmap v197: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.5 MiB/s wr, 812 op/s 2024-04-25T02:01:47.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:47 smithi102 ceph-mon[27796]: pgmap v198: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.1 MiB/s wr, 681 op/s 2024-04-25T02:01:47.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:47 smithi110 ceph-mon[32848]: pgmap v198: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.1 MiB/s wr, 681 op/s 2024-04-25T02:01:47.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:47 smithi193 ceph-mon[32981]: pgmap v198: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 6.1 MiB/s wr, 681 op/s 2024-04-25T02:01:48.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 355.0 s, 2540.2 tps, lat 39.172 ms stddev 45.546 2024-04-25T02:01:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:49 smithi110 ceph-mon[32848]: pgmap v199: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 9.1 MiB/s wr, 985 op/s 2024-04-25T02:01:49.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:49 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:01:49.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:49 smithi193 ceph-mon[32981]: pgmap v199: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 9.1 MiB/s wr, 985 op/s 2024-04-25T02:01:49.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:49 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:01:49.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:49 smithi102 ceph-mon[27796]: pgmap v199: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 9.1 MiB/s wr, 985 op/s 2024-04-25T02:01:49.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:49 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:01:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.236+0000 7f2821a00640 1 -- 172.21.15.102:0/792051207 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c10d150 msgr2=0x7f281c10d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.236+0000 7f2821a00640 1 --2- 172.21.15.102:0/792051207 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c10d150 0x7f281c10d5b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2804009860 tx=0x7f280402a340 comp rx=0 tx=0).stop 2024-04-25T02:01:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.237+0000 7f2821a00640 1 -- 172.21.15.102:0/792051207 shutdown_connections 2024-04-25T02:01:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.237+0000 7f2821a00640 1 --2- 172.21.15.102:0/792051207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f281c112eb0 0x7f281c1152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.237+0000 7f2821a00640 1 --2- 172.21.15.102:0/792051207 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c10d150 0x7f281c10d5b0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.237+0000 7f2821a00640 1 --2- 172.21.15.102:0/792051207 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f281c108fd0 0x7f281c10cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.237+0000 7f2821a00640 1 -- 172.21.15.102:0/792051207 >> 172.21.15.102:0/792051207 conn(0x7f281c06cad0 msgr2=0x7f281c06cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.237+0000 7f2821a00640 1 -- 172.21.15.102:0/792051207 shutdown_connections 2024-04-25T02:01:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.237+0000 7f2821a00640 1 -- 172.21.15.102:0/792051207 wait complete. 2024-04-25T02:01:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.238+0000 7f2821a00640 1 Processor -- start 2024-04-25T02:01:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.239+0000 7f2821a00640 1 -- start start 2024-04-25T02:01:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.239+0000 7f2821a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c108fd0 0x7f281c19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.240+0000 7f2821a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f281c10d150 0x7f281c19d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.240+0000 7f2821a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f281c112eb0 0x7f281c19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.240+0000 7f2821a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f281c117f50 con 0x7f281c10d150 2024-04-25T02:01:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.240+0000 7f2821a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f281c117dd0 con 0x7f281c108fd0 2024-04-25T02:01:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.240+0000 7f2821a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f281c1180d0 con 0x7f281c112eb0 2024-04-25T02:01:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.240+0000 7f281be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c108fd0 0x7f281c19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.240+0000 7f281be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c108fd0 0x7f281c19d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36274/0 (socket says 172.21.15.102:36274) 2024-04-25T02:01:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.240+0000 7f281be00640 1 -- 172.21.15.102:0/1893490987 learned_addr learned my addr 172.21.15.102:0/1893490987 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.241+0000 7f2821000640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f281c112eb0 0x7f281c19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.241+0000 7f2813400640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f281c10d150 0x7f281c19d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.241+0000 7f281be00640 1 -- 172.21.15.102:0/1893490987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f281c112eb0 msgr2=0x7f281c19df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.241+0000 7f281be00640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f281c112eb0 0x7f281c19df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.241+0000 7f281be00640 1 -- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f281c10d150 msgr2=0x7f281c19d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.241+0000 7f281be00640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f281c10d150 0x7f281c19d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.241+0000 7f281be00640 1 -- 172.21.15.102:0/1893490987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281c1b2ef0 con 0x7f281c108fd0 2024-04-25T02:01:51.250 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.242+0000 7f281be00640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c108fd0 0x7f281c19d4b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f280c009fa0 tx=0x7f280c00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:51.250 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.242+0000 7f2819600640 1 -- 172.21.15.102:0/1893490987 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c01b070 con 0x7f281c108fd0 2024-04-25T02:01:51.250 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.242+0000 7f2819600640 1 -- 172.21.15.102:0/1893490987 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f280c00ba50 con 0x7f281c108fd0 2024-04-25T02:01:51.250 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.242+0000 7f2819600640 1 -- 172.21.15.102:0/1893490987 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c016c10 con 0x7f281c108fd0 2024-04-25T02:01:51.250 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.242+0000 7f2821a00640 1 -- 172.21.15.102:0/1893490987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f281c1b31e0 con 0x7f281c108fd0 2024-04-25T02:01:51.250 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.243+0000 7f2821a00640 1 -- 172.21.15.102:0/1893490987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f281c1b36b0 con 0x7f281c108fd0 2024-04-25T02:01:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.248+0000 7f2819600640 1 -- 172.21.15.102:0/1893490987 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f280c01e070 con 0x7f281c108fd0 2024-04-25T02:01:51.254 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.249+0000 7f2825e36740 1 Processor -- start 2024-04-25T02:01:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.249+0000 7f2825e36740 1 -- start start 2024-04-25T02:01:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f2825e36740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d11f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f2825e36740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d129c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f2825e36740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d12c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f2825e36740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56290d082ab0 con 0x56290d11efb0 2024-04-25T02:01:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f2825e36740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56290d02b640 con 0x56290d11f8d0 2024-04-25T02:01:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f2825e36740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56290d07ea70 con 0x56290d12a190 2024-04-25T02:01:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f281be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d129c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f281be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d129c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36284/0 (socket says 172.21.15.102:36284) 2024-04-25T02:01:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f281be00640 1 -- 172.21.15.102:0/911648781 learned_addr learned my addr 172.21.15.102:0/911648781 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.250+0000 7f2821000640 1 --2- 172.21.15.102:0/911648781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d11f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.251+0000 7f2813400640 1 --2- 172.21.15.102:0/911648781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d12c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.251+0000 7f2819600640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f28000a2d70 0x7f28000a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.251+0000 7f281be00640 1 -- 172.21.15.102:0/911648781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 msgr2=0x56290d12c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.251+0000 7f281be00640 1 --2- 172.21.15.102:0/911648781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d12c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.251+0000 7f281be00640 1 -- 172.21.15.102:0/911648781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 msgr2=0x56290d11f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.251+0000 7f281be00640 1 --2- 172.21.15.102:0/911648781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d11f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.251+0000 7f281be00640 1 -- 172.21.15.102:0/911648781 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56290cd97210 con 0x56290d11f8d0 2024-04-25T02:01:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.251+0000 7f2819600640 1 -- 172.21.15.102:0/1893490987 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f280c0cf460 con 0x7f281c108fd0 2024-04-25T02:01:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.251+0000 7f2821000640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f28000a2d70 0x7f28000a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.252+0000 7f281be00640 1 --2- 172.21.15.102:0/911648781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d129c50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f280c09b940 tx=0x7f280c05bb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3a823bc38e66eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.252+0000 7f2811600640 1 -- 172.21.15.102:0/911648781 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c09c0b0 con 0x56290d11f8d0 2024-04-25T02:01:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.252+0000 7f2811600640 1 -- 172.21.15.102:0/911648781 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f280c09b9b0 con 0x56290d11f8d0 2024-04-25T02:01:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.252+0000 7f2811600640 1 -- 172.21.15.102:0/911648781 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c0cce60 con 0x56290d11f8d0 2024-04-25T02:01:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.253+0000 7f2825e36740 1 -- 172.21.15.102:0/911648781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 msgr2=0x56290d129c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.253+0000 7f2825e36740 1 --2- 172.21.15.102:0/911648781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d129c50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f280c09b940 tx=0x7f280c05bb00 comp rx=0 tx=0).stop 2024-04-25T02:01:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.253+0000 7f2821000640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f28000a2d70 0x7f28000a5230 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f28140058c0 tx=0x7f2814004730 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.255+0000 7f2825e36740 1 -- 172.21.15.102:0/911648781 shutdown_connections 2024-04-25T02:01:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.255+0000 7f2825e36740 1 --2- 172.21.15.102:0/911648781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d12c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.255+0000 7f2825e36740 1 --2- 172.21.15.102:0/911648781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d129c50 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.255+0000 7f2825e36740 1 --2- 172.21.15.102:0/911648781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d11f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.255+0000 7f2825e36740 1 -- 172.21.15.102:0/911648781 >> 172.21.15.102:0/911648781 conn(0x56290d11b4c0 msgr2=0x56290d11b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.256+0000 7f2825e36740 1 -- 172.21.15.102:0/911648781 shutdown_connections 2024-04-25T02:01:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.256+0000 7f2825e36740 1 -- 172.21.15.102:0/911648781 wait complete. 2024-04-25T02:01:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.256+0000 7f2825e36740 1 -- start start 2024-04-25T02:01:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.257+0000 7f2825e36740 1 Processor -- start 2024-04-25T02:01:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.257+0000 7f2825e36740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d1270a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.257+0000 7f2825e36740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d1275e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.257+0000 7f2825e36740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d127b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.257+0000 7f2825e36740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56290d12cd60 con 0x56290d11efb0 2024-04-25T02:01:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.257+0000 7f2825e36740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56290d03e990 con 0x56290d11f8d0 2024-04-25T02:01:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.257+0000 7f2825e36740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56290d12cee0 con 0x56290d12a190 2024-04-25T02:01:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.257+0000 7f2825e36740 10 client.0 _ll_register_callbacks cb 0x56290d040950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.258+0000 7f2821000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d127b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.258+0000 7f2821000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d127b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52260/0 (socket says 172.21.15.102:52260) 2024-04-25T02:01:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.258+0000 7f2821000640 1 -- 172.21.15.102:0/301955456 learned_addr learned my addr 172.21.15.102:0/301955456 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.258+0000 7f281be00640 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d1270a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.258+0000 7f2813400640 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d1275e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.259+0000 7f281be00640 1 -- 172.21.15.102:0/301955456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 msgr2=0x56290d127b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.259+0000 7f281be00640 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d127b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.259+0000 7f281be00640 1 -- 172.21.15.102:0/301955456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 msgr2=0x56290d1275e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.259+0000 7f281be00640 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d1275e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.259+0000 7f281be00640 1 -- 172.21.15.102:0/301955456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56290ce63620 con 0x56290d11efb0 2024-04-25T02:01:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.260+0000 7f281be00640 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d1270a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f280c09b940 tx=0x7f280c0d2f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ea58b0ff2394be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.260+0000 7f280a000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:01:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.261+0000 7f280a000640 1 -- 172.21.15.102:0/301955456 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c052950 con 0x56290d11efb0 2024-04-25T02:01:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.261+0000 7f280a000640 1 -- 172.21.15.102:0/301955456 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f280c0860b0 con 0x56290d11efb0 2024-04-25T02:01:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.261+0000 7f2825e36740 1 -- 172.21.15.102:0/301955456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f280c09b9b0 con 0x56290d11efb0 2024-04-25T02:01:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.261+0000 7f280a000640 1 -- 172.21.15.102:0/301955456 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f280c09fd60 con 0x56290d11efb0 2024-04-25T02:01:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.261+0000 7f2825e36740 10 client.16461 fetch_fsmap learned FSMap version 36 2024-04-25T02:01:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.261+0000 7f2825e36740 1 -- 172.21.15.102:0/301955456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56290d12cd60 con 0x56290d11efb0 2024-04-25T02:01:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.262+0000 7f280a000640 1 -- 172.21.15.102:0/301955456 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f280c09b9b0 con 0x56290d11efb0 2024-04-25T02:01:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.262+0000 7f280a000640 1 -- 172.21.15.102:0/301955456 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f280c09cce0 con 0x56290d11efb0 2024-04-25T02:01:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.262+0000 7f2825e36740 10 client.16461 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:01:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.262+0000 7f2825e36740 10 client.16461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.262+0000 7f2825e36740 20 client.16461 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.262+0000 7f2825e36740 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56290d139d30 0x56290d13c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.262+0000 7f2825e36740 4 client.16461 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:01:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.262+0000 7f2825e36740 1 -- 172.21.15.102:0/301955456 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56290d08fff0 con 0x56290d139d30 2024-04-25T02:01:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.264+0000 7f2813400640 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56290d139d30 0x56290d13c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.265+0000 7f2813400640 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56290d139d30 0x56290d13c120 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68c2820fd0c93aae server_cookie=713f68842ed17e90 in_seq=0 out_seq=0 2024-04-25T02:01:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.265+0000 7f280a000640 10 client.16461 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f280a000640 1 -- 172.21.15.102:0/301955456 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56290d08fff0 con 0x56290d139d30 2024-04-25T02:01:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f280a000640 10 client.16461 handle_command_reply: tid=0 2024-04-25T02:01:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 client.16461 shutdown 2024-04-25T02:01:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 -- 172.21.15.102:0/301955456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 msgr2=0x56290d1270a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d1270a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f280c09b940 tx=0x7f280c0d2f30 comp rx=0 tx=0).stop 2024-04-25T02:01:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 -- 172.21.15.102:0/301955456 shutdown_connections 2024-04-25T02:01:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56290d139d30 0x56290d13c120 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d127b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11f8d0 0x56290d1275e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 --2- 172.21.15.102:0/301955456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d1270a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f280a000640 0 client.16461 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 -- 172.21.15.102:0/301955456 >> 172.21.15.102:0/301955456 conn(0x56290d11b4c0 msgr2=0x56290d1211f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.266+0000 7f2825e36740 1 -- 172.21.15.102:0/301955456 shutdown_connections 2024-04-25T02:01:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.267+0000 7f2825e36740 1 -- 172.21.15.102:0/301955456 wait complete. 2024-04-25T02:01:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.267+0000 7f2825e36740 20 client.16461 trim_cache size 0 max 16384 2024-04-25T02:01:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.291+0000 7f2825e36740 1 Processor -- start 2024-04-25T02:01:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.291+0000 7f2825e36740 1 -- start start 2024-04-25T02:01:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.292+0000 7f2825e36740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11efb0 0x56290d135c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.292+0000 7f2825e36740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11f8d0 0x56290d121070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.292+0000 7f2825e36740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d127940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.292+0000 7f2825e36740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f281c1be9a0 con 0x56290d11f8d0 2024-04-25T02:01:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.292+0000 7f2825e36740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f281c1bc1b0 con 0x56290d11efb0 2024-04-25T02:01:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.292+0000 7f2825e36740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f281c1ba000 con 0x56290d12a190 2024-04-25T02:01:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.292+0000 7f281be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11efb0 0x56290d135c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.292+0000 7f281be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11efb0 0x56290d135c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36302/0 (socket says 172.21.15.102:36302) 2024-04-25T02:01:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.292+0000 7f281be00640 1 -- 172.21.15.102:0/184246041 learned_addr learned my addr 172.21.15.102:0/184246041 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.293+0000 7f2813400640 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11f8d0 0x56290d121070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.293+0000 7f2821000640 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d127940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.293+0000 7f281be00640 1 -- 172.21.15.102:0/184246041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 msgr2=0x56290d127940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.293+0000 7f281be00640 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d127940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.293+0000 7f281be00640 1 -- 172.21.15.102:0/184246041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11f8d0 msgr2=0x56290d121070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.293+0000 7f281be00640 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11f8d0 0x56290d121070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.293+0000 7f281be00640 1 -- 172.21.15.102:0/184246041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56290d11f700 con 0x56290d11efb0 2024-04-25T02:01:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.293+0000 7f2813400640 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11f8d0 0x56290d121070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:01:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.293+0000 7f281be00640 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11efb0 0x56290d135c20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f280c000f80 tx=0x7f280c00ce80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ee24ffbdc59c63c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.294+0000 7f2810c00640 1 -- 172.21.15.102:0/184246041 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c097c50 con 0x56290d11efb0 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.294+0000 7f2810c00640 1 -- 172.21.15.102:0/184246041 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f280c052950 con 0x56290d11efb0 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.294+0000 7f2810c00640 1 -- 172.21.15.102:0/184246041 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f280c09af20 con 0x56290d11efb0 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.295+0000 7f2825e36740 1 -- 172.21.15.102:0/184246041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11efb0 msgr2=0x56290d135c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.295+0000 7f2825e36740 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11efb0 0x56290d135c20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f280c000f80 tx=0x7f280c00ce80 comp rx=0 tx=0).stop 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.295+0000 7f2825e36740 1 -- 172.21.15.102:0/184246041 shutdown_connections 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.295+0000 7f2825e36740 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d12a190 0x56290d127940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.295+0000 7f2825e36740 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11f8d0 0x56290d121070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.295+0000 7f2825e36740 1 --2- 172.21.15.102:0/184246041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d11efb0 0x56290d135c20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.295+0000 7f2825e36740 1 -- 172.21.15.102:0/184246041 >> 172.21.15.102:0/184246041 conn(0x56290d11b4c0 msgr2=0x56290d120960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.295+0000 7f2825e36740 1 -- 172.21.15.102:0/184246041 shutdown_connections 2024-04-25T02:01:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.295+0000 7f2825e36740 1 -- 172.21.15.102:0/184246041 wait complete. 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.296+0000 7f2825e36740 1 -- start start 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.296+0000 7f2825e36740 1 Processor -- start 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2825e36740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d132560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2825e36740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d11f8d0 0x56290d123a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2825e36740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d12a190 0x56290d123f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2825e36740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f281c1ba670 con 0x56290d11efb0 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2825e36740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f281c1bbae0 con 0x56290d12a190 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2825e36740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f281c1bad40 con 0x56290d11f8d0 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2825e36740 10 client.0 _ll_register_callbacks cb 0x7f281400a7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:01:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2821000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d12a190 0x56290d123f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f281be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d132560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2821000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d12a190 0x56290d123f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36316/0 (socket says 172.21.15.102:36316) 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2821000640 1 -- 172.21.15.102:0/1372955814 learned_addr learned my addr 172.21.15.102:0/1372955814 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2813400640 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d11f8d0 0x56290d123a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2821000640 1 -- 172.21.15.102:0/1372955814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d11f8d0 msgr2=0x56290d123a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2821000640 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d11f8d0 0x56290d123a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2821000640 1 -- 172.21.15.102:0/1372955814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 msgr2=0x56290d132560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2821000640 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d132560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.297+0000 7f2821000640 1 -- 172.21.15.102:0/1372955814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56290d124760 con 0x56290d12a190 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.298+0000 7f2821000640 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d12a190 0x56290d123f80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2814004320 tx=0x7f28140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d82bb4b8da37ee6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.298+0000 7f280a000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.298+0000 7f280a000640 1 -- 172.21.15.102:0/1372955814 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281402f060 con 0x56290d12a190 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.298+0000 7f2825e36740 1 -- 172.21.15.102:0/1372955814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f280c052950 con 0x56290d12a190 2024-04-25T02:01:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.298+0000 7f280a000640 1 -- 172.21.15.102:0/1372955814 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7f281402a3f0 con 0x56290d12a190 2024-04-25T02:01:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.298+0000 7f280a000640 1 -- 172.21.15.102:0/1372955814 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f281402a5f0 con 0x56290d12a190 2024-04-25T02:01:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.298+0000 7f2825e36740 10 client.27029 fetch_fsmap learned FSMap version 36 2024-04-25T02:01:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.298+0000 7f2825e36740 1 -- 172.21.15.102:0/1372955814 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56290d1a0100 con 0x56290d12a190 2024-04-25T02:01:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.298+0000 7f280a000640 1 -- 172.21.15.102:0/1372955814 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f280c052950 con 0x56290d12a190 2024-04-25T02:01:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.299+0000 7f280a000640 1 -- 172.21.15.102:0/1372955814 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f2814029030 con 0x56290d12a190 2024-04-25T02:01:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.299+0000 7f2825e36740 10 client.27029 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:01:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.299+0000 7f2825e36740 10 client.27029 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:01:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.299+0000 7f2825e36740 20 client.27029 populate_metadata read hostname 'smithi102' 2024-04-25T02:01:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.299+0000 7f2825e36740 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56290d1a0890 0x56290d1a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:01:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.299+0000 7f2825e36740 4 client.27029 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:01:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.299+0000 7f2825e36740 1 -- 172.21.15.102:0/1372955814 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56290d08a340 con 0x56290d1a0890 2024-04-25T02:01:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.300+0000 7f281be00640 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56290d1a0890 0x56290d1a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:01:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.301+0000 7f281be00640 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56290d1a0890 0x56290d1a2cb0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef1df3de7277213a server_cookie=56b187f9e7e79bca in_seq=0 out_seq=0 2024-04-25T02:01:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.301+0000 7f280a000640 10 client.27029 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.312+0000 7f280a000640 1 -- 172.21.15.102:0/1372955814 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56290d08a340 con 0x56290d1a0890 2024-04-25T02:01:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.312+0000 7f280a000640 10 client.27029 handle_command_reply: tid=0 2024-04-25T02:01:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.312+0000 7f2825e36740 1 client.27029 shutdown 2024-04-25T02:01:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.313+0000 7f2825e36740 1 -- 172.21.15.102:0/1372955814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d12a190 msgr2=0x56290d123f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.313+0000 7f2825e36740 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d12a190 0x56290d123f80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2814004320 tx=0x7f28140046c0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f2825e36740 1 -- 172.21.15.102:0/1372955814 shutdown_connections 2024-04-25T02:01:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f2825e36740 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56290d1a0890 0x56290d1a2cb0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f2825e36740 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56290d12a190 0x56290d123f80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f2825e36740 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56290d11f8d0 0x56290d123a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f2825e36740 1 --2- 172.21.15.102:0/1372955814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290d11efb0 0x56290d132560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f2825e36740 1 -- 172.21.15.102:0/1372955814 >> 172.21.15.102:0/1372955814 conn(0x56290d11b4c0 msgr2=0x56290d125570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f280a000640 0 client.27029 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:01:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f2825e36740 1 -- 172.21.15.102:0/1372955814 shutdown_connections 2024-04-25T02:01:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f2825e36740 1 -- 172.21.15.102:0/1372955814 wait complete. 2024-04-25T02:01:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.314+0000 7f2825e36740 20 client.27029 trim_cache size 0 max 16384 2024-04-25T02:01:51.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:01:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:01:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:01:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:01:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.316+0000 7f2821a00640 1 -- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f28000a2d70 msgr2=0x7f28000a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.316+0000 7f2821a00640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f28000a2d70 0x7f28000a5230 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f28140058c0 tx=0x7f2814004730 comp rx=0 tx=0).stop 2024-04-25T02:01:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.316+0000 7f2821a00640 1 -- 172.21.15.102:0/1893490987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c108fd0 msgr2=0x7f281c19d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:01:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.316+0000 7f2821a00640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c108fd0 0x7f281c19d4b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f280c009fa0 tx=0x7f280c00c800 comp rx=0 tx=0).stop 2024-04-25T02:01:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.317+0000 7f2821a00640 1 -- 172.21.15.102:0/1893490987 shutdown_connections 2024-04-25T02:01:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.317+0000 7f2821a00640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f28000a2d70 0x7f28000a5230 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.317+0000 7f2821a00640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f281c112eb0 0x7f281c19df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.317+0000 7f2821a00640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f281c10d150 0x7f281c19d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.317+0000 7f2821a00640 1 --2- 172.21.15.102:0/1893490987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f281c108fd0 0x7f281c19d4b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:01:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.317+0000 7f2821a00640 1 -- 172.21.15.102:0/1893490987 >> 172.21.15.102:0/1893490987 conn(0x7f281c06cad0 msgr2=0x7f281c113d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:01:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.317+0000 7f2821a00640 1 -- 172.21.15.102:0/1893490987 shutdown_connections 2024-04-25T02:01:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:01:51.317+0000 7f2821a00640 1 -- 172.21.15.102:0/1893490987 wait complete. 2024-04-25T02:01:51.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:51 smithi110 ceph-mon[32848]: pgmap v200: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 6.4 MiB/s wr, 718 op/s 2024-04-25T02:01:51.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:51 smithi193 ceph-mon[32981]: pgmap v200: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 6.4 MiB/s wr, 718 op/s 2024-04-25T02:01:51.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:51 smithi102 ceph-mon[27796]: pgmap v200: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 6.4 MiB/s wr, 718 op/s 2024-04-25T02:01:52.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:52 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:01:52.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:52 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:01:52.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:52 smithi110 ceph-mon[32848]: scrub summary: idle 2024-04-25T02:01:52.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:52 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:01:52.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:52 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:01:52.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:52 smithi193 ceph-mon[32981]: scrub summary: idle 2024-04-25T02:01:52.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:52 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:01:52.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:52 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:01:52.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:52 smithi102 ceph-mon[27796]: scrub summary: idle 2024-04-25T02:01:53.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 360.0 s, 2447.2 tps, lat 40.773 ms stddev 47.738 2024-04-25T02:01:53.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:53 smithi110 ceph-mon[32848]: pgmap v201: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 8.2 MiB/s wr, 892 op/s 2024-04-25T02:01:53.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:53 smithi193 ceph-mon[32981]: pgmap v201: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 8.2 MiB/s wr, 892 op/s 2024-04-25T02:01:53.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:53 smithi102 ceph-mon[27796]: pgmap v201: 97 pgs: 97 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 8.2 MiB/s wr, 892 op/s 2024-04-25T02:01:55.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:55 smithi110 ceph-mon[32848]: pgmap v202: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 6.7 MiB/s wr, 708 op/s 2024-04-25T02:01:55.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:55 smithi193 ceph-mon[32981]: pgmap v202: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 6.7 MiB/s wr, 708 op/s 2024-04-25T02:01:55.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:55 smithi102 ceph-mon[27796]: pgmap v202: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 6.7 MiB/s wr, 708 op/s 2024-04-25T02:01:57.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:57 smithi110 ceph-mon[32848]: pgmap v203: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.1 MiB/s wr, 600 op/s 2024-04-25T02:01:57.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:57 smithi193 ceph-mon[32981]: pgmap v203: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.1 MiB/s wr, 600 op/s 2024-04-25T02:01:57.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:57 smithi102 ceph-mon[27796]: pgmap v203: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.1 MiB/s wr, 600 op/s 2024-04-25T02:01:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 365.0 s, 2659.6 tps, lat 37.846 ms stddev 46.537 2024-04-25T02:01:59.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:01:59 smithi110 ceph-mon[32848]: pgmap v204: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 8.7 MiB/s wr, 885 op/s 2024-04-25T02:01:59.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:01:59 smithi193 ceph-mon[32981]: pgmap v204: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 8.7 MiB/s wr, 885 op/s 2024-04-25T02:01:59.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:01:59 smithi102 ceph-mon[27796]: pgmap v204: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 8.7 MiB/s wr, 885 op/s 2024-04-25T02:02:01.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:01 smithi110 ceph-mon[32848]: pgmap v205: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.7 MiB/s wr, 581 op/s 2024-04-25T02:02:01.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:01 smithi193 ceph-mon[32981]: pgmap v205: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.7 MiB/s wr, 581 op/s 2024-04-25T02:02:01.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:01 smithi102 ceph-mon[27796]: pgmap v205: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.7 MiB/s wr, 581 op/s 2024-04-25T02:02:03.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 370.0 s, 1613.6 tps, lat 58.832 ms stddev 143.828 2024-04-25T02:02:03.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:03 smithi110 ceph-mon[32848]: pgmap v206: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.9 MiB/s wr, 799 op/s 2024-04-25T02:02:03.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:03 smithi193 ceph-mon[32981]: pgmap v206: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.9 MiB/s wr, 799 op/s 2024-04-25T02:02:03.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:03 smithi102 ceph-mon[27796]: pgmap v206: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.9 MiB/s wr, 799 op/s 2024-04-25T02:02:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.479+0000 7faf57e00640 1 -- 172.21.15.102:0/3803222446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58103020 msgr2=0x7faf5810b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.479+0000 7faf57e00640 1 --2- 172.21.15.102:0/3803222446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58103020 0x7faf5810b500 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7faf40009900 tx=0x7faf4002a480 comp rx=0 tx=0).stop 2024-04-25T02:02:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.479+0000 7faf57e00640 1 -- 172.21.15.102:0/3803222446 shutdown_connections 2024-04-25T02:02:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.479+0000 7faf57e00640 1 --2- 172.21.15.102:0/3803222446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf5810bb20 0x7faf5810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.479+0000 7faf57e00640 1 --2- 172.21.15.102:0/3803222446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58103020 0x7faf5810b500 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.479+0000 7faf57e00640 1 --2- 172.21.15.102:0/3803222446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf58102700 0x7faf58102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.479+0000 7faf57e00640 1 -- 172.21.15.102:0/3803222446 >> 172.21.15.102:0/3803222446 conn(0x7faf580fbf60 msgr2=0x7faf580fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:04.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.479+0000 7faf57e00640 1 -- 172.21.15.102:0/3803222446 shutdown_connections 2024-04-25T02:02:04.485 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.479+0000 7faf57e00640 1 -- 172.21.15.102:0/3803222446 wait complete. 2024-04-25T02:02:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.480+0000 7faf57e00640 1 Processor -- start 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.480+0000 7faf57e00640 1 -- start start 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf57e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58102700 0x7faf58100350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf57e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf58103020 0x7faf58100890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf57e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf5810bb20 0x7faf58100dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf57e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faf58111950 con 0x7faf5810bb20 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf57e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf581117d0 con 0x7faf58103020 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf57e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faf58111ad0 con 0x7faf58102700 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf56a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58102700 0x7faf58100350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf4fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf58103020 0x7faf58100890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf56a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58102700 0x7faf58100350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40588/0 (socket says 172.21.15.102:40588) 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf56a00640 1 -- 172.21.15.102:0/2059293630 learned_addr learned my addr 172.21.15.102:0/2059293630 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf57400640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf5810bb20 0x7faf58100dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf4fe00640 1 -- 172.21.15.102:0/2059293630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58102700 msgr2=0x7faf58100350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf4fe00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58102700 0x7faf58100350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf4fe00640 1 -- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf5810bb20 msgr2=0x7faf58100dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf4fe00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf5810bb20 0x7faf58100dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf4fe00640 1 -- 172.21.15.102:0/2059293630 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf581a6390 con 0x7faf58103020 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.481+0000 7faf56a00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58102700 0x7faf58100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.482+0000 7faf4fe00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf58103020 0x7faf58100890 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7faf4002a9a0 tx=0x7faf40003a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.482+0000 7faf4f400640 1 -- 172.21.15.102:0/2059293630 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4003f070 con 0x7faf58103020 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.482+0000 7faf4f400640 1 -- 172.21.15.102:0/2059293630 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf40031c90 con 0x7faf58103020 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.482+0000 7faf4f400640 1 -- 172.21.15.102:0/2059293630 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40035030 con 0x7faf58103020 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.482+0000 7faf57e00640 1 -- 172.21.15.102:0/2059293630 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf581a6620 con 0x7faf58103020 2024-04-25T02:02:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.482+0000 7faf57e00640 1 -- 172.21.15.102:0/2059293630 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf581a6ac0 con 0x7faf58103020 2024-04-25T02:02:04.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.485+0000 7faf4f400640 1 -- 172.21.15.102:0/2059293630 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7faf40042070 con 0x7faf58103020 2024-04-25T02:02:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.486+0000 7faf60447740 1 Processor -- start 2024-04-25T02:02:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.486+0000 7faf60447740 1 -- start start 2024-04-25T02:02:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf60447740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf60447740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da1578d0 0x5570da161c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf60447740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da162190 0x5570da164580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf60447740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5570da0baab0 con 0x5570da1578d0 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf60447740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5570da063640 con 0x5570da156fb0 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf60447740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5570da0b6a70 con 0x5570da162190 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf57400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf4fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da162190 0x5570da164580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf57400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41004/0 (socket says 172.21.15.102:41004) 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf57400640 1 -- 172.21.15.102:0/151668760 learned_addr learned my addr 172.21.15.102:0/151668760 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf4f400640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf3c0a2d70 0x7faf3c0a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf56a00640 1 --2- 172.21.15.102:0/151668760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da1578d0 0x5570da161c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.487+0000 7faf4f400640 1 -- 172.21.15.102:0/2059293630 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7faf400efbb0 con 0x7faf58103020 2024-04-25T02:02:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf57400640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf3c0a2d70 0x7faf3c0a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf4fe00640 1 -- 172.21.15.102:0/151668760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 msgr2=0x5570da157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf4fe00640 1 --2- 172.21.15.102:0/151668760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf4fe00640 1 -- 172.21.15.102:0/151668760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da1578d0 msgr2=0x5570da161c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf4fe00640 1 --2- 172.21.15.102:0/151668760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da1578d0 0x5570da161c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf4fe00640 1 -- 172.21.15.102:0/151668760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570d9dcf210 con 0x5570da162190 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf4fe00640 1 --2- 172.21.15.102:0/151668760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da162190 0x5570da164580 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7faf400ed2d0 tx=0x7faf4003b6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e73221a77bd351c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf47400640 1 -- 172.21.15.102:0/151668760 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40049070 con 0x5570da162190 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf47400640 1 -- 172.21.15.102:0/151668760 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf40031e40 con 0x5570da162190 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.488+0000 7faf47400640 1 -- 172.21.15.102:0/151668760 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf400b87c0 con 0x5570da162190 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf60447740 1 -- 172.21.15.102:0/151668760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da162190 msgr2=0x5570da164580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf57400640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf3c0a2d70 0x7faf3c0a5230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faf500049d0 tx=0x7faf50004ab0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf60447740 1 --2- 172.21.15.102:0/151668760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da162190 0x5570da164580 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7faf400ed2d0 tx=0x7faf4003b6b0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf60447740 1 -- 172.21.15.102:0/151668760 shutdown_connections 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf60447740 1 --2- 172.21.15.102:0/151668760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da162190 0x5570da164580 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf60447740 1 --2- 172.21.15.102:0/151668760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da1578d0 0x5570da161c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf60447740 1 --2- 172.21.15.102:0/151668760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf60447740 1 -- 172.21.15.102:0/151668760 >> 172.21.15.102:0/151668760 conn(0x5570da1534c0 msgr2=0x5570da1538a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf60447740 1 -- 172.21.15.102:0/151668760 shutdown_connections 2024-04-25T02:02:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.489+0000 7faf60447740 1 -- 172.21.15.102:0/151668760 wait complete. 2024-04-25T02:02:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.490+0000 7faf60447740 1 -- start start 2024-04-25T02:02:04.496 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.490+0000 7faf60447740 1 Processor -- start 2024-04-25T02:02:04.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf60447740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da15f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf60447740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 0x5570da15f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf60447740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da15fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf60447740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5570da0b6a70 con 0x5570da162190 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf60447740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5570da076990 con 0x5570da156fb0 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf60447740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5570da164b80 con 0x5570da1578d0 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf60447740 10 client.0 _ll_register_callbacks cb 0x5570da078950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf56a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da15f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf56a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da15f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41012/0 (socket says 172.21.15.102:41012) 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf56a00640 1 -- 172.21.15.102:0/3721780794 learned_addr learned my addr 172.21.15.102:0/3721780794 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf4fe00640 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 0x5570da15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf57400640 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da15fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf56a00640 1 -- 172.21.15.102:0/3721780794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 msgr2=0x5570da15f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.491+0000 7faf56a00640 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 0x5570da15f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf56a00640 1 -- 172.21.15.102:0/3721780794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 msgr2=0x5570da15fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf56a00640 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da15fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf56a00640 1 -- 172.21.15.102:0/3721780794 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570d9e9b620 con 0x5570da156fb0 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf57400640 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da15fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf56a00640 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da15f110 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7faf48012d20 tx=0x7faf4800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3d1f51a4b1fdcbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf3be00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf3be00640 1 -- 172.21.15.102:0/3721780794 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4800fe20 con 0x5570da156fb0 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf60447740 1 -- 172.21.15.102:0/3721780794 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5570da0c7ff0 con 0x5570da156fb0 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf3be00640 1 -- 172.21.15.102:0/3721780794 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7faf48010430 con 0x5570da156fb0 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.492+0000 7faf3be00640 1 -- 172.21.15.102:0/3721780794 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7faf48010630 con 0x5570da156fb0 2024-04-25T02:02:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf60447740 10 client.27041 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf60447740 1 -- 172.21.15.102:0/3721780794 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5570da170910 con 0x5570da156fb0 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf3be00640 1 -- 172.21.15.102:0/3721780794 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5570da0c7ff0 con 0x5570da156fb0 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf3be00640 1 -- 172.21.15.102:0/3721780794 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7faf4801c840 con 0x5570da156fb0 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf60447740 10 client.27041 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf60447740 10 client.27041 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf60447740 20 client.27041 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf60447740 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5570da171e90 0x5570da1742a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf60447740 4 client.27041 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.493+0000 7faf60447740 1 -- 172.21.15.102:0/3721780794 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570da0c2340 con 0x5570da171e90 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.494+0000 7faf4fe00640 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5570da171e90 0x5570da1742a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.494+0000 7faf4fe00640 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5570da171e90 0x5570da1742a0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=295d4da4b3b44cc7 server_cookie=7a7f2177f6e1da95 in_seq=0 out_seq=0 2024-04-25T02:02:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.494+0000 7faf3be00640 10 client.27041 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf3be00640 1 -- 172.21.15.102:0/3721780794 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570da0c2340 con 0x5570da171e90 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf3be00640 10 client.27041 handle_command_reply: tid=0 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 client.27041 shutdown 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 -- 172.21.15.102:0/3721780794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 msgr2=0x5570da15f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da15f110 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7faf48012d20 tx=0x7faf4800f790 comp rx=0 tx=0).stop 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 -- 172.21.15.102:0/3721780794 shutdown_connections 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5570da171e90 0x5570da1742a0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da15fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 0x5570da15f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 --2- 172.21.15.102:0/3721780794 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da15f110 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf3be00640 0 client.27041 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 -- 172.21.15.102:0/3721780794 >> 172.21.15.102:0/3721780794 conn(0x5570da1534c0 msgr2=0x5570da159240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:04.503 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 -- 172.21.15.102:0/3721780794 shutdown_connections 2024-04-25T02:02:04.504 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 1 -- 172.21.15.102:0/3721780794 wait complete. 2024-04-25T02:02:04.504 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.495+0000 7faf60447740 20 client.27041 trim_cache size 0 max 16384 2024-04-25T02:02:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.512+0000 7faf60447740 1 Processor -- start 2024-04-25T02:02:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf60447740 1 -- start start 2024-04-25T02:02:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf60447740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da159970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf60447740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 0x5570da15f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf60447740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da179b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf60447740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faf581b1e10 con 0x5570da162190 2024-04-25T02:02:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf60447740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf581af620 con 0x5570da156fb0 2024-04-25T02:02:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf60447740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faf581ad470 con 0x5570da1578d0 2024-04-25T02:02:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf56a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf4fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 0x5570da15f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf56a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41018/0 (socket says 172.21.15.102:41018) 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf56a00640 1 -- 172.21.15.102:0/4224605636 learned_addr learned my addr 172.21.15.102:0/4224605636 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.513+0000 7faf57400640 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da179b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf56a00640 1 -- 172.21.15.102:0/4224605636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 msgr2=0x5570da15f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf56a00640 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 0x5570da15f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf56a00640 1 -- 172.21.15.102:0/4224605636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 msgr2=0x5570da179b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf56a00640 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da179b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf56a00640 1 -- 172.21.15.102:0/4224605636 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570da1649a0 con 0x5570da156fb0 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf57400640 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da179b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf4fe00640 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 0x5570da15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:04.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf56a00640 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da159970 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7faf480098e0 tx=0x7faf4800fc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3155bee6a379d808 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf46a00640 1 -- 172.21.15.102:0/4224605636 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48000e60 con 0x5570da156fb0 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf46a00640 1 -- 172.21.15.102:0/4224605636 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf48010430 con 0x5570da156fb0 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.514+0000 7faf46a00640 1 -- 172.21.15.102:0/4224605636 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48029070 con 0x5570da156fb0 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.515+0000 7faf60447740 1 -- 172.21.15.102:0/4224605636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 msgr2=0x5570da159970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.515+0000 7faf60447740 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da159970 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7faf480098e0 tx=0x7faf4800fc90 comp rx=0 tx=0).stop 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.515+0000 7faf60447740 1 -- 172.21.15.102:0/4224605636 shutdown_connections 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.515+0000 7faf60447740 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da179b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.515+0000 7faf60447740 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da1578d0 0x5570da15f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.515+0000 7faf60447740 1 --2- 172.21.15.102:0/4224605636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da156fb0 0x5570da159970 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.515+0000 7faf60447740 1 -- 172.21.15.102:0/4224605636 >> 172.21.15.102:0/4224605636 conn(0x5570da1534c0 msgr2=0x5570da152620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.515+0000 7faf60447740 1 -- 172.21.15.102:0/4224605636 shutdown_connections 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.515+0000 7faf60447740 1 -- 172.21.15.102:0/4224605636 wait complete. 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.516+0000 7faf60447740 1 -- start start 2024-04-25T02:02:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.516+0000 7faf60447740 1 Processor -- start 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf60447740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da156fb0 0x5570da165b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf60447740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da1578d0 0x5570da166080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf60447740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da1d73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf60447740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faf581adae0 con 0x5570da162190 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf60447740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7faf581aef50 con 0x5570da1578d0 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf60447740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faf581ae1b0 con 0x5570da156fb0 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf60447740 10 client.0 _ll_register_callbacks cb 0x7faf48009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf56a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da156fb0 0x5570da165b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf4fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da1578d0 0x5570da166080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf56a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da156fb0 0x5570da165b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40616/0 (socket says 172.21.15.102:40616) 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf56a00640 1 -- 172.21.15.102:0/1009139715 learned_addr learned my addr 172.21.15.102:0/1009139715 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf57400640 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da1d73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf4fe00640 1 -- 172.21.15.102:0/1009139715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da156fb0 msgr2=0x5570da165b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf4fe00640 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da156fb0 0x5570da165b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf4fe00640 1 -- 172.21.15.102:0/1009139715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 msgr2=0x5570da1d73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf4fe00640 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da1d73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.517+0000 7faf4fe00640 1 -- 172.21.15.102:0/1009139715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570da166950 con 0x5570da1578d0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf4fe00640 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da1578d0 0x5570da166080 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7faf400c0610 tx=0x7faf400bc020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9bef7ca8a039f77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf3be00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf3be00640 1 -- 172.21.15.102:0/1009139715 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf40049040 con 0x5570da1578d0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf60447740 1 -- 172.21.15.102:0/1009139715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf48010430 con 0x5570da1578d0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf3be00640 1 -- 172.21.15.102:0/1009139715 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8469+0+0 (secure 0 0 0) 0x7faf400b87c0 con 0x5570da1578d0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf3be00640 1 -- 172.21.15.102:0/1009139715 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7faf400c0450 con 0x5570da1578d0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf60447740 10 client.27047 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf60447740 1 -- 172.21.15.102:0/1009139715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5570da1d8a30 con 0x5570da1578d0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf3be00640 1 -- 172.21.15.102:0/1009139715 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf48010430 con 0x5570da1578d0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf3be00640 1 -- 172.21.15.102:0/1009139715 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7faf400dace0 con 0x5570da1578d0 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf60447740 10 client.27047 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf60447740 10 client.27047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.518+0000 7faf60447740 20 client.27047 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.519+0000 7faf60447740 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5570da1d8f70 0x5570da1db360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.519+0000 7faf60447740 4 client.27047 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.519+0000 7faf60447740 1 -- 172.21.15.102:0/1009139715 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5570da0407a0 con 0x5570da1d8f70 2024-04-25T02:02:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.519+0000 7faf56a00640 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5570da1d8f70 0x5570da1db360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.519+0000 7faf56a00640 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5570da1d8f70 0x5570da1db360 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a25cf0b0fe0ad9e6 server_cookie=7b624f1bf8e7f8e6 in_seq=0 out_seq=0 2024-04-25T02:02:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.519+0000 7faf3be00640 10 client.27047 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:04.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf3be00640 1 -- 172.21.15.102:0/1009139715 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5570da0407a0 con 0x5570da1d8f70 2024-04-25T02:02:04.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf3be00640 10 client.27047 handle_command_reply: tid=0 2024-04-25T02:02:04.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 client.27047 shutdown 2024-04-25T02:02:04.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 -- 172.21.15.102:0/1009139715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da1578d0 msgr2=0x5570da166080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:04.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da1578d0 0x5570da166080 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7faf400c0610 tx=0x7faf400bc020 comp rx=0 tx=0).stop 2024-04-25T02:02:04.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 -- 172.21.15.102:0/1009139715 shutdown_connections 2024-04-25T02:02:04.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5570da1d8f70 0x5570da1db360 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5570da162190 0x5570da1d73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5570da1578d0 0x5570da166080 secure :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7faf400c0610 tx=0x7faf400bc020 comp rx=0 tx=0).stop 2024-04-25T02:02:04.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 --2- 172.21.15.102:0/1009139715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5570da156fb0 0x5570da165b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf3be00640 0 client.27047 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:04.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 -- 172.21.15.102:0/1009139715 >> 172.21.15.102:0/1009139715 conn(0x5570da1534c0 msgr2=0x5570da160940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:04.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 -- 172.21.15.102:0/1009139715 shutdown_connections 2024-04-25T02:02:04.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.527+0000 7faf60447740 1 -- 172.21.15.102:0/1009139715 wait complete. 2024-04-25T02:02:04.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf60447740 20 client.27047 trim_cache size 0 max 16384 2024-04-25T02:02:04.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:02:04.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:02:04.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:02:04.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:02:04.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 -- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf3c0a2d70 msgr2=0x7faf3c0a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:04.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf3c0a2d70 0x7faf3c0a5230 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faf500049d0 tx=0x7faf50004ab0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 -- 172.21.15.102:0/2059293630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf58103020 msgr2=0x7faf58100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:04.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf58103020 0x7faf58100890 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7faf4002a9a0 tx=0x7faf40003a70 comp rx=0 tx=0).stop 2024-04-25T02:02:04.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 -- 172.21.15.102:0/2059293630 shutdown_connections 2024-04-25T02:02:04.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7faf3c0a2d70 0x7faf3c0a5230 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf5810bb20 0x7faf58100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf58103020 0x7faf58100890 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 --2- 172.21.15.102:0/2059293630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faf58102700 0x7faf58100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.528+0000 7faf57e00640 1 -- 172.21.15.102:0/2059293630 >> 172.21.15.102:0/2059293630 conn(0x7faf580fbf60 msgr2=0x7faf580fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.529+0000 7faf57e00640 1 -- 172.21.15.102:0/2059293630 shutdown_connections 2024-04-25T02:02:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:04.529+0000 7faf57e00640 1 -- 172.21.15.102:0/2059293630 wait complete. 2024-04-25T02:02:05.491 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:05 smithi110 ceph-mon[32848]: pgmap v207: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.8 MiB/s wr, 718 op/s 2024-04-25T02:02:05.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:05 smithi193 ceph-mon[32981]: pgmap v207: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.8 MiB/s wr, 718 op/s 2024-04-25T02:02:05.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:05 smithi102 ceph-mon[27796]: pgmap v207: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.8 MiB/s wr, 718 op/s 2024-04-25T02:02:07.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:07 smithi193 ceph-mon[32981]: pgmap v208: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 5.5 MiB/s wr, 594 op/s 2024-04-25T02:02:07.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:07 smithi102 ceph-mon[27796]: pgmap v208: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 5.5 MiB/s wr, 594 op/s 2024-04-25T02:02:07.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:07 smithi110 ceph-mon[32848]: pgmap v208: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 5.5 MiB/s wr, 594 op/s 2024-04-25T02:02:08.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 375.0 s, 146.4 tps, lat 646.600 ms stddev 381.450 2024-04-25T02:02:08.509 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:08 smithi193 ceph-mon[32981]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T02:02:08.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T02:02:08.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T02:02:08.673 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:08 smithi102 ceph-mon[27796]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T02:02:08.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:08 smithi110 ceph-mon[32848]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T02:02:08.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.806+0000 7f2418c00640 1 -- 172.21.15.102:0/1913441575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 msgr2=0x7f2414103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:08.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.806+0000 7f2418c00640 1 --2- 172.21.15.102:0/1913441575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 0x7f2414103d80 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009900 tx=0x7f23fc02a410 comp rx=0 tx=0).stop 2024-04-25T02:02:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.806+0000 7f2418c00640 1 -- 172.21.15.102:0/1913441575 shutdown_connections 2024-04-25T02:02:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.806+0000 7f2418c00640 1 --2- 172.21.15.102:0/1913441575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24141042c0 0x7f24141107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.806+0000 7f2418c00640 1 --2- 172.21.15.102:0/1913441575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 0x7f2414103d80 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.806+0000 7f2418c00640 1 --2- 172.21.15.102:0/1913441575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2414109910 0x7f2414109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.806+0000 7f2418c00640 1 -- 172.21.15.102:0/1913441575 >> 172.21.15.102:0/1913441575 conn(0x7f24140ff3f0 msgr2=0x7f2414101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.807+0000 7f2418c00640 1 -- 172.21.15.102:0/1913441575 shutdown_connections 2024-04-25T02:02:08.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.807+0000 7f2418c00640 1 -- 172.21.15.102:0/1913441575 wait complete. 2024-04-25T02:02:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2418c00640 1 Processor -- start 2024-04-25T02:02:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2418c00640 1 -- start start 2024-04-25T02:02:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2418c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 0x7f241419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2418c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24141042c0 0x7f241419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2418c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2414109910 0x7f241419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2418c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2414113d70 con 0x7f24141042c0 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2418c00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f2414113bf0 con 0x7f2414103900 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2418c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2414113ef0 con 0x7f2414109910 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2411e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 0x7f241419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2411e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 0x7f241419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41040/0 (socket says 172.21.15.102:41040) 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.808+0000 7f2411e00640 1 -- 172.21.15.102:0/567017910 learned_addr learned my addr 172.21.15.102:0/567017910 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f2411400640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24141042c0 0x7f241419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f2412800640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2414109910 0x7f241419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f2411e00640 1 -- 172.21.15.102:0/567017910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2414109910 msgr2=0x7f241419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f2411e00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2414109910 0x7f241419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f2411e00640 1 -- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24141042c0 msgr2=0x7f241419a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f2411e00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24141042c0 0x7f241419a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f2411e00640 1 -- 172.21.15.102:0/567017910 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241419f200 con 0x7f2414103900 2024-04-25T02:02:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f2411400640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24141042c0 0x7f241419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:08.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f2411e00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 0x7f241419a210 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f240400e9f0 tx=0x7f240400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:08.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f240aa00640 1 -- 172.21.15.102:0/567017910 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240400cd80 con 0x7f2414103900 2024-04-25T02:02:08.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f240aa00640 1 -- 172.21.15.102:0/567017910 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2404010910 con 0x7f2414103900 2024-04-25T02:02:08.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.809+0000 7f240aa00640 1 -- 172.21.15.102:0/567017910 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2404018990 con 0x7f2414103900 2024-04-25T02:02:08.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.810+0000 7f2418c00640 1 -- 172.21.15.102:0/567017910 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241419f4f0 con 0x7f2414103900 2024-04-25T02:02:08.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.810+0000 7f2418c00640 1 -- 172.21.15.102:0/567017910 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24141a6d80 con 0x7f2414103900 2024-04-25T02:02:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.814+0000 7f241bba2740 1 Processor -- start 2024-04-25T02:02:08.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.814+0000 7f241bba2740 1 -- start start 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.814+0000 7f240aa00640 1 -- 172.21.15.102:0/567017910 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2404010ac0 con 0x7f2414103900 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f241bba2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f241bba2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f241bba2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d4ff3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f241bba2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559f3d456560 con 0x559f3d4fcff0 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f241bba2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559f3d3fe930 con 0x559f3d4f2920 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f241bba2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f3d452520 con 0x559f3d4f2000 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2411e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2412800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2411e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fcab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41042/0 (socket says 172.21.15.102:41042) 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2411e00640 1 -- 172.21.15.102:0/877554712 learned_addr learned my addr 172.21.15.102:0/877554712 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:08.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2411400640 1 --2- 172.21.15.102:0/877554712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d4ff3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2411e00640 1 -- 172.21.15.102:0/877554712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 msgr2=0x559f3d4f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2411e00640 1 --2- 172.21.15.102:0/877554712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2411e00640 1 -- 172.21.15.102:0/877554712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 msgr2=0x559f3d4ff3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2411e00640 1 --2- 172.21.15.102:0/877554712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d4ff3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.815+0000 7f2411e00640 1 -- 172.21.15.102:0/877554712 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f3d2347d0 con 0x559f3d4f2920 2024-04-25T02:02:08.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.816+0000 7f2411e00640 1 --2- 172.21.15.102:0/877554712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fcab0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f24040cce40 tx=0x7f24040d7690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0195976e01e8339 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:08.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.816+0000 7f2403000640 1 -- 172.21.15.102:0/877554712 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24040de070 con 0x559f3d4f2920 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.816+0000 7f2403000640 1 -- 172.21.15.102:0/877554712 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24040da920 con 0x559f3d4f2920 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.816+0000 7f2403000640 1 -- 172.21.15.102:0/877554712 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24040d1570 con 0x559f3d4f2920 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f240aa00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23f40a2d80 0x7f23f40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f241bba2740 1 -- 172.21.15.102:0/877554712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 msgr2=0x559f3d4fcab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f241bba2740 1 --2- 172.21.15.102:0/877554712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fcab0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f24040cce40 tx=0x7f24040d7690 comp rx=0 tx=0).stop 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f240aa00640 1 -- 172.21.15.102:0/567017910 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f24040cf7a0 con 0x7f2414103900 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f2412800640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23f40a2d80 0x7f23f40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f241bba2740 1 -- 172.21.15.102:0/877554712 shutdown_connections 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f241bba2740 1 --2- 172.21.15.102:0/877554712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d4ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f241bba2740 1 --2- 172.21.15.102:0/877554712 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fcab0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f241bba2740 1 --2- 172.21.15.102:0/877554712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f241bba2740 1 -- 172.21.15.102:0/877554712 >> 172.21.15.102:0/877554712 conn(0x559f3d4ee320 msgr2=0x559f3d4ee700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:08.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f241bba2740 1 -- 172.21.15.102:0/877554712 shutdown_connections 2024-04-25T02:02:08.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.817+0000 7f241bba2740 1 -- 172.21.15.102:0/877554712 wait complete. 2024-04-25T02:02:08.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.818+0000 7f2412800640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23f40a2d80 0x7f23f40a5240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f240c005fd0 tx=0x7f240c009500 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:08.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.818+0000 7f241bba2740 1 -- start start 2024-04-25T02:02:08.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.818+0000 7f241bba2740 1 Processor -- start 2024-04-25T02:02:08.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.818+0000 7f241bba2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.818+0000 7f241bba2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f241bba2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d4fa990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f241bba2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559f3d4ffde0 con 0x559f3d4fcff0 2024-04-25T02:02:08.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f241bba2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559f3d411d90 con 0x559f3d4f2920 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f241bba2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f3d4fff60 con 0x559f3d4f2000 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f241bba2740 10 client.0 _ll_register_callbacks cb 0x559f3d411780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40650/0 (socket says 172.21.15.102:40650) 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411e00640 1 -- 172.21.15.102:0/2847681603 learned_addr learned my addr 172.21.15.102:0/2847681603 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411400640 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2412800640 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d4fa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411e00640 1 -- 172.21.15.102:0/2847681603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 msgr2=0x559f3d4fa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411e00640 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411e00640 1 -- 172.21.15.102:0/2847681603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 msgr2=0x559f3d4fa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411e00640 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d4fa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411e00640 1 -- 172.21.15.102:0/2847681603 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f3d16a3f0 con 0x559f3d4f2000 2024-04-25T02:02:08.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f2411e00640 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f9f10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2404098670 tx=0x7f2404018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71df3e3fa76439e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.819+0000 7f23fb400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f23fb400640 1 -- 172.21.15.102:0/2847681603 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24040de070 con 0x559f3d4f2000 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f23fb400640 1 -- 172.21.15.102:0/2847681603 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f24040d5790 con 0x559f3d4f2000 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f241bba2740 1 -- 172.21.15.102:0/2847681603 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24040da920 con 0x559f3d4f2000 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f23fb400640 1 -- 172.21.15.102:0/2847681603 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2404097f40 con 0x559f3d4f2000 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f241bba2740 10 client.27019 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f241bba2740 1 -- 172.21.15.102:0/2847681603 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559f3d12a0f0 con 0x559f3d4f2000 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f23fb400640 1 -- 172.21.15.102:0/2847681603 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24040da920 con 0x559f3d4f2000 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f23fb400640 1 -- 172.21.15.102:0/2847681603 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f2404098310 con 0x559f3d4f2000 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f241bba2740 10 client.27019 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f241bba2740 10 client.27019 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f241bba2740 20 client.27019 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f241bba2740 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559f3d50d1a0 0x559f3d50f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f241bba2740 4 client.27019 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.820+0000 7f241bba2740 1 -- 172.21.15.102:0/2847681603 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f3d469d90 con 0x559f3d50d1a0 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.821+0000 7f2411400640 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559f3d50d1a0 0x559f3d50f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.821+0000 7f2411400640 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559f3d50d1a0 0x559f3d50f590 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77fe1e1138d968e3 server_cookie=3f85d75ad9e495c3 in_seq=0 out_seq=0 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.821+0000 7f23fb400640 10 client.27019 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f23fb400640 1 -- 172.21.15.102:0/2847681603 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f3d469d90 con 0x559f3d50d1a0 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f23fb400640 10 client.27019 handle_command_reply: tid=0 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f241bba2740 1 client.27019 shutdown 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f241bba2740 1 -- 172.21.15.102:0/2847681603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 msgr2=0x559f3d4f9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f241bba2740 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f9f10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2404098670 tx=0x7f2404018420 comp rx=0 tx=0).stop 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f241bba2740 1 -- 172.21.15.102:0/2847681603 shutdown_connections 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f241bba2740 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559f3d50d1a0 0x559f3d50f590 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f241bba2740 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d4fa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f241bba2740 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4fa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f241bba2740 1 --2- 172.21.15.102:0/2847681603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4f9f10 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.822+0000 7f241bba2740 1 -- 172.21.15.102:0/2847681603 >> 172.21.15.102:0/2847681603 conn(0x559f3d4ee320 msgr2=0x559f3d4f4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.823+0000 7f23fb400640 0 client.27019 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.823+0000 7f241bba2740 1 -- 172.21.15.102:0/2847681603 shutdown_connections 2024-04-25T02:02:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.823+0000 7f241bba2740 1 -- 172.21.15.102:0/2847681603 wait complete. 2024-04-25T02:02:08.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.823+0000 7f241bba2740 20 client.27019 trim_cache size 0 max 16384 2024-04-25T02:02:08.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.845+0000 7f241bba2740 1 Processor -- start 2024-04-25T02:02:08.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.845+0000 7f241bba2740 1 -- start start 2024-04-25T02:02:08.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f241bba2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4e92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f241bba2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4ed300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f241bba2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d5121d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f241bba2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24141b2190 con 0x559f3d4fcff0 2024-04-25T02:02:08.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f241bba2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f24141af9a0 con 0x559f3d4f2920 2024-04-25T02:02:08.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f241bba2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f24141ad810 con 0x559f3d4f2000 2024-04-25T02:02:08.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4e92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2412800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d5121d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4ed300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4e92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40656/0 (socket says 172.21.15.102:40656) 2024-04-25T02:02:08.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411e00640 1 -- 172.21.15.102:0/3505709200 learned_addr learned my addr 172.21.15.102:0/3505709200 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:08.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411e00640 1 -- 172.21.15.102:0/3505709200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 msgr2=0x559f3d4ed300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:08.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411e00640 1 --2- 172.21.15.102:0/3505709200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4ed300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411e00640 1 -- 172.21.15.102:0/3505709200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 msgr2=0x559f3d5121d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411e00640 1 --2- 172.21.15.102:0/3505709200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d5121d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411e00640 1 -- 172.21.15.102:0/3505709200 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f3d4ff970 con 0x559f3d4f2000 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.846+0000 7f2411400640 1 --2- 172.21.15.102:0/3505709200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4ed300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.847+0000 7f2411e00640 1 --2- 172.21.15.102:0/3505709200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4e92e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2404098100 tx=0x7f24040ccf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57e7c6450375ffb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.847+0000 7f2402600640 1 -- 172.21.15.102:0/3505709200 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24040de070 con 0x559f3d4f2000 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.847+0000 7f2402600640 1 -- 172.21.15.102:0/3505709200 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f240409c3a0 con 0x559f3d4f2000 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.847+0000 7f2402600640 1 -- 172.21.15.102:0/3505709200 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240409b600 con 0x559f3d4f2000 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 -- 172.21.15.102:0/3505709200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 msgr2=0x559f3d4e92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 --2- 172.21.15.102:0/3505709200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4e92e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2404098100 tx=0x7f24040ccf80 comp rx=0 tx=0).stop 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 -- 172.21.15.102:0/3505709200 shutdown_connections 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 --2- 172.21.15.102:0/3505709200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d5121d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 --2- 172.21.15.102:0/3505709200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d4ed300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 --2- 172.21.15.102:0/3505709200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d4e92e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 -- 172.21.15.102:0/3505709200 >> 172.21.15.102:0/3505709200 conn(0x559f3d4ee320 msgr2=0x559f3d4ecbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:08.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 -- 172.21.15.102:0/3505709200 shutdown_connections 2024-04-25T02:02:08.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 -- 172.21.15.102:0/3505709200 wait complete. 2024-04-25T02:02:08.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 -- start start 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.848+0000 7f241bba2740 1 Processor -- start 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f241bba2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d5121d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f241bba2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d512710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f241bba2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d50f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f241bba2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24141ade60 con 0x559f3d4fcff0 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f241bba2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f24141af2d0 con 0x559f3d4f2920 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f241bba2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f24141ae530 con 0x559f3d4f2000 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f241bba2740 10 client.0 _ll_register_callbacks cb 0x7f240400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2411400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d512710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2411400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d512710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41076/0 (socket says 172.21.15.102:41076) 2024-04-25T02:02:08.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2412800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d50f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2411400640 1 -- 172.21.15.102:0/593636720 learned_addr learned my addr 172.21.15.102:0/593636720 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2411e00640 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d5121d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2411400640 1 -- 172.21.15.102:0/593636720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 msgr2=0x559f3d5121d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2411400640 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d5121d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2411400640 1 -- 172.21.15.102:0/593636720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 msgr2=0x559f3d50f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2411400640 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d50f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.849+0000 7f2411400640 1 -- 172.21.15.102:0/593636720 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f3d50ff60 con 0x559f3d4f2920 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.850+0000 7f2411400640 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d512710 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f23fc000c00 tx=0x7f23fc005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=787a3219daf78d57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.850+0000 7f23fb400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.850+0000 7f23fb400640 1 -- 172.21.15.102:0/593636720 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc035040 con 0x559f3d4f2920 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.850+0000 7f241bba2740 1 -- 172.21.15.102:0/593636720 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f240409c3a0 con 0x559f3d4f2920 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.850+0000 7f23fb400640 1 -- 172.21.15.102:0/593636720 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f23fc0051b0 con 0x559f3d4f2920 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.850+0000 7f23fb400640 1 -- 172.21.15.102:0/593636720 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f23fc0053b0 con 0x559f3d4f2920 2024-04-25T02:02:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.850+0000 7f241bba2740 10 client.27065 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.850+0000 7f241bba2740 1 -- 172.21.15.102:0/593636720 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559f3d573050 con 0x559f3d4f2920 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.851+0000 7f23fb400640 1 -- 172.21.15.102:0/593636720 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f240409c3a0 con 0x559f3d4f2920 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.851+0000 7f23fb400640 1 -- 172.21.15.102:0/593636720 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f23fc004a60 con 0x559f3d4f2920 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.851+0000 7f241bba2740 10 client.27065 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.851+0000 7f241bba2740 10 client.27065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.851+0000 7f241bba2740 20 client.27065 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.851+0000 7f241bba2740 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559f3d573810 0x559f3d575c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.851+0000 7f241bba2740 4 client.27065 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.851+0000 7f241bba2740 1 -- 172.21.15.102:0/593636720 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559f3d463aa0 con 0x559f3d573810 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.851+0000 7f2411e00640 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559f3d573810 0x559f3d575c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f2411e00640 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559f3d573810 0x559f3d575c30 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a875dd968278cafe server_cookie=e7e8e59bb6d4b177 in_seq=0 out_seq=0 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f23fb400640 10 client.27065 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f23fb400640 1 -- 172.21.15.102:0/593636720 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559f3d463aa0 con 0x559f3d573810 2024-04-25T02:02:08.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f23fb400640 10 client.27065 handle_command_reply: tid=0 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 client.27065 shutdown 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 -- 172.21.15.102:0/593636720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 msgr2=0x559f3d512710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d512710 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f23fc000c00 tx=0x7f23fc005040 comp rx=0 tx=0).stop 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 -- 172.21.15.102:0/593636720 shutdown_connections 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559f3d573810 0x559f3d575c30 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559f3d4fcff0 0x559f3d50f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559f3d4f2920 0x559f3d512710 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f23fb400640 0 client.27065 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 --2- 172.21.15.102:0/593636720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f3d4f2000 0x559f3d5121d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 -- 172.21.15.102:0/593636720 >> 172.21.15.102:0/593636720 conn(0x559f3d4ee320 msgr2=0x559f3d508a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 -- 172.21.15.102:0/593636720 shutdown_connections 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 1 -- 172.21.15.102:0/593636720 wait complete. 2024-04-25T02:02:08.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.852+0000 7f241bba2740 20 client.27065 trim_cache size 0 max 16384 2024-04-25T02:02:08.862 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:02:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "status": "no active scrubs running", 2024-04-25T02:02:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "scrubs": {} 2024-04-25T02:02:08.862 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 -- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23f40a2d80 msgr2=0x7f23f40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23f40a2d80 0x7f23f40a5240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f240c005fd0 tx=0x7f240c009500 comp rx=0 tx=0).stop 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 -- 172.21.15.102:0/567017910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 msgr2=0x7f241419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 0x7f241419a210 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f240400e9f0 tx=0x7f240400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 -- 172.21.15.102:0/567017910 shutdown_connections 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f23f40a2d80 0x7f23f40a5240 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2414109910 0x7f241419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24141042c0 0x7f241419a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 --2- 172.21.15.102:0/567017910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2414103900 0x7f241419a210 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 -- 172.21.15.102:0/567017910 >> 172.21.15.102:0/567017910 conn(0x7f24140ff3f0 msgr2=0x7f24140ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:08.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 -- 172.21.15.102:0/567017910 shutdown_connections 2024-04-25T02:02:08.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:08.853+0000 7f2418c00640 1 -- 172.21.15.102:0/567017910 wait complete. 2024-04-25T02:02:08.878 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T02:02:08.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T02:02:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.070+0000 7f311aa00640 1 -- 172.21.15.102:0/1156863598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3114073b10 msgr2=0x7f3114073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.070+0000 7f311aa00640 1 --2- 172.21.15.102:0/1156863598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3114073b10 0x7f3114073ef0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3104009860 tx=0x7f310402a3d0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.071+0000 7f311aa00640 1 -- 172.21.15.102:0/1156863598 shutdown_connections 2024-04-25T02:02:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.071+0000 7f311aa00640 1 --2- 172.21.15.102:0/1156863598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f311410dff0 0x7f3114110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.071+0000 7f311aa00640 1 --2- 172.21.15.102:0/1156863598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3114074430 0x7f311410dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.071+0000 7f311aa00640 1 --2- 172.21.15.102:0/1156863598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3114073b10 0x7f3114073ef0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.071+0000 7f311aa00640 1 -- 172.21.15.102:0/1156863598 >> 172.21.15.102:0/1156863598 conn(0x7f3114078bd0 msgr2=0x7f3114078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.071+0000 7f311aa00640 1 -- 172.21.15.102:0/1156863598 shutdown_connections 2024-04-25T02:02:09.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.071+0000 7f311aa00640 1 -- 172.21.15.102:0/1156863598 wait complete. 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.072+0000 7f311aa00640 1 Processor -- start 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.072+0000 7f311aa00640 1 -- start start 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f311aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3114073b10 0x7f311419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f311aa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3114074430 0x7f311419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f311aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f311410dff0 0x7f311419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f311aa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3114113da0 con 0x7f3114073b10 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f311aa00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3114113c20 con 0x7f3114074430 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f311aa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3114113f20 con 0x7f311410dff0 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3113400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3114074430 0x7f311419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3113400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3114074430 0x7f311419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41102/0 (socket says 172.21.15.102:41102) 2024-04-25T02:02:09.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3113e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3114073b10 0x7f311419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3113400640 1 -- 172.21.15.102:0/314482189 learned_addr learned my addr 172.21.15.102:0/314482189 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3118a00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f311410dff0 0x7f311419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3113400640 1 -- 172.21.15.102:0/314482189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f311410dff0 msgr2=0x7f311419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3113400640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f311410dff0 0x7f311419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3113400640 1 -- 172.21.15.102:0/314482189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3114073b10 msgr2=0x7f311419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3113400640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3114073b10 0x7f311419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.073+0000 7f3113400640 1 -- 172.21.15.102:0/314482189 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311419f2b0 con 0x7f3114074430 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.074+0000 7f3113400640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3114074430 0x7f311419a800 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f310000e9c0 tx=0x7f310000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.074+0000 7f3110c00640 1 -- 172.21.15.102:0/314482189 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310000cde0 con 0x7f3114074430 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.074+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311419f5a0 con 0x7f3114074430 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.074+0000 7f3110c00640 1 -- 172.21.15.102:0/314482189 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31000035f0 con 0x7f3114074430 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.074+0000 7f3110c00640 1 -- 172.21.15.102:0/314482189 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100010650 con 0x7f3114074430 2024-04-25T02:02:09.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.074+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31141a6e30 con 0x7f3114074430 2024-04-25T02:02:09.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.076+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3114067cd0 con 0x7f3114074430 2024-04-25T02:02:09.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.079+0000 7f3110c00640 1 -- 172.21.15.102:0/314482189 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f31000037a0 con 0x7f3114074430 2024-04-25T02:02:09.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.080+0000 7f3110c00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f30f80a2d80 0x7f30f80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.081+0000 7f3110c00640 1 -- 172.21.15.102:0/314482189 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f31000cf770 con 0x7f3114074430 2024-04-25T02:02:09.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.081+0000 7f3113e00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f30f80a2d80 0x7f30f80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.082+0000 7f3113e00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f30f80a2d80 0x7f30f80a5240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f310402a9c0 tx=0x7f3104005c90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:09.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.084+0000 7f3110c00640 1 -- 172.21.15.102:0/314482189 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f310009c310 con 0x7f3114074430 2024-04-25T02:02:09.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.401+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f311402d170 con 0x7f3114074430 2024-04-25T02:02:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.402+0000 7f3110c00640 1 -- 172.21.15.102:0/314482189 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11907 (secure 0 0 0) 0x7f310009bc10 con 0x7f3114074430 2024-04-25T02:02:09.407 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:02:09.407 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T02:01:47.813106+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":141,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24541,24587,24533]},"id":1}]} 2024-04-25T02:02:09.407 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 36 2024-04-25T02:02:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f30f80a2d80 msgr2=0x7f30f80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f30f80a2d80 0x7f30f80a5240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f310402a9c0 tx=0x7f3104005c90 comp rx=0 tx=0).stop 2024-04-25T02:02:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3114074430 msgr2=0x7f311419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3114074430 0x7f311419a800 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f310000e9c0 tx=0x7f310000ee80 comp rx=0 tx=0).stop 2024-04-25T02:02:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 shutdown_connections 2024-04-25T02:02:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f30f80a2d80 0x7f30f80a5240 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f311410dff0 0x7f311419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3114074430 0x7f311419a800 secure :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f310000e9c0 tx=0x7f310000ee80 comp rx=0 tx=0).stop 2024-04-25T02:02:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 --2- 172.21.15.102:0/314482189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3114073b10 0x7f311419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.405+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 >> 172.21.15.102:0/314482189 conn(0x7f3114078bd0 msgr2=0x7f31140fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:09.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.406+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 shutdown_connections 2024-04-25T02:02:09.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.406+0000 7f311aa00640 1 -- 172.21.15.102:0/314482189 wait complete. 2024-04-25T02:02:09.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T02:02:09.583 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:09 smithi102 ceph-mon[27796]: pgmap v210: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.4 MiB/s wr, 495 op/s 2024-04-25T02:02:09.642 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:09 smithi110 ceph-mon[32848]: pgmap v210: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.4 MiB/s wr, 495 op/s 2024-04-25T02:02:09.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:09 smithi193 ceph-mon[32981]: pgmap v210: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.4 MiB/s wr, 495 op/s 2024-04-25T02:02:09.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.821+0000 7f99dfa00640 1 -- 172.21.15.102:0/522351485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99d81020d0 msgr2=0x7f99d810e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.821+0000 7f99dfa00640 1 --2- 172.21.15.102:0/522351485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99d81020d0 0x7f99d810e5e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009860 tx=0x7f99d002a360 comp rx=0 tx=0).stop 2024-04-25T02:02:09.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.821+0000 7f99dfa00640 1 -- 172.21.15.102:0/522351485 shutdown_connections 2024-04-25T02:02:09.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.821+0000 7f99dfa00640 1 --2- 172.21.15.102:0/522351485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99d81020d0 0x7f99d810e5e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.821+0000 7f99dfa00640 1 --2- 172.21.15.102:0/522351485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99d8101710 0x7f99d8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.821+0000 7f99dfa00640 1 --2- 172.21.15.102:0/522351485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99d8107720 0x7f99d8107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.821+0000 7f99dfa00640 1 -- 172.21.15.102:0/522351485 >> 172.21.15.102:0/522351485 conn(0x7f99d8078b90 msgr2=0x7f99d8078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:09.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.822+0000 7f99dfa00640 1 -- 172.21.15.102:0/522351485 shutdown_connections 2024-04-25T02:02:09.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.830+0000 7f99dfa00640 1 -- 172.21.15.102:0/522351485 wait complete. 2024-04-25T02:02:09.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.832+0000 7f99dfa00640 1 Processor -- start 2024-04-25T02:02:09.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.832+0000 7f99dfa00640 1 -- start start 2024-04-25T02:02:09.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.833+0000 7f99dfa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99d8101710 0x7f99d8072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.834+0000 7f99dfa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99d81020d0 0x7f99d806c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.834+0000 7f99dd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99d8101710 0x7f99d8072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.834+0000 7f99dd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99d8101710 0x7f99d8072f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40698/0 (socket says 172.21.15.102:40698) 2024-04-25T02:02:09.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.835+0000 7f99dd000640 1 -- 172.21.15.102:0/2935339022 learned_addr learned my addr 172.21.15.102:0/2935339022 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.835+0000 7f99dfa00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99d8107720 0x7f99d806c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.835+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8111b30 con 0x7f99d81020d0 2024-04-25T02:02:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.835+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99d81119b0 con 0x7f99d8107720 2024-04-25T02:02:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.835+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99d8111cb0 con 0x7f99d8101710 2024-04-25T02:02:09.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.836+0000 7f99dda00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99d8107720 0x7f99d806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.837+0000 7f99dd000640 1 -- 172.21.15.102:0/2935339022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99d8107720 msgr2=0x7f99d806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.837+0000 7f99dd000640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99d8107720 0x7f99d806c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.837+0000 7f99dd000640 1 -- 172.21.15.102:0/2935339022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99d81020d0 msgr2=0x7f99d806c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.837+0000 7f99dd000640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99d81020d0 0x7f99d806c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.837+0000 7f99dd000640 1 -- 172.21.15.102:0/2935339022 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d806cb10 con 0x7f99d8101710 2024-04-25T02:02:09.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.838+0000 7f99dd000640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99d8101710 0x7f99d8072f40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f99c000d980 tx=0x7f99c000de40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:09.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.840+0000 7f99d5600640 1 -- 172.21.15.102:0/2935339022 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0017070 con 0x7f99d8101710 2024-04-25T02:02:09.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.841+0000 7f99d5600640 1 -- 172.21.15.102:0/2935339022 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c0009c30 con 0x7f99d8101710 2024-04-25T02:02:09.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.841+0000 7f99d5600640 1 -- 172.21.15.102:0/2935339022 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c000f620 con 0x7f99d8101710 2024-04-25T02:02:09.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.841+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d806ce00 con 0x7f99d8101710 2024-04-25T02:02:09.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.841+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d8070d90 con 0x7f99d8101710 2024-04-25T02:02:09.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.846+0000 7f99e2784740 1 Processor -- start 2024-04-25T02:02:09.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.847+0000 7f99e2784740 1 -- start start 2024-04-25T02:02:09.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.847+0000 7f99e2784740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965fe2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.851+0000 7f99e2784740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fddee0 0x564965fde360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.851+0000 7f99e2784740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fde8a0 0x564965fea5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.851+0000 7f99e2784740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x564965f394e0 con 0x564965fde8a0 2024-04-25T02:02:09.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.851+0000 7f99e2784740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564965ee1a70 con 0x564965fe2680 2024-04-25T02:02:09.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.851+0000 7f99e2784740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564965f354a0 con 0x564965fddee0 2024-04-25T02:02:09.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.852+0000 7f99dda00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965fe2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.852+0000 7f99dda00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965fe2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41128/0 (socket says 172.21.15.102:41128) 2024-04-25T02:02:09.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.852+0000 7f99dda00640 1 -- 172.21.15.102:0/1391477491 learned_addr learned my addr 172.21.15.102:0/1391477491 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.854+0000 7f99dda00640 1 -- 172.21.15.102:0/1391477491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fddee0 msgr2=0x564965fde360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:02:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.854+0000 7f99dda00640 1 --2- 172.21.15.102:0/1391477491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fddee0 0x564965fde360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.854+0000 7f99dda00640 1 -- 172.21.15.102:0/1391477491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fde8a0 msgr2=0x564965fea5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.854+0000 7f99dda00640 1 --2- 172.21.15.102:0/1391477491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fde8a0 0x564965fea5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.854+0000 7f99dda00640 1 -- 172.21.15.102:0/1391477491 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564965c4d3f0 con 0x564965fe2680 2024-04-25T02:02:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.854+0000 7f99d5600640 1 -- 172.21.15.102:0/2935339022 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f99c0010460 con 0x7f99d8101710 2024-04-25T02:02:09.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.854+0000 7f99dda00640 1 --2- 172.21.15.102:0/1391477491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965fe2a60 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009860 tx=0x7f99d00304e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aee02b1542e5b8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:09.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.855+0000 7f99ce000640 1 -- 172.21.15.102:0/1391477491 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d0037040 con 0x564965fe2680 2024-04-25T02:02:09.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.855+0000 7f99ce000640 1 -- 172.21.15.102:0/1391477491 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99d0030590 con 0x564965fe2680 2024-04-25T02:02:09.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.855+0000 7f99ce000640 1 -- 172.21.15.102:0/1391477491 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d00053b0 con 0x564965fe2680 2024-04-25T02:02:09.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.864+0000 7f99e2784740 1 -- 172.21.15.102:0/1391477491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 msgr2=0x564965fe2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.864+0000 7f99e2784740 1 --2- 172.21.15.102:0/1391477491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965fe2a60 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009860 tx=0x7f99d00304e0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.864+0000 7f99e2784740 1 -- 172.21.15.102:0/1391477491 shutdown_connections 2024-04-25T02:02:09.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.864+0000 7f99e2784740 1 --2- 172.21.15.102:0/1391477491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fde8a0 0x564965fea5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.864+0000 7f99e2784740 1 --2- 172.21.15.102:0/1391477491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fddee0 0x564965fde360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.864+0000 7f99e2784740 1 --2- 172.21.15.102:0/1391477491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965fe2a60 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.864+0000 7f99e2784740 1 -- 172.21.15.102:0/1391477491 >> 172.21.15.102:0/1391477491 conn(0x564965fd0390 msgr2=0x564965fd0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:09.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.864+0000 7f99d5600640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99bc0a2e40 0x7f99bc0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.864+0000 7f99e2784740 1 -- 172.21.15.102:0/1391477491 shutdown_connections 2024-04-25T02:02:09.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.865+0000 7f99e2784740 1 -- 172.21.15.102:0/1391477491 wait complete. 2024-04-25T02:02:09.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.865+0000 7f99d5600640 1 -- 172.21.15.102:0/2935339022 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f99c00cf130 con 0x7f99d8101710 2024-04-25T02:02:09.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.866+0000 7f99e2784740 1 -- start start 2024-04-25T02:02:09.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.866+0000 7f99e2784740 1 Processor -- start 2024-04-25T02:02:09.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.867+0000 7f99e2784740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 0x564965fe7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.868+0000 7f99e2784740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fde8a0 0x564965fe7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.868+0000 7f99e2784740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fe2680 0x564965fe8500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.869+0000 7f99d7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fde8a0 0x564965fe7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.869+0000 7f99d7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fde8a0 0x564965fe7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41130/0 (socket says 172.21.15.102:41130) 2024-04-25T02:02:09.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.869+0000 7f99d7e00640 1 -- 172.21.15.102:0/817245344 learned_addr learned my addr 172.21.15.102:0/817245344 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.870+0000 7f99dda00640 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fe2680 0x564965fe8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.870+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x564965feab70 con 0x564965fddee0 2024-04-25T02:02:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.871+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x564965ef4ec0 con 0x564965fde8a0 2024-04-25T02:02:09.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.871+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564965feacf0 con 0x564965fe2680 2024-04-25T02:02:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.872+0000 7f99d7e00640 1 -- 172.21.15.102:0/817245344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fe2680 msgr2=0x564965fe8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:09.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.872+0000 7f99dd000640 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 0x564965fe7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.872+0000 7f99d7e00640 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fe2680 0x564965fe8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.872+0000 7f99d7e00640 1 -- 172.21.15.102:0/817245344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 msgr2=0x564965fe7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.872+0000 7f99d7e00640 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 0x564965fe7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.872+0000 7f99d7e00640 1 -- 172.21.15.102:0/817245344 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564965d177d0 con 0x564965fde8a0 2024-04-25T02:02:09.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.872+0000 7f99dd000640 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 0x564965fe7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:02:09.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.873+0000 7f99dda00640 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fe2680 0x564965fe8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:09.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.873+0000 7f99e2784740 10 client.0 _ll_register_callbacks cb 0x564965f313f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:09.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.874+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99d0030590 con 0x564965fde8a0 2024-04-25T02:02:09.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.874+0000 7f99d7e00640 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fde8a0 0x564965fe7fc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f99c800f360 tx=0x7f99c800f820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=958cf1c4da7816e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:09.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.874+0000 7f99d7e00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99bc0a2e40 0x7f99bc0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.874+0000 7f99c6a00640 10 client.27083 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:09.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.875+0000 7f99c6a00640 1 -- 172.21.15.102:0/817245344 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c8011db0 con 0x564965fde8a0 2024-04-25T02:02:09.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.875+0000 7f99c6a00640 1 -- 172.21.15.102:0/817245344 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f99c801c410 con 0x564965fde8a0 2024-04-25T02:02:09.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.875+0000 7f99c6a00640 1 -- 172.21.15.102:0/817245344 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f99c801c6b0 con 0x564965fde8a0 2024-04-25T02:02:09.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.875+0000 7f99e2784740 10 client.27083 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:09.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.876+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=36}) -- 0x564965ff6d10 con 0x564965fde8a0 2024-04-25T02:02:09.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.876+0000 7f99c6a00640 1 -- 172.21.15.102:0/817245344 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f99c800d8b0 con 0x564965fde8a0 2024-04-25T02:02:09.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.877+0000 7f99d7e00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99bc0a2e40 0x7f99bc0a5300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99c801d000 tx=0x7f99c8023000 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.877+0000 7f99e2784740 10 client.27083 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.877+0000 7f99e2784740 10 client.27083 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T02:02:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.877+0000 7f99e2784740 20 client.27083 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.877+0000 7f99e2784740 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564965ff8130 0x564965ffa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.877+0000 7f99e2784740 4 client.27083 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:09.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.877+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564965f46a20 con 0x564965ff8130 2024-04-25T02:02:09.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.880+0000 7f99dd000640 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564965ff8130 0x564965ffa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.881+0000 7f99c6a00640 10 client.27083 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T02:02:09.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.881+0000 7f99dd000640 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564965ff8130 0x564965ffa5f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=631fb4dc60b88983 server_cookie=654a1039adb7cd43 in_seq=0 out_seq=0 2024-04-25T02:02:09.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.883+0000 7f99c6a00640 1 -- 172.21.15.102:0/817245344 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564965f46a20 con 0x564965ff8130 2024-04-25T02:02:09.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.883+0000 7f99c6a00640 10 client.27083 handle_command_reply: tid=0 2024-04-25T02:02:09.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.883+0000 7f99e2784740 1 client.27083 shutdown 2024-04-25T02:02:09.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fde8a0 msgr2=0x564965fe7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fde8a0 0x564965fe7fc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f99c800f360 tx=0x7f99c800f820 comp rx=0 tx=0).stop 2024-04-25T02:02:09.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 shutdown_connections 2024-04-25T02:02:09.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fe2680 0x564965fe8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x564965ff8130 0x564965ffa5f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99c6a00640 0 client.27083 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T02:02:09.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fde8a0 0x564965fe7fc0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 --2- 172.21.15.102:0/817245344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 0x564965fe7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 >> 172.21.15.102:0/817245344 conn(0x564965fd0390 msgr2=0x564965fdd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:09.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 shutdown_connections 2024-04-25T02:02:09.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.884+0000 7f99e2784740 1 -- 172.21.15.102:0/817245344 wait complete. 2024-04-25T02:02:09.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.885+0000 7f99e2784740 20 client.27083 trim_cache size 0 max 16384 2024-04-25T02:02:09.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.921+0000 7f99e2784740 1 Processor -- start 2024-04-25T02:02:09.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.921+0000 7f99e2784740 1 -- start start 2024-04-25T02:02:09.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.921+0000 7f99e2784740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fddee0 0x564965fe3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.921+0000 7f99e2784740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fde8a0 0x564965ff3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.921+0000 7f99e2784740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fe2680 0x564965fe5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.922+0000 7f99dd000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fddee0 0x564965fe3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.922+0000 7f99dd000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fddee0 0x564965fe3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41146/0 (socket says 172.21.15.102:41146) 2024-04-25T02:02:09.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.922+0000 7f99dd000640 1 -- 172.21.15.102:0/1953045204 learned_addr learned my addr 172.21.15.102:0/1953045204 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:09.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.922+0000 7f99dda00640 1 --2- 172.21.15.102:0/1953045204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fe2680 0x564965fe5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.923+0000 7f99e2784740 1 -- 172.21.15.102:0/1953045204 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99d81b6600 con 0x564965fe2680 2024-04-25T02:02:09.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.922+0000 7f99d7e00640 1 --2- 172.21.15.102:0/1953045204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fde8a0 0x564965ff3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.923+0000 7f99e2784740 1 -- 172.21.15.102:0/1953045204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99d81b3e10 con 0x564965fddee0 2024-04-25T02:02:09.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.923+0000 7f99e2784740 1 -- 172.21.15.102:0/1953045204 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99d81b1ca0 con 0x564965fde8a0 2024-04-25T02:02:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.923+0000 7f99dd000640 1 -- 172.21.15.102:0/1953045204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fde8a0 msgr2=0x564965ff3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.924+0000 7f99dd000640 1 --2- 172.21.15.102:0/1953045204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fde8a0 0x564965ff3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.924+0000 7f99dd000640 1 -- 172.21.15.102:0/1953045204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fe2680 msgr2=0x564965fe5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.924+0000 7f99dd000640 1 --2- 172.21.15.102:0/1953045204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fe2680 0x564965fe5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.924+0000 7f99dd000640 1 -- 172.21.15.102:0/1953045204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564965fd1880 con 0x564965fddee0 2024-04-25T02:02:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.926+0000 7f99dd000640 1 --2- 172.21.15.102:0/1953045204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fddee0 0x564965fe3a30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f99c009b2d0 tx=0x7f99c00ce040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9b6054793548cf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:09.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.926+0000 7f99cd600640 1 -- 172.21.15.102:0/1953045204 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0020040 con 0x564965fddee0 2024-04-25T02:02:09.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.926+0000 7f99cd600640 1 -- 172.21.15.102:0/1953045204 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99c0015070 con 0x564965fddee0 2024-04-25T02:02:09.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.926+0000 7f99cd600640 1 -- 172.21.15.102:0/1953045204 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0097c40 con 0x564965fddee0 2024-04-25T02:02:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.927+0000 7f99e2784740 1 -- 172.21.15.102:0/1953045204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fddee0 msgr2=0x564965fe3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.927+0000 7f99e2784740 1 --2- 172.21.15.102:0/1953045204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fddee0 0x564965fe3a30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f99c009b2d0 tx=0x7f99c00ce040 comp rx=0 tx=0).stop 2024-04-25T02:02:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.928+0000 7f99e2784740 1 -- 172.21.15.102:0/1953045204 shutdown_connections 2024-04-25T02:02:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.928+0000 7f99e2784740 1 --2- 172.21.15.102:0/1953045204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fe2680 0x564965fe5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.928+0000 7f99e2784740 1 --2- 172.21.15.102:0/1953045204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fde8a0 0x564965ff3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.928+0000 7f99e2784740 1 --2- 172.21.15.102:0/1953045204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fddee0 0x564965fe3a30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.928+0000 7f99e2784740 1 -- 172.21.15.102:0/1953045204 >> 172.21.15.102:0/1953045204 conn(0x564965fd0390 msgr2=0x564965fe3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.928+0000 7f99e2784740 1 -- 172.21.15.102:0/1953045204 shutdown_connections 2024-04-25T02:02:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.928+0000 7f99e2784740 1 -- 172.21.15.102:0/1953045204 wait complete. 2024-04-25T02:02:09.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.929+0000 7f99e2784740 1 -- start start 2024-04-25T02:02:09.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.929+0000 7f99e2784740 1 Processor -- start 2024-04-25T02:02:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.932+0000 7f99e2784740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 0x564966000460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.932+0000 7f99e2784740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fde8a0 0x564965feffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.932+0000 7f99e2784740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965ffef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.933+0000 7f99e2784740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99d81b22f0 con 0x564965fddee0 2024-04-25T02:02:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.933+0000 7f99dda00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965ffef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.933+0000 7f99dda00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965ffef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41154/0 (socket says 172.21.15.102:41154) 2024-04-25T02:02:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.933+0000 7f99dda00640 1 -- 172.21.15.102:0/2927167171 learned_addr learned my addr 172.21.15.102:0/2927167171 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:09.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.933+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99d81b3740 con 0x564965fe2680 2024-04-25T02:02:09.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.934+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99d81b29a0 con 0x564965fde8a0 2024-04-25T02:02:09.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.934+0000 7f99e2784740 10 client.0 _ll_register_callbacks cb 0x7f99c8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.934+0000 7f99dda00640 1 -- 172.21.15.102:0/2927167171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fde8a0 msgr2=0x564965feffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.934+0000 7f99dda00640 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fde8a0 0x564965feffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.934+0000 7f99dda00640 1 -- 172.21.15.102:0/2927167171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 msgr2=0x564966000460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.934+0000 7f99dda00640 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 0x564966000460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.934+0000 7f99dda00640 1 -- 172.21.15.102:0/2927167171 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564965fff4b0 con 0x564965fe2680 2024-04-25T02:02:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.935+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99c0015070 con 0x564965fe2680 2024-04-25T02:02:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.936+0000 7f99c6a00640 10 client.27089 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.936+0000 7f99dda00640 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965ffef70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f99d0005b60 tx=0x7f99d0004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d40d99e841d5fc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:09.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.938+0000 7f99c6a00640 1 -- 172.21.15.102:0/2927167171 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d0054050 con 0x564965fe2680 2024-04-25T02:02:09.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.939+0000 7f99c6a00640 1 -- 172.21.15.102:0/2927167171 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f99d00037f0 con 0x564965fe2680 2024-04-25T02:02:09.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.939+0000 7f99c6a00640 1 -- 172.21.15.102:0/2927167171 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f99d0037040 con 0x564965fe2680 2024-04-25T02:02:09.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.940+0000 7f99e2784740 10 client.27089 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:09.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.940+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=36}) -- 0x56496605ec90 con 0x564965fe2680 2024-04-25T02:02:09.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.941+0000 7f99c6a00640 1 -- 172.21.15.102:0/2927167171 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f99d0057020 con 0x564965fe2680 2024-04-25T02:02:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.941+0000 7f99e2784740 10 client.27089 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.941+0000 7f99e2784740 10 client.27089 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-04-25T02:02:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.941+0000 7f99e2784740 20 client.27089 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.941+0000 7f99e2784740 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x56496605f340 0x564966061800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.941+0000 7f99e2784740 4 client.27089 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:02:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.941+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 --> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564965f40d70 con 0x56496605f340 2024-04-25T02:02:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.942+0000 7f99dd000640 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x56496605f340 0x564966061800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:09.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.943+0000 7f99dd000640 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x56496605f340 0x564966061800 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=350a500b1bf17de2 server_cookie=61cf7b39720bc4f in_seq=0 out_seq=0 2024-04-25T02:02:09.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.943+0000 7f99c6a00640 10 client.27089 ms_handle_connect on v2:172.21.15.110:6836/1533256041 2024-04-25T02:02:09.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.943+0000 7f99c6a00640 1 -- 172.21.15.102:0/2927167171 <== mds.2 v2:172.21.15.110:6836/1533256041 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564965f40d70 con 0x56496605f340 2024-04-25T02:02:09.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.943+0000 7f99c6a00640 10 client.27089 handle_command_reply: tid=0 2024-04-25T02:02:09.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.943+0000 7f99e2784740 1 client.27089 shutdown 2024-04-25T02:02:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.945+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 msgr2=0x564965ffef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.945+0000 7f99e2784740 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965ffef70 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f99d0005b60 tx=0x7f99d0004610 comp rx=0 tx=0).stop 2024-04-25T02:02:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.945+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 shutdown_connections 2024-04-25T02:02:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.945+0000 7f99e2784740 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.110:6836/1533256041,v1:172.21.15.110:6837/1533256041] conn(0x56496605f340 0x564966061800 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.945+0000 7f99e2784740 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x564965fe2680 0x564965ffef70 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.945+0000 7f99e2784740 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564965fde8a0 0x564965feffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.945+0000 7f99e2784740 1 --2- 172.21.15.102:0/2927167171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564965fddee0 0x564966000460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.945+0000 7f99c6a00640 0 client.27089 ms_handle_reset on v2:172.21.15.110:6836/1533256041 2024-04-25T02:02:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.945+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 >> 172.21.15.102:0/2927167171 conn(0x564965fd0390 msgr2=0x564965fe6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.946+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 shutdown_connections 2024-04-25T02:02:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.946+0000 7f99e2784740 1 -- 172.21.15.102:0/2927167171 wait complete. 2024-04-25T02:02:09.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.946+0000 7f99e2784740 20 client.27089 trim_cache size 0 max 16384 2024-04-25T02:02:09.953 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:02:09.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.952+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99bc0a2e40 msgr2=0x7f99bc0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.952+0000 7f99dfa00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99bc0a2e40 0x7f99bc0a5300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f99c801d000 tx=0x7f99c8023000 comp rx=0 tx=0).stop 2024-04-25T02:02:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.952+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99d8101710 msgr2=0x7f99d8072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.952+0000 7f99dfa00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99d8101710 0x7f99d8072f40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f99c000d980 tx=0x7f99c000de40 comp rx=0 tx=0).stop 2024-04-25T02:02:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.953+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 shutdown_connections 2024-04-25T02:02:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.953+0000 7f99dfa00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99bc0a2e40 0x7f99bc0a5300 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.953+0000 7f99dfa00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99d8107720 0x7f99d806c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.953+0000 7f99dfa00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99d81020d0 0x7f99d806c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.953+0000 7f99dfa00640 1 --2- 172.21.15.102:0/2935339022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99d8101710 0x7f99d8072f40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.953+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 >> 172.21.15.102:0/2935339022 conn(0x7f99d8078b90 msgr2=0x7f99d80ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.953+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 shutdown_connections 2024-04-25T02:02:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:09.953+0000 7f99dfa00640 1 -- 172.21.15.102:0/2935339022 wait complete. 2024-04-25T02:02:10.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T02:02:10.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.380+0000 7f371ae00640 1 -- 172.21.15.102:0/847934721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37141015e0 msgr2=0x7f3714101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.380+0000 7f371ae00640 1 --2- 172.21.15.102:0/847934721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37141015e0 0x7f3714101a60 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3700009930 tx=0x7f370002a440 comp rx=0 tx=0).stop 2024-04-25T02:02:10.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.380+0000 7f371ae00640 1 -- 172.21.15.102:0/847934721 shutdown_connections 2024-04-25T02:02:10.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.380+0000 7f371ae00640 1 --2- 172.21.15.102:0/847934721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3714101fa0 0x7f371410e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.380+0000 7f371ae00640 1 --2- 172.21.15.102:0/847934721 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37141015e0 0x7f3714101a60 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.380+0000 7f371ae00640 1 --2- 172.21.15.102:0/847934721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3714107680 0x7f3714107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.380+0000 7f371ae00640 1 -- 172.21.15.102:0/847934721 >> 172.21.15.102:0/847934721 conn(0x7f37140fd110 msgr2=0x7f37140ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.382+0000 7f371ae00640 1 -- 172.21.15.102:0/847934721 shutdown_connections 2024-04-25T02:02:10.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.382+0000 7f371ae00640 1 -- 172.21.15.102:0/847934721 wait complete. 2024-04-25T02:02:10.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.383+0000 7f371ae00640 1 Processor -- start 2024-04-25T02:02:10.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.383+0000 7f371ae00640 1 -- start start 2024-04-25T02:02:10.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.384+0000 7f371ae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37141015e0 0x7f3714073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.384+0000 7f371ae00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3714101fa0 0x7f3714073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.385+0000 7f371ae00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3714107680 0x7f37140744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.385+0000 7f3719000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3714101fa0 0x7f3714073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.385+0000 7f3719000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3714101fa0 0x7f3714073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40736/0 (socket says 172.21.15.102:40736) 2024-04-25T02:02:10.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.385+0000 7f3719000640 1 -- 172.21.15.102:0/2804517621 learned_addr learned my addr 172.21.15.102:0/2804517621 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.385+0000 7f371ae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3714111a50 con 0x7f37141015e0 2024-04-25T02:02:10.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.385+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f37141118d0 con 0x7f3714107680 2024-04-25T02:02:10.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.385+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3714111bd0 con 0x7f3714101fa0 2024-04-25T02:02:10.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.385+0000 7f371a400640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3714107680 0x7f37140744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.387+0000 7f3719000640 1 -- 172.21.15.102:0/2804517621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3714107680 msgr2=0x7f37140744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.387+0000 7f3719000640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3714107680 0x7f37140744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.387+0000 7f3719000640 1 -- 172.21.15.102:0/2804517621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37141015e0 msgr2=0x7f3714073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.387+0000 7f3719000640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37141015e0 0x7f3714073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.387+0000 7f3719000640 1 -- 172.21.15.102:0/2804517621 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37141a6210 con 0x7f3714101fa0 2024-04-25T02:02:10.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.387+0000 7f371a400640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3714107680 0x7f37140744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:10.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.388+0000 7f3719000640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3714101fa0 0x7f3714073f80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f370002aa30 tx=0x7f3700030940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.388+0000 7f3712000640 1 -- 172.21.15.102:0/2804517621 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370003f070 con 0x7f3714101fa0 2024-04-25T02:02:10.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.388+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37141a64a0 con 0x7f3714101fa0 2024-04-25T02:02:10.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.389+0000 7f3712000640 1 -- 172.21.15.102:0/2804517621 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3700032e40 con 0x7f3714101fa0 2024-04-25T02:02:10.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.389+0000 7f3712000640 1 -- 172.21.15.102:0/2804517621 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370003f070 con 0x7f3714101fa0 2024-04-25T02:02:10.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.391+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37141a69e0 con 0x7f3714101fa0 2024-04-25T02:02:10.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.396+0000 7f3712000640 1 -- 172.21.15.102:0/2804517621 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3700042070 con 0x7f3714101fa0 2024-04-25T02:02:10.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.402+0000 7f371f38a740 1 Processor -- start 2024-04-25T02:02:10.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.402+0000 7f371f38a740 1 -- start start 2024-04-25T02:02:10.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.403+0000 7f371f38a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bacfb0 0x563cb7bad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.403+0000 7f371f38a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.403+0000 7f371f38a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bb7fa0 0x563cb7bba390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.403+0000 7f371f38a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x563cb7b094e0 con 0x563cb7bacfb0 2024-04-25T02:02:10.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.403+0000 7f371f38a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563cb7ab1a70 con 0x563cb7bb7fa0 2024-04-25T02:02:10.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.403+0000 7f371f38a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563cb7b054a0 con 0x563cb7bad8d0 2024-04-25T02:02:10.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.404+0000 7f3719a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.404+0000 7f3712000640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f36fc0a2f80 0x7f36fc0a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.404+0000 7f3719a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40742/0 (socket says 172.21.15.102:40742) 2024-04-25T02:02:10.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.404+0000 7f3719a00640 1 -- 172.21.15.102:0/1894141464 learned_addr learned my addr 172.21.15.102:0/1894141464 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.404+0000 7f3712000640 1 -- 172.21.15.102:0/2804517621 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f37000efa00 con 0x7f3714101fa0 2024-04-25T02:02:10.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.405+0000 7f3719000640 1 --2- 172.21.15.102:0/1894141464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bb7fa0 0x563cb7bba390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.405+0000 7f3719a00640 1 -- 172.21.15.102:0/1894141464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bb7fa0 msgr2=0x563cb7bba390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.405+0000 7f3719a00640 1 --2- 172.21.15.102:0/1894141464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bb7fa0 0x563cb7bba390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.405+0000 7f3719a00640 1 -- 172.21.15.102:0/1894141464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bacfb0 msgr2=0x563cb7bad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.405+0000 7f3719a00640 1 --2- 172.21.15.102:0/1894141464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bacfb0 0x563cb7bad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.405+0000 7f3719a00640 1 -- 172.21.15.102:0/1894141464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cb781d3f0 con 0x563cb7bad8d0 2024-04-25T02:02:10.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.406+0000 7f3719a00640 1 --2- 172.21.15.102:0/1894141464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb7a60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f370800bb20 tx=0x7f370800d2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2f51a2b4740e9e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.407+0000 7f3706a00640 1 -- 172.21.15.102:0/1894141464 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3708016020 con 0x563cb7bad8d0 2024-04-25T02:02:10.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.407+0000 7f3706a00640 1 -- 172.21.15.102:0/1894141464 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f370800dd10 con 0x563cb7bad8d0 2024-04-25T02:02:10.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.408+0000 7f371f38a740 1 -- 172.21.15.102:0/1894141464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 msgr2=0x563cb7bb7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.408+0000 7f371f38a740 1 --2- 172.21.15.102:0/1894141464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb7a60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f370800bb20 tx=0x7f370800d2e0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.408+0000 7f371f38a740 1 -- 172.21.15.102:0/1894141464 shutdown_connections 2024-04-25T02:02:10.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.408+0000 7f371f38a740 1 --2- 172.21.15.102:0/1894141464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bb7fa0 0x563cb7bba390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.408+0000 7f371f38a740 1 --2- 172.21.15.102:0/1894141464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb7a60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.408+0000 7f371f38a740 1 --2- 172.21.15.102:0/1894141464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bacfb0 0x563cb7bad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.408+0000 7f371f38a740 1 -- 172.21.15.102:0/1894141464 >> 172.21.15.102:0/1894141464 conn(0x563cb7ba92d0 msgr2=0x563cb7ba96b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.409+0000 7f371f38a740 1 -- 172.21.15.102:0/1894141464 shutdown_connections 2024-04-25T02:02:10.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.409+0000 7f371f38a740 1 -- 172.21.15.102:0/1894141464 wait complete. 2024-04-25T02:02:10.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.410+0000 7f371f38a740 1 -- start start 2024-04-25T02:02:10.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.411+0000 7f371f38a740 1 Processor -- start 2024-04-25T02:02:10.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.413+0000 7f371a400640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f36fc0a2f80 0x7f36fc0a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.414+0000 7f371f38a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 0x563cb7bb4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.415+0000 7f371a400640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f36fc0a2f80 0x7f36fc0a5440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f370c00b7b0 tx=0x7f370c00d040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.415+0000 7f3719a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 0x563cb7bb4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.415+0000 7f3719a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 0x563cb7bb4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40752/0 (socket says 172.21.15.102:40752) 2024-04-25T02:02:10.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.415+0000 7f3719a00640 1 -- 172.21.15.102:0/3457915588 learned_addr learned my addr 172.21.15.102:0/3457915588 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.415+0000 7f371f38a740 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.416+0000 7f371f38a740 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 0x563cb7bb5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.416+0000 7f3719000640 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.417+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x563cb7bcbe40 con 0x563cb7bb7fa0 2024-04-25T02:02:10.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.417+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x563cb7ac4ec0 con 0x563cb7bad8d0 2024-04-25T02:02:10.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.417+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563cb7bcbfc0 con 0x563cb7bacfb0 2024-04-25T02:02:10.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.418+0000 7f371f38a740 10 client.0 _ll_register_callbacks cb 0x563cb7b013f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:10.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.418+0000 7f371a400640 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 0x563cb7bb5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.419+0000 7f3719000640 1 -- 172.21.15.102:0/3457915588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 msgr2=0x563cb7bb4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.419+0000 7f3719000640 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 0x563cb7bb4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.419+0000 7f3719000640 1 -- 172.21.15.102:0/3457915588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 msgr2=0x563cb7bb5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.419+0000 7f3719000640 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 0x563cb7bb5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.419+0000 7f3719000640 1 -- 172.21.15.102:0/3457915588 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cb78e77d0 con 0x563cb7bad8d0 2024-04-25T02:02:10.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.420+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f370800dd10 con 0x563cb7bad8d0 2024-04-25T02:02:10.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.420+0000 7f3719000640 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb5340 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f37000ed0d0 tx=0x7f370003d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e73e5562bdb9d54e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.420+0000 7f36fb400640 10 client.27101 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:10.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.421+0000 7f36fb400640 1 -- 172.21.15.102:0/3457915588 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700005d50 con 0x563cb7bad8d0 2024-04-25T02:02:10.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.421+0000 7f36fb400640 1 -- 172.21.15.102:0/3457915588 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f37000ed350 con 0x563cb7bad8d0 2024-04-25T02:02:10.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.422+0000 7f36fb400640 1 -- 172.21.15.102:0/3457915588 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f37000bd2e0 con 0x563cb7bad8d0 2024-04-25T02:02:10.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.422+0000 7f371f38a740 10 client.27101 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:10.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.422+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=36}) -- 0x563cb7bd7ee0 con 0x563cb7bad8d0 2024-04-25T02:02:10.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.423+0000 7f36fb400640 1 -- 172.21.15.102:0/3457915588 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f37000b9250 con 0x563cb7bad8d0 2024-04-25T02:02:10.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.423+0000 7f371f38a740 10 client.27101 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:10.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.423+0000 7f371f38a740 10 client.27101 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T02:02:10.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.423+0000 7f371f38a740 20 client.27101 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:10.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.423+0000 7f371f38a740 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x563cb7bd9280 0x563cb7bdb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.423+0000 7f371f38a740 4 client.27101 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:10.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.423+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cb7b16a20 con 0x563cb7bd9280 2024-04-25T02:02:10.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.423+0000 7f3719a00640 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x563cb7bd9280 0x563cb7bdb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.425+0000 7f3719a00640 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x563cb7bd9280 0x563cb7bdb710 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f9963ed807bc9fe server_cookie=ed80f77c25efc399 in_seq=0 out_seq=0 2024-04-25T02:02:10.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.425+0000 7f36fb400640 10 client.27101 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T02:02:10.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.425+0000 7f36fb400640 1 -- 172.21.15.102:0/3457915588 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563cb7b16a20 con 0x563cb7bd9280 2024-04-25T02:02:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.426+0000 7f36fb400640 10 client.27101 handle_command_reply: tid=0 2024-04-25T02:02:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.427+0000 7f371f38a740 1 client.27101 shutdown 2024-04-25T02:02:10.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.427+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 msgr2=0x563cb7bb5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.428+0000 7f371f38a740 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb5340 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f37000ed0d0 tx=0x7f370003d000 comp rx=0 tx=0).stop 2024-04-25T02:02:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.428+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 shutdown_connections 2024-04-25T02:02:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.428+0000 7f371f38a740 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x563cb7bd9280 0x563cb7bdb710 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.429+0000 7f371f38a740 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 0x563cb7bb5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.429+0000 7f36fb400640 0 client.27101 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T02:02:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.429+0000 7f371f38a740 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb5340 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.429+0000 7f371f38a740 1 --2- 172.21.15.102:0/3457915588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 0x563cb7bb4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.429+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 >> 172.21.15.102:0/3457915588 conn(0x563cb7ba92d0 msgr2=0x563cb7baf0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.429+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 shutdown_connections 2024-04-25T02:02:10.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.430+0000 7f371f38a740 1 -- 172.21.15.102:0/3457915588 wait complete. 2024-04-25T02:02:10.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.430+0000 7f371f38a740 20 client.27101 trim_cache size 0 max 16384 2024-04-25T02:02:10.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.477+0000 7f371f38a740 1 Processor -- start 2024-04-25T02:02:10.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.477+0000 7f371f38a740 1 -- start start 2024-04-25T02:02:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.478+0000 7f371f38a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bacfb0 0x563cb7bd4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.478+0000 7f371f38a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.478+0000 7f371f38a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bb7fa0 0x563cb7b9f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.478+0000 7f371f38a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3714071e10 con 0x563cb7bacfb0 2024-04-25T02:02:10.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.478+0000 7f371f38a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f371406f620 con 0x563cb7bb7fa0 2024-04-25T02:02:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.478+0000 7f371f38a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f371406d4d0 con 0x563cb7bad8d0 2024-04-25T02:02:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.479+0000 7f3719000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.479+0000 7f3719000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40754/0 (socket says 172.21.15.102:40754) 2024-04-25T02:02:10.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.479+0000 7f3719000640 1 -- 172.21.15.102:0/3612731573 learned_addr learned my addr 172.21.15.102:0/3612731573 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.480+0000 7f3719a00640 1 --2- 172.21.15.102:0/3612731573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bacfb0 0x563cb7bd4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.480+0000 7f3719000640 1 -- 172.21.15.102:0/3612731573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bb7fa0 msgr2=0x563cb7b9f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.480+0000 7f3719000640 1 --2- 172.21.15.102:0/3612731573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bb7fa0 0x563cb7b9f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.480+0000 7f3719000640 1 -- 172.21.15.102:0/3612731573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bacfb0 msgr2=0x563cb7bd4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.480+0000 7f3719000640 1 --2- 172.21.15.102:0/3612731573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bacfb0 0x563cb7bd4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.480+0000 7f3719000640 1 -- 172.21.15.102:0/3612731573 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cb7bad700 con 0x563cb7bad8d0 2024-04-25T02:02:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.480+0000 7f3719000640 1 --2- 172.21.15.102:0/3612731573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb3f60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3700005ab0 tx=0x7f3700045000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d07f4e13e9c43423 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.481+0000 7f3706000640 1 -- 172.21.15.102:0/3612731573 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700005d50 con 0x563cb7bad8d0 2024-04-25T02:02:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.481+0000 7f3706000640 1 -- 172.21.15.102:0/3612731573 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f370800dd10 con 0x563cb7bad8d0 2024-04-25T02:02:10.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.481+0000 7f3706000640 1 -- 172.21.15.102:0/3612731573 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37000bc9b0 con 0x563cb7bad8d0 2024-04-25T02:02:10.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.482+0000 7f371f38a740 1 -- 172.21.15.102:0/3612731573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 msgr2=0x563cb7bb3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.482+0000 7f371f38a740 1 --2- 172.21.15.102:0/3612731573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb3f60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3700005ab0 tx=0x7f3700045000 comp rx=0 tx=0).stop 2024-04-25T02:02:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.483+0000 7f371f38a740 1 -- 172.21.15.102:0/3612731573 shutdown_connections 2024-04-25T02:02:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.483+0000 7f371f38a740 1 --2- 172.21.15.102:0/3612731573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bb7fa0 0x563cb7b9f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.483+0000 7f371f38a740 1 --2- 172.21.15.102:0/3612731573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bad8d0 0x563cb7bb3f60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.483+0000 7f371f38a740 1 --2- 172.21.15.102:0/3612731573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bacfb0 0x563cb7bd4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.483+0000 7f371f38a740 1 -- 172.21.15.102:0/3612731573 >> 172.21.15.102:0/3612731573 conn(0x563cb7ba92d0 msgr2=0x563cb7b9f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.483+0000 7f371f38a740 1 -- 172.21.15.102:0/3612731573 shutdown_connections 2024-04-25T02:02:10.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.483+0000 7f371f38a740 1 -- 172.21.15.102:0/3612731573 wait complete. 2024-04-25T02:02:10.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.484+0000 7f371f38a740 1 -- start start 2024-04-25T02:02:10.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.485+0000 7f371f38a740 1 Processor -- start 2024-04-25T02:02:10.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.486+0000 7f371f38a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 0x563cb7bb1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.486+0000 7f371f38a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.486+0000 7f371f38a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 0x563cb7bb2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.487+0000 7f371f38a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f371406db20 con 0x563cb7bb7fa0 2024-04-25T02:02:10.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.487+0000 7f371f38a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f371406ef50 con 0x563cb7bad8d0 2024-04-25T02:02:10.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.487+0000 7f371f38a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f371406e1b0 con 0x563cb7bacfb0 2024-04-25T02:02:10.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.487+0000 7f371f38a740 10 client.0 _ll_register_callbacks cb 0x7f3700009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:10.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.487+0000 7f3719000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.487+0000 7f3719000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41198/0 (socket says 172.21.15.102:41198) 2024-04-25T02:02:10.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.487+0000 7f3719000640 1 -- 172.21.15.102:0/2429124091 learned_addr learned my addr 172.21.15.102:0/2429124091 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.488+0000 7f371a400640 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 0x563cb7bb2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.488+0000 7f3719000640 1 -- 172.21.15.102:0/2429124091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 msgr2=0x563cb7bb1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.488+0000 7f3719000640 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 0x563cb7bb1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.488+0000 7f3719000640 1 -- 172.21.15.102:0/2429124091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 msgr2=0x563cb7bb2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.488+0000 7f3719000640 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 0x563cb7bb2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.488+0000 7f3719000640 1 -- 172.21.15.102:0/2429124091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cb7c3ea00 con 0x563cb7bad8d0 2024-04-25T02:02:10.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.489+0000 7f36fb400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:10.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.489+0000 7f3719000640 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb2210 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f37000bc7c0 tx=0x7f37000efc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca3350edb5424d0c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.493+0000 7f36fb400640 1 -- 172.21.15.102:0/2429124091 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37000bc9b0 con 0x563cb7bad8d0 2024-04-25T02:02:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.493+0000 7f36fb400640 1 -- 172.21.15.102:0/2429124091 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f37000dab50 con 0x563cb7bad8d0 2024-04-25T02:02:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.493+0000 7f371f38a740 1 -- 172.21.15.102:0/2429124091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f370800dd10 con 0x563cb7bad8d0 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.494+0000 7f36fb400640 1 -- 172.21.15.102:0/2429124091 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3700004250 con 0x563cb7bad8d0 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.494+0000 7f371f38a740 10 client.27104 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.494+0000 7f371f38a740 1 -- 172.21.15.102:0/2429124091 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563cb7c3ea00 con 0x563cb7bad8d0 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.495+0000 7f36fb400640 1 -- 172.21.15.102:0/2429124091 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37000cc020 con 0x563cb7bad8d0 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.496+0000 7f36fb400640 1 -- 172.21.15.102:0/2429124091 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f37000c2190 con 0x563cb7bad8d0 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.496+0000 7f371f38a740 10 client.27104 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.496+0000 7f371f38a740 10 client.27104 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.496+0000 7f371f38a740 20 client.27104 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.496+0000 7f371f38a740 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x563cb7bd9280 0x563cb7c42040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.497+0000 7f371f38a740 4 client.27104 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.497+0000 7f371f38a740 1 -- 172.21.15.102:0/2429124091 --> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563cb7b10d70 con 0x563cb7bd9280 2024-04-25T02:02:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.497+0000 7f3719a00640 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x563cb7bd9280 0x563cb7c42040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.502+0000 7f3719a00640 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x563cb7bd9280 0x563cb7c42040 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=85f3bd707267a8f7 server_cookie=e9a4d77a63edbab5 in_seq=0 out_seq=0 2024-04-25T02:02:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.502+0000 7f36fb400640 10 client.27104 ms_handle_connect on v2:172.21.15.193:6838/2484219231 2024-04-25T02:02:10.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.502+0000 7f36fb400640 1 -- 172.21.15.102:0/2429124091 <== mds.1 v2:172.21.15.193:6838/2484219231 1 ==== command_reply(tid 0: 0 ) ==== 8+0+863 (crc 0 0 0) 0x563cb7b10d70 con 0x563cb7bd9280 2024-04-25T02:02:10.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.502+0000 7f36fb400640 10 client.27104 handle_command_reply: tid=0 2024-04-25T02:02:10.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.502+0000 7f371f38a740 1 client.27104 shutdown 2024-04-25T02:02:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.503+0000 7f371f38a740 1 -- 172.21.15.102:0/2429124091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 msgr2=0x563cb7bb2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.503+0000 7f371f38a740 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb2210 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f37000bc7c0 tx=0x7f37000efc50 comp rx=0 tx=0).stop 2024-04-25T02:02:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.504+0000 7f371f38a740 1 -- 172.21.15.102:0/2429124091 shutdown_connections 2024-04-25T02:02:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.504+0000 7f371f38a740 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.193:6838/2484219231,v1:172.21.15.193:6839/2484219231] conn(0x563cb7bd9280 0x563cb7c42040 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.504+0000 7f371f38a740 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563cb7bb7fa0 0x563cb7bb2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.504+0000 7f371f38a740 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x563cb7bad8d0 0x563cb7bb2210 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.504+0000 7f371f38a740 1 --2- 172.21.15.102:0/2429124091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cb7bacfb0 0x563cb7bb1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.504+0000 7f371f38a740 1 -- 172.21.15.102:0/2429124091 >> 172.21.15.102:0/2429124091 conn(0x563cb7ba92d0 msgr2=0x563cb7bb4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.504+0000 7f36fb400640 0 client.27104 ms_handle_reset on v2:172.21.15.193:6838/2484219231 2024-04-25T02:02:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.504+0000 7f371f38a740 1 -- 172.21.15.102:0/2429124091 shutdown_connections 2024-04-25T02:02:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.504+0000 7f371f38a740 1 -- 172.21.15.102:0/2429124091 wait complete. 2024-04-25T02:02:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.505+0000 7f371f38a740 20 client.27104 trim_cache size 0 max 16384 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: "damage_type": "backtrace", 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 420237867, 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: "ino": 2199023256061, 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004A" 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: "damage_type": "backtrace", 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 595209436, 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: "ino": 1099511629289, 2024-04-25T02:02:10.510 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000C" 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: "damage_type": "backtrace", 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 1771500608, 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: "ino": 1099511629288, 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000B" 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: "damage_type": "backtrace", 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 3430452798, 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: "ino": 2199023256068, 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: "path": "/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004D" 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:02:10.511 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-04-25T02:02:10.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.506+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f36fc0a2f80 msgr2=0x7f36fc0a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.506+0000 7f371ae00640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f36fc0a2f80 0x7f36fc0a5440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f370c00b7b0 tx=0x7f370c00d040 comp rx=0 tx=0).stop 2024-04-25T02:02:10.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.506+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3714101fa0 msgr2=0x7f3714073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.506+0000 7f371ae00640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3714101fa0 0x7f3714073f80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f370002aa30 tx=0x7f3700030940 comp rx=0 tx=0).stop 2024-04-25T02:02:10.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.507+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 shutdown_connections 2024-04-25T02:02:10.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.507+0000 7f371ae00640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f36fc0a2f80 0x7f36fc0a5440 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.507+0000 7f371ae00640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3714107680 0x7f37140744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.507+0000 7f371ae00640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3714101fa0 0x7f3714073f80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.507+0000 7f371ae00640 1 --2- 172.21.15.102:0/2804517621 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37141015e0 0x7f3714073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.507+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 >> 172.21.15.102:0/2804517621 conn(0x7f37140fd110 msgr2=0x7f37141084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.507+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 shutdown_connections 2024-04-25T02:02:10.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.507+0000 7f371ae00640 1 -- 172.21.15.102:0/2804517621 wait complete. 2024-04-25T02:02:10.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T02:02:10.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:10 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/314482189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:02:10.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:10 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/314482189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:02:10.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:10 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/314482189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:02:10.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.824+0000 7f5139000640 1 -- 172.21.15.102:0/4110904083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5134108970 msgr2=0x7f513410c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.824+0000 7f5139000640 1 --2- 172.21.15.102:0/4110904083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5134108970 0x7f513410c960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f511c009900 tx=0x7f511c02a410 comp rx=0 tx=0).stop 2024-04-25T02:02:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.824+0000 7f5139000640 1 -- 172.21.15.102:0/4110904083 shutdown_connections 2024-04-25T02:02:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.824+0000 7f5139000640 1 --2- 172.21.15.102:0/4110904083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513410d040 0x7f5134114c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.824+0000 7f5139000640 1 --2- 172.21.15.102:0/4110904083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5134108970 0x7f513410c960 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.824+0000 7f5139000640 1 --2- 172.21.15.102:0/4110904083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134107fc0 0x7f51341083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.824+0000 7f5139000640 1 -- 172.21.15.102:0/4110904083 >> 172.21.15.102:0/4110904083 conn(0x7f5134075c80 msgr2=0x7f51340780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.824+0000 7f5139000640 1 -- 172.21.15.102:0/4110904083 shutdown_connections 2024-04-25T02:02:10.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.825+0000 7f5139000640 1 -- 172.21.15.102:0/4110904083 wait complete. 2024-04-25T02:02:10.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.826+0000 7f5139000640 1 Processor -- start 2024-04-25T02:02:10.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.826+0000 7f5139000640 1 -- start start 2024-04-25T02:02:10.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.827+0000 7f5139000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134107fc0 0x7f5134072f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.827+0000 7f5139000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513410d040 0x7f513406c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.827+0000 7f5139000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f513406c550 0x7f513406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5139000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5134118100 con 0x7f5134107fc0 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5139000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5134117f80 con 0x7f513410d040 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5139000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5134118280 con 0x7f513406c550 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5131e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134107fc0 0x7f5134072f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5131e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134107fc0 0x7f5134072f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50024/0 (socket says 172.21.15.102:50024) 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5131e00640 1 -- 172.21.15.102:0/3515177273 learned_addr learned my addr 172.21.15.102:0/3515177273 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5132800640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f513406c550 0x7f513406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5132800640 1 -- 172.21.15.102:0/3515177273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513410d040 msgr2=0x7f513406c010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5132800640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513410d040 0x7f513406c010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5132800640 1 -- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134107fc0 msgr2=0x7f5134072f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5132800640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134107fc0 0x7f5134072f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5132800640 1 -- 172.21.15.102:0/3515177273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5134070ab0 con 0x7f513406c550 2024-04-25T02:02:10.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.828+0000 7f5131e00640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134107fc0 0x7f5134072f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:10.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.829+0000 7f5132800640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f513406c550 0x7f513406ca00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f512c009fd0 tx=0x7f512c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.829+0000 7f512aa00640 1 -- 172.21.15.102:0/3515177273 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c00ce20 con 0x7f513406c550 2024-04-25T02:02:10.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.829+0000 7f512aa00640 1 -- 172.21.15.102:0/3515177273 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512c005020 con 0x7f513406c550 2024-04-25T02:02:10.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.829+0000 7f512aa00640 1 -- 172.21.15.102:0/3515177273 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c010720 con 0x7f513406c550 2024-04-25T02:02:10.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.829+0000 7f5139000640 1 -- 172.21.15.102:0/3515177273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5134070da0 con 0x7f513406c550 2024-04-25T02:02:10.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.829+0000 7f5139000640 1 -- 172.21.15.102:0/3515177273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51340712a0 con 0x7f513406c550 2024-04-25T02:02:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.833+0000 7f512aa00640 1 -- 172.21.15.102:0/3515177273 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f512c005540 con 0x7f513406c550 2024-04-25T02:02:10.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.833+0000 7f513be26740 1 Processor -- start 2024-04-25T02:02:10.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f513be26740 1 -- start start 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f513be26740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b0b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f513be26740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b16070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f513be26740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b189a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f513be26740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a1b1a704e0 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f513be26740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a1b1a18a70 con 0x55a1b1b0bee0 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f513be26740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a1b1a6c4a0 con 0x55a1b1b165b0 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f512be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b0b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f512be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b189a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f5131e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b16070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.834+0000 7f512be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b189a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40812/0 (socket says 172.21.15.102:40812) 2024-04-25T02:02:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f512be00640 1 -- 172.21.15.102:0/887066958 learned_addr learned my addr 172.21.15.102:0/887066958 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f512aa00640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f51180a2e80 0x7f51180a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f5131e00640 1 -- 172.21.15.102:0/887066958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 msgr2=0x55a1b1b189a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f5131e00640 1 --2- 172.21.15.102:0/887066958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b189a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f5131e00640 1 -- 172.21.15.102:0/887066958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 msgr2=0x55a1b1b0b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f5131e00640 1 --2- 172.21.15.102:0/887066958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b0b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f5131e00640 1 -- 172.21.15.102:0/887066958 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1b17843f0 con 0x55a1b1b0bee0 2024-04-25T02:02:10.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f512be00640 1 --2- 172.21.15.102:0/887066958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b0b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:10.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f512aa00640 1 -- 172.21.15.102:0/3515177273 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f512c0cfab0 con 0x7f513406c550 2024-04-25T02:02:10.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.835+0000 7f5131e00640 1 --2- 172.21.15.102:0/887066958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b16070 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f51240114f0 tx=0x7f51240119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23431ea58a1633cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.836+0000 7f5123400640 1 -- 172.21.15.102:0/887066958 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512401e070 con 0x55a1b1b0bee0 2024-04-25T02:02:10.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.836+0000 7f5123400640 1 -- 172.21.15.102:0/887066958 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51240035f0 con 0x55a1b1b0bee0 2024-04-25T02:02:10.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.836+0000 7f5123400640 1 -- 172.21.15.102:0/887066958 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512400f820 con 0x55a1b1b0bee0 2024-04-25T02:02:10.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.837+0000 7f513be26740 1 -- 172.21.15.102:0/887066958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 msgr2=0x55a1b1b16070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.837+0000 7f513be26740 1 --2- 172.21.15.102:0/887066958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b16070 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f51240114f0 tx=0x7f51240119b0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.837+0000 7f5132800640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f51180a2e80 0x7f51180a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.838+0000 7f513be26740 1 -- 172.21.15.102:0/887066958 shutdown_connections 2024-04-25T02:02:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.838+0000 7f513be26740 1 --2- 172.21.15.102:0/887066958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b189a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.838+0000 7f513be26740 1 --2- 172.21.15.102:0/887066958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b16070 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.838+0000 7f513be26740 1 --2- 172.21.15.102:0/887066958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b0b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.838+0000 7f513be26740 1 -- 172.21.15.102:0/887066958 >> 172.21.15.102:0/887066958 conn(0x55a1b1b078e0 msgr2=0x55a1b1b07cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.838+0000 7f513be26740 1 -- 172.21.15.102:0/887066958 shutdown_connections 2024-04-25T02:02:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.838+0000 7f5132800640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f51180a2e80 0x7f51180a5340 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f512c098980 tx=0x7f512c005f70 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.838+0000 7f513be26740 1 -- 172.21.15.102:0/887066958 wait complete. 2024-04-25T02:02:10.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.839+0000 7f513be26740 1 -- start start 2024-04-25T02:02:10.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.839+0000 7f513be26740 1 Processor -- start 2024-04-25T02:02:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.840+0000 7f513be26740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b135a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f513be26740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f513be26740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b165b0 0x55a1b1b14020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f513be26740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a1b1b193a0 con 0x55a1b1b0bee0 2024-04-25T02:02:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f513be26740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a1b1a2bec0 con 0x55a1b1b165b0 2024-04-25T02:02:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f513be26740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a1b1b19520 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f512be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f513be26740 10 client.0 _ll_register_callbacks cb 0x55a1b1a683f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f512be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40824/0 (socket says 172.21.15.102:40824) 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f5131e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f512be00640 1 -- 172.21.15.102:0/2784211406 learned_addr learned my addr 172.21.15.102:0/2784211406 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f512be00640 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b165b0 0x55a1b1b14020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f512be00640 1 -- 172.21.15.102:0/2784211406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b165b0 msgr2=0x55a1b1b14020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f512be00640 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b165b0 0x55a1b1b14020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f512be00640 1 -- 172.21.15.102:0/2784211406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 msgr2=0x55a1b1b13ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f512be00640 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.841+0000 7f512be00640 1 -- 172.21.15.102:0/2784211406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1b184e7d0 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.842+0000 7f512be00640 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b135a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f511c005ab0 tx=0x7f511c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d551ce37862f6dd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.842+0000 7f5117e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.842+0000 7f5117e00640 1 -- 172.21.15.102:0/2784211406 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c035070 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.842+0000 7f5117e00640 1 -- 172.21.15.102:0/2784211406 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f511c003c80 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.842+0000 7f513be26740 1 -- 172.21.15.102:0/2784211406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51240035f0 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.843+0000 7f5117e00640 1 -- 172.21.15.102:0/2784211406 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f511c0315b0 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.843+0000 7f513be26740 10 client.27070 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.843+0000 7f513be26740 1 -- 172.21.15.102:0/2784211406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a1b1b256d0 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.843+0000 7f5117e00640 1 -- 172.21.15.102:0/2784211406 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51240035f0 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.844+0000 7f5117e00640 1 -- 172.21.15.102:0/2784211406 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f511c03a040 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.844+0000 7f513be26740 10 client.27070 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.844+0000 7f513be26740 10 client.27070 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.844+0000 7f513be26740 20 client.27070 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.844+0000 7f513be26740 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a1b1b26b90 0x55a1b1b28fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.844+0000 7f513be26740 4 client.27070 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.844+0000 7f513be26740 1 -- 172.21.15.102:0/2784211406 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1b1a7da20 con 0x55a1b1b26b90 2024-04-25T02:02:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.844+0000 7f5131e00640 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a1b1b26b90 0x55a1b1b28fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.845+0000 7f5131e00640 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a1b1b26b90 0x55a1b1b28fd0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a294d3052719503 server_cookie=e8ef0a7b2779b99f in_seq=0 out_seq=0 2024-04-25T02:02:10.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.845+0000 7f5117e00640 10 client.27070 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:10.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.845+0000 7f5117e00640 1 -- 172.21.15.102:0/2784211406 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a1b1a7da20 con 0x55a1b1b26b90 2024-04-25T02:02:10.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.845+0000 7f5117e00640 10 client.27070 handle_command_reply: tid=0 2024-04-25T02:02:10.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f513be26740 1 client.27070 shutdown 2024-04-25T02:02:10.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f513be26740 1 -- 172.21.15.102:0/2784211406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b0b5c0 msgr2=0x55a1b1b135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f513be26740 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b135a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f511c005ab0 tx=0x7f511c009260 comp rx=0 tx=0).stop 2024-04-25T02:02:10.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f513be26740 1 -- 172.21.15.102:0/2784211406 shutdown_connections 2024-04-25T02:02:10.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f513be26740 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a1b1b26b90 0x55a1b1b28fd0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f513be26740 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b165b0 0x55a1b1b14020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f5117e00640 0 client.27070 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:10.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f513be26740 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f513be26740 1 --2- 172.21.15.102:0/2784211406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b135a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.846+0000 7f513be26740 1 -- 172.21.15.102:0/2784211406 >> 172.21.15.102:0/2784211406 conn(0x55a1b1b078e0 msgr2=0x55a1b1b0d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.847+0000 7f513be26740 1 -- 172.21.15.102:0/2784211406 shutdown_connections 2024-04-25T02:02:10.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.847+0000 7f513be26740 1 -- 172.21.15.102:0/2784211406 wait complete. 2024-04-25T02:02:10.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.847+0000 7f513be26740 20 client.27070 trim_cache size 0 max 16384 2024-04-25T02:02:10.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.866+0000 7f513be26740 1 Processor -- start 2024-04-25T02:02:10.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.866+0000 7f513be26740 1 -- start start 2024-04-25T02:02:10.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.866+0000 7f513be26740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b22290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.866+0000 7f513be26740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.866+0000 7f513be26740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b2e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.866+0000 7f513be26740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f51341bed90 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.866+0000 7f513be26740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f51341bc5a0 con 0x55a1b1b0bee0 2024-04-25T02:02:10.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.866+0000 7f513be26740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f51341ba410 con 0x55a1b1b165b0 2024-04-25T02:02:10.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f5131e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f5131e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41240/0 (socket says 172.21.15.102:41240) 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f5131e00640 1 -- 172.21.15.102:0/2005026566 learned_addr learned my addr 172.21.15.102:0/2005026566 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f512be00640 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b22290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f512be00640 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b2e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f5131e00640 1 -- 172.21.15.102:0/2005026566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 msgr2=0x55a1b1b2e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f5131e00640 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b2e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f5131e00640 1 -- 172.21.15.102:0/2005026566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 msgr2=0x55a1b1b22290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f5131e00640 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b22290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f5131e00640 1 -- 172.21.15.102:0/2005026566 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1b1b189a0 con 0x55a1b1b0bee0 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f512be00640 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b2e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.867+0000 7f5131e00640 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13e40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5124009230 tx=0x7f5124005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=486f08b8c1903fba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.868+0000 7f5122a00640 1 -- 172.21.15.102:0/2005026566 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512402e070 con 0x55a1b1b0bee0 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.868+0000 7f5122a00640 1 -- 172.21.15.102:0/2005026566 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512401f040 con 0x55a1b1b0bee0 2024-04-25T02:02:10.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.868+0000 7f5122a00640 1 -- 172.21.15.102:0/2005026566 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124029e60 con 0x55a1b1b0bee0 2024-04-25T02:02:10.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.869+0000 7f513be26740 1 -- 172.21.15.102:0/2005026566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 msgr2=0x55a1b1b13e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.869+0000 7f513be26740 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13e40 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5124009230 tx=0x7f5124005f90 comp rx=0 tx=0).stop 2024-04-25T02:02:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.869+0000 7f513be26740 1 -- 172.21.15.102:0/2005026566 shutdown_connections 2024-04-25T02:02:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.869+0000 7f513be26740 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b2e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.869+0000 7f513be26740 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b13e40 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.869+0000 7f513be26740 1 --2- 172.21.15.102:0/2005026566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b22290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.869+0000 7f513be26740 1 -- 172.21.15.102:0/2005026566 >> 172.21.15.102:0/2005026566 conn(0x55a1b1b078e0 msgr2=0x55a1b1b0d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.869+0000 7f513be26740 1 -- 172.21.15.102:0/2005026566 shutdown_connections 2024-04-25T02:02:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.869+0000 7f513be26740 1 -- 172.21.15.102:0/2005026566 wait complete. 2024-04-25T02:02:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.870+0000 7f513be26740 1 -- start start 2024-04-25T02:02:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.870+0000 7f513be26740 1 Processor -- start 2024-04-25T02:02:10.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f513be26740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b1e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f513be26740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b1ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f513be26740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b19f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f513be26740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f51341baa60 con 0x55a1b1b0bee0 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f513be26740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f51341bbed0 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f513be26740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f51341bb130 con 0x55a1b1b165b0 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f513be26740 10 client.0 _ll_register_callbacks cb 0x7f512c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f512be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b1e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f512be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b1e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:41254/0 (socket says 172.21.15.102:41254) 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f512be00640 1 -- 172.21.15.102:0/1790049051 learned_addr learned my addr 172.21.15.102:0/1790049051 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f5131e00640 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b1ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.871+0000 7f512be00640 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b19f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f512be00640 1 -- 172.21.15.102:0/1790049051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 msgr2=0x55a1b1b19f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f512be00640 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b19f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f512be00640 1 -- 172.21.15.102:0/1790049051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 msgr2=0x55a1b1b1ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f512be00640 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b1ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f512be00640 1 -- 172.21.15.102:0/1790049051 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1b1b1a720 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f5131e00640 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b1ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f512be00640 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b1e7d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f511c001eb0 tx=0x7f511c040040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33a8f5d264a27ea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f5117e00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f5117e00640 1 -- 172.21.15.102:0/1790049051 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c035040 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f5117e00640 1 -- 172.21.15.102:0/1790049051 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8557+0+0 (secure 0 0 0) 0x7f511c003a80 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f513be26740 1 -- 172.21.15.102:0/1790049051 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f512401f040 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f5117e00640 1 -- 172.21.15.102:0/1790049051 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f511c003c80 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f513be26740 10 client.27119 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.872+0000 7f513be26740 1 -- 172.21.15.102:0/1790049051 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a1b1b8cde0 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.873+0000 7f5117e00640 1 -- 172.21.15.102:0/1790049051 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f512401f040 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.873+0000 7f5117e00640 1 -- 172.21.15.102:0/1790049051 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f511c031ac0 con 0x55a1b1b0b5c0 2024-04-25T02:02:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.873+0000 7f513be26740 10 client.27119 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.873+0000 7f513be26740 10 client.27119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.873+0000 7f513be26740 20 client.27119 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.873+0000 7f513be26740 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a1b1b26b90 0x55a1b1b8f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.873+0000 7f513be26740 4 client.27119 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T02:02:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.873+0000 7f513be26740 1 -- 172.21.15.102:0/1790049051 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a1b1a77d70 con 0x55a1b1b26b90 2024-04-25T02:02:10.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.873+0000 7f5131e00640 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a1b1b26b90 0x55a1b1b8f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f5131e00640 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a1b1b26b90 0x55a1b1b8f600 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64086b2e39493426 server_cookie=af8b4516029efb9d in_seq=0 out_seq=0 2024-04-25T02:02:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f5117e00640 10 client.27119 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f5117e00640 1 -- 172.21.15.102:0/1790049051 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a1b1a77d70 con 0x55a1b1b26b90 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f5117e00640 10 client.27119 handle_command_reply: tid=0 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f513be26740 1 client.27119 shutdown 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f513be26740 1 -- 172.21.15.102:0/1790049051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0b5c0 msgr2=0x55a1b1b1e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f513be26740 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b1e7d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f511c001eb0 tx=0x7f511c040040 comp rx=0 tx=0).stop 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f513be26740 1 -- 172.21.15.102:0/1790049051 shutdown_connections 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f513be26740 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a1b1b26b90 0x55a1b1b8f600 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f513be26740 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1b1b165b0 0x55a1b1b19f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f513be26740 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a1b1b0bee0 0x55a1b1b1ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.875+0000 7f513be26740 1 --2- 172.21.15.102:0/1790049051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a1b1b0b5c0 0x55a1b1b1e7d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.874+0000 7f5117e00640 0 client.27119 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.875+0000 7f513be26740 1 -- 172.21.15.102:0/1790049051 >> 172.21.15.102:0/1790049051 conn(0x55a1b1b078e0 msgr2=0x55a1b1b062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.875+0000 7f513be26740 1 -- 172.21.15.102:0/1790049051 shutdown_connections 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.875+0000 7f513be26740 1 -- 172.21.15.102:0/1790049051 wait complete. 2024-04-25T02:02:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.875+0000 7f513be26740 20 client.27119 trim_cache size 0 max 16384 2024-04-25T02:02:10.883 INFO:teuthology.orchestra.run.smithi102.stdout:[] 2024-04-25T02:02:10.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.875+0000 7f5139000640 1 -- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f51180a2e80 msgr2=0x7f51180a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.875+0000 7f5139000640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f51180a2e80 0x7f51180a5340 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f512c098980 tx=0x7f512c005f70 comp rx=0 tx=0).stop 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.875+0000 7f5139000640 1 -- 172.21.15.102:0/3515177273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f513406c550 msgr2=0x7f513406ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.875+0000 7f5139000640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f513406c550 0x7f513406ca00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f512c009fd0 tx=0x7f512c00ef10 comp rx=0 tx=0).stop 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.876+0000 7f5139000640 1 -- 172.21.15.102:0/3515177273 shutdown_connections 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.876+0000 7f5139000640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f51180a2e80 0x7f51180a5340 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.876+0000 7f5139000640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f513406c550 0x7f513406ca00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.876+0000 7f5139000640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f513410d040 0x7f513406c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.876+0000 7f5139000640 1 --2- 172.21.15.102:0/3515177273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5134107fc0 0x7f5134072f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.876+0000 7f5139000640 1 -- 172.21.15.102:0/3515177273 >> 172.21.15.102:0/3515177273 conn(0x7f5134075c80 msgr2=0x7f5134077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.876+0000 7f5139000640 1 -- 172.21.15.102:0/3515177273 shutdown_connections 2024-04-25T02:02:10.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:02:10.876+0000 7f5139000640 1 -- 172.21.15.102:0/3515177273 wait complete. 2024-04-25T02:02:10.904 ERROR:tasks.fwd_scrub:rank 1 damaged: [{'damage_type': 'backtrace', 'id': 420237867, 'ino': 2199023256061, 'path': '/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004A'}, {'damage_type': 'backtrace', 'id': 595209436, 'ino': 1099511629289, 'path': '/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000C'}, {'damage_type': 'backtrace', 'id': 1771500608, 'ino': 1099511629288, 'path': '/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_subtrans/000B'}, {'damage_type': 'backtrace', 'id': 3430452798, 'ino': 2199023256068, 'path': '/volumes/qa/sv_1/81669b28-eb2a-4861-893e-c7b62ed09a76/postgres/data/pg_wal/00000001000000000000004D'}] 2024-04-25T02:02:10.905 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-04-25T02:02:11.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:11 smithi102 ceph-mon[27796]: pgmap v211: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.4 MiB/s wr, 495 op/s 2024-04-25T02:02:11.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:11 smithi110 ceph-mon[32848]: pgmap v211: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.4 MiB/s wr, 495 op/s 2024-04-25T02:02:11.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:11 smithi193 ceph-mon[32981]: pgmap v211: 97 pgs: 97 active+clean; 4.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 4.4 MiB/s wr, 495 op/s 2024-04-25T02:02:13.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 380.0 s, 1048.9 tps, lat 104.002 ms stddev 259.317 2024-04-25T02:02:13.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:13 smithi102 ceph-mon[27796]: pgmap v212: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 MiB/s wr, 1.10k op/s 2024-04-25T02:02:13.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:13 smithi110 ceph-mon[32848]: pgmap v212: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 MiB/s wr, 1.10k op/s 2024-04-25T02:02:13.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:13 smithi193 ceph-mon[32981]: pgmap v212: 97 pgs: 97 active+clean; 4.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 MiB/s wr, 1.10k op/s 2024-04-25T02:02:15.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:15 smithi102 ceph-mon[27796]: pgmap v213: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 32 MiB/s wr, 1.53k op/s 2024-04-25T02:02:15.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:15 smithi110 ceph-mon[32848]: pgmap v213: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 32 MiB/s wr, 1.53k op/s 2024-04-25T02:02:15.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:15 smithi193 ceph-mon[32981]: pgmap v213: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 32 MiB/s wr, 1.53k op/s 2024-04-25T02:02:17.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:17 smithi102 ceph-mon[27796]: pgmap v214: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 32 MiB/s wr, 1.53k op/s 2024-04-25T02:02:17.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:17 smithi102 ceph-mon[27796]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T02:02:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.704+0000 7f7246a00640 1 -- 172.21.15.102:0/2604060815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7240111460 msgr2=0x7f7240119030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.704+0000 7f7246a00640 1 --2- 172.21.15.102:0/2604060815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7240111460 0x7f7240119030 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f7238009be0 tx=0x7f723802a6d0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.704+0000 7f7246a00640 1 -- 172.21.15.102:0/2604060815 shutdown_connections 2024-04-25T02:02:17.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.704+0000 7f7246a00640 1 --2- 172.21.15.102:0/2604060815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7240111460 0x7f7240119030 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.704+0000 7f7246a00640 1 --2- 172.21.15.102:0/2604060815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f724010cd40 0x7f7240110d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.704+0000 7f7246a00640 1 --2- 172.21.15.102:0/2604060815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f724010c390 0x7f724010c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.704+0000 7f7246a00640 1 -- 172.21.15.102:0/2604060815 >> 172.21.15.102:0/2604060815 conn(0x7f724006c9c0 msgr2=0x7f724010a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.704+0000 7f7246a00640 1 -- 172.21.15.102:0/2604060815 shutdown_connections 2024-04-25T02:02:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.705+0000 7f7246a00640 1 -- 172.21.15.102:0/2604060815 wait complete. 2024-04-25T02:02:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.706+0000 7f7246a00640 1 Processor -- start 2024-04-25T02:02:17.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.706+0000 7f7246a00640 1 -- start start 2024-04-25T02:02:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f7246a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f724010c390 0x7f72401a8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f7246a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240111460 0x7f72401a8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f7246a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72401a2780 0x7f72401a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f7246a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f724011c500 con 0x7f7240111460 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f7246a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f724011c380 con 0x7f72401a2780 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f7246a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f724011c680 con 0x7f724010c390 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f723f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240111460 0x7f72401a8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f723fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f724010c390 0x7f72401a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f723fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f724010c390 0x7f72401a8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:39420/0 (socket says 172.21.15.102:39420) 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.707+0000 7f723fe00640 1 -- 172.21.15.102:0/3387618074 learned_addr learned my addr 172.21.15.102:0/3387618074 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.708+0000 7f7244a00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72401a2780 0x7f72401a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.708+0000 7f723fe00640 1 -- 172.21.15.102:0/3387618074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72401a2780 msgr2=0x7f72401a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.708+0000 7f723fe00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72401a2780 0x7f72401a2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.708+0000 7f723fe00640 1 -- 172.21.15.102:0/3387618074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240111460 msgr2=0x7f72401a8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.708+0000 7f723fe00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240111460 0x7f72401a8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.708+0000 7f723fe00640 1 -- 172.21.15.102:0/3387618074 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72401a3310 con 0x7f724010c390 2024-04-25T02:02:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.709+0000 7f723fe00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f724010c390 0x7f72401a8610 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f723000e9c0 tx=0x7f723000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.709+0000 7f723cc00640 1 -- 172.21.15.102:0/3387618074 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723000ccc0 con 0x7f724010c390 2024-04-25T02:02:17.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.709+0000 7f723cc00640 1 -- 172.21.15.102:0/3387618074 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7230005500 con 0x7f724010c390 2024-04-25T02:02:17.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.709+0000 7f723cc00640 1 -- 172.21.15.102:0/3387618074 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230010430 con 0x7f724010c390 2024-04-25T02:02:17.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.709+0000 7f7246a00640 1 -- 172.21.15.102:0/3387618074 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72401a3600 con 0x7f724010c390 2024-04-25T02:02:17.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.709+0000 7f7246a00640 1 -- 172.21.15.102:0/3387618074 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72401af830 con 0x7f724010c390 2024-04-25T02:02:17.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.715+0000 7f723cc00640 1 -- 172.21.15.102:0/3387618074 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f72300046c0 con 0x7f724010c390 2024-04-25T02:02:17.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.715+0000 7f72497b2740 1 Processor -- start 2024-04-25T02:02:17.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.716+0000 7f72497b2740 1 -- start start 2024-04-25T02:02:17.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.717+0000 7f72497b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.717+0000 7f72497b2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197ebaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.717+0000 7f72497b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 0x5602197ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.717+0000 7f72497b2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560219744ab0 con 0x5602197e1720 2024-04-25T02:02:17.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.717+0000 7f72497b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5602196ed640 con 0x5602197e0e00 2024-04-25T02:02:17.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.717+0000 7f72497b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560219740a70 con 0x5602197ebfe0 2024-04-25T02:02:17.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.718+0000 7f7244a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.718+0000 7f7244a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56860/0 (socket says 172.21.15.102:56860) 2024-04-25T02:02:17.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.718+0000 7f7244a00640 1 -- 172.21.15.102:0/3682335347 learned_addr learned my addr 172.21.15.102:0/3682335347 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:17.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.718+0000 7f723f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 0x5602197ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.718+0000 7f723fe00640 1 --2- 172.21.15.102:0/3682335347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197ebaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.718+0000 7f723cc00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f72240a2e20 0x7f72240a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.719+0000 7f723f400640 1 -- 172.21.15.102:0/3682335347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 msgr2=0x5602197e11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:17.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.719+0000 7f723f400640 1 --2- 172.21.15.102:0/3682335347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.719+0000 7f723f400640 1 -- 172.21.15.102:0/3682335347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 msgr2=0x5602197ebaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:17.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.719+0000 7f723f400640 1 --2- 172.21.15.102:0/3682335347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197ebaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.719+0000 7f723f400640 1 -- 172.21.15.102:0/3682335347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560219459210 con 0x5602197ebfe0 2024-04-25T02:02:17.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.719+0000 7f723cc00640 1 -- 172.21.15.102:0/3387618074 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f72300cf7d0 con 0x7f724010c390 2024-04-25T02:02:17.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.719+0000 7f7244a00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f72240a2e20 0x7f72240a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.719+0000 7f723f400640 1 --2- 172.21.15.102:0/3682335347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 0x5602197ee3d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f72401a3ad0 tx=0x7f722c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8072218608279208 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:17.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.720+0000 7f7235200640 1 -- 172.21.15.102:0/3682335347 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c004050 con 0x5602197ebfe0 2024-04-25T02:02:17.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.720+0000 7f7235200640 1 -- 172.21.15.102:0/3682335347 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f722c005500 con 0x5602197ebfe0 2024-04-25T02:02:17.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.720+0000 7f7235200640 1 -- 172.21.15.102:0/3682335347 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c004d20 con 0x5602197ebfe0 2024-04-25T02:02:17.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f72497b2740 1 -- 172.21.15.102:0/3682335347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 msgr2=0x5602197ee3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f72497b2740 1 --2- 172.21.15.102:0/3682335347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 0x5602197ee3d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f72401a3ad0 tx=0x7f722c011810 comp rx=0 tx=0).stop 2024-04-25T02:02:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f7244a00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f72240a2e20 0x7f72240a52e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f72380099a0 tx=0x7f7238009500 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f72497b2740 1 -- 172.21.15.102:0/3682335347 shutdown_connections 2024-04-25T02:02:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f72497b2740 1 --2- 172.21.15.102:0/3682335347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 0x5602197ee3d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f72497b2740 1 --2- 172.21.15.102:0/3682335347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197ebaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f72497b2740 1 --2- 172.21.15.102:0/3682335347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f72497b2740 1 -- 172.21.15.102:0/3682335347 >> 172.21.15.102:0/3682335347 conn(0x5602197dd310 msgr2=0x5602197dd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f72497b2740 1 -- 172.21.15.102:0/3682335347 shutdown_connections 2024-04-25T02:02:17.727 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.721+0000 7f72497b2740 1 -- 172.21.15.102:0/3682335347 wait complete. 2024-04-25T02:02:17.727 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.722+0000 7f72497b2740 1 -- start start 2024-04-25T02:02:17.727 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.722+0000 7f72497b2740 1 Processor -- start 2024-04-25T02:02:17.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f72497b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e0e00 0x5602197e8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f72497b2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f72497b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x5602197e9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f72497b2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5602197eec10 con 0x5602197e1720 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f72497b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560219700990 con 0x5602197ebfe0 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f72497b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5602197eed90 con 0x5602197e0e00 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f72497b2740 10 client.0 _ll_register_callbacks cb 0x560219702950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f7244a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x5602197e9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f7244a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x5602197e9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56876/0 (socket says 172.21.15.102:56876) 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f7244a00640 1 -- 172.21.15.102:0/548149302 learned_addr learned my addr 172.21.15.102:0/548149302 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.723+0000 7f723fe00640 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e0e00 0x5602197e8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.724+0000 7f723f400640 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.724+0000 7f7244a00640 1 -- 172.21.15.102:0/548149302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e0e00 msgr2=0x5602197e8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.724+0000 7f7244a00640 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e0e00 0x5602197e8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.724+0000 7f7244a00640 1 -- 172.21.15.102:0/548149302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 msgr2=0x5602197e9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.724+0000 7f7244a00640 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.724+0000 7f7244a00640 1 -- 172.21.15.102:0/548149302 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560219525620 con 0x5602197ebfe0 2024-04-25T02:02:17.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.724+0000 7f723f400640 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:02:17.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.725+0000 7f7244a00640 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x5602197e9a40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7238033000 tx=0x7f7238009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6569264e1f37cf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.725+0000 7f7229600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.725+0000 7f7229600640 1 -- 172.21.15.102:0/548149302 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238036040 con 0x5602197ebfe0 2024-04-25T02:02:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.725+0000 7f7229600640 1 -- 172.21.15.102:0/548149302 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f7238003d80 con 0x5602197ebfe0 2024-04-25T02:02:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.726+0000 7f72497b2740 1 -- 172.21.15.102:0/548149302 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f722c005500 con 0x5602197ebfe0 2024-04-25T02:02:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.726+0000 7f7229600640 1 -- 172.21.15.102:0/548149302 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7238004040 con 0x5602197ebfe0 2024-04-25T02:02:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.726+0000 7f72497b2740 10 client.27131 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.726+0000 7f72497b2740 1 -- 172.21.15.102:0/548149302 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560219803340 con 0x5602197ebfe0 2024-04-25T02:02:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.726+0000 7f7229600640 1 -- 172.21.15.102:0/548149302 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72380046c0 con 0x5602197ebfe0 2024-04-25T02:02:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.726+0000 7f7229600640 1 -- 172.21.15.102:0/548149302 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f7238054030 con 0x5602197ebfe0 2024-04-25T02:02:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.727+0000 7f72497b2740 10 client.27131 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.727+0000 7f72497b2740 10 client.27131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.727+0000 7f72497b2740 20 client.27131 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.727+0000 7f72497b2740 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5602198048c0 0x560219806d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.727+0000 7f72497b2740 4 client.27131 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.727+0000 7f72497b2740 1 -- 172.21.15.102:0/548149302 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f722c005500 con 0x5602198048c0 2024-04-25T02:02:17.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.728+0000 7f723fe00640 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5602198048c0 0x560219806d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.729+0000 7f723fe00640 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5602198048c0 0x560219806d10 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=311b1b7f10e8a57 server_cookie=9b56507a1225647 in_seq=0 out_seq=0 2024-04-25T02:02:17.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.729+0000 7f7229600640 10 client.27131 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.729+0000 7f7229600640 1 -- 172.21.15.102:0/548149302 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f722c005500 con 0x5602198048c0 2024-04-25T02:02:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f7229600640 10 client.27131 handle_command_reply: tid=0 2024-04-25T02:02:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f72497b2740 1 client.27131 shutdown 2024-04-25T02:02:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f72497b2740 1 -- 172.21.15.102:0/548149302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 msgr2=0x5602197e9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f72497b2740 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x5602197e9a40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7238033000 tx=0x7f7238009280 comp rx=0 tx=0).stop 2024-04-25T02:02:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f72497b2740 1 -- 172.21.15.102:0/548149302 shutdown_connections 2024-04-25T02:02:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f72497b2740 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5602198048c0 0x560219806d10 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f72497b2740 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x5602197e9a40 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f72497b2740 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f72497b2740 1 --2- 172.21.15.102:0/548149302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e0e00 0x5602197e8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f72497b2740 1 -- 172.21.15.102:0/548149302 >> 172.21.15.102:0/548149302 conn(0x5602197dd310 msgr2=0x5602197e3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.730+0000 7f7229600640 0 client.27131 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:17.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.731+0000 7f72497b2740 1 -- 172.21.15.102:0/548149302 shutdown_connections 2024-04-25T02:02:17.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.731+0000 7f72497b2740 1 -- 172.21.15.102:0/548149302 wait complete. 2024-04-25T02:02:17.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.731+0000 7f72497b2740 20 client.27131 trim_cache size 0 max 16384 2024-04-25T02:02:17.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:17 smithi110 ceph-mon[32848]: pgmap v214: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 32 MiB/s wr, 1.53k op/s 2024-04-25T02:02:17.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:17 smithi110 ceph-mon[32848]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T02:02:17.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:17 smithi193 ceph-mon[32981]: pgmap v214: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 32 MiB/s wr, 1.53k op/s 2024-04-25T02:02:17.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:17 smithi193 ceph-mon[32981]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T02:02:17.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.758+0000 7f72497b2740 1 Processor -- start 2024-04-25T02:02:17.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.758+0000 7f72497b2740 1 -- start start 2024-04-25T02:02:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.759+0000 7f72497b2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e0e00 0x560219800320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.759+0000 7f72497b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e1720 0x5602197e9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.759+0000 7f72497b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x56021980c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.759+0000 7f72497b2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f72401b64a0 con 0x5602197e0e00 2024-04-25T02:02:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.759+0000 7f72497b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f72401b3cb0 con 0x5602197ebfe0 2024-04-25T02:02:17.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.759+0000 7f72497b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f72401b1bc0 con 0x5602197e1720 2024-04-25T02:02:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.760+0000 7f7244a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x56021980c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.760+0000 7f7244a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x56021980c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56880/0 (socket says 172.21.15.102:56880) 2024-04-25T02:02:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.760+0000 7f723f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e1720 0x5602197e9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.760+0000 7f723f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e1720 0x5602197e9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:39446/0 (socket says 172.21.15.102:39446) 2024-04-25T02:02:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.760+0000 7f723f400640 1 -- 172.21.15.102:0/1319870815 learned_addr learned my addr 172.21.15.102:0/1319870815 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.760+0000 7f723fe00640 1 --2- 172.21.15.102:0/1319870815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e0e00 0x560219800320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.761+0000 7f7244a00640 1 -- 172.21.15.102:0/1319870815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e1720 msgr2=0x5602197e9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.761+0000 7f7244a00640 1 --2- 172.21.15.102:0/1319870815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e1720 0x5602197e9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.761+0000 7f7244a00640 1 -- 172.21.15.102:0/1319870815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e0e00 msgr2=0x560219800320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.761+0000 7f7244a00640 1 --2- 172.21.15.102:0/1319870815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e0e00 0x560219800320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.762+0000 7f7244a00640 1 -- 172.21.15.102:0/1319870815 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602197ddee0 con 0x5602197ebfe0 2024-04-25T02:02:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.762+0000 7f723fe00640 1 --2- 172.21.15.102:0/1319870815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e0e00 0x560219800320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.762+0000 7f7244a00640 1 --2- 172.21.15.102:0/1319870815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x56021980c620 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7238006010 tx=0x7f7238030fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=253bec986d651713 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.763+0000 7f722be00640 1 -- 172.21.15.102:0/1319870815 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238061020 con 0x5602197ebfe0 2024-04-25T02:02:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.763+0000 7f722be00640 1 -- 172.21.15.102:0/1319870815 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7238003d80 con 0x5602197ebfe0 2024-04-25T02:02:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.763+0000 7f722be00640 1 -- 172.21.15.102:0/1319870815 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238036070 con 0x5602197ebfe0 2024-04-25T02:02:17.769 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.764+0000 7f72497b2740 1 -- 172.21.15.102:0/1319870815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 msgr2=0x56021980c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:17.769 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.764+0000 7f72497b2740 1 --2- 172.21.15.102:0/1319870815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x56021980c620 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7238006010 tx=0x7f7238030fa0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.769 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.765+0000 7f72497b2740 1 -- 172.21.15.102:0/1319870815 shutdown_connections 2024-04-25T02:02:17.769 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.765+0000 7f72497b2740 1 --2- 172.21.15.102:0/1319870815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197ebfe0 0x56021980c620 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.769 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.765+0000 7f72497b2740 1 --2- 172.21.15.102:0/1319870815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197e1720 0x5602197e9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.765+0000 7f72497b2740 1 --2- 172.21.15.102:0/1319870815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e0e00 0x560219800320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.765+0000 7f72497b2740 1 -- 172.21.15.102:0/1319870815 >> 172.21.15.102:0/1319870815 conn(0x5602197dd310 msgr2=0x5602197ea7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.765+0000 7f72497b2740 1 -- 172.21.15.102:0/1319870815 shutdown_connections 2024-04-25T02:02:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.765+0000 7f72497b2740 1 -- 172.21.15.102:0/1319870815 wait complete. 2024-04-25T02:02:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.766+0000 7f72497b2740 1 -- start start 2024-04-25T02:02:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.766+0000 7f72497b2740 1 Processor -- start 2024-04-25T02:02:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.767+0000 7f72497b2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.767+0000 7f72497b2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.767+0000 7f72497b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 0x5602197e68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.767+0000 7f72497b2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f72401b2210 con 0x5602197e1720 2024-04-25T02:02:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.767+0000 7f72497b2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f72401b35e0 con 0x5602197e0e00 2024-04-25T02:02:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.767+0000 7f72497b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f72401b2860 con 0x5602197ebfe0 2024-04-25T02:02:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.767+0000 7f72497b2740 10 client.0 _ll_register_callbacks cb 0x7f722c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:17.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.767+0000 7f723fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.768+0000 7f723fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56896/0 (socket says 172.21.15.102:56896) 2024-04-25T02:02:17.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.768+0000 7f723fe00640 1 -- 172.21.15.102:0/4011162040 learned_addr learned my addr 172.21.15.102:0/4011162040 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:17.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.768+0000 7f7244a00640 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 0x5602197e68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.768+0000 7f723fe00640 1 -- 172.21.15.102:0/4011162040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 msgr2=0x5602197e68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:17.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.768+0000 7f723fe00640 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 0x5602197e68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.768+0000 7f723fe00640 1 -- 172.21.15.102:0/4011162040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 msgr2=0x5602197e6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.768+0000 7f723f400640 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.768+0000 7f723fe00640 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.768+0000 7f723fe00640 1 -- 172.21.15.102:0/4011162040 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56021986a040 con 0x5602197e0e00 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.769+0000 7f723f400640 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.769+0000 7f723fe00640 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e5e40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f723000c6a0 tx=0x7f7230009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f67b2dd548134f2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.769+0000 7f7229600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.769+0000 7f7229600640 1 -- 172.21.15.102:0/4011162040 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72300a4900 con 0x5602197e0e00 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.769+0000 7f7229600640 1 -- 172.21.15.102:0/4011162040 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f7230098300 con 0x5602197e0e00 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.769+0000 7f72497b2740 1 -- 172.21.15.102:0/4011162040 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7238003d80 con 0x5602197e0e00 2024-04-25T02:02:17.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f7229600640 1 -- 172.21.15.102:0/4011162040 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f72300cd140 con 0x5602197e0e00 2024-04-25T02:02:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f72497b2740 10 client.27137 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f72497b2740 1 -- 172.21.15.102:0/4011162040 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56021986b390 con 0x5602197e0e00 2024-04-25T02:02:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f7229600640 1 -- 172.21.15.102:0/4011162040 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7238003d80 con 0x5602197e0e00 2024-04-25T02:02:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f7229600640 1 -- 172.21.15.102:0/4011162040 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f72300cfbc0 con 0x5602197e0e00 2024-04-25T02:02:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f72497b2740 10 client.27137 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f72497b2740 10 client.27137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:17.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f72497b2740 20 client.27137 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f72497b2740 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56021986baf0 0x56021986df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f72497b2740 4 client.27137 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.770+0000 7f72497b2740 1 -- 172.21.15.102:0/4011162040 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560219751ff0 con 0x56021986baf0 2024-04-25T02:02:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.771+0000 7f723f400640 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56021986baf0 0x56021986df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.772+0000 7f723f400640 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56021986baf0 0x56021986df10 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c9d20a96d23f590 server_cookie=dab1fdf9cc6dc915 in_seq=0 out_seq=0 2024-04-25T02:02:17.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.772+0000 7f7229600640 10 client.27137 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.782+0000 7f7229600640 1 -- 172.21.15.102:0/4011162040 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560219751ff0 con 0x56021986baf0 2024-04-25T02:02:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.782+0000 7f7229600640 10 client.27137 handle_command_reply: tid=0 2024-04-25T02:02:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.782+0000 7f72497b2740 1 client.27137 shutdown 2024-04-25T02:02:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.782+0000 7f72497b2740 1 -- 172.21.15.102:0/4011162040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 msgr2=0x5602197e5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.782+0000 7f72497b2740 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e5e40 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f723000c6a0 tx=0x7f7230009f90 comp rx=0 tx=0).stop 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f72497b2740 1 -- 172.21.15.102:0/4011162040 shutdown_connections 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f72497b2740 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56021986baf0 0x56021986df10 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f72497b2740 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602197ebfe0 0x5602197e68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f72497b2740 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5602197e1720 0x5602197e6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f72497b2740 1 --2- 172.21.15.102:0/4011162040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5602197e0e00 0x5602197e5e40 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f72497b2740 1 -- 172.21.15.102:0/4011162040 >> 172.21.15.102:0/4011162040 conn(0x5602197dd310 msgr2=0x5602197eb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f7229600640 0 client.27137 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f72497b2740 1 -- 172.21.15.102:0/4011162040 shutdown_connections 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f72497b2740 1 -- 172.21.15.102:0/4011162040 wait complete. 2024-04-25T02:02:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.783+0000 7f72497b2740 20 client.27137 trim_cache size 0 max 16384 2024-04-25T02:02:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:02:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:02:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:02:17.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.784+0000 7f7246a00640 1 -- 172.21.15.102:0/3387618074 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f72240a2e20 msgr2=0x7f72240a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.784+0000 7f7246a00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f72240a2e20 0x7f72240a52e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f72380099a0 tx=0x7f7238009500 comp rx=0 tx=0).stop 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.784+0000 7f7246a00640 1 -- 172.21.15.102:0/3387618074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f724010c390 msgr2=0x7f72401a8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.784+0000 7f7246a00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f724010c390 0x7f72401a8610 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f723000e9c0 tx=0x7f723000ee80 comp rx=0 tx=0).stop 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.785+0000 7f7246a00640 1 -- 172.21.15.102:0/3387618074 shutdown_connections 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.785+0000 7f7246a00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f72240a2e20 0x7f72240a52e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.785+0000 7f7246a00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72401a2780 0x7f72401a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.785+0000 7f7246a00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7240111460 0x7f72401a8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.785+0000 7f7246a00640 1 --2- 172.21.15.102:0/3387618074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f724010c390 0x7f72401a8610 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:17.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.785+0000 7f7246a00640 1 -- 172.21.15.102:0/3387618074 >> 172.21.15.102:0/3387618074 conn(0x7f724006c9c0 msgr2=0x7f724010f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.785+0000 7f7246a00640 1 -- 172.21.15.102:0/3387618074 shutdown_connections 2024-04-25T02:02:17.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:17.785+0000 7f7246a00640 1 -- 172.21.15.102:0/3387618074 wait complete. 2024-04-25T02:02:18.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 385.0 s, 2136.4 tps, lat 47.371 ms stddev 55.226 2024-04-25T02:02:19.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:19 smithi102 ceph-mon[27796]: pgmap v216: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 34 MiB/s wr, 1.72k op/s 2024-04-25T02:02:19.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:19 smithi110 ceph-mon[32848]: pgmap v216: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 34 MiB/s wr, 1.72k op/s 2024-04-25T02:02:19.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:19 smithi193 ceph-mon[32981]: pgmap v216: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 34 MiB/s wr, 1.72k op/s 2024-04-25T02:02:21.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:21 smithi102 ceph-mon[27796]: pgmap v217: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 34 MiB/s wr, 1.72k op/s 2024-04-25T02:02:21.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:21 smithi110 ceph-mon[32848]: pgmap v217: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 34 MiB/s wr, 1.72k op/s 2024-04-25T02:02:21.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:21 smithi193 ceph-mon[32981]: pgmap v217: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 34 MiB/s wr, 1.72k op/s 2024-04-25T02:02:23.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 390.0 s, 2933.6 tps, lat 34.188 ms stddev 41.300 2024-04-25T02:02:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:23 smithi102 ceph-mon[27796]: pgmap v218: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 18 MiB/s wr, 1.12k op/s 2024-04-25T02:02:23.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:23 smithi110 ceph-mon[32848]: pgmap v218: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 18 MiB/s wr, 1.12k op/s 2024-04-25T02:02:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:23 smithi193 ceph-mon[32981]: pgmap v218: 97 pgs: 97 active+clean; 4.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 18 MiB/s wr, 1.12k op/s 2024-04-25T02:02:25.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:25 smithi102 ceph-mon[27796]: pgmap v219: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 7.3 MiB/s wr, 740 op/s 2024-04-25T02:02:25.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:25 smithi110 ceph-mon[32848]: pgmap v219: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 7.3 MiB/s wr, 740 op/s 2024-04-25T02:02:25.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:25 smithi193 ceph-mon[32981]: pgmap v219: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 7.3 MiB/s wr, 740 op/s 2024-04-25T02:02:27.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:27 smithi102 ceph-mon[27796]: pgmap v220: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 7.3 MiB/s wr, 740 op/s 2024-04-25T02:02:27.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:27 smithi110 ceph-mon[32848]: pgmap v220: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 7.3 MiB/s wr, 740 op/s 2024-04-25T02:02:27.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:27 smithi193 ceph-mon[32981]: pgmap v220: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 7.3 MiB/s wr, 740 op/s 2024-04-25T02:02:28.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 395.0 s, 2344.8 tps, lat 42.164 ms stddev 51.799 2024-04-25T02:02:29.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:29 smithi102 ceph-mon[27796]: pgmap v221: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.3 MiB/s wr, 728 op/s 2024-04-25T02:02:29.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:29 smithi110 ceph-mon[32848]: pgmap v221: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.3 MiB/s wr, 728 op/s 2024-04-25T02:02:29.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:29 smithi193 ceph-mon[32981]: pgmap v221: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.3 MiB/s wr, 728 op/s 2024-04-25T02:02:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.952+0000 7f565f600640 1 -- 172.21.15.102:0/3561173997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56581125c0 msgr2=0x7f56581149b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.952+0000 7f565f600640 1 --2- 172.21.15.102:0/3561173997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56581125c0 0x7f56581149b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5650009ca0 tx=0x7f565002a8c0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.952+0000 7f565f600640 1 -- 172.21.15.102:0/3561173997 shutdown_connections 2024-04-25T02:02:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.952+0000 7f565f600640 1 --2- 172.21.15.102:0/3561173997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56581125c0 0x7f56581149b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.952+0000 7f565f600640 1 --2- 172.21.15.102:0/3561173997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5658076590 0x7f5658112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.952+0000 7f565f600640 1 --2- 172.21.15.102:0/3561173997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5658075c70 0x7f5658076050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.958 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.952+0000 7f565f600640 1 -- 172.21.15.102:0/3561173997 >> 172.21.15.102:0/3561173997 conn(0x7f56580ff430 msgr2=0x7f5658101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.953+0000 7f565f600640 1 -- 172.21.15.102:0/3561173997 shutdown_connections 2024-04-25T02:02:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.953+0000 7f565f600640 1 -- 172.21.15.102:0/3561173997 wait complete. 2024-04-25T02:02:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.954+0000 7f565f600640 1 Processor -- start 2024-04-25T02:02:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.954+0000 7f565f600640 1 -- start start 2024-04-25T02:02:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f565f600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5658075c70 0x7f565819e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f565f600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5658076590 0x7f565819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f565f600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56581125c0 0x7f56581a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.960 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f565f600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5658118150 con 0x7f5658076590 2024-04-25T02:02:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f565f600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5658117fd0 con 0x7f56581125c0 2024-04-25T02:02:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f565f600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56581182d0 con 0x7f5658075c70 2024-04-25T02:02:30.961 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f565cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5658075c70 0x7f565819e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f565cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5658075c70 0x7f565819e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58468/0 (socket says 172.21.15.102:58468) 2024-04-25T02:02:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f5657e00640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5658076590 0x7f565819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.955+0000 7f565cc00640 1 -- 172.21.15.102:0/3146714216 learned_addr learned my addr 172.21.15.102:0/3146714216 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.956+0000 7f565d600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56581125c0 0x7f56581a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.956+0000 7f5657e00640 1 -- 172.21.15.102:0/3146714216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5658075c70 msgr2=0x7f565819e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.956+0000 7f5657e00640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5658075c70 0x7f565819e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.956+0000 7f5657e00640 1 -- 172.21.15.102:0/3146714216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56581125c0 msgr2=0x7f56581a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.956+0000 7f5657e00640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56581125c0 0x7f56581a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.962 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.956+0000 7f5657e00640 1 -- 172.21.15.102:0/3146714216 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56581a3500 con 0x7f5658076590 2024-04-25T02:02:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.956+0000 7f5657e00640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5658076590 0x7f565819ea50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f564400e9f0 tx=0x7f564400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.957+0000 7f5655600640 1 -- 172.21.15.102:0/3146714216 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564400ccc0 con 0x7f5658076590 2024-04-25T02:02:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.957+0000 7f565f600640 1 -- 172.21.15.102:0/3146714216 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56581a37f0 con 0x7f5658076590 2024-04-25T02:02:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.957+0000 7f5655600640 1 -- 172.21.15.102:0/3146714216 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5644005020 con 0x7f5658076590 2024-04-25T02:02:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.957+0000 7f5655600640 1 -- 172.21.15.102:0/3146714216 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564400ccc0 con 0x7f5658076590 2024-04-25T02:02:30.963 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.957+0000 7f565f600640 1 -- 172.21.15.102:0/3146714216 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56581ab080 con 0x7f5658076590 2024-04-25T02:02:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5662401740 1 Processor -- start 2024-04-25T02:02:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5655600640 1 -- 172.21.15.102:0/3146714216 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f56440051d0 con 0x7f5658076590 2024-04-25T02:02:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5662401740 1 -- start start 2024-04-25T02:02:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5662401740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aa9e00 0x55b717aaa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5662401740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aaa720 0x55b717ab4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5662401740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5662401740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b717a0dab0 con 0x55b717aa9e00 2024-04-25T02:02:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5662401740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b7179b6640 con 0x55b717ab4fe0 2024-04-25T02:02:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5662401740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b717a09a70 con 0x55b717aaa720 2024-04-25T02:02:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.963+0000 7f5657e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f565cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aaa720 0x55b717ab4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5657e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34096/0 (socket says 172.21.15.102:34096) 2024-04-25T02:02:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5657e00640 1 -- 172.21.15.102:0/473162389 learned_addr learned my addr 172.21.15.102:0/473162389 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f565d600640 1 --2- 172.21.15.102:0/473162389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aa9e00 0x55b717aaa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5657e00640 1 -- 172.21.15.102:0/473162389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aaa720 msgr2=0x55b717ab4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5657e00640 1 --2- 172.21.15.102:0/473162389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aaa720 0x55b717ab4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5657e00640 1 -- 172.21.15.102:0/473162389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aa9e00 msgr2=0x55b717aaa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5657e00640 1 --2- 172.21.15.102:0/473162389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aa9e00 0x55b717aaa1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5657e00640 1 -- 172.21.15.102:0/473162389 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b717722210 con 0x55b717ab4fe0 2024-04-25T02:02:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f565d600640 1 --2- 172.21.15.102:0/473162389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aa9e00 0x55b717aaa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:02:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5655600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56400a3020 0x7f56400a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5657e00640 1 --2- 172.21.15.102:0/473162389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab73d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f56440cc5a0 tx=0x7f56440d75f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad561c5d2b6a05a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.964+0000 7f5655600640 1 -- 172.21.15.102:0/3146714216 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f56440cef70 con 0x7f5658076590 2024-04-25T02:02:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.965+0000 7f5649c00640 1 -- 172.21.15.102:0/473162389 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56440976d0 con 0x55b717ab4fe0 2024-04-25T02:02:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.965+0000 7f565d600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56400a3020 0x7f56400a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.965+0000 7f5649c00640 1 -- 172.21.15.102:0/473162389 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f564409bb30 con 0x55b717ab4fe0 2024-04-25T02:02:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.965+0000 7f5649c00640 1 -- 172.21.15.102:0/473162389 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564409ad90 con 0x55b717ab4fe0 2024-04-25T02:02:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.966+0000 7f5662401740 1 -- 172.21.15.102:0/473162389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 msgr2=0x55b717ab73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.966+0000 7f5662401740 1 --2- 172.21.15.102:0/473162389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab73d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f56440cc5a0 tx=0x7f56440d75f0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.967+0000 7f5662401740 1 -- 172.21.15.102:0/473162389 shutdown_connections 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.967+0000 7f5662401740 1 --2- 172.21.15.102:0/473162389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab73d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.967+0000 7f5662401740 1 --2- 172.21.15.102:0/473162389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aaa720 0x55b717ab4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.967+0000 7f5662401740 1 --2- 172.21.15.102:0/473162389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aa9e00 0x55b717aaa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.967+0000 7f5662401740 1 -- 172.21.15.102:0/473162389 >> 172.21.15.102:0/473162389 conn(0x55b717aa6310 msgr2=0x55b717aa66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.967+0000 7f565d600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56400a3020 0x7f56400a54e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5650004af0 tx=0x7f5650005d80 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.967+0000 7f5662401740 1 -- 172.21.15.102:0/473162389 shutdown_connections 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.967+0000 7f5662401740 1 -- 172.21.15.102:0/473162389 wait complete. 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.968+0000 7f5662401740 1 -- start start 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.968+0000 7f5662401740 1 Processor -- start 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.968+0000 7f5662401740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa9e00 0x55b717ab1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.968+0000 7f5662401740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aaa720 0x55b717ab2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.968+0000 7f5662401740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717ab4fe0 0x55b717ab2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f5662401740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b717ab7b50 con 0x55b717ab4fe0 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f5662401740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b7179c9990 con 0x55b717aaa720 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f5662401740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b717ab7cd0 con 0x55b717aa9e00 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f5662401740 10 client.0 _ll_register_callbacks cb 0x55b7179cb950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f565cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa9e00 0x55b717ab1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f565cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa9e00 0x55b717ab1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:58490/0 (socket says 172.21.15.102:58490) 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f565cc00640 1 -- 172.21.15.102:0/3463654990 learned_addr learned my addr 172.21.15.102:0/3463654990 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f565d600640 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717ab4fe0 0x55b717ab2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f5657e00640 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aaa720 0x55b717ab2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f565cc00640 1 -- 172.21.15.102:0/3463654990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aaa720 msgr2=0x55b717ab2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f565cc00640 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aaa720 0x55b717ab2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f565cc00640 1 -- 172.21.15.102:0/3463654990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717ab4fe0 msgr2=0x55b717ab2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f565cc00640 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717ab4fe0 0x55b717ab2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.969+0000 7f565cc00640 1 -- 172.21.15.102:0/3463654990 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7177ee620 con 0x55b717aa9e00 2024-04-25T02:02:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.971+0000 7f565cc00640 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa9e00 0x55b717ab1ff0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f564c012860 tx=0x7f564c012d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f099700e10d97f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.971+0000 7f563e000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:02:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.971+0000 7f563e000640 1 -- 172.21.15.102:0/3463654990 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c00f920 con 0x55b717aa9e00 2024-04-25T02:02:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.971+0000 7f5662401740 1 -- 172.21.15.102:0/3463654990 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f564409bb30 con 0x55b717aa9e00 2024-04-25T02:02:30.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.971+0000 7f563e000640 1 -- 172.21.15.102:0/3463654990 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f564c00fac0 con 0x55b717aa9e00 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.972+0000 7f563e000640 1 -- 172.21.15.102:0/3463654990 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f564c00fcc0 con 0x55b717aa9e00 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.972+0000 7f5662401740 10 client.27106 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.972+0000 7f5662401740 1 -- 172.21.15.102:0/3463654990 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b717ac3b10 con 0x55b717aa9e00 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.972+0000 7f563e000640 1 -- 172.21.15.102:0/3463654990 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f564409bb30 con 0x55b717aa9e00 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.973+0000 7f563e000640 1 -- 172.21.15.102:0/3463654990 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f564c010430 con 0x55b717aa9e00 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.973+0000 7f5662401740 10 client.27106 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.973+0000 7f5662401740 10 client.27106 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.973+0000 7f5662401740 20 client.27106 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.973+0000 7f5662401740 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b717ac4fd0 0x55b717ac73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.973+0000 7f5662401740 4 client.27106 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.973+0000 7f5662401740 1 -- 172.21.15.102:0/3463654990 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b717a1aff0 con 0x55b717ac4fd0 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.973+0000 7f5657e00640 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b717ac4fd0 0x55b717ac73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.974+0000 7f5657e00640 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b717ac4fd0 0x55b717ac73c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0fa4093176495ce server_cookie=4e0ff1703929710b in_seq=0 out_seq=0 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.974+0000 7f563e000640 10 client.27106 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.974+0000 7f563e000640 1 -- 172.21.15.102:0/3463654990 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b717a1aff0 con 0x55b717ac4fd0 2024-04-25T02:02:30.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.974+0000 7f563e000640 10 client.27106 handle_command_reply: tid=0 2024-04-25T02:02:30.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f5662401740 1 client.27106 shutdown 2024-04-25T02:02:30.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f5662401740 1 -- 172.21.15.102:0/3463654990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa9e00 msgr2=0x55b717ab1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:30.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f5662401740 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa9e00 0x55b717ab1ff0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f564c012860 tx=0x7f564c012d20 comp rx=0 tx=0).stop 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f5662401740 1 -- 172.21.15.102:0/3463654990 shutdown_connections 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f5662401740 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b717ac4fd0 0x55b717ac73c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f5662401740 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717ab4fe0 0x55b717ab2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f5662401740 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aaa720 0x55b717ab2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f5662401740 1 --2- 172.21.15.102:0/3463654990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa9e00 0x55b717ab1ff0 secure :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f564c012860 tx=0x7f564c012d20 comp rx=0 tx=0).stop 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f5662401740 1 -- 172.21.15.102:0/3463654990 >> 172.21.15.102:0/3463654990 conn(0x55b717aa6310 msgr2=0x55b717aac090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.975+0000 7f563e000640 0 client.27106 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.976+0000 7f5662401740 1 -- 172.21.15.102:0/3463654990 shutdown_connections 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.976+0000 7f5662401740 1 -- 172.21.15.102:0/3463654990 wait complete. 2024-04-25T02:02:30.983 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.976+0000 7f5662401740 20 client.27106 trim_cache size 0 max 16384 2024-04-25T02:02:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.996+0000 7f5662401740 1 Processor -- start 2024-04-25T02:02:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.997+0000 7f5662401740 1 -- start start 2024-04-25T02:02:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.997+0000 7f5662401740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717ac0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.997+0000 7f5662401740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.998+0000 7f565cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717ac0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.998+0000 7f565cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717ac0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48486/0 (socket says 172.21.15.102:48486) 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.998+0000 7f565cc00640 1 -- 172.21.15.102:0/3966185004 learned_addr learned my addr 172.21.15.102:0/3966185004 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.998+0000 7f5662401740 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa5300 0x55b717acccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.998+0000 7f5657e00640 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.998+0000 7f5662401740 1 -- 172.21.15.102:0/3966185004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f56581b63a0 con 0x55b717aaa720 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.998+0000 7f5662401740 1 -- 172.21.15.102:0/3966185004 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f56581b3bb0 con 0x55b717ab4fe0 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.998+0000 7f5662401740 1 -- 172.21.15.102:0/3966185004 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56581b1a20 con 0x55b717aa5300 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.999+0000 7f565d600640 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa5300 0x55b717acccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.999+0000 7f5657e00640 1 -- 172.21.15.102:0/3966185004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa5300 msgr2=0x55b717acccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.999+0000 7f5657e00640 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa5300 0x55b717acccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.999+0000 7f5657e00640 1 -- 172.21.15.102:0/3966185004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 msgr2=0x55b717ac0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.999+0000 7f5657e00640 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717ac0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.999+0000 7f5657e00640 1 -- 172.21.15.102:0/3966185004 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b717ab0fe0 con 0x55b717ab4fe0 2024-04-25T02:02:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:30.999+0000 7f565cc00640 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717ac0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.000+0000 7f5657e00640 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab28c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x55b717aa74a0 tx=0x7f5644097040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c94c68f3c9ce4fdb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.000+0000 7f5649200640 1 -- 172.21.15.102:0/3966185004 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564401c040 con 0x55b717ab4fe0 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.000+0000 7f5649200640 1 -- 172.21.15.102:0/3966185004 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f564409b430 con 0x55b717ab4fe0 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.000+0000 7f5649200640 1 -- 172.21.15.102:0/3966185004 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564409af70 con 0x55b717ab4fe0 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.000+0000 7f5662401740 1 -- 172.21.15.102:0/3966185004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 msgr2=0x55b717ab28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.000+0000 7f5662401740 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab28c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x55b717aa74a0 tx=0x7f5644097040 comp rx=0 tx=0).stop 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.001+0000 7f5662401740 1 -- 172.21.15.102:0/3966185004 shutdown_connections 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.001+0000 7f5662401740 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717aa5300 0x55b717acccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.001+0000 7f5662401740 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717ab4fe0 0x55b717ab28c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.001+0000 7f5662401740 1 --2- 172.21.15.102:0/3966185004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717ac0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.001+0000 7f5662401740 1 -- 172.21.15.102:0/3966185004 >> 172.21.15.102:0/3966185004 conn(0x55b717aa6310 msgr2=0x55b717aa4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.001+0000 7f5662401740 1 -- 172.21.15.102:0/3966185004 shutdown_connections 2024-04-25T02:02:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.001+0000 7f5662401740 1 -- 172.21.15.102:0/3966185004 wait complete. 2024-04-25T02:02:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.001+0000 7f5662401740 1 -- start start 2024-04-25T02:02:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.001+0000 7f5662401740 1 Processor -- start 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.002+0000 7f5662401740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aa5300 0x55b717aaf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.002+0000 7f5662401740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717aaf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.002+0000 7f5662401740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717ab4fe0 0x55b717b2a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.002+0000 7f565cc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aa5300 0x55b717aaf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f5662401740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f56581b2070 con 0x55b717aaa720 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f5662401740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f56581b34e0 con 0x55b717aa5300 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f5662401740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f56581b2740 con 0x55b717ab4fe0 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f565cc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aa5300 0x55b717aaf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34116/0 (socket says 172.21.15.102:34116) 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f565cc00640 1 -- 172.21.15.102:0/2693061838 learned_addr learned my addr 172.21.15.102:0/2693061838 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f5662401740 10 client.0 _ll_register_callbacks cb 0x7f564c009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f565d600640 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717ab4fe0 0x55b717b2a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f565cc00640 1 -- 172.21.15.102:0/2693061838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717ab4fe0 msgr2=0x55b717b2a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f5657e00640 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717aaf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f565cc00640 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717ab4fe0 0x55b717b2a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f565cc00640 1 -- 172.21.15.102:0/2693061838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 msgr2=0x55b717aaf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f565cc00640 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717aaf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f565cc00640 1 -- 172.21.15.102:0/2693061838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b717b2aa60 con 0x55b717aa5300 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f565d600640 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717ab4fe0 0x55b717b2a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.003+0000 7f5657e00640 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717aaf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.004+0000 7f565cc00640 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aa5300 0x55b717aaf3c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f564c005e00 tx=0x7f564c0122e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f33d15b311c6aa61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.004+0000 7f563e000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.004+0000 7f563e000640 1 -- 172.21.15.102:0/2693061838 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c0123d0 con 0x55b717aa5300 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.004+0000 7f563e000640 1 -- 172.21.15.102:0/2693061838 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f564c00f920 con 0x55b717aa5300 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.004+0000 7f5662401740 1 -- 172.21.15.102:0/2693061838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b717a15340 con 0x55b717aa5300 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.004+0000 7f563e000640 1 -- 172.21.15.102:0/2693061838 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f564c00fb20 con 0x55b717aa5300 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.004+0000 7f5662401740 10 client.27155 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.004+0000 7f5662401740 1 -- 172.21.15.102:0/2693061838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b717b2bd10 con 0x55b717aa5300 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.005+0000 7f563e000640 1 -- 172.21.15.102:0/2693061838 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b717a15340 con 0x55b717aa5300 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.005+0000 7f563e000640 1 -- 172.21.15.102:0/2693061838 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f564c0038c0 con 0x55b717aa5300 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.005+0000 7f5662401740 10 client.27155 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.005+0000 7f5662401740 10 client.27155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.005+0000 7f5662401740 20 client.27155 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.005+0000 7f5662401740 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b717b2c4f0 0x55b717b2e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.005+0000 7f5662401740 4 client.27155 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.005+0000 7f5662401740 1 -- 172.21.15.102:0/2693061838 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b7179937a0 con 0x55b717b2c4f0 2024-04-25T02:02:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.005+0000 7f5657e00640 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b717b2c4f0 0x55b717b2e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.007+0000 7f563e000640 10 client.27155 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.007+0000 7f5657e00640 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b717b2c4f0 0x55b717b2e910 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca099a6bd0cf054d server_cookie=75363ae9a1d44478 in_seq=0 out_seq=0 2024-04-25T02:02:31.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.015+0000 7f563e000640 1 -- 172.21.15.102:0/2693061838 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b7179937a0 con 0x55b717b2c4f0 2024-04-25T02:02:31.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.015+0000 7f563e000640 10 client.27155 handle_command_reply: tid=0 2024-04-25T02:02:31.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.015+0000 7f5662401740 1 client.27155 shutdown 2024-04-25T02:02:31.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.016+0000 7f5662401740 1 -- 172.21.15.102:0/2693061838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aa5300 msgr2=0x55b717aaf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:31.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.016+0000 7f5662401740 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aa5300 0x55b717aaf3c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f564c005e00 tx=0x7f564c0122e0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.016+0000 7f5662401740 1 -- 172.21.15.102:0/2693061838 shutdown_connections 2024-04-25T02:02:31.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.016+0000 7f5662401740 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b717b2c4f0 0x55b717b2e910 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.016+0000 7f5662401740 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b717ab4fe0 0x55b717b2a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.016+0000 7f5662401740 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b717aaa720 0x55b717aaf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.016+0000 7f5662401740 1 --2- 172.21.15.102:0/2693061838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b717aa5300 0x55b717aaf3c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.016+0000 7f5662401740 1 -- 172.21.15.102:0/2693061838 >> 172.21.15.102:0/2693061838 conn(0x55b717aa6310 msgr2=0x55b717aa56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:31.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.016+0000 7f563e000640 0 client.27155 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:31.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.017+0000 7f5662401740 1 -- 172.21.15.102:0/2693061838 shutdown_connections 2024-04-25T02:02:31.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.017+0000 7f5662401740 1 -- 172.21.15.102:0/2693061838 wait complete. 2024-04-25T02:02:31.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.017+0000 7f5662401740 20 client.27155 trim_cache size 0 max 16384 2024-04-25T02:02:31.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:02:31.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:02:31.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:02:31.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 -- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56400a3020 msgr2=0x7f56400a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56400a3020 0x7f56400a54e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5650004af0 tx=0x7f5650005d80 comp rx=0 tx=0).stop 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 -- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5658076590 msgr2=0x7f565819ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5658076590 0x7f565819ea50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f564400e9f0 tx=0x7f564400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 -- 172.21.15.102:0/3146714216 shutdown_connections 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f56400a3020 0x7f56400a54e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56581125c0 0x7f56581a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5658076590 0x7f565819ea50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 --2- 172.21.15.102:0/3146714216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5658075c70 0x7f565819e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 -- 172.21.15.102:0/3146714216 >> 172.21.15.102:0/3146714216 conn(0x7f56580ff430 msgr2=0x7f5658101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 -- 172.21.15.102:0/3146714216 shutdown_connections 2024-04-25T02:02:31.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:31.018+0000 7f565f600640 1 -- 172.21.15.102:0/3146714216 wait complete. 2024-04-25T02:02:31.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:31 smithi102 ceph-mon[27796]: pgmap v222: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.4 MiB/s wr, 638 op/s 2024-04-25T02:02:31.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:31 smithi110 ceph-mon[32848]: pgmap v222: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.4 MiB/s wr, 638 op/s 2024-04-25T02:02:31.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:31 smithi193 ceph-mon[32981]: pgmap v222: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.4 MiB/s wr, 638 op/s 2024-04-25T02:02:33.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 400.0 s, 2118.8 tps, lat 47.414 ms stddev 57.554 2024-04-25T02:02:33.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:33 smithi102 ceph-mon[27796]: pgmap v223: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.3 MiB/s wr, 833 op/s 2024-04-25T02:02:33.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:33 smithi110 ceph-mon[32848]: pgmap v223: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.3 MiB/s wr, 833 op/s 2024-04-25T02:02:33.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:33 smithi193 ceph-mon[32981]: pgmap v223: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.3 MiB/s wr, 833 op/s 2024-04-25T02:02:35.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:35 smithi102 ceph-mon[27796]: pgmap v224: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.9 MiB/s wr, 710 op/s 2024-04-25T02:02:35.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:35 smithi110 ceph-mon[32848]: pgmap v224: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.9 MiB/s wr, 710 op/s 2024-04-25T02:02:35.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:35 smithi193 ceph-mon[32981]: pgmap v224: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.9 MiB/s wr, 710 op/s 2024-04-25T02:02:37.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:37 smithi102 ceph-mon[27796]: pgmap v225: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.4 MiB/s wr, 576 op/s 2024-04-25T02:02:37.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:37 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:02:37.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:37 smithi110 ceph-mon[32848]: pgmap v225: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.4 MiB/s wr, 576 op/s 2024-04-25T02:02:37.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:37 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:02:37.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:37 smithi193 ceph-mon[32981]: pgmap v225: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.4 MiB/s wr, 576 op/s 2024-04-25T02:02:37.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:37 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:02:38.073 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 405.0 s, 1402.0 tps, lat 71.489 ms stddev 143.469 2024-04-25T02:02:39.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:39 smithi102 ceph-mon[27796]: pgmap v226: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 1.89k op/s 2024-04-25T02:02:39.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:39 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:02:39.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:39 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:02:39.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:39 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:02:39.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:39 smithi110 ceph-mon[32848]: pgmap v226: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 1.89k op/s 2024-04-25T02:02:39.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:39 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:02:39.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:39 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:02:39.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:39 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:02:39.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:39 smithi193 ceph-mon[32981]: pgmap v226: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 1.89k op/s 2024-04-25T02:02:39.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:39 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:02:39.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:39 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:02:39.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:39 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:02:41.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:41 smithi102 ceph-mon[27796]: pgmap v227: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 MiB/s wr, 1.61k op/s 2024-04-25T02:02:41.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:41 smithi110 ceph-mon[32848]: pgmap v227: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 MiB/s wr, 1.61k op/s 2024-04-25T02:02:41.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:41 smithi193 ceph-mon[32981]: pgmap v227: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 MiB/s wr, 1.61k op/s 2024-04-25T02:02:43.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 410.0 s, 1991.4 tps, lat 49.882 ms stddev 56.592 2024-04-25T02:02:43.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:43 smithi102 ceph-mon[27796]: pgmap v228: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-25T02:02:43.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:43 smithi110 ceph-mon[32848]: pgmap v228: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-25T02:02:43.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:43 smithi193 ceph-mon[32981]: pgmap v228: 97 pgs: 97 active+clean; 4.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 MiB/s wr, 1.82k op/s 2024-04-25T02:02:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.220+0000 7febc3a00640 1 -- 172.21.15.102:0/1518612242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 msgr2=0x7febbc115a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.220+0000 7febc3a00640 1 --2- 172.21.15.102:0/1518612242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 0x7febbc115a80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7febb4009ca0 tx=0x7febb402a8c0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.220+0000 7febc3a00640 1 -- 172.21.15.102:0/1518612242 shutdown_connections 2024-04-25T02:02:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.220+0000 7febc3a00640 1 --2- 172.21.15.102:0/1518612242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 0x7febbc115a80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.220+0000 7febc3a00640 1 --2- 172.21.15.102:0/1518612242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febbc10aa90 0x7febbc113150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.220+0000 7febc3a00640 1 --2- 172.21.15.102:0/1518612242 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febbc10a0e0 0x7febbc10a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.220+0000 7febc3a00640 1 -- 172.21.15.102:0/1518612242 >> 172.21.15.102:0/1518612242 conn(0x7febbc06ccb0 msgr2=0x7febbc072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.221+0000 7febc3a00640 1 -- 172.21.15.102:0/1518612242 shutdown_connections 2024-04-25T02:02:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.221+0000 7febc3a00640 1 -- 172.21.15.102:0/1518612242 wait complete. 2024-04-25T02:02:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.223+0000 7febc3a00640 1 Processor -- start 2024-04-25T02:02:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.223+0000 7febc3a00640 1 -- start start 2024-04-25T02:02:44.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.224+0000 7febc3a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febbc10a0e0 0x7febbc1a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.224+0000 7febc3a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febbc10aa90 0x7febbc1a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.224+0000 7febc3a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 0x7febbc1a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 0x7febbc1a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 0x7febbc1a6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50142/0 (socket says 172.21.15.102:50142) 2024-04-25T02:02:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1a00640 1 -- 172.21.15.102:0/1347986568 learned_addr learned my addr 172.21.15.102:0/1347986568 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7febbc119260 con 0x7febbc10a0e0 2024-04-25T02:02:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1190e0 con 0x7febbc10aa90 2024-04-25T02:02:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1193e0 con 0x7febbc113690 2024-04-25T02:02:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febb3e00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febbc10aa90 0x7febbc1a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1a00640 1 -- 172.21.15.102:0/1347986568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febbc10aa90 msgr2=0x7febbc1a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1a00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febbc10aa90 0x7febbc1a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1000640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febbc10a0e0 0x7febbc1a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1a00640 1 -- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febbc10a0e0 msgr2=0x7febbc1a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1a00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febbc10a0e0 0x7febbc1a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1a00640 1 -- 172.21.15.102:0/1347986568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc1a6930 con 0x7febbc113690 2024-04-25T02:02:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.225+0000 7febc1000640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febbc10a0e0 0x7febbc1a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:02:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.226+0000 7febc1a00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 0x7febbc1a6250 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7febb4030c30 tx=0x7febb4030c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:44.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.227+0000 7febba000640 1 -- 172.21.15.102:0/1347986568 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb4003b70 con 0x7febbc113690 2024-04-25T02:02:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.227+0000 7febba000640 1 -- 172.21.15.102:0/1347986568 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febb402f380 con 0x7febbc113690 2024-04-25T02:02:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.228+0000 7febba000640 1 -- 172.21.15.102:0/1347986568 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb4004550 con 0x7febbc113690 2024-04-25T02:02:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.228+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febbc1a6bc0 con 0x7febbc113690 2024-04-25T02:02:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.228+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febbc19ef80 con 0x7febbc113690 2024-04-25T02:02:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.234+0000 7febc678c740 1 Processor -- start 2024-04-25T02:02:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.236+0000 7febc678c740 1 -- start start 2024-04-25T02:02:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.236+0000 7febc678c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 0x55fb66d0f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.236+0000 7febc678c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d222b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.236+0000 7febc678c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d227f0 0x55fb66d24be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.236+0000 7febc678c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55fb66c72ab0 con 0x55fb66d0f660 2024-04-25T02:02:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.236+0000 7febc678c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fb66c1b640 con 0x55fb66d227f0 2024-04-25T02:02:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.236+0000 7febc678c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fb66c6ea70 con 0x55fb66d0ed40 2024-04-25T02:02:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.235+0000 7febba000640 1 -- 172.21.15.102:0/1347986568 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7febb40046f0 con 0x7febbc113690 2024-04-25T02:02:44.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.237+0000 7febb3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 0x55fb66d0f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.237+0000 7febb3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 0x55fb66d0f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50150/0 (socket says 172.21.15.102:50150) 2024-04-25T02:02:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.237+0000 7febb3e00640 1 -- 172.21.15.102:0/1876891612 learned_addr learned my addr 172.21.15.102:0/1876891612 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.237+0000 7febb3e00640 1 --2- 172.21.15.102:0/1876891612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d227f0 0x55fb66d24be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.238+0000 7febc1000640 1 --2- 172.21.15.102:0/1876891612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d222b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.238+0000 7febc1000640 1 -- 172.21.15.102:0/1876891612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 msgr2=0x55fb66d0f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.238+0000 7febc1000640 1 --2- 172.21.15.102:0/1876891612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 0x55fb66d0f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.238+0000 7febc1000640 1 -- 172.21.15.102:0/1876891612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d227f0 msgr2=0x55fb66d24be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.238+0000 7febc1000640 1 --2- 172.21.15.102:0/1876891612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d227f0 0x55fb66d24be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.238+0000 7febc1000640 1 -- 172.21.15.102:0/1876891612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb66987210 con 0x55fb66d0f660 2024-04-25T02:02:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.239+0000 7febc1000640 1 --2- 172.21.15.102:0/1876891612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d222b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7febac009c40 tx=0x7febac010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22988e5e1125a7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.240+0000 7febb2000640 1 -- 172.21.15.102:0/1876891612 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac004030 con 0x55fb66d0f660 2024-04-25T02:02:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.240+0000 7febb2000640 1 -- 172.21.15.102:0/1876891612 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febac0041d0 con 0x55fb66d0f660 2024-04-25T02:02:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.240+0000 7febb2000640 1 -- 172.21.15.102:0/1876891612 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac0126b0 con 0x55fb66d0f660 2024-04-25T02:02:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.241+0000 7febc678c740 1 -- 172.21.15.102:0/1876891612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 msgr2=0x55fb66d222b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:44.247 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.241+0000 7febc678c740 1 --2- 172.21.15.102:0/1876891612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d222b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7febac009c40 tx=0x7febac010740 comp rx=0 tx=0).stop 2024-04-25T02:02:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.243+0000 7febc678c740 1 -- 172.21.15.102:0/1876891612 shutdown_connections 2024-04-25T02:02:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.243+0000 7febc678c740 1 --2- 172.21.15.102:0/1876891612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d227f0 0x55fb66d24be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.243+0000 7febc678c740 1 --2- 172.21.15.102:0/1876891612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d222b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.243+0000 7febba000640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feb9c0a2f80 0x7feb9c0a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.244+0000 7febc678c740 1 --2- 172.21.15.102:0/1876891612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 0x55fb66d0f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.244+0000 7febba000640 1 -- 172.21.15.102:0/1347986568 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7febb40f0620 con 0x7febbc113690 2024-04-25T02:02:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.244+0000 7febc678c740 1 -- 172.21.15.102:0/1876891612 >> 172.21.15.102:0/1876891612 conn(0x55fb66d0b310 msgr2=0x55fb66d0b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:44.250 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.244+0000 7febb3e00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feb9c0a2f80 0x7feb9c0a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.246+0000 7febb3e00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feb9c0a2f80 0x7feb9c0a5440 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55fb66d0c4a0 tx=0x7feba4004730 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.246+0000 7febc678c740 1 -- 172.21.15.102:0/1876891612 shutdown_connections 2024-04-25T02:02:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.247+0000 7febc678c740 1 -- 172.21.15.102:0/1876891612 wait complete. 2024-04-25T02:02:44.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.248+0000 7febc678c740 1 -- start start 2024-04-25T02:02:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.248+0000 7febc678c740 1 Processor -- start 2024-04-25T02:02:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc678c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d1c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc678c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d1c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc678c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d227f0 0x55fb66d22070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc678c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55fb66d11ce0 con 0x55fb66d0f660 2024-04-25T02:02:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc678c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55fb66c2e990 con 0x55fb66d0ed40 2024-04-25T02:02:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc1000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d1c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc678c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fb66d11e60 con 0x55fb66d227f0 2024-04-25T02:02:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febb3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d1c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc1a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d227f0 0x55fb66d22070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc1000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d1c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:60602/0 (socket says 172.21.15.102:60602) 2024-04-25T02:02:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc1a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d227f0 0x55fb66d22070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50158/0 (socket says 172.21.15.102:50158) 2024-04-25T02:02:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.249+0000 7febc1000640 1 -- 172.21.15.102:0/4077549073 learned_addr learned my addr 172.21.15.102:0/4077549073 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.250+0000 7febc678c740 10 client.0 _ll_register_callbacks cb 0x55fb66c30950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.250+0000 7febc1a00640 1 -- 172.21.15.102:0/4077549073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 msgr2=0x55fb66d1c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.250+0000 7febc1a00640 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d1c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.250+0000 7febc1a00640 1 -- 172.21.15.102:0/4077549073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 msgr2=0x55fb66d1c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.250+0000 7febc1a00640 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d1c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.250+0000 7febc1a00640 1 -- 172.21.15.102:0/4077549073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb66a53620 con 0x55fb66d227f0 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.250+0000 7febb3e00640 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d1c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.251+0000 7febc1a00640 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d227f0 0x55fb66d22070 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7febb40ede20 tx=0x7febb40bcab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6280695a2f43658 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.251+0000 7febaaa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:02:44.258 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.251+0000 7febaaa00640 1 -- 172.21.15.102:0/4077549073 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb40d3070 con 0x55fb66d227f0 2024-04-25T02:02:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.251+0000 7febaaa00640 1 -- 172.21.15.102:0/4077549073 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7febb40edf90 con 0x55fb66d227f0 2024-04-25T02:02:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.251+0000 7febc678c740 1 -- 172.21.15.102:0/4077549073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febac0041d0 con 0x55fb66d227f0 2024-04-25T02:02:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.252+0000 7febaaa00640 1 -- 172.21.15.102:0/4077549073 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7febb40ee190 con 0x55fb66d227f0 2024-04-25T02:02:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.252+0000 7febc678c740 10 client.27124 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.252+0000 7febc678c740 1 -- 172.21.15.102:0/4077549073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55fb66d11ce0 con 0x55fb66d227f0 2024-04-25T02:02:44.259 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.252+0000 7febaaa00640 1 -- 172.21.15.102:0/4077549073 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febac0041d0 con 0x55fb66d227f0 2024-04-25T02:02:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.253+0000 7febaaa00640 1 -- 172.21.15.102:0/4077549073 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7febb4004b40 con 0x55fb66d227f0 2024-04-25T02:02:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.253+0000 7febc678c740 10 client.27124 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.253+0000 7febc678c740 10 client.27124 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.254+0000 7febc678c740 20 client.27124 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.254+0000 7febc678c740 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fb66d3af80 0x55fb66d3d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.254+0000 7febc678c740 4 client.27124 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:44.260 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.255+0000 7febc678c740 1 -- 172.21.15.102:0/4077549073 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb66c7fff0 con 0x55fb66d3af80 2024-04-25T02:02:44.261 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.255+0000 7febc1000640 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fb66d3af80 0x55fb66d3d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.262 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.256+0000 7febc1000640 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fb66d3af80 0x55fb66d3d3e0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d50d23f14f346356 server_cookie=c1d360073140ae54 in_seq=0 out_seq=0 2024-04-25T02:02:44.262 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.256+0000 7febaaa00640 10 client.27124 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:44.263 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.257+0000 7febaaa00640 1 -- 172.21.15.102:0/4077549073 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb66c7fff0 con 0x55fb66d3af80 2024-04-25T02:02:44.263 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.257+0000 7febaaa00640 10 client.27124 handle_command_reply: tid=0 2024-04-25T02:02:44.263 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.257+0000 7febc678c740 1 client.27124 shutdown 2024-04-25T02:02:44.263 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.257+0000 7febc678c740 1 -- 172.21.15.102:0/4077549073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d227f0 msgr2=0x55fb66d22070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.257+0000 7febc678c740 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d227f0 0x55fb66d22070 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7febb40ede20 tx=0x7febb40bcab0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.258+0000 7febc678c740 1 -- 172.21.15.102:0/4077549073 shutdown_connections 2024-04-25T02:02:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.258+0000 7febc678c740 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fb66d3af80 0x55fb66d3d3e0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.258+0000 7febc678c740 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d227f0 0x55fb66d22070 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.258+0000 7febc678c740 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d0f660 0x55fb66d1c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.258+0000 7febc678c740 1 --2- 172.21.15.102:0/4077549073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d1c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.258+0000 7febc678c740 1 -- 172.21.15.102:0/4077549073 >> 172.21.15.102:0/4077549073 conn(0x55fb66d0b310 msgr2=0x55fb66d11090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.258+0000 7febaaa00640 0 client.27124 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.259+0000 7febc678c740 1 -- 172.21.15.102:0/4077549073 shutdown_connections 2024-04-25T02:02:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.259+0000 7febc678c740 1 -- 172.21.15.102:0/4077549073 wait complete. 2024-04-25T02:02:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.260+0000 7febc678c740 20 client.27124 trim_cache size 0 max 16384 2024-04-25T02:02:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.289+0000 7febc678c740 1 Processor -- start 2024-04-25T02:02:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.289+0000 7febc678c740 1 -- start start 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.289+0000 7febc678c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d42cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.289+0000 7febc678c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0f660 0x55fb66d43230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc678c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 0x55fb66d4e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc678c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1b6340 con 0x55fb66d227f0 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc678c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1b3b50 con 0x55fb66d0ed40 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc678c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1b19c0 con 0x55fb66d0f660 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc1000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d42cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febb3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0f660 0x55fb66d43230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc1000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d42cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:60614/0 (socket says 172.21.15.102:60614) 2024-04-25T02:02:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febb3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0f660 0x55fb66d43230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50168/0 (socket says 172.21.15.102:50168) 2024-04-25T02:02:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc1000640 1 -- 172.21.15.102:0/655821894 learned_addr learned my addr 172.21.15.102:0/655821894 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc1a00640 1 --2- 172.21.15.102:0/655821894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 0x55fb66d4e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc1000640 1 -- 172.21.15.102:0/655821894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0f660 msgr2=0x55fb66d43230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc1000640 1 --2- 172.21.15.102:0/655821894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0f660 0x55fb66d43230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc1000640 1 -- 172.21.15.102:0/655821894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 msgr2=0x55fb66d4e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.290+0000 7febc1000640 1 --2- 172.21.15.102:0/655821894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 0x55fb66d4e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.291+0000 7febc1000640 1 -- 172.21.15.102:0/655821894 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb66d0f490 con 0x55fb66d0ed40 2024-04-25T02:02:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.291+0000 7febc1a00640 1 --2- 172.21.15.102:0/655821894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 0x55fb66d4e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:02:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.291+0000 7febc1000640 1 --2- 172.21.15.102:0/655821894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d42cf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7febac009b60 tx=0x7febac003ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f514a322679b910 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.292+0000 7febb1600640 1 -- 172.21.15.102:0/655821894 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac0052d0 con 0x55fb66d0ed40 2024-04-25T02:02:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.292+0000 7febb1600640 1 -- 172.21.15.102:0/655821894 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febac003ad0 con 0x55fb66d0ed40 2024-04-25T02:02:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.292+0000 7febb1600640 1 -- 172.21.15.102:0/655821894 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac02a3d0 con 0x55fb66d0ed40 2024-04-25T02:02:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.294+0000 7febc678c740 1 -- 172.21.15.102:0/655821894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 msgr2=0x55fb66d42cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.294+0000 7febc678c740 1 --2- 172.21.15.102:0/655821894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d42cf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7febac009b60 tx=0x7febac003ea0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.294+0000 7febc678c740 1 -- 172.21.15.102:0/655821894 shutdown_connections 2024-04-25T02:02:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.294+0000 7febc678c740 1 --2- 172.21.15.102:0/655821894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 0x55fb66d4e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.295+0000 7febc678c740 1 --2- 172.21.15.102:0/655821894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0f660 0x55fb66d43230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.295+0000 7febc678c740 1 --2- 172.21.15.102:0/655821894 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0ed40 0x55fb66d42cf0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.295+0000 7febc678c740 1 -- 172.21.15.102:0/655821894 >> 172.21.15.102:0/655821894 conn(0x55fb66d0b310 msgr2=0x55fb66d0a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.295+0000 7febc678c740 1 -- 172.21.15.102:0/655821894 shutdown_connections 2024-04-25T02:02:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.296+0000 7febc678c740 1 -- 172.21.15.102:0/655821894 wait complete. 2024-04-25T02:02:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.297+0000 7febc678c740 1 -- start start 2024-04-25T02:02:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.297+0000 7febc678c740 1 Processor -- start 2024-04-25T02:02:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febc678c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 0x55fb66d49e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febc678c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0f660 0x55fb66d4a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febc678c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 0x55fb66d4a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febc678c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1b2010 con 0x55fb66d227f0 2024-04-25T02:02:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febc678c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1b3480 con 0x55fb66d0f660 2024-04-25T02:02:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febc678c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1b26e0 con 0x55fb66d0ed40 2024-04-25T02:02:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febc678c740 10 client.0 _ll_register_callbacks cb 0x7febb4031aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febb3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0f660 0x55fb66d4a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febb3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0f660 0x55fb66d4a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:60630/0 (socket says 172.21.15.102:60630) 2024-04-25T02:02:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.298+0000 7febb3e00640 1 -- 172.21.15.102:0/2591515378 learned_addr learned my addr 172.21.15.102:0/2591515378 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.299+0000 7febb3e00640 1 -- 172.21.15.102:0/2591515378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 msgr2=0x55fb66d49e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:02:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.299+0000 7febb3e00640 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 0x55fb66d49e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.299+0000 7febb3e00640 1 -- 172.21.15.102:0/2591515378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 msgr2=0x55fb66d4a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:02:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.299+0000 7febc1a00640 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 0x55fb66d4a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.299+0000 7febb3e00640 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 0x55fb66d4a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.299+0000 7febb3e00640 1 -- 172.21.15.102:0/2591515378 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb66da0140 con 0x55fb66d0f660 2024-04-25T02:02:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.300+0000 7febb3e00640 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0f660 0x55fb66d4a3a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7feba400ffd0 tx=0x7feba4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0d9411eab22207f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.300+0000 7febaaa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:02:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.300+0000 7febaaa00640 1 -- 172.21.15.102:0/2591515378 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba4013a80 con 0x55fb66d0f660 2024-04-25T02:02:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.301+0000 7febc678c740 1 -- 172.21.15.102:0/2591515378 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febac003ad0 con 0x55fb66d0f660 2024-04-25T02:02:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.301+0000 7febaaa00640 1 -- 172.21.15.102:0/2591515378 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7feba401c070 con 0x55fb66d0f660 2024-04-25T02:02:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.301+0000 7febaaa00640 1 -- 172.21.15.102:0/2591515378 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7feba4013a80 con 0x55fb66d0f660 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.301+0000 7febc678c740 10 client.27173 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.301+0000 7febc678c740 1 -- 172.21.15.102:0/2591515378 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55fb66da1440 con 0x55fb66d0f660 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.301+0000 7febaaa00640 1 -- 172.21.15.102:0/2591515378 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febac003ad0 con 0x55fb66d0f660 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.302+0000 7febaaa00640 1 -- 172.21.15.102:0/2591515378 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7feba402ebe0 con 0x55fb66d0f660 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.302+0000 7febc678c740 10 client.27173 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.302+0000 7febc678c740 10 client.27173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.302+0000 7febc678c740 20 client.27173 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.302+0000 7febc678c740 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fb66da1c80 0x55fb66da40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.302+0000 7febc678c740 4 client.27173 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.302+0000 7febc678c740 1 -- 172.21.15.102:0/2591515378 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fb66c7a340 con 0x55fb66da1c80 2024-04-25T02:02:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.302+0000 7febc1000640 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fb66da1c80 0x55fb66da40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.303+0000 7febc1000640 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fb66da1c80 0x55fb66da40a0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aebfc4c5a522b207 server_cookie=c08eede4cd10078e in_seq=0 out_seq=0 2024-04-25T02:02:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.303+0000 7febaaa00640 10 client.27173 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febaaa00640 1 -- 172.21.15.102:0/2591515378 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fb66c7a340 con 0x55fb66da1c80 2024-04-25T02:02:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febaaa00640 10 client.27173 handle_command_reply: tid=0 2024-04-25T02:02:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febc678c740 1 client.27173 shutdown 2024-04-25T02:02:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febc678c740 1 -- 172.21.15.102:0/2591515378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0f660 msgr2=0x55fb66d4a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febc678c740 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0f660 0x55fb66d4a3a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7feba400ffd0 tx=0x7feba4005f90 comp rx=0 tx=0).stop 2024-04-25T02:02:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febc678c740 1 -- 172.21.15.102:0/2591515378 shutdown_connections 2024-04-25T02:02:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febc678c740 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55fb66da1c80 0x55fb66da40a0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febc678c740 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb66d227f0 0x55fb66d4a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febc678c740 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb66d0f660 0x55fb66d4a3a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febc678c740 1 --2- 172.21.15.102:0/2591515378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fb66d0ed40 0x55fb66d49e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febc678c740 1 -- 172.21.15.102:0/2591515378 >> 172.21.15.102:0/2591515378 conn(0x55fb66d0b310 msgr2=0x55fb66d0a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.364+0000 7febaaa00640 0 client.27173 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.365+0000 7febc678c740 1 -- 172.21.15.102:0/2591515378 shutdown_connections 2024-04-25T02:02:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.365+0000 7febc678c740 1 -- 172.21.15.102:0/2591515378 wait complete. 2024-04-25T02:02:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.365+0000 7febc678c740 20 client.27173 trim_cache size 0 max 16384 2024-04-25T02:02:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:02:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:02:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:02:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:02:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.367+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feb9c0a2f80 msgr2=0x7feb9c0a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.367+0000 7febc3a00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feb9c0a2f80 0x7feb9c0a5440 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55fb66d0c4a0 tx=0x7feba4004730 comp rx=0 tx=0).stop 2024-04-25T02:02:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.367+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 msgr2=0x7febbc1a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.367+0000 7febc3a00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 0x7febbc1a6250 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7febb4030c30 tx=0x7febb4030c60 comp rx=0 tx=0).stop 2024-04-25T02:02:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.368+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 shutdown_connections 2024-04-25T02:02:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.368+0000 7febc3a00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7feb9c0a2f80 0x7feb9c0a5440 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.368+0000 7febc3a00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7febbc113690 0x7febbc1a6250 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.368+0000 7febc3a00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febbc10aa90 0x7febbc1a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.368+0000 7febc3a00640 1 --2- 172.21.15.102:0/1347986568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febbc10a0e0 0x7febbc1a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.368+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 >> 172.21.15.102:0/1347986568 conn(0x7febbc06ccb0 msgr2=0x7febbc070790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:44.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.369+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 shutdown_connections 2024-04-25T02:02:44.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:44.369+0000 7febc3a00640 1 -- 172.21.15.102:0/1347986568 wait complete. 2024-04-25T02:02:45.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:45 smithi110 ceph-mon[32848]: pgmap v229: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-25T02:02:45.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:45 smithi193 ceph-mon[32981]: pgmap v229: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-25T02:02:45.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:45 smithi102 ceph-mon[27796]: pgmap v229: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-25T02:02:47.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:47 smithi110 ceph-mon[32848]: pgmap v230: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 MiB/s wr, 1.62k op/s 2024-04-25T02:02:47.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:47 smithi193 ceph-mon[32981]: pgmap v230: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 MiB/s wr, 1.62k op/s 2024-04-25T02:02:47.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:47 smithi102 ceph-mon[27796]: pgmap v230: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 MiB/s wr, 1.62k op/s 2024-04-25T02:02:48.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 415.0 s, 2196.4 tps, lat 44.026 ms stddev 151.638 2024-04-25T02:02:49.730 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:49 smithi102 ceph-mon[27796]: pgmap v231: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 1.90k op/s 2024-04-25T02:02:49.741 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:49 smithi110 ceph-mon[32848]: pgmap v231: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 1.90k op/s 2024-04-25T02:02:49.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:49 smithi193 ceph-mon[32981]: pgmap v231: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 25 MiB/s wr, 1.90k op/s 2024-04-25T02:02:51.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:51 smithi110 ceph-mon[32848]: pgmap v232: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 5.8 MiB/s wr, 585 op/s 2024-04-25T02:02:51.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:51 smithi193 ceph-mon[32981]: pgmap v232: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 5.8 MiB/s wr, 585 op/s 2024-04-25T02:02:51.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:51 smithi102 ceph-mon[27796]: pgmap v232: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 5.8 MiB/s wr, 585 op/s 2024-04-25T02:02:53.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 420.0 s, 2274.8 tps, lat 45.365 ms stddev 77.748 2024-04-25T02:02:53.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:53 smithi110 ceph-mon[32848]: pgmap v233: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 725 op/s 2024-04-25T02:02:53.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:53 smithi193 ceph-mon[32981]: pgmap v233: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 725 op/s 2024-04-25T02:02:53.927 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:53 smithi102 ceph-mon[27796]: pgmap v233: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.2 MiB/s wr, 725 op/s 2024-04-25T02:02:54.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:54 smithi193 ceph-mon[32981]: pgmap v234: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 605 op/s 2024-04-25T02:02:54.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:54 smithi102 ceph-mon[27796]: pgmap v234: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 605 op/s 2024-04-25T02:02:54.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:54 smithi110 ceph-mon[32848]: pgmap v234: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 5.8 MiB/s wr, 605 op/s 2024-04-25T02:02:57.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:56 smithi102 ceph-mon[27796]: pgmap v235: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 504 op/s 2024-04-25T02:02:57.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:56 smithi110 ceph-mon[32848]: pgmap v235: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 504 op/s 2024-04-25T02:02:57.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:56 smithi193 ceph-mon[32981]: pgmap v235: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 4.8 MiB/s wr, 504 op/s 2024-04-25T02:02:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.558+0000 7f00a2000640 1 -- 172.21.15.102:0/1376668000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f009c109930 msgr2=0x7f009c109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.558+0000 7f00a2000640 1 --2- 172.21.15.102:0/1376668000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f009c109930 0x7f009c109d10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f008c009930 tx=0x7f008c02a440 comp rx=0 tx=0).stop 2024-04-25T02:02:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.558+0000 7f00a2000640 1 -- 172.21.15.102:0/1376668000 shutdown_connections 2024-04-25T02:02:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.558+0000 7f00a2000640 1 --2- 172.21.15.102:0/1376668000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f009c1042e0 0x7f009c1107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.558+0000 7f00a2000640 1 --2- 172.21.15.102:0/1376668000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f009c103920 0x7f009c103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.558+0000 7f00a2000640 1 --2- 172.21.15.102:0/1376668000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f009c109930 0x7f009c109d10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.558+0000 7f00a2000640 1 -- 172.21.15.102:0/1376668000 >> 172.21.15.102:0/1376668000 conn(0x7f009c0ff430 msgr2=0x7f009c101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.558+0000 7f00a2000640 1 -- 172.21.15.102:0/1376668000 shutdown_connections 2024-04-25T02:02:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.559+0000 7f00a2000640 1 -- 172.21.15.102:0/1376668000 wait complete. 2024-04-25T02:02:57.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.559+0000 7f00a2000640 1 Processor -- start 2024-04-25T02:02:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f00a2000640 1 -- start start 2024-04-25T02:02:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f00a2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f009c103920 0x7f009c119c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f00a2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f009c1042e0 0x7f009c11a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f00a2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f009c109930 0x7f009c10f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f00a2000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f009c077720 con 0x7f009c103920 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f00a2000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f009c0775a0 con 0x7f009c109930 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f00a2000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f009c0778a0 con 0x7f009c1042e0 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f009be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f009c109930 0x7f009c10f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f009be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f009c109930 0x7f009c10f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34770/0 (socket says 172.21.15.102:34770) 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.561+0000 7f009be00640 1 -- 172.21.15.102:0/1874157986 learned_addr learned my addr 172.21.15.102:0/1874157986 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.560+0000 7f009b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f009c103920 0x7f009c119c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.561+0000 7f009aa00640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f009c1042e0 0x7f009c11a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.561+0000 7f009aa00640 1 -- 172.21.15.102:0/1874157986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f009c109930 msgr2=0x7f009c10f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.561+0000 7f009aa00640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f009c109930 0x7f009c10f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.561+0000 7f009aa00640 1 -- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f009c103920 msgr2=0x7f009c119c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.561+0000 7f009aa00640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f009c103920 0x7f009c119c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.561+0000 7f009aa00640 1 -- 172.21.15.102:0/1874157986 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c1100a0 con 0x7f009c1042e0 2024-04-25T02:02:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.561+0000 7f009b400640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f009c103920 0x7f009c119c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.561+0000 7f009be00640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f009c109930 0x7f009c10f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.562+0000 7f009aa00640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f009c1042e0 0x7f009c11a1b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f008800e9f0 tx=0x7f008800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.562+0000 7f0093e00640 1 -- 172.21.15.102:0/1874157986 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008800ccc0 con 0x7f009c1042e0 2024-04-25T02:02:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.562+0000 7f0093e00640 1 -- 172.21.15.102:0/1874157986 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0088005020 con 0x7f009c1042e0 2024-04-25T02:02:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.562+0000 7f0093e00640 1 -- 172.21.15.102:0/1874157986 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00880049e0 con 0x7f009c1042e0 2024-04-25T02:02:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.562+0000 7f00a2000640 1 -- 172.21.15.102:0/1874157986 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c110390 con 0x7f009c1042e0 2024-04-25T02:02:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.562+0000 7f00a2000640 1 -- 172.21.15.102:0/1874157986 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c115720 con 0x7f009c1042e0 2024-04-25T02:02:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.567+0000 7f0093e00640 1 -- 172.21.15.102:0/1874157986 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0088005560 con 0x7f009c1042e0 2024-04-25T02:02:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.567+0000 7f00a4dff740 1 Processor -- start 2024-04-25T02:02:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.568+0000 7f00a4dff740 1 -- start start 2024-04-25T02:02:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.568+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d7131e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.568+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d713720 0x56367d71daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.568+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d71dfe0 0x56367d7203d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.568+0000 7f00a4dff740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56367d676ab0 con 0x56367d713720 2024-04-25T02:02:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.568+0000 7f00a4dff740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56367d61f640 con 0x56367d712e00 2024-04-25T02:02:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.568+0000 7f00a4dff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56367d672a70 con 0x56367d71dfe0 2024-04-25T02:02:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d71dfe0 0x56367d7203d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d7131e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d7131e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34774/0 (socket says 172.21.15.102:34774) 2024-04-25T02:02:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009be00640 1 -- 172.21.15.102:0/2035896807 learned_addr learned my addr 172.21.15.102:0/2035896807 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009b400640 1 --2- 172.21.15.102:0/2035896807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d713720 0x56367d71daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f0093e00640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00800a2e20 0x7f00800a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009be00640 1 -- 172.21.15.102:0/2035896807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d71dfe0 msgr2=0x56367d7203d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009be00640 1 --2- 172.21.15.102:0/2035896807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d71dfe0 0x56367d7203d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009be00640 1 -- 172.21.15.102:0/2035896807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d713720 msgr2=0x56367d71daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f0093e00640 1 -- 172.21.15.102:0/1874157986 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f00880cf360 con 0x7f009c1042e0 2024-04-25T02:02:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009be00640 1 --2- 172.21.15.102:0/2035896807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d713720 0x56367d71daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009be00640 1 -- 172.21.15.102:0/2035896807 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56367d38b210 con 0x56367d712e00 2024-04-25T02:02:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.569+0000 7f009b400640 1 --2- 172.21.15.102:0/2035896807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d713720 0x56367d71daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:02:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.570+0000 7f009be00640 1 --2- 172.21.15.102:0/2035896807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d7131e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f009400e830 tx=0x7f009400ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc06481ad0ba1afc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.570+0000 7f009be00640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00800a2e20 0x7f00800a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.570+0000 7f0087e00640 1 -- 172.21.15.102:0/2035896807 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094011ce0 con 0x56367d712e00 2024-04-25T02:02:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.570+0000 7f0087e00640 1 -- 172.21.15.102:0/2035896807 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f009401c930 con 0x56367d712e00 2024-04-25T02:02:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.570+0000 7f0087e00640 1 -- 172.21.15.102:0/2035896807 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009400fb40 con 0x56367d712e00 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.571+0000 7f00a4dff740 1 -- 172.21.15.102:0/2035896807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 msgr2=0x56367d7131e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.571+0000 7f009be00640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00800a2e20 0x7f00800a52e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f009401b000 tx=0x7f009400fce0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.571+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2035896807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d7131e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f009400e830 tx=0x7f009400ecf0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.572+0000 7f00a4dff740 1 -- 172.21.15.102:0/2035896807 shutdown_connections 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.572+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2035896807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d71dfe0 0x56367d7203d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.572+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2035896807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d713720 0x56367d71daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.572+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2035896807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d7131e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.572+0000 7f00a4dff740 1 -- 172.21.15.102:0/2035896807 >> 172.21.15.102:0/2035896807 conn(0x56367d70f310 msgr2=0x56367d70f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.572+0000 7f00a4dff740 1 -- 172.21.15.102:0/2035896807 shutdown_connections 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.572+0000 7f00a4dff740 1 -- 172.21.15.102:0/2035896807 wait complete. 2024-04-25T02:02:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.572+0000 7f00a4dff740 1 -- start start 2024-04-25T02:02:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.573+0000 7f00a4dff740 1 Processor -- start 2024-04-25T02:02:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.573+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d71aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.573+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d71b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.573+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d71dfe0 0x56367d71ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.573+0000 7f00a4dff740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56367d720c10 con 0x56367d71dfe0 2024-04-25T02:02:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.573+0000 7f00a4dff740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56367d632990 con 0x56367d712e00 2024-04-25T02:02:57.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.573+0000 7f00a4dff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56367d720d90 con 0x56367d713720 2024-04-25T02:02:57.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.573+0000 7f00a4dff740 10 client.0 _ll_register_callbacks cb 0x56367d634950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:57.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d71b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d71b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50698/0 (socket says 172.21.15.102:50698) 2024-04-25T02:02:57.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009aa00640 1 -- 172.21.15.102:0/2690842451 learned_addr learned my addr 172.21.15.102:0/2690842451 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:57.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009b400640 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d71aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009be00640 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d71dfe0 0x56367d71ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009aa00640 1 -- 172.21.15.102:0/2690842451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 msgr2=0x56367d71aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:57.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009aa00640 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d71aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009aa00640 1 -- 172.21.15.102:0/2690842451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d71dfe0 msgr2=0x56367d71ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:57.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009aa00640 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d71dfe0 0x56367d71ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.574+0000 7f009aa00640 1 -- 172.21.15.102:0/2690842451 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56367d457620 con 0x56367d713720 2024-04-25T02:02:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.575+0000 7f009aa00640 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d71b530 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f008800e9c0 tx=0x7f0088097be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca02d3f6d74bc874 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.575+0000 7f0084c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:02:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.575+0000 7f0084c00640 1 -- 172.21.15.102:0/2690842451 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008801c070 con 0x56367d713720 2024-04-25T02:02:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.575+0000 7f0084c00640 1 -- 172.21.15.102:0/2690842451 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f0088004de0 con 0x56367d713720 2024-04-25T02:02:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.575+0000 7f00a4dff740 1 -- 172.21.15.102:0/2690842451 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f009401c930 con 0x56367d713720 2024-04-25T02:02:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.575+0000 7f0084c00640 1 -- 172.21.15.102:0/2690842451 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f00880b4050 con 0x56367d713720 2024-04-25T02:02:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.575+0000 7f00a4dff740 10 client.27139 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.575+0000 7f00a4dff740 1 -- 172.21.15.102:0/2690842451 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56367d72cbd0 con 0x56367d713720 2024-04-25T02:02:57.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.576+0000 7f0084c00640 1 -- 172.21.15.102:0/2690842451 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f009401c930 con 0x56367d713720 2024-04-25T02:02:57.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.576+0000 7f0084c00640 1 -- 172.21.15.102:0/2690842451 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f008809bae0 con 0x56367d713720 2024-04-25T02:02:57.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.576+0000 7f00a4dff740 10 client.27139 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:57.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.576+0000 7f00a4dff740 10 client.27139 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.576+0000 7f00a4dff740 20 client.27139 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.576+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56367d72e130 0x56367d730520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.576+0000 7f00a4dff740 4 client.27139 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:02:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.576+0000 7f00a4dff740 1 -- 172.21.15.102:0/2690842451 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56367d683ff0 con 0x56367d72e130 2024-04-25T02:02:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.576+0000 7f009b400640 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56367d72e130 0x56367d730520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.577+0000 7f009b400640 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56367d72e130 0x56367d730520 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcd059e53240913 server_cookie=2aeef43db00ec19d in_seq=0 out_seq=0 2024-04-25T02:02:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.577+0000 7f0084c00640 10 client.27139 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.578+0000 7f0084c00640 1 -- 172.21.15.102:0/2690842451 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56367d683ff0 con 0x56367d72e130 2024-04-25T02:02:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.578+0000 7f0084c00640 10 client.27139 handle_command_reply: tid=0 2024-04-25T02:02:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.578+0000 7f00a4dff740 1 client.27139 shutdown 2024-04-25T02:02:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.579+0000 7f00a4dff740 1 -- 172.21.15.102:0/2690842451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 msgr2=0x56367d71b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.579+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d71b530 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f008800e9c0 tx=0x7f0088097be0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.579+0000 7f00a4dff740 1 -- 172.21.15.102:0/2690842451 shutdown_connections 2024-04-25T02:02:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.579+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56367d72e130 0x56367d730520 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.579+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d71dfe0 0x56367d71ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.579+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d71b530 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.579+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2690842451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d712e00 0x56367d71aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.579+0000 7f00a4dff740 1 -- 172.21.15.102:0/2690842451 >> 172.21.15.102:0/2690842451 conn(0x56367d70f310 msgr2=0x56367d715090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.580+0000 7f0084c00640 0 client.27139 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.580+0000 7f00a4dff740 1 -- 172.21.15.102:0/2690842451 shutdown_connections 2024-04-25T02:02:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.580+0000 7f00a4dff740 1 -- 172.21.15.102:0/2690842451 wait complete. 2024-04-25T02:02:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.580+0000 7f00a4dff740 20 client.27139 trim_cache size 0 max 16384 2024-04-25T02:02:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.607+0000 7f00a4dff740 1 Processor -- start 2024-04-25T02:02:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.607+0000 7f00a4dff740 1 -- start start 2024-04-25T02:02:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.607+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d712e00 0x56367d729ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.607+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d713720 0x56367d71b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.607+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d71dfe0 0x56367d735df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.607+0000 7f00a4dff740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1be9f0 con 0x56367d71dfe0 2024-04-25T02:02:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.607+0000 7f00a4dff740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1bc200 con 0x56367d713720 2024-04-25T02:02:57.612 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.607+0000 7f00a4dff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1ba070 con 0x56367d712e00 2024-04-25T02:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.608+0000 7f009b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d712e00 0x56367d729ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.608+0000 7f009b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d712e00 0x56367d729ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50710/0 (socket says 172.21.15.102:50710) 2024-04-25T02:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.608+0000 7f009b400640 1 -- 172.21.15.102:0/2403217451 learned_addr learned my addr 172.21.15.102:0/2403217451 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.608+0000 7f009b400640 1 -- 172.21.15.102:0/2403217451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d713720 msgr2=0x56367d71b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.608+0000 7f009b400640 1 --2- 172.21.15.102:0/2403217451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d713720 0x56367d71b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.608+0000 7f009b400640 1 -- 172.21.15.102:0/2403217451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d71dfe0 msgr2=0x56367d735df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.608+0000 7f009b400640 1 --2- 172.21.15.102:0/2403217451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d71dfe0 0x56367d735df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.608+0000 7f009b400640 1 -- 172.21.15.102:0/2403217451 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56367d70a620 con 0x56367d712e00 2024-04-25T02:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.609+0000 7f009b400640 1 --2- 172.21.15.102:0/2403217451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d712e00 0x56367d729ad0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f008c009930 tx=0x7f008c02abb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23dc083f75d388f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.609+0000 7f0087400640 1 -- 172.21.15.102:0/2403217451 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c035040 con 0x56367d712e00 2024-04-25T02:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.609+0000 7f0087400640 1 -- 172.21.15.102:0/2403217451 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008c04c070 con 0x56367d712e00 2024-04-25T02:02:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.609+0000 7f0087400640 1 -- 172.21.15.102:0/2403217451 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c031630 con 0x56367d712e00 2024-04-25T02:02:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.610+0000 7f00a4dff740 1 -- 172.21.15.102:0/2403217451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d712e00 msgr2=0x56367d729ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.610+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2403217451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d712e00 0x56367d729ad0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f008c009930 tx=0x7f008c02abb0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.610+0000 7f00a4dff740 1 -- 172.21.15.102:0/2403217451 shutdown_connections 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.610+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2403217451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d71dfe0 0x56367d735df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.610+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2403217451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d713720 0x56367d71b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.610+0000 7f00a4dff740 1 --2- 172.21.15.102:0/2403217451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d712e00 0x56367d729ad0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.610+0000 7f00a4dff740 1 -- 172.21.15.102:0/2403217451 >> 172.21.15.102:0/2403217451 conn(0x56367d70f310 msgr2=0x56367d716300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.610+0000 7f00a4dff740 1 -- 172.21.15.102:0/2403217451 shutdown_connections 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.610+0000 7f00a4dff740 1 -- 172.21.15.102:0/2403217451 wait complete. 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.611+0000 7f00a4dff740 1 -- start start 2024-04-25T02:02:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.611+0000 7f00a4dff740 1 Processor -- start 2024-04-25T02:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d712e00 0x56367d721c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d722190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f00a4dff740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d71dfe0 0x56367d7226d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f00a4dff740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1ba6c0 con 0x56367d712e00 2024-04-25T02:02:57.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f00a4dff740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1bbb30 con 0x56367d71dfe0 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f00a4dff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1bad90 con 0x56367d713720 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f00a4dff740 10 client.0 _ll_register_callbacks cb 0x7f00880046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d722190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d722190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50716/0 (socket says 172.21.15.102:50716) 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009aa00640 1 -- 172.21.15.102:0/1839042370 learned_addr learned my addr 172.21.15.102:0/1839042370 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009b400640 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d712e00 0x56367d721c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009be00640 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d71dfe0 0x56367d7226d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009aa00640 1 -- 172.21.15.102:0/1839042370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d71dfe0 msgr2=0x56367d7226d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009aa00640 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d71dfe0 0x56367d7226d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009aa00640 1 -- 172.21.15.102:0/1839042370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d712e00 msgr2=0x56367d721c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:02:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009aa00640 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d712e00 0x56367d721c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.612+0000 7f009aa00640 1 -- 172.21.15.102:0/1839042370 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56367d7938b0 con 0x56367d713720 2024-04-25T02:02:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.613+0000 7f009aa00640 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d722190 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f00880ccb60 tx=0x7f0088005440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0da373ea95cac58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:02:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.613+0000 7f0084c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:02:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.613+0000 7f0084c00640 1 -- 172.21.15.102:0/1839042370 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00880b8060 con 0x56367d713720 2024-04-25T02:02:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.613+0000 7f00a4dff740 1 -- 172.21.15.102:0/1839042370 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f008c04c070 con 0x56367d713720 2024-04-25T02:02:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.614+0000 7f0084c00640 1 -- 172.21.15.102:0/1839042370 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 8513+0+0 (secure 0 0 0) 0x7f008809cbb0 con 0x56367d713720 2024-04-25T02:02:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.614+0000 7f0084c00640 1 -- 172.21.15.102:0/1839042370 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f008801c070 con 0x56367d713720 2024-04-25T02:02:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.614+0000 7f00a4dff740 10 client.27145 fetch_fsmap learned FSMap version 36 2024-04-25T02:02:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.614+0000 7f00a4dff740 1 -- 172.21.15.102:0/1839042370 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56367d794bc0 con 0x56367d713720 2024-04-25T02:02:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.616+0000 7f0084c00640 1 -- 172.21.15.102:0/1839042370 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008c04c070 con 0x56367d713720 2024-04-25T02:02:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.616+0000 7f0084c00640 1 -- 172.21.15.102:0/1839042370 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f008809acd0 con 0x56367d713720 2024-04-25T02:02:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.617+0000 7f00a4dff740 10 client.27145 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:02:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.617+0000 7f00a4dff740 10 client.27145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:02:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.617+0000 7f00a4dff740 20 client.27145 populate_metadata read hostname 'smithi102' 2024-04-25T02:02:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.617+0000 7f00a4dff740 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56367d795390 0x56367d7977b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:02:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.617+0000 7f00a4dff740 4 client.27145 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:02:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.617+0000 7f00a4dff740 1 -- 172.21.15.102:0/1839042370 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56367d67e340 con 0x56367d795390 2024-04-25T02:02:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.617+0000 7f009b400640 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56367d795390 0x56367d7977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:02:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.618+0000 7f009b400640 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56367d795390 0x56367d7977b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a4a6ccc4f3085a5 server_cookie=d9c559bdeffb9aca in_seq=0 out_seq=0 2024-04-25T02:02:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.618+0000 7f0084c00640 10 client.27145 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.626+0000 7f0084c00640 1 -- 172.21.15.102:0/1839042370 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56367d67e340 con 0x56367d795390 2024-04-25T02:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.626+0000 7f0084c00640 10 client.27145 handle_command_reply: tid=0 2024-04-25T02:02:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.627+0000 7f00a4dff740 1 client.27145 shutdown 2024-04-25T02:02:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 -- 172.21.15.102:0/1839042370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 msgr2=0x56367d722190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d722190 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f00880ccb60 tx=0x7f0088005440 comp rx=0 tx=0).stop 2024-04-25T02:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 -- 172.21.15.102:0/1839042370 shutdown_connections 2024-04-25T02:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56367d795390 0x56367d7977b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56367d71dfe0 0x56367d7226d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56367d713720 0x56367d722190 secure :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f00880ccb60 tx=0x7f0088005440 comp rx=0 tx=0).stop 2024-04-25T02:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 --2- 172.21.15.102:0/1839042370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56367d712e00 0x56367d721c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 -- 172.21.15.102:0/1839042370 >> 172.21.15.102:0/1839042370 conn(0x56367d70f310 msgr2=0x56367d714800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f0084c00640 0 client.27145 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 -- 172.21.15.102:0/1839042370 shutdown_connections 2024-04-25T02:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.628+0000 7f00a4dff740 1 -- 172.21.15.102:0/1839042370 wait complete. 2024-04-25T02:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.629+0000 7f00a4dff740 20 client.27145 trim_cache size 0 max 16384 2024-04-25T02:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:02:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:02:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.629+0000 7f00a2000640 1 -- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00800a2e20 msgr2=0x7f00800a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.629+0000 7f00a2000640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00800a2e20 0x7f00800a52e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f009401b000 tx=0x7f009400fce0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.629+0000 7f00a2000640 1 -- 172.21.15.102:0/1874157986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f009c1042e0 msgr2=0x7f009c11a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:02:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.629+0000 7f00a2000640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f009c1042e0 0x7f009c11a1b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f008800e9f0 tx=0x7f008800eeb0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.630+0000 7f00a2000640 1 -- 172.21.15.102:0/1874157986 shutdown_connections 2024-04-25T02:02:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.630+0000 7f00a2000640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00800a2e20 0x7f00800a52e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.630+0000 7f00a2000640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f009c109930 0x7f009c10f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.630+0000 7f00a2000640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f009c1042e0 0x7f009c11a1b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.630+0000 7f00a2000640 1 --2- 172.21.15.102:0/1874157986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f009c103920 0x7f009c119c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:02:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.630+0000 7f00a2000640 1 -- 172.21.15.102:0/1874157986 >> 172.21.15.102:0/1874157986 conn(0x7f009c0ff430 msgr2=0x7f009c108be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:02:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.630+0000 7f00a2000640 1 -- 172.21.15.102:0/1874157986 shutdown_connections 2024-04-25T02:02:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:02:57.630+0000 7f00a2000640 1 -- 172.21.15.102:0/1874157986 wait complete. 2024-04-25T02:02:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 425.0 s, 1740.2 tps, lat 57.576 ms stddev 67.251 2024-04-25T02:02:59.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:02:58 smithi102 ceph-mon[27796]: pgmap v236: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.6 MiB/s wr, 796 op/s 2024-04-25T02:02:59.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:02:58 smithi110 ceph-mon[32848]: pgmap v236: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.6 MiB/s wr, 796 op/s 2024-04-25T02:02:59.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:02:58 smithi193 ceph-mon[32981]: pgmap v236: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.6 MiB/s wr, 796 op/s 2024-04-25T02:03:01.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:00 smithi102 ceph-mon[27796]: pgmap v237: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.9 MiB/s wr, 521 op/s 2024-04-25T02:03:01.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:00 smithi110 ceph-mon[32848]: pgmap v237: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.9 MiB/s wr, 521 op/s 2024-04-25T02:03:01.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:00 smithi193 ceph-mon[32981]: pgmap v237: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.9 MiB/s wr, 521 op/s 2024-04-25T02:03:03.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 430.0 s, 880.4 tps, lat 110.548 ms stddev 194.391 2024-04-25T02:03:03.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:02 smithi102 ceph-mon[27796]: pgmap v238: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.1 MiB/s wr, 664 op/s 2024-04-25T02:03:03.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:02 smithi110 ceph-mon[32848]: pgmap v238: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.1 MiB/s wr, 664 op/s 2024-04-25T02:03:03.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:02 smithi193 ceph-mon[32981]: pgmap v238: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.1 MiB/s wr, 664 op/s 2024-04-25T02:03:05.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:04 smithi102 ceph-mon[27796]: pgmap v239: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.7 MiB/s wr, 638 op/s 2024-04-25T02:03:05.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:04 smithi110 ceph-mon[32848]: pgmap v239: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.7 MiB/s wr, 638 op/s 2024-04-25T02:03:05.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:04 smithi193 ceph-mon[32981]: pgmap v239: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.7 MiB/s wr, 638 op/s 2024-04-25T02:03:07.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:06 smithi102 ceph-mon[27796]: pgmap v240: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-04-25T02:03:07.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:06 smithi110 ceph-mon[32848]: pgmap v240: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-04-25T02:03:07.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:06 smithi193 ceph-mon[32981]: pgmap v240: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 5.9 MiB/s wr, 548 op/s 2024-04-25T02:03:08.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 435.0 s, 1379.2 tps, lat 74.109 ms stddev 95.687 2024-04-25T02:03:08.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:07 smithi102 ceph-mon[27796]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T02:03:08.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:07 smithi110 ceph-mon[32848]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T02:03:08.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:07 smithi193 ceph-mon[32981]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T02:03:09.180 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:08 smithi102 ceph-mon[27796]: pgmap v242: 97 pgs: 16 active+clean+snaptrim_wait, 74 active+clean, 7 active+clean+snaptrim; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 29 MiB/s wr, 2.04k op/s 2024-04-25T02:03:09.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:08 smithi110 ceph-mon[32848]: pgmap v242: 97 pgs: 16 active+clean+snaptrim_wait, 74 active+clean, 7 active+clean+snaptrim; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 29 MiB/s wr, 2.04k op/s 2024-04-25T02:03:09.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:08 smithi193 ceph-mon[32981]: pgmap v242: 97 pgs: 16 active+clean+snaptrim_wait, 74 active+clean, 7 active+clean+snaptrim; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 29 MiB/s wr, 2.04k op/s 2024-04-25T02:03:10.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.794+0000 7f70ec600640 1 -- 172.21.15.102:0/1295240039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e41007c0 msgr2=0x7f70e4100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:10.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.794+0000 7f70ec600640 1 --2- 172.21.15.102:0/1295240039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e41007c0 0x7f70e4100ba0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009900 tx=0x7f70d802a530 comp rx=0 tx=0).stop 2024-04-25T02:03:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.794+0000 7f70ec600640 1 -- 172.21.15.102:0/1295240039 shutdown_connections 2024-04-25T02:03:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.794+0000 7f70ec600640 1 --2- 172.21.15.102:0/1295240039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70e4105ae0 0x7f70e4105f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.794+0000 7f70ec600640 1 --2- 172.21.15.102:0/1295240039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e41010e0 0x7f70e41055a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.794+0000 7f70ec600640 1 --2- 172.21.15.102:0/1295240039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e41007c0 0x7f70e4100ba0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.794+0000 7f70ec600640 1 -- 172.21.15.102:0/1295240039 >> 172.21.15.102:0/1295240039 conn(0x7f70e40777c0 msgr2=0x7f70e4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:10.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.795+0000 7f70ec600640 1 -- 172.21.15.102:0/1295240039 shutdown_connections 2024-04-25T02:03:10.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.795+0000 7f70ec600640 1 -- 172.21.15.102:0/1295240039 wait complete. 2024-04-25T02:03:10.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.797+0000 7f70ec600640 1 Processor -- start 2024-04-25T02:03:10.801 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.797+0000 7f70ec600640 1 -- start start 2024-04-25T02:03:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.797+0000 7f70ec600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e41007c0 0x7f70e4072de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70ec600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70e41010e0 0x7f70e4073320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70ec600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4105ae0 0x7f70e406cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70ec600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70e406b8c0 con 0x7f70e41010e0 2024-04-25T02:03:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70ec600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f70e406b740 con 0x7f70e4105ae0 2024-04-25T02:03:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70ec600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70e406ba40 con 0x7f70e41007c0 2024-04-25T02:03:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70e9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e41007c0 0x7f70e4072de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70e9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e41007c0 0x7f70e4072de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:57112/0 (socket says 172.21.15.102:57112) 2024-04-25T02:03:10.803 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70e9c00640 1 -- 172.21.15.102:0/3172459904 learned_addr learned my addr 172.21.15.102:0/3172459904 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70e9200640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70e41010e0 0x7f70e4073320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.798+0000 7f70ea600640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4105ae0 0x7f70e406cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.804 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.799+0000 7f70e9200640 1 -- 172.21.15.102:0/3172459904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e41007c0 msgr2=0x7f70e4072de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.799+0000 7f70e9200640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e41007c0 0x7f70e4072de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.799+0000 7f70e9200640 1 -- 172.21.15.102:0/3172459904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4105ae0 msgr2=0x7f70e406cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.799+0000 7f70e9200640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4105ae0 0x7f70e406cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.799+0000 7f70e9200640 1 -- 172.21.15.102:0/3172459904 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e406d7c0 con 0x7f70e41010e0 2024-04-25T02:03:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.800+0000 7f70e9200640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70e41010e0 0x7f70e4073320 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f70d000e9f0 tx=0x7f70d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:10.806 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.800+0000 7f70e2000640 1 -- 172.21.15.102:0/3172459904 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d000ccc0 con 0x7f70e41010e0 2024-04-25T02:03:10.806 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.800+0000 7f70e2000640 1 -- 172.21.15.102:0/3172459904 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d0010910 con 0x7f70e41010e0 2024-04-25T02:03:10.806 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.800+0000 7f70ec600640 1 -- 172.21.15.102:0/3172459904 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e406dab0 con 0x7f70e41010e0 2024-04-25T02:03:10.806 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.800+0000 7f70e2000640 1 -- 172.21.15.102:0/3172459904 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0018960 con 0x7f70e41010e0 2024-04-25T02:03:10.806 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.800+0000 7f70ec600640 1 -- 172.21.15.102:0/3172459904 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e41b3900 con 0x7f70e41010e0 2024-04-25T02:03:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.821+0000 7f70ef3b4740 1 Processor -- start 2024-04-25T02:03:10.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.821+0000 7f70e2000640 1 -- 172.21.15.102:0/3172459904 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f70d0010ac0 con 0x7f70e41010e0 2024-04-25T02:03:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.826+0000 7f70ef3b4740 1 -- start start 2024-04-25T02:03:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.826+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d4f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.826+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4f720 0x55c4e4d59aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.826+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d5c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.826+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c4e4cb2ab0 con 0x55c4e4d4ee00 2024-04-25T02:03:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.826+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c4e4c5b640 con 0x55c4e4d59fe0 2024-04-25T02:03:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.826+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c4e4caea70 con 0x55c4e4d4f720 2024-04-25T02:03:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.827+0000 7f70ea600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d4f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.832 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.827+0000 7f70e9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4f720 0x55c4e4d59aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.827+0000 7f70e9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4f720 0x55c4e4d59aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:57124/0 (socket says 172.21.15.102:57124) 2024-04-25T02:03:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.827+0000 7f70e9c00640 1 -- 172.21.15.102:0/1146552765 learned_addr learned my addr 172.21.15.102:0/1146552765 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.828+0000 7f70e9c00640 1 -- 172.21.15.102:0/1146552765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 msgr2=0x55c4e4d5c3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:03:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.828+0000 7f70e9c00640 1 --2- 172.21.15.102:0/1146552765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d5c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.828+0000 7f70e9c00640 1 -- 172.21.15.102:0/1146552765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4ee00 msgr2=0x55c4e4d4f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:10.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.828+0000 7f70e9c00640 1 --2- 172.21.15.102:0/1146552765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d4f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.828+0000 7f70e9c00640 1 -- 172.21.15.102:0/1146552765 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4e49c7210 con 0x55c4e4d4f720 2024-04-25T02:03:10.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.829+0000 7f70e9c00640 1 --2- 172.21.15.102:0/1146552765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4f720 0x55c4e4d59aa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f70d8006010 tx=0x7f70d8004170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45cc2e09289537cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.829+0000 7f70e2000640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70cc0a2da0 0x7f70cc0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.830+0000 7f70e2000640 1 -- 172.21.15.102:0/3172459904 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8557+0+0 (secure 0 0 0) 0x7f70d00cf780 con 0x7f70e41010e0 2024-04-25T02:03:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.830+0000 7f70ea600640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70cc0a2da0 0x7f70cc0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.830+0000 7f70d6a00640 1 -- 172.21.15.102:0/1146552765 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8040070 con 0x55c4e4d4f720 2024-04-25T02:03:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.830+0000 7f70d6a00640 1 -- 172.21.15.102:0/1146552765 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d8003ad0 con 0x55c4e4d4f720 2024-04-25T02:03:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.830+0000 7f70d6a00640 1 -- 172.21.15.102:0/1146552765 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8035040 con 0x55c4e4d4f720 2024-04-25T02:03:10.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.831+0000 7f70ea600640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70cc0a2da0 0x7f70cc0a5260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f70dc00ae40 tx=0x7f70dc00a4d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:10.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.834+0000 7f70ef3b4740 1 -- 172.21.15.102:0/1146552765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4f720 msgr2=0x55c4e4d59aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:10.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.834+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/1146552765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4f720 0x55c4e4d59aa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f70d8006010 tx=0x7f70d8004170 comp rx=0 tx=0).stop 2024-04-25T02:03:10.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.834+0000 7f70ef3b4740 1 -- 172.21.15.102:0/1146552765 shutdown_connections 2024-04-25T02:03:10.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.834+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/1146552765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d5c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.834+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/1146552765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4f720 0x55c4e4d59aa0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.834+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/1146552765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d4f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.834+0000 7f70ef3b4740 1 -- 172.21.15.102:0/1146552765 >> 172.21.15.102:0/1146552765 conn(0x55c4e4d4b310 msgr2=0x55c4e4d4b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:10.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.834+0000 7f70ef3b4740 1 -- 172.21.15.102:0/1146552765 shutdown_connections 2024-04-25T02:03:10.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.835+0000 7f70ef3b4740 1 -- 172.21.15.102:0/1146552765 wait complete. 2024-04-25T02:03:10.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.836+0000 7f70ef3b4740 1 -- start start 2024-04-25T02:03:10.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.836+0000 7f70ef3b4740 1 Processor -- start 2024-04-25T02:03:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d56ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d57a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c4e4d5cbb0 con 0x55c4e4d4f720 2024-04-25T02:03:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c4e4c6e990 con 0x55c4e4d4ee00 2024-04-25T02:03:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c4e4d5cd30 con 0x55c4e4d59fe0 2024-04-25T02:03:10.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70ef3b4740 10 client.0 _ll_register_callbacks cb 0x55c4e4c70950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70e9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70e9c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d56ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70e9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49812/0 (socket says 172.21.15.102:49812) 2024-04-25T02:03:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.837+0000 7f70e9200640 1 -- 172.21.15.102:0/4032543262 learned_addr learned my addr 172.21.15.102:0/4032543262 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.838+0000 7f70ea600640 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d57a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.838+0000 7f70e9c00640 1 -- 172.21.15.102:0/4032543262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d59fe0 msgr2=0x55c4e4d57a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.838+0000 7f70e9c00640 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d57a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.838+0000 7f70e9c00640 1 -- 172.21.15.102:0/4032543262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 msgr2=0x55c4e4d57530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:10.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.838+0000 7f70e9c00640 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.838+0000 7f70e9c00640 1 -- 172.21.15.102:0/4032543262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4e4a93620 con 0x55c4e4d4ee00 2024-04-25T02:03:10.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.838+0000 7f70e9200640 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:03:10.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.838+0000 7f70e9c00640 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d56ff0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f70d80041e0 tx=0x7f70d802acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e43bd87a992247b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:10.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.838+0000 7f70cb400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:03:10.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.839+0000 7f70cb400640 1 -- 172.21.15.102:0/4032543262 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8035040 con 0x55c4e4d4ee00 2024-04-25T02:03:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.839+0000 7f70cb400640 1 -- 172.21.15.102:0/4032543262 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8557+0+0 (secure 0 0 0) 0x7f70d80035f0 con 0x55c4e4d4ee00 2024-04-25T02:03:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.839+0000 7f70ef3b4740 1 -- 172.21.15.102:0/4032543262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70d8003ad0 con 0x55c4e4d4ee00 2024-04-25T02:03:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.839+0000 7f70cb400640 1 -- 172.21.15.102:0/4032543262 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f70d8054050 con 0x55c4e4d4ee00 2024-04-25T02:03:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.839+0000 7f70ef3b4740 10 client.27197 fetch_fsmap learned FSMap version 36 2024-04-25T02:03:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.839+0000 7f70ef3b4740 1 -- 172.21.15.102:0/4032543262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c4e4d68b50 con 0x55c4e4d4ee00 2024-04-25T02:03:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.842+0000 7f70cb400640 1 -- 172.21.15.102:0/4032543262 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d8003ad0 con 0x55c4e4d4ee00 2024-04-25T02:03:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.842+0000 7f70cb400640 1 -- 172.21.15.102:0/4032543262 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f70d8004730 con 0x55c4e4d4ee00 2024-04-25T02:03:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.842+0000 7f70ef3b4740 10 client.27197 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:03:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.842+0000 7f70ef3b4740 10 client.27197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.842+0000 7f70ef3b4740 20 client.27197 populate_metadata read hostname 'smithi102' 2024-04-25T02:03:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.842+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c4e4d69ea0 0x55c4e4d6c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.842+0000 7f70ef3b4740 4 client.27197 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:03:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.842+0000 7f70ef3b4740 1 -- 172.21.15.102:0/4032543262 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4e4cbfff0 con 0x55c4e4d69ea0 2024-04-25T02:03:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.844+0000 7f70e9200640 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c4e4d69ea0 0x55c4e4d6c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.844+0000 7f70e9200640 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c4e4d69ea0 0x55c4e4d6c290 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6adb2b350aa9d06 server_cookie=5f899bd408863186 in_seq=0 out_seq=0 2024-04-25T02:03:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.844+0000 7f70cb400640 10 client.27197 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.845+0000 7f70cb400640 1 -- 172.21.15.102:0/4032543262 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c4e4cbfff0 con 0x55c4e4d69ea0 2024-04-25T02:03:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.845+0000 7f70cb400640 10 client.27197 handle_command_reply: tid=0 2024-04-25T02:03:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.845+0000 7f70ef3b4740 1 client.27197 shutdown 2024-04-25T02:03:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.846+0000 7f70ef3b4740 1 -- 172.21.15.102:0/4032543262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4ee00 msgr2=0x55c4e4d56ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:10.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.846+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d56ff0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f70d80041e0 tx=0x7f70d802acb0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.847+0000 7f70ef3b4740 1 -- 172.21.15.102:0/4032543262 shutdown_connections 2024-04-25T02:03:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.847+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c4e4d69ea0 0x55c4e4d6c290 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.847+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d57a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.847+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.847+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/4032543262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d56ff0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.847+0000 7f70cb400640 0 client.27197 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.847+0000 7f70ef3b4740 1 -- 172.21.15.102:0/4032543262 >> 172.21.15.102:0/4032543262 conn(0x55c4e4d4b310 msgr2=0x55c4e4d51090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.847+0000 7f70ef3b4740 1 -- 172.21.15.102:0/4032543262 shutdown_connections 2024-04-25T02:03:10.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.848+0000 7f70ef3b4740 1 -- 172.21.15.102:0/4032543262 wait complete. 2024-04-25T02:03:10.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.848+0000 7f70ef3b4740 20 client.27197 trim_cache size 0 max 16384 2024-04-25T02:03:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.884+0000 7f70ef3b4740 1 Processor -- start 2024-04-25T02:03:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.884+0000 7f70ef3b4740 1 -- start start 2024-04-25T02:03:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.885+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d517c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.886+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.886+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.886+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70e41bed20 con 0x55c4e4d59fe0 2024-04-25T02:03:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.886+0000 7f70ea600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.886+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f70e41bc530 con 0x55c4e4d4f720 2024-04-25T02:03:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.886+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70e41ba3a0 con 0x55c4e4d4ee00 2024-04-25T02:03:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.886+0000 7f70ea600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49816/0 (socket says 172.21.15.102:49816) 2024-04-25T02:03:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.886+0000 7f70e9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.887+0000 7f70ea600640 1 -- 172.21.15.102:0/242933946 learned_addr learned my addr 172.21.15.102:0/242933946 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.887+0000 7f70e9200640 1 --2- 172.21.15.102:0/242933946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.887+0000 7f70e9c00640 1 -- 172.21.15.102:0/242933946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4f720 msgr2=0x55c4e4d57890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:10.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.888+0000 7f70e9c00640 1 --2- 172.21.15.102:0/242933946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.888+0000 7f70e9c00640 1 -- 172.21.15.102:0/242933946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d59fe0 msgr2=0x55c4e4d6f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.888+0000 7f70e9c00640 1 --2- 172.21.15.102:0/242933946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.888+0000 7f70e9c00640 1 -- 172.21.15.102:0/242933946 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4e4d4bc90 con 0x55c4e4d4ee00 2024-04-25T02:03:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.889+0000 7f70e9c00640 1 --2- 172.21.15.102:0/242933946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d517c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f70d8004b00 tx=0x7f70d8004170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b1cc5296f3f049 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.889+0000 7f70d6000640 1 -- 172.21.15.102:0/242933946 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8035040 con 0x55c4e4d4ee00 2024-04-25T02:03:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.889+0000 7f70d6000640 1 -- 172.21.15.102:0/242933946 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d8005850 con 0x55c4e4d4ee00 2024-04-25T02:03:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.889+0000 7f70d6000640 1 -- 172.21.15.102:0/242933946 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8040040 con 0x55c4e4d4ee00 2024-04-25T02:03:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.891+0000 7f70ef3b4740 1 -- 172.21.15.102:0/242933946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 msgr2=0x55c4e4d517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.891+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/242933946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d517c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f70d8004b00 tx=0x7f70d8004170 comp rx=0 tx=0).stop 2024-04-25T02:03:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.891+0000 7f70ef3b4740 1 -- 172.21.15.102:0/242933946 shutdown_connections 2024-04-25T02:03:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.891+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/242933946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.891+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/242933946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d4f720 0x55c4e4d57890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.891+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/242933946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d517c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.891+0000 7f70ef3b4740 1 -- 172.21.15.102:0/242933946 >> 172.21.15.102:0/242933946 conn(0x55c4e4d4b310 msgr2=0x55c4e4d65810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:10.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.892+0000 7f70ef3b4740 1 -- 172.21.15.102:0/242933946 shutdown_connections 2024-04-25T02:03:10.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.892+0000 7f70ef3b4740 1 -- 172.21.15.102:0/242933946 wait complete. 2024-04-25T02:03:10.899 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.893+0000 7f70ef3b4740 1 -- start start 2024-04-25T02:03:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.893+0000 7f70ef3b4740 1 Processor -- start 2024-04-25T02:03:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d6f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 0x55c4e4d6ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ef3b4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70e41ba9f0 con 0x55c4e4d4f720 2024-04-25T02:03:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f70e41bbe60 con 0x55c4e4d59fe0 2024-04-25T02:03:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ef3b4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70e41bb0c0 con 0x55c4e4d4ee00 2024-04-25T02:03:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ef3b4740 10 client.0 _ll_register_callbacks cb 0x7f70d8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:10.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ea600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ea600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46042/0 (socket says 172.21.15.102:46042) 2024-04-25T02:03:10.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.894+0000 7f70ea600640 1 -- 172.21.15.102:0/2208107458 learned_addr learned my addr 172.21.15.102:0/2208107458 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:10.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.895+0000 7f70e9c00640 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d6f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.896+0000 7f70ea600640 1 -- 172.21.15.102:0/2208107458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 msgr2=0x55c4e4d6f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.896+0000 7f70ea600640 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d6f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.896+0000 7f70ea600640 1 -- 172.21.15.102:0/2208107458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 msgr2=0x55c4e4d6ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.896+0000 7f70ea600640 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 0x55c4e4d6ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.896+0000 7f70ea600640 1 -- 172.21.15.102:0/2208107458 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4e4d6d7d0 con 0x55c4e4d59fe0 2024-04-25T02:03:10.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.896+0000 7f70e9c00640 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d6f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:03:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.897+0000 7f70ea600640 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6cfd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f70dc01ecb0 tx=0x7f70dc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f163cb73d7f8f546 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.897+0000 7f70cb400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:03:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.897+0000 7f70cb400640 1 -- 172.21.15.102:0/2208107458 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70dc02c060 con 0x55c4e4d59fe0 2024-04-25T02:03:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.898+0000 7f70cb400640 1 -- 172.21.15.102:0/2208107458 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8557+0+0 (secure 0 0 0) 0x7f70dc01cc50 con 0x55c4e4d59fe0 2024-04-25T02:03:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.898+0000 7f70ef3b4740 1 -- 172.21.15.102:0/2208107458 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70d8005850 con 0x55c4e4d59fe0 2024-04-25T02:03:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.898+0000 7f70cb400640 1 -- 172.21.15.102:0/2208107458 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f70dc0273f0 con 0x55c4e4d59fe0 2024-04-25T02:03:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.898+0000 7f70ef3b4740 10 client.27203 fetch_fsmap learned FSMap version 36 2024-04-25T02:03:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.898+0000 7f70ef3b4740 1 -- 172.21.15.102:0/2208107458 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c4e4de1130 con 0x55c4e4d59fe0 2024-04-25T02:03:10.907 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.899+0000 7f70cb400640 1 -- 172.21.15.102:0/2208107458 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d8005850 con 0x55c4e4d59fe0 2024-04-25T02:03:10.907 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.899+0000 7f70cb400640 1 -- 172.21.15.102:0/2208107458 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7f70dc026070 con 0x55c4e4d59fe0 2024-04-25T02:03:10.907 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.901+0000 7f70ef3b4740 10 client.27203 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:03:10.907 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.901+0000 7f70ef3b4740 10 client.27203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:10.907 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.901+0000 7f70ef3b4740 20 client.27203 populate_metadata read hostname 'smithi102' 2024-04-25T02:03:10.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.901+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c4e4de1890 0x55c4e4de3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:10.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.901+0000 7f70ef3b4740 4 client.27203 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:03:10.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.901+0000 7f70ef3b4740 1 -- 172.21.15.102:0/2208107458 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c4e4cba340 con 0x55c4e4de1890 2024-04-25T02:03:10.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.901+0000 7f70e9c00640 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c4e4de1890 0x55c4e4de3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:10.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.902+0000 7f70e9c00640 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c4e4de1890 0x55c4e4de3cf0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b68cdc0ed979755 server_cookie=4edf0370eb3b292a in_seq=0 out_seq=0 2024-04-25T02:03:10.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.902+0000 7f70cb400640 10 client.27203 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:10.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.921+0000 7f70cb400640 1 -- 172.21.15.102:0/2208107458 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c4e4cba340 con 0x55c4e4de1890 2024-04-25T02:03:10.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.921+0000 7f70cb400640 10 client.27203 handle_command_reply: tid=0 2024-04-25T02:03:10.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.921+0000 7f70ef3b4740 1 client.27203 shutdown 2024-04-25T02:03:10.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.922+0000 7f70ef3b4740 1 -- 172.21.15.102:0/2208107458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 msgr2=0x55c4e4d6cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:10.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.922+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6cfd0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f70dc01ecb0 tx=0x7f70dc004a40 comp rx=0 tx=0).stop 2024-04-25T02:03:10.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.923+0000 7f70ef3b4740 1 -- 172.21.15.102:0/2208107458 shutdown_connections 2024-04-25T02:03:10.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.923+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c4e4de1890 0x55c4e4de3cf0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.923+0000 7f70cb400640 0 client.27203 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.923+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c4e4d59fe0 0x55c4e4d6cfd0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.923+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c4e4d4f720 0x55c4e4d6ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.923+0000 7f70ef3b4740 1 --2- 172.21.15.102:0/2208107458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4e4d4ee00 0x55c4e4d6f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.923+0000 7f70ef3b4740 1 -- 172.21.15.102:0/2208107458 >> 172.21.15.102:0/2208107458 conn(0x55c4e4d4b310 msgr2=0x55c4e4d49bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:10.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.923+0000 7f70ef3b4740 1 -- 172.21.15.102:0/2208107458 shutdown_connections 2024-04-25T02:03:10.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.924+0000 7f70ef3b4740 1 -- 172.21.15.102:0/2208107458 wait complete. 2024-04-25T02:03:10.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.924+0000 7f70ef3b4740 20 client.27203 trim_cache size 0 max 16384 2024-04-25T02:03:10.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:03:10.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:03:10.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:03:10.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:03:10.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.925+0000 7f70ec600640 1 -- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70cc0a2da0 msgr2=0x7f70cc0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.925+0000 7f70ec600640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70cc0a2da0 0x7f70cc0a5260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f70dc00ae40 tx=0x7f70dc00a4d0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.925+0000 7f70ec600640 1 -- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70e41010e0 msgr2=0x7f70e4073320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.925+0000 7f70ec600640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70e41010e0 0x7f70e4073320 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f70d000e9f0 tx=0x7f70d000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.928+0000 7f70ec600640 1 -- 172.21.15.102:0/3172459904 shutdown_connections 2024-04-25T02:03:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.928+0000 7f70ec600640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70cc0a2da0 0x7f70cc0a5260 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.928+0000 7f70ec600640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4105ae0 0x7f70e406cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.928+0000 7f70ec600640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70e41010e0 0x7f70e4073320 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.928+0000 7f70ec600640 1 --2- 172.21.15.102:0/3172459904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70e41007c0 0x7f70e4072de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:10.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.928+0000 7f70ec600640 1 -- 172.21.15.102:0/3172459904 >> 172.21.15.102:0/3172459904 conn(0x7f70e40777c0 msgr2=0x7f70e4103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:10.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.928+0000 7f70ec600640 1 -- 172.21.15.102:0/3172459904 shutdown_connections 2024-04-25T02:03:10.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:10.928+0000 7f70ec600640 1 -- 172.21.15.102:0/3172459904 wait complete. 2024-04-25T02:03:11.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:10 smithi102 ceph-mon[27796]: pgmap v243: 97 pgs: 16 active+clean+snaptrim_wait, 74 active+clean, 7 active+clean+snaptrim; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 29 MiB/s wr, 2.04k op/s 2024-04-25T02:03:11.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:10 smithi110 ceph-mon[32848]: pgmap v243: 97 pgs: 16 active+clean+snaptrim_wait, 74 active+clean, 7 active+clean+snaptrim; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 29 MiB/s wr, 2.04k op/s 2024-04-25T02:03:11.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:10 smithi193 ceph-mon[32981]: pgmap v243: 97 pgs: 16 active+clean+snaptrim_wait, 74 active+clean, 7 active+clean+snaptrim; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 29 MiB/s wr, 2.04k op/s 2024-04-25T02:03:13.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 440.0 s, 1676.8 tps, lat 59.611 ms stddev 84.458 2024-04-25T02:03:13.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:12 smithi102 ceph-mon[27796]: pgmap v244: 97 pgs: 16 active+clean+snaptrim_wait, 74 active+clean, 7 active+clean+snaptrim; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 30 MiB/s wr, 2.05k op/s 2024-04-25T02:03:13.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:12 smithi110 ceph-mon[32848]: pgmap v244: 97 pgs: 16 active+clean+snaptrim_wait, 74 active+clean, 7 active+clean+snaptrim; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 30 MiB/s wr, 2.05k op/s 2024-04-25T02:03:13.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:12 smithi193 ceph-mon[32981]: pgmap v244: 97 pgs: 16 active+clean+snaptrim_wait, 74 active+clean, 7 active+clean+snaptrim; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 30 MiB/s wr, 2.05k op/s 2024-04-25T02:03:15.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:14 smithi110 ceph-mon[32848]: pgmap v245: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-25T02:03:15.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:14 smithi193 ceph-mon[32981]: pgmap v245: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-25T02:03:15.427 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:14 smithi102 ceph-mon[27796]: pgmap v245: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-25T02:03:17.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:16 smithi110 ceph-mon[32848]: pgmap v246: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-25T02:03:17.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:16 smithi193 ceph-mon[32981]: pgmap v246: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-25T02:03:17.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:16 smithi102 ceph-mon[27796]: pgmap v246: 97 pgs: 97 active+clean; 4.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 34 MiB/s wr, 2.06k op/s 2024-04-25T02:03:18.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 445.0 s, 2116.4 tps, lat 47.161 ms stddev 55.233 2024-04-25T02:03:18.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:18 smithi102 ceph-mon[27796]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T02:03:18.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:18 smithi110 ceph-mon[32848]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T02:03:18.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:18 smithi193 ceph-mon[32981]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T02:03:19.580 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:19 smithi102 ceph-mon[27796]: pgmap v248: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 MiB/s wr, 665 op/s 2024-04-25T02:03:19.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:19 smithi110 ceph-mon[32848]: pgmap v248: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 MiB/s wr, 665 op/s 2024-04-25T02:03:19.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:19 smithi193 ceph-mon[32981]: pgmap v248: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 MiB/s wr, 665 op/s 2024-04-25T02:03:21.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:21 smithi102 ceph-mon[27796]: pgmap v249: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 MiB/s wr, 665 op/s 2024-04-25T02:03:21.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:21 smithi110 ceph-mon[32848]: pgmap v249: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 MiB/s wr, 665 op/s 2024-04-25T02:03:21.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:21 smithi193 ceph-mon[32981]: pgmap v249: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 MiB/s wr, 665 op/s 2024-04-25T02:03:23.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 450.0 s, 1941.0 tps, lat 50.847 ms stddev 64.435 2024-04-25T02:03:23.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:23 smithi102 ceph-mon[27796]: pgmap v250: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 18 MiB/s wr, 681 op/s 2024-04-25T02:03:23.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:23 smithi110 ceph-mon[32848]: pgmap v250: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 18 MiB/s wr, 681 op/s 2024-04-25T02:03:23.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:23 smithi193 ceph-mon[32981]: pgmap v250: 97 pgs: 97 active+clean; 4.2 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 18 MiB/s wr, 681 op/s 2024-04-25T02:03:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.092+0000 7fba14800640 1 -- 172.21.15.102:0/3642585250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1032f0 msgr2=0x7fba0c10b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.092+0000 7fba14800640 1 --2- 172.21.15.102:0/3642585250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1032f0 0x7fba0c10b7d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc009900 tx=0x7fb9fc02a510 comp rx=0 tx=0).stop 2024-04-25T02:03:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.092+0000 7fba14800640 1 -- 172.21.15.102:0/3642585250 shutdown_connections 2024-04-25T02:03:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.092+0000 7fba14800640 1 --2- 172.21.15.102:0/3642585250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c10bd10 0x7fba0c10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.098 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.092+0000 7fba14800640 1 --2- 172.21.15.102:0/3642585250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1032f0 0x7fba0c10b7d0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.092+0000 7fba14800640 1 --2- 172.21.15.102:0/3642585250 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba0c1029d0 0x7fba0c102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.092+0000 7fba14800640 1 -- 172.21.15.102:0/3642585250 >> 172.21.15.102:0/3642585250 conn(0x7fba0c0777c0 msgr2=0x7fba0c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.092+0000 7fba14800640 1 -- 172.21.15.102:0/3642585250 shutdown_connections 2024-04-25T02:03:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.092+0000 7fba14800640 1 -- 172.21.15.102:0/3642585250 wait complete. 2024-04-25T02:03:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.093+0000 7fba14800640 1 Processor -- start 2024-04-25T02:03:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.094+0000 7fba14800640 1 -- start start 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.094+0000 7fba14800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1029d0 0x7fba0c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.094+0000 7fba14800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba0c1032f0 0x7fba0c198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.094+0000 7fba14800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c10bd10 0x7fba0c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.094+0000 7fba14800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c111b10 con 0x7fba0c1032f0 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.094+0000 7fba14800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c111990 con 0x7fba0c1029d0 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.094+0000 7fba14800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c111c90 con 0x7fba0c10bd10 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba11400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba0c1032f0 0x7fba0c198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba11e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1029d0 0x7fba0c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba11e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1029d0 0x7fba0c197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:47680/0 (socket says 172.21.15.102:47680) 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba11e00640 1 -- 172.21.15.102:0/247059031 learned_addr learned my addr 172.21.15.102:0/247059031 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba12800640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c10bd10 0x7fba0c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba11e00640 1 -- 172.21.15.102:0/247059031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c10bd10 msgr2=0x7fba0c19c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:24.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba11e00640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c10bd10 0x7fba0c19c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba11e00640 1 -- 172.21.15.102:0/247059031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba0c1032f0 msgr2=0x7fba0c198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba11e00640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba0c1032f0 0x7fba0c198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.095+0000 7fba11e00640 1 -- 172.21.15.102:0/247059031 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0c19cf40 con 0x7fba0c1029d0 2024-04-25T02:03:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.096+0000 7fba11e00640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1029d0 0x7fba0c197fd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fba0000e9f0 tx=0x7fba0000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.096+0000 7fba06a00640 1 -- 172.21.15.102:0/247059031 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0000cd80 con 0x7fba0c1029d0 2024-04-25T02:03:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.096+0000 7fba06a00640 1 -- 172.21.15.102:0/247059031 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba00010910 con 0x7fba0c1029d0 2024-04-25T02:03:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.096+0000 7fba06a00640 1 -- 172.21.15.102:0/247059031 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00018980 con 0x7fba0c1029d0 2024-04-25T02:03:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.096+0000 7fba14800640 1 -- 172.21.15.102:0/247059031 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0c19d230 con 0x7fba0c1029d0 2024-04-25T02:03:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.096+0000 7fba14800640 1 -- 172.21.15.102:0/247059031 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba0c19d670 con 0x7fba0c1029d0 2024-04-25T02:03:24.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.101+0000 7fba1759d740 1 Processor -- start 2024-04-25T02:03:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.102+0000 7fba06a00640 1 -- 172.21.15.102:0/247059031 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fba00010ac0 con 0x7fba0c1029d0 2024-04-25T02:03:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.102+0000 7fba1759d740 1 -- start start 2024-04-25T02:03:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.102+0000 7fba1759d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05e9f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.103+0000 7fba1759d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.103+0000 7fba1759d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05eac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.103+0000 7fba1759d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ad05e02ab0 con 0x55ad05ea9fe0 2024-04-25T02:03:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.103+0000 7fba1759d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ad05dab640 con 0x55ad05e9ee00 2024-04-25T02:03:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.103+0000 7fba1759d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ad05dfea70 con 0x55ad05e9f720 2024-04-25T02:03:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.103+0000 7fba12800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05e9f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.103+0000 7fba11e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.103+0000 7fba11e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33192/0 (socket says 172.21.15.102:33192) 2024-04-25T02:03:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.103+0000 7fba11e00640 1 -- 172.21.15.102:0/2820068684 learned_addr learned my addr 172.21.15.102:0/2820068684 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.104+0000 7fba11400640 1 --2- 172.21.15.102:0/2820068684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05eac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.104+0000 7fba11e00640 1 -- 172.21.15.102:0/2820068684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 msgr2=0x55ad05e9f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.104+0000 7fba11e00640 1 --2- 172.21.15.102:0/2820068684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05e9f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.104+0000 7fba11e00640 1 -- 172.21.15.102:0/2820068684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 msgr2=0x55ad05eac3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.104+0000 7fba11e00640 1 --2- 172.21.15.102:0/2820068684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05eac3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.104+0000 7fba11e00640 1 -- 172.21.15.102:0/2820068684 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad05b17210 con 0x55ad05e9f720 2024-04-25T02:03:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.104+0000 7fba11e00640 1 --2- 172.21.15.102:0/2820068684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea9aa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fba000cd0e0 tx=0x7fba000d84f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9b34b30cd5048c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.105+0000 7fb9fb000640 1 -- 172.21.15.102:0/2820068684 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba000e3070 con 0x55ad05e9f720 2024-04-25T02:03:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.105+0000 7fb9fb000640 1 -- 172.21.15.102:0/2820068684 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba000d8e00 con 0x55ad05e9f720 2024-04-25T02:03:24.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.105+0000 7fb9fb000640 1 -- 172.21.15.102:0/2820068684 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba000cfe60 con 0x55ad05e9f720 2024-04-25T02:03:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.106+0000 7fba1759d740 1 -- 172.21.15.102:0/2820068684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 msgr2=0x55ad05ea9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.106+0000 7fba1759d740 1 --2- 172.21.15.102:0/2820068684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea9aa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fba000cd0e0 tx=0x7fba000d84f0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.106+0000 7fba1759d740 1 -- 172.21.15.102:0/2820068684 shutdown_connections 2024-04-25T02:03:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.106+0000 7fba1759d740 1 --2- 172.21.15.102:0/2820068684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05eac3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.106+0000 7fba1759d740 1 --2- 172.21.15.102:0/2820068684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea9aa0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.106+0000 7fba1759d740 1 --2- 172.21.15.102:0/2820068684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05e9f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.106+0000 7fba1759d740 1 -- 172.21.15.102:0/2820068684 >> 172.21.15.102:0/2820068684 conn(0x55ad05e9b310 msgr2=0x55ad05e9b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.106+0000 7fba1759d740 1 -- 172.21.15.102:0/2820068684 shutdown_connections 2024-04-25T02:03:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.106+0000 7fba1759d740 1 -- 172.21.15.102:0/2820068684 wait complete. 2024-04-25T02:03:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.107+0000 7fba06a00640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb9f40a2d60 0x7fb9f40a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.107+0000 7fba06a00640 1 -- 172.21.15.102:0/247059031 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7fba000cfab0 con 0x7fba0c1029d0 2024-04-25T02:03:24.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.107+0000 7fba1759d740 1 -- start start 2024-04-25T02:03:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.108+0000 7fba11400640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb9f40a2d60 0x7fb9f40a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.108+0000 7fba1759d740 1 Processor -- start 2024-04-25T02:03:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.109+0000 7fba11400640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb9f40a2d60 0x7fb9f40a5220 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55ad05e9c730 tx=0x7fb9fc005e70 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.109+0000 7fba1759d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.109+0000 7fba1759d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05e9f720 0x55ad05ea7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.109+0000 7fba1759d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05ea9fe0 0x55ad05ea7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.109+0000 7fba1759d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55ad05eacc10 con 0x55ad05e9f720 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.109+0000 7fba1759d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55ad05dbe990 con 0x55ad05e9ee00 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.109+0000 7fba1759d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ad05eacd90 con 0x55ad05ea9fe0 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.109+0000 7fba1759d740 10 client.0 _ll_register_callbacks cb 0x55ad05dc0950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba12800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05ea9fe0 0x55ad05ea7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:47708/0 (socket says 172.21.15.102:47708) 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba12800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05ea9fe0 0x55ad05ea7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33208/0 (socket says 172.21.15.102:33208) 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11e00640 1 -- 172.21.15.102:0/16039212 learned_addr learned my addr 172.21.15.102:0/16039212 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11400640 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05e9f720 0x55ad05ea7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11e00640 1 -- 172.21.15.102:0/16039212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05ea9fe0 msgr2=0x55ad05ea7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11e00640 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05ea9fe0 0x55ad05ea7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11e00640 1 -- 172.21.15.102:0/16039212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05e9f720 msgr2=0x55ad05ea7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11e00640 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05e9f720 0x55ad05ea7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11e00640 1 -- 172.21.15.102:0/16039212 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad05be3620 con 0x55ad05e9ee00 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fba11e00640 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea7030 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fba00098940 tx=0x7fba000d8fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2880c4ec99fe9eb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fb9ef400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fb9ef400640 1 -- 172.21.15.102:0/16039212 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba000cfe60 con 0x55ad05e9ee00 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.110+0000 7fb9ef400640 1 -- 172.21.15.102:0/16039212 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7fba000cd110 con 0x55ad05e9ee00 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fba1759d740 1 -- 172.21.15.102:0/16039212 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba000d8e00 con 0x55ad05e9ee00 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fb9ef400640 1 -- 172.21.15.102:0/16039212 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fba00098210 con 0x55ad05e9ee00 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fba1759d740 10 client.27215 fetch_fsmap learned FSMap version 36 2024-04-25T02:03:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fba1759d740 1 -- 172.21.15.102:0/16039212 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ad05ec9bb0 con 0x55ad05e9ee00 2024-04-25T02:03:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fb9ef400640 1 -- 172.21.15.102:0/16039212 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba000d8e00 con 0x55ad05e9ee00 2024-04-25T02:03:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fb9ef400640 1 -- 172.21.15.102:0/16039212 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fba000985e0 con 0x55ad05e9ee00 2024-04-25T02:03:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fba1759d740 10 client.27215 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:03:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fba1759d740 10 client.27215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fba1759d740 20 client.27215 populate_metadata read hostname 'smithi102' 2024-04-25T02:03:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fba1759d740 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ad05ecaf00 0x55ad05ecd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fba1759d740 4 client.27215 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:03:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.111+0000 7fba1759d740 1 -- 172.21.15.102:0/16039212 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad05e0fff0 con 0x55ad05ecaf00 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.112+0000 7fba11400640 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ad05ecaf00 0x55ad05ecd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.112+0000 7fba11400640 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ad05ecaf00 0x55ad05ecd2f0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18fc9fc7fec9e318 server_cookie=8428a0657dd4a0f2 in_seq=0 out_seq=0 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.112+0000 7fb9ef400640 10 client.27215 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fb9ef400640 1 -- 172.21.15.102:0/16039212 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad05e0fff0 con 0x55ad05ecaf00 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fb9ef400640 10 client.27215 handle_command_reply: tid=0 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 client.27215 shutdown 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 -- 172.21.15.102:0/16039212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 msgr2=0x55ad05ea7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea7030 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fba00098940 tx=0x7fba000d8fb0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 -- 172.21.15.102:0/16039212 shutdown_connections 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ad05ecaf00 0x55ad05ecd2f0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05ea9fe0 0x55ad05ea7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05e9f720 0x55ad05ea7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 --2- 172.21.15.102:0/16039212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea7030 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 -- 172.21.15.102:0/16039212 >> 172.21.15.102:0/16039212 conn(0x55ad05e9b310 msgr2=0x55ad05ea1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:24.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fb9ef400640 0 client.27215 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:24.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 -- 172.21.15.102:0/16039212 shutdown_connections 2024-04-25T02:03:24.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.113+0000 7fba1759d740 1 -- 172.21.15.102:0/16039212 wait complete. 2024-04-25T02:03:24.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.114+0000 7fba1759d740 20 client.27215 trim_cache size 0 max 16384 2024-04-25T02:03:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.136+0000 7fba1759d740 1 Processor -- start 2024-04-25T02:03:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba1759d740 1 -- start start 2024-04-25T02:03:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba1759d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba1759d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba1759d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05ecff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba1759d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c06fef0 con 0x55ad05ea9fe0 2024-04-25T02:03:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba1759d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c06d700 con 0x55ad05e9ee00 2024-04-25T02:03:24.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba1759d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c06b570 con 0x55ad05e9f720 2024-04-25T02:03:24.144 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba12800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05ecff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba11e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba12800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05ecff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42290/0 (socket says 172.21.15.102:42290) 2024-04-25T02:03:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba12800640 1 -- 172.21.15.102:0/2947507064 learned_addr learned my addr 172.21.15.102:0/2947507064 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.137+0000 7fba11400640 1 --2- 172.21.15.102:0/2947507064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fba12800640 1 -- 172.21.15.102:0/2947507064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 msgr2=0x55ad05ea78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fba12800640 1 --2- 172.21.15.102:0/2947507064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fba12800640 1 -- 172.21.15.102:0/2947507064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 msgr2=0x55ad05ea2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fba12800640 1 --2- 172.21.15.102:0/2947507064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fba12800640 1 -- 172.21.15.102:0/2947507064 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad05e9bc90 con 0x55ad05ea9fe0 2024-04-25T02:03:24.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fba11e00640 1 --2- 172.21.15.102:0/2947507064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:24.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fba12800640 1 --2- 172.21.15.102:0/2947507064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05ecff30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fba08009ee0 tx=0x7fba080127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba01ac4f1fab4978 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:24.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fb9fa600640 1 -- 172.21.15.102:0/2947507064 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba08020070 con 0x55ad05ea9fe0 2024-04-25T02:03:24.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fb9fa600640 1 -- 172.21.15.102:0/2947507064 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba0801b8f0 con 0x55ad05ea9fe0 2024-04-25T02:03:24.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.138+0000 7fb9fa600640 1 -- 172.21.15.102:0/2947507064 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba08010a20 con 0x55ad05ea9fe0 2024-04-25T02:03:24.149 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.139+0000 7fba1759d740 1 -- 172.21.15.102:0/2947507064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 msgr2=0x55ad05ecff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.139+0000 7fba1759d740 1 --2- 172.21.15.102:0/2947507064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05ecff30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fba08009ee0 tx=0x7fba080127c0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.139+0000 7fba1759d740 1 -- 172.21.15.102:0/2947507064 shutdown_connections 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.139+0000 7fba1759d740 1 --2- 172.21.15.102:0/2947507064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05ecff30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.139+0000 7fba1759d740 1 --2- 172.21.15.102:0/2947507064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9f720 0x55ad05ea78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.139+0000 7fba1759d740 1 --2- 172.21.15.102:0/2947507064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9ee00 0x55ad05ea2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.139+0000 7fba1759d740 1 -- 172.21.15.102:0/2947507064 >> 172.21.15.102:0/2947507064 conn(0x55ad05e9b310 msgr2=0x55ad05e9a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.139+0000 7fba1759d740 1 -- 172.21.15.102:0/2947507064 shutdown_connections 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.139+0000 7fba1759d740 1 -- 172.21.15.102:0/2947507064 wait complete. 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.140+0000 7fba1759d740 1 -- start start 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.140+0000 7fba1759d740 1 Processor -- start 2024-04-25T02:03:24.150 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.140+0000 7fba1759d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9ee00 0x55ad05ecdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.140+0000 7fba1759d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9f720 0x55ad05ece260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.140+0000 7fba1759d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05f300e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.140+0000 7fba1759d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c06bbc0 con 0x55ad05ea9fe0 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.140+0000 7fba1759d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c06d030 con 0x55ad05e9f720 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.140+0000 7fba1759d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba0c06c290 con 0x55ad05e9ee00 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.140+0000 7fba1759d740 10 client.0 _ll_register_callbacks cb 0x7fba000e1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9ee00 0x55ad05ecdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9ee00 0x55ad05ecdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33218/0 (socket says 172.21.15.102:33218) 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11e00640 1 -- 172.21.15.102:0/1596514807 learned_addr learned my addr 172.21.15.102:0/1596514807 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11400640 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9f720 0x55ad05ece260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba12800640 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05f300e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11400640 1 -- 172.21.15.102:0/1596514807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9ee00 msgr2=0x55ad05ecdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:24.151 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11400640 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9ee00 0x55ad05ecdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11400640 1 -- 172.21.15.102:0/1596514807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 msgr2=0x55ad05f300e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11400640 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05f300e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11400640 1 -- 172.21.15.102:0/1596514807 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad05f30620 con 0x55ad05e9f720 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fba11400640 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9f720 0x55ad05ece260 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x55ad05e9c4a0 tx=0x7fb9fc02a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2be55253096d029 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.141+0000 7fb9ef400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.142+0000 7fb9ef400640 1 -- 172.21.15.102:0/1596514807 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc035050 con 0x55ad05e9f720 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.142+0000 7fb9ef400640 1 -- 172.21.15.102:0/1596514807 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7fb9fc0043d0 con 0x55ad05e9f720 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.142+0000 7fba1759d740 1 -- 172.21.15.102:0/1596514807 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba0801b8f0 con 0x55ad05e9f720 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.142+0000 7fb9ef400640 1 -- 172.21.15.102:0/1596514807 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb9fc05b050 con 0x55ad05e9f720 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.142+0000 7fba1759d740 10 client.27221 fetch_fsmap learned FSMap version 36 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.142+0000 7fba1759d740 1 -- 172.21.15.102:0/1596514807 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ad05f317d0 con 0x55ad05e9f720 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.142+0000 7fb9ef400640 1 -- 172.21.15.102:0/1596514807 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba0801b8f0 con 0x55ad05e9f720 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.142+0000 7fb9ef400640 1 -- 172.21.15.102:0/1596514807 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 36) ==== 8362+0+0 (secure 0 0 0) 0x7fb9fc046040 con 0x55ad05e9f720 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.143+0000 7fba1759d740 10 client.27221 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T02:03:24.152 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.143+0000 7fba1759d740 10 client.27221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.143+0000 7fba1759d740 20 client.27221 populate_metadata read hostname 'smithi102' 2024-04-25T02:03:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.143+0000 7fba1759d740 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ad05f31f60 0x55ad05f34380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.143+0000 7fba1759d740 4 client.27221 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:03:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.143+0000 7fba1759d740 1 -- 172.21.15.102:0/1596514807 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ad05e0a340 con 0x55ad05f31f60 2024-04-25T02:03:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.143+0000 7fba11e00640 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ad05f31f60 0x55ad05f34380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.145+0000 7fba11e00640 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ad05f31f60 0x55ad05f34380 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2d362cc5cabf93c server_cookie=4121a1da098ca06c in_seq=0 out_seq=0 2024-04-25T02:03:24.153 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.145+0000 7fb9ef400640 10 client.27221 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fb9ef400640 1 -- 172.21.15.102:0/1596514807 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ad05e0a340 con 0x55ad05f31f60 2024-04-25T02:03:24.161 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fb9ef400640 10 client.27221 handle_command_reply: tid=0 2024-04-25T02:03:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fba1759d740 1 client.27221 shutdown 2024-04-25T02:03:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fba1759d740 1 -- 172.21.15.102:0/1596514807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9f720 msgr2=0x55ad05ece260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fba1759d740 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9f720 0x55ad05ece260 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x55ad05e9c4a0 tx=0x7fb9fc02a510 comp rx=0 tx=0).stop 2024-04-25T02:03:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fba1759d740 1 -- 172.21.15.102:0/1596514807 shutdown_connections 2024-04-25T02:03:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fba1759d740 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55ad05f31f60 0x55ad05f34380 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fba1759d740 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad05ea9fe0 0x55ad05f300e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.162 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fba1759d740 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55ad05e9f720 0x55ad05ece260 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fba1759d740 1 --2- 172.21.15.102:0/1596514807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad05e9ee00 0x55ad05ecdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fba1759d740 1 -- 172.21.15.102:0/1596514807 >> 172.21.15.102:0/1596514807 conn(0x55ad05e9b310 msgr2=0x55ad05e99bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.154+0000 7fb9ef400640 0 client.27221 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba1759d740 1 -- 172.21.15.102:0/1596514807 shutdown_connections 2024-04-25T02:03:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba1759d740 1 -- 172.21.15.102:0/1596514807 wait complete. 2024-04-25T02:03:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba1759d740 20 client.27221 trim_cache size 0 max 16384 2024-04-25T02:03:24.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:03:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:03:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:03:24.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:03:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 -- 172.21.15.102:0/247059031 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb9f40a2d60 msgr2=0x7fb9f40a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb9f40a2d60 0x7fb9f40a5220 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55ad05e9c730 tx=0x7fb9fc005e70 comp rx=0 tx=0).stop 2024-04-25T02:03:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 -- 172.21.15.102:0/247059031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1029d0 msgr2=0x7fba0c197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1029d0 0x7fba0c197fd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fba0000e9f0 tx=0x7fba0000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 -- 172.21.15.102:0/247059031 shutdown_connections 2024-04-25T02:03:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb9f40a2d60 0x7fb9f40a5220 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c10bd10 0x7fba0c19c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba0c1032f0 0x7fba0c198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 --2- 172.21.15.102:0/247059031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba0c1029d0 0x7fba0c197fd0 secure :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fba0000e9f0 tx=0x7fba0000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:03:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.155+0000 7fba14800640 1 -- 172.21.15.102:0/247059031 >> 172.21.15.102:0/247059031 conn(0x7fba0c0777c0 msgr2=0x7fba0c0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.156+0000 7fba14800640 1 -- 172.21.15.102:0/247059031 shutdown_connections 2024-04-25T02:03:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:24.156+0000 7fba14800640 1 -- 172.21.15.102:0/247059031 wait complete. 2024-04-25T02:03:25.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:25 smithi102 ceph-mon[27796]: pgmap v251: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 667 op/s 2024-04-25T02:03:25.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:25 smithi110 ceph-mon[32848]: pgmap v251: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 667 op/s 2024-04-25T02:03:25.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:25 smithi193 ceph-mon[32981]: pgmap v251: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 667 op/s 2024-04-25T02:03:27.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:27 smithi102 ceph-mon[27796]: pgmap v252: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 667 op/s 2024-04-25T02:03:27.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:27 smithi110 ceph-mon[32848]: pgmap v252: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 667 op/s 2024-04-25T02:03:27.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:27 smithi193 ceph-mon[32981]: pgmap v252: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 667 op/s 2024-04-25T02:03:28.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 455.0 s, 1860.6 tps, lat 54.988 ms stddev 67.371 2024-04-25T02:03:28.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:28 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:03:28.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:28 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:03:28.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:28 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:03:29.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:29 smithi102 ceph-mon[27796]: pgmap v253: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 14 MiB/s wr, 683 op/s 2024-04-25T02:03:29.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:29 smithi110 ceph-mon[32848]: pgmap v253: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 14 MiB/s wr, 683 op/s 2024-04-25T02:03:29.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:29 smithi193 ceph-mon[32981]: pgmap v253: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 14 MiB/s wr, 683 op/s 2024-04-25T02:03:31.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:31 smithi102 ceph-mon[27796]: pgmap v254: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 597 op/s 2024-04-25T02:03:31.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:31 smithi110 ceph-mon[32848]: pgmap v254: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 597 op/s 2024-04-25T02:03:31.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:31 smithi193 ceph-mon[32981]: pgmap v254: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 597 op/s 2024-04-25T02:03:33.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 460.0 s, 1602.8 tps, lat 62.220 ms stddev 69.653 2024-04-25T02:03:33.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:33 smithi102 ceph-mon[27796]: pgmap v255: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 15 MiB/s wr, 756 op/s 2024-04-25T02:03:33.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:33 smithi110 ceph-mon[32848]: pgmap v255: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 15 MiB/s wr, 756 op/s 2024-04-25T02:03:33.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:33 smithi193 ceph-mon[32981]: pgmap v255: 97 pgs: 97 active+clean; 4.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 15 MiB/s wr, 756 op/s 2024-04-25T02:03:35.677 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:35 smithi102 ceph-mon[27796]: pgmap v256: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 13 MiB/s wr, 701 op/s 2024-04-25T02:03:35.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:35 smithi110 ceph-mon[32848]: pgmap v256: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 13 MiB/s wr, 701 op/s 2024-04-25T02:03:35.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:35 smithi193 ceph-mon[32981]: pgmap v256: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 13 MiB/s wr, 701 op/s 2024-04-25T02:03:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.346+0000 7fd7e7800640 1 -- 172.21.15.102:0/1010150921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0101750 msgr2=0x7fd7e0101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.346+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1010150921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0101750 0x7fd7e0101bd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd7d000fb40 tx=0x7fd7d002e5c0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.347+0000 7fd7e7800640 1 -- 172.21.15.102:0/1010150921 shutdown_connections 2024-04-25T02:03:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.347+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1010150921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e0102110 0x7fd7e010e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.347+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1010150921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0101750 0x7fd7e0101bd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.347+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1010150921 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7e0107760 0x7fd7e0107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.347+0000 7fd7e7800640 1 -- 172.21.15.102:0/1010150921 >> 172.21.15.102:0/1010150921 conn(0x7fd7e0078bd0 msgr2=0x7fd7e0078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.347+0000 7fd7e7800640 1 -- 172.21.15.102:0/1010150921 shutdown_connections 2024-04-25T02:03:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.347+0000 7fd7e7800640 1 -- 172.21.15.102:0/1010150921 wait complete. 2024-04-25T02:03:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.348+0000 7fd7e7800640 1 Processor -- start 2024-04-25T02:03:37.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.348+0000 7fd7e7800640 1 -- start start 2024-04-25T02:03:37.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.348+0000 7fd7e7800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e0101750 0x7fd7e0195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.348+0000 7fd7e7800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0102110 0x7fd7e0196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.348+0000 7fd7e7800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7e0107760 0x7fd7e019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7e7800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e010f9c0 con 0x7fd7e0101750 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7e7800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e010f840 con 0x7fd7e0107760 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7e7800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e010fb40 con 0x7fd7e0102110 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7d7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0102110 0x7fd7e0196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7d7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0102110 0x7fd7e0196320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55340/0 (socket says 172.21.15.102:55340) 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7d7e00640 1 -- 172.21.15.102:0/1227848125 learned_addr learned my addr 172.21.15.102:0/1227848125 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7e5800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7e0107760 0x7fd7e019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e0101750 0x7fd7e0195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7d7e00640 1 -- 172.21.15.102:0/1227848125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7e0107760 msgr2=0x7fd7e019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:37.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7e0107760 0x7fd7e019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7d7e00640 1 -- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e0101750 msgr2=0x7fd7e0195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e0101750 0x7fd7e0195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.349+0000 7fd7d7e00640 1 -- 172.21.15.102:0/1227848125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e019add0 con 0x7fd7e0102110 2024-04-25T02:03:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.350+0000 7fd7e5800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7e0107760 0x7fd7e019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.350+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0102110 0x7fd7e0196320 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd7d002ebb0 tx=0x7fd7d000f340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:37.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.350+0000 7fd7de000640 1 -- 172.21.15.102:0/1227848125 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0044030 con 0x7fd7e0102110 2024-04-25T02:03:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.350+0000 7fd7de000640 1 -- 172.21.15.102:0/1227848125 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7d0004950 con 0x7fd7e0102110 2024-04-25T02:03:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.350+0000 7fd7e7800640 1 -- 172.21.15.102:0/1227848125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e019b060 con 0x7fd7e0102110 2024-04-25T02:03:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.350+0000 7fd7de000640 1 -- 172.21.15.102:0/1227848125 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0040440 con 0x7fd7e0102110 2024-04-25T02:03:37.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.350+0000 7fd7e7800640 1 -- 172.21.15.102:0/1227848125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e01a2950 con 0x7fd7e0102110 2024-04-25T02:03:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.355+0000 7fd7de000640 1 -- 172.21.15.102:0/1227848125 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd7d0047030 con 0x7fd7e0102110 2024-04-25T02:03:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.356+0000 7fd7ea76e740 1 Processor -- start 2024-04-25T02:03:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.357+0000 7fd7ea76e740 1 -- start start 2024-04-25T02:03:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed444e00 0x5595ed4451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 0x5595ed4523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7ea76e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5595ed3a8ab0 con 0x5595ed445720 2024-04-25T02:03:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7ea76e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5595ed351640 con 0x5595ed44ffe0 2024-04-25T02:03:37.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7ea76e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5595ed3a4a70 con 0x5595ed444e00 2024-04-25T02:03:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7e4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7e5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed444e00 0x5595ed4451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7e4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53308/0 (socket says 172.21.15.102:53308) 2024-04-25T02:03:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.358+0000 7fd7e4e00640 1 -- 172.21.15.102:0/3918864113 learned_addr learned my addr 172.21.15.102:0/3918864113 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.359+0000 7fd7e4e00640 1 -- 172.21.15.102:0/3918864113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed444e00 msgr2=0x5595ed4451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.359+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/3918864113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed444e00 0x5595ed4451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.359+0000 7fd7e4e00640 1 -- 172.21.15.102:0/3918864113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 msgr2=0x5595ed4523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.359+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/3918864113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 0x5595ed4523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.359+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/3918864113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 0x5595ed4523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.359+0000 7fd7e4e00640 1 -- 172.21.15.102:0/3918864113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595ed0bd210 con 0x5595ed445720 2024-04-25T02:03:37.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.359+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/3918864113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 0x5595ed4523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:03:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.359+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/3918864113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44faa0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009c70 tx=0x7fd7c8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461d58c548e9c676 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.360+0000 7fd7de000640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7c40a2ea0 0x7fd7c40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.360+0000 7fd7d6000640 1 -- 172.21.15.102:0/3918864113 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8004230 con 0x5595ed445720 2024-04-25T02:03:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.360+0000 7fd7d6000640 1 -- 172.21.15.102:0/3918864113 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7c80043d0 con 0x5595ed445720 2024-04-25T02:03:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.360+0000 7fd7d6000640 1 -- 172.21.15.102:0/3918864113 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8019820 con 0x5595ed445720 2024-04-25T02:03:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.360+0000 7fd7de000640 1 -- 172.21.15.102:0/1227848125 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7fd7d00f3400 con 0x7fd7e0102110 2024-04-25T02:03:37.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.361+0000 7fd7e5800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7c40a2ea0 0x7fd7c40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.361+0000 7fd7ea76e740 1 -- 172.21.15.102:0/3918864113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 msgr2=0x5595ed44faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.361+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/3918864113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44faa0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009c70 tx=0x7fd7c8011810 comp rx=0 tx=0).stop 2024-04-25T02:03:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.362+0000 7fd7ea76e740 1 -- 172.21.15.102:0/3918864113 shutdown_connections 2024-04-25T02:03:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.362+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/3918864113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 0x5595ed4523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.362+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/3918864113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44faa0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.362+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/3918864113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed444e00 0x5595ed4451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.362+0000 7fd7ea76e740 1 -- 172.21.15.102:0/3918864113 >> 172.21.15.102:0/3918864113 conn(0x5595ed441310 msgr2=0x5595ed4416f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:37.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.362+0000 7fd7e5800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7c40a2ea0 0x7fd7c40a5360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8005fd0 tx=0x7fd7d80095f0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:37.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.362+0000 7fd7ea76e740 1 -- 172.21.15.102:0/3918864113 shutdown_connections 2024-04-25T02:03:37.368 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.362+0000 7fd7ea76e740 1 -- 172.21.15.102:0/3918864113 wait complete. 2024-04-25T02:03:37.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.364+0000 7fd7ea76e740 1 -- start start 2024-04-25T02:03:37.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.364+0000 7fd7ea76e740 1 Processor -- start 2024-04-25T02:03:37.369 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed44ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 0x5595ed44d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7ea76e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5595ed452c10 con 0x5595ed445720 2024-04-25T02:03:37.370 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7ea76e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5595ed364990 con 0x5595ed444e00 2024-04-25T02:03:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7ea76e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5595ed452d90 con 0x5595ed44ffe0 2024-04-25T02:03:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7ea76e740 10 client.0 _ll_register_callbacks cb 0x5595ed366950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7e4e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed44ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.371 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7d7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.365+0000 7fd7e4e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed44ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:43118/0 (socket says 172.21.15.102:43118) 2024-04-25T02:03:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.366+0000 7fd7e4e00640 1 -- 172.21.15.102:0/2546625438 learned_addr learned my addr 172.21.15.102:0/2546625438 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.366+0000 7fd7e5800640 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 0x5595ed44d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.366+0000 7fd7e4e00640 1 -- 172.21.15.102:0/2546625438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 msgr2=0x5595ed44d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.366+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 0x5595ed44d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.366+0000 7fd7e4e00640 1 -- 172.21.15.102:0/2546625438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 msgr2=0x5595ed44d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.366+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.366+0000 7fd7e4e00640 1 -- 172.21.15.102:0/2546625438 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595ed189620 con 0x5595ed444e00 2024-04-25T02:03:37.372 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.366+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.367+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed44ce90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8019480 tx=0x7fd7c80056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c8afe1800357285 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.367+0000 7fd7cea00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:03:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.367+0000 7fd7cea00640 1 -- 172.21.15.102:0/2546625438 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8027070 con 0x5595ed444e00 2024-04-25T02:03:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.367+0000 7fd7cea00640 1 -- 172.21.15.102:0/2546625438 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7fd7c8003800 con 0x5595ed444e00 2024-04-25T02:03:37.373 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.367+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2546625438 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7c80043d0 con 0x5595ed444e00 2024-04-25T02:03:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.367+0000 7fd7cea00640 1 -- 172.21.15.102:0/2546625438 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd7c8031050 con 0x5595ed444e00 2024-04-25T02:03:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.367+0000 7fd7ea76e740 10 client.27230 fetch_fsmap learned FSMap version 37 2024-04-25T02:03:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.367+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2546625438 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5595ed45e9e0 con 0x5595ed444e00 2024-04-25T02:03:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.368+0000 7fd7cea00640 1 -- 172.21.15.102:0/2546625438 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7c80043d0 con 0x5595ed444e00 2024-04-25T02:03:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.369+0000 7fd7cea00640 1 -- 172.21.15.102:0/2546625438 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7fd7c8018070 con 0x5595ed444e00 2024-04-25T02:03:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.369+0000 7fd7ea76e740 10 client.27230 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T02:03:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.369+0000 7fd7ea76e740 10 client.27230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:37.374 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.369+0000 7fd7ea76e740 20 client.27230 populate_metadata read hostname 'smithi102' 2024-04-25T02:03:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.369+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595ed45fde0 0x5595ed4621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.369+0000 7fd7ea76e740 4 client.27230 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:03:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.369+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2546625438 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595ed3b5ff0 con 0x5595ed45fde0 2024-04-25T02:03:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.369+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595ed45fde0 0x5595ed4621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.370+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595ed45fde0 0x5595ed4621d0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e998c618338b4caf server_cookie=33bcb2e90419aab8 in_seq=0 out_seq=0 2024-04-25T02:03:37.375 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.370+0000 7fd7cea00640 10 client.27230 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.370+0000 7fd7cea00640 1 -- 172.21.15.102:0/2546625438 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595ed3b5ff0 con 0x5595ed45fde0 2024-04-25T02:03:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.370+0000 7fd7cea00640 10 client.27230 handle_command_reply: tid=0 2024-04-25T02:03:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7ea76e740 1 client.27230 shutdown 2024-04-25T02:03:37.376 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2546625438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 msgr2=0x5595ed44ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed44ce90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8019480 tx=0x7fd7c80056d0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2546625438 shutdown_connections 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595ed45fde0 0x5595ed4621d0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 0x5595ed44d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2546625438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed44ce90 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2546625438 >> 172.21.15.102:0/2546625438 conn(0x5595ed441310 msgr2=0x5595ed447040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.371+0000 7fd7cea00640 0 client.27230 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.372+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2546625438 shutdown_connections 2024-04-25T02:03:37.377 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.372+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2546625438 wait complete. 2024-04-25T02:03:37.378 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.372+0000 7fd7ea76e740 20 client.27230 trim_cache size 0 max 16384 2024-04-25T02:03:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.398+0000 7fd7ea76e740 1 Processor -- start 2024-04-25T02:03:37.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.398+0000 7fd7ea76e740 1 -- start start 2024-04-25T02:03:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.399+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed45bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.399+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.399+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 0x5595ed44d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.399+0000 7fd7e4e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed45bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.399+0000 7fd7e4e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed45bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:43134/0 (socket says 172.21.15.102:43134) 2024-04-25T02:03:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.399+0000 7fd7e5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 0x5595ed44d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.399+0000 7fd7e5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 0x5595ed44d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:55366/0 (socket says 172.21.15.102:55366) 2024-04-25T02:03:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.399+0000 7fd7e4e00640 1 -- 172.21.15.102:0/2841639800 learned_addr learned my addr 172.21.15.102:0/2841639800 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.399+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2841639800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc0198e0 con 0x5595ed445720 2024-04-25T02:03:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.400+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2841639800 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc0170f0 con 0x5595ed444e00 2024-04-25T02:03:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.400+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2841639800 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc014f60 con 0x5595ed44ffe0 2024-04-25T02:03:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.400+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/2841639800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.400+0000 7fd7e4e00640 1 -- 172.21.15.102:0/2841639800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 msgr2=0x5595ed44d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.400+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/2841639800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 0x5595ed44d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.401+0000 7fd7e4e00640 1 -- 172.21.15.102:0/2841639800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 msgr2=0x5595ed44be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.401+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/2841639800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.401+0000 7fd7e4e00640 1 -- 172.21.15.102:0/2841639800 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595ed43c620 con 0x5595ed444e00 2024-04-25T02:03:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.401+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/2841639800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed45bb00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8000c00 tx=0x7fd7c8019fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b5c4d4f24407263 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.401+0000 7fd7d5600640 1 -- 172.21.15.102:0/2841639800 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c803b070 con 0x5595ed444e00 2024-04-25T02:03:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.402+0000 7fd7d5600640 1 -- 172.21.15.102:0/2841639800 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7c8003ce0 con 0x5595ed444e00 2024-04-25T02:03:37.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.402+0000 7fd7d5600640 1 -- 172.21.15.102:0/2841639800 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8015070 con 0x5595ed444e00 2024-04-25T02:03:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2841639800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 msgr2=0x5595ed45bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2841639800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed45bb00 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8000c00 tx=0x7fd7c8019fa0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2841639800 shutdown_connections 2024-04-25T02:03:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2841639800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed44ffe0 0x5595ed44d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2841639800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed445720 0x5595ed44be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/2841639800 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed444e00 0x5595ed45bb00 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2841639800 >> 172.21.15.102:0/2841639800 conn(0x5595ed441310 msgr2=0x5595ed44b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2841639800 shutdown_connections 2024-04-25T02:03:37.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 -- 172.21.15.102:0/2841639800 wait complete. 2024-04-25T02:03:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.403+0000 7fd7ea76e740 1 -- start start 2024-04-25T02:03:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.405+0000 7fd7ea76e740 1 Processor -- start 2024-04-25T02:03:37.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.405+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed444e00 0x5595ed452fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.405+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed445720 0x5595ed4534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.405+0000 7fd7ea76e740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 0x5595ed453a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.405+0000 7fd7ea76e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc0155b0 con 0x5595ed444e00 2024-04-25T02:03:37.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.405+0000 7fd7ea76e740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc016a20 con 0x5595ed44ffe0 2024-04-25T02:03:37.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7ea76e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc015c80 con 0x5595ed445720 2024-04-25T02:03:37.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7ea76e740 10 client.0 _ll_register_callbacks cb 0x7fd7d8009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed444e00 0x5595ed452fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7d7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed445720 0x5595ed4534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed444e00 0x5595ed452fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53332/0 (socket says 172.21.15.102:53332) 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e4e00640 1 -- 172.21.15.102:0/279213979 learned_addr learned my addr 172.21.15.102:0/279213979 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e5800640 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 0x5595ed453a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e4e00640 1 -- 172.21.15.102:0/279213979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed445720 msgr2=0x5595ed4534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed445720 0x5595ed4534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e4e00640 1 -- 172.21.15.102:0/279213979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 msgr2=0x5595ed453a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 0x5595ed453a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e4e00640 1 -- 172.21.15.102:0/279213979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595ed4d5370 con 0x5595ed444e00 2024-04-25T02:03:37.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.406+0000 7fd7e4e00640 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed444e00 0x5595ed452fa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8019420 tx=0x7fd7c8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3feb99a009b84ec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:37.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.407+0000 7fd7cea00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:03:37.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.407+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed445720 0x5595ed4534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:03:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.412+0000 7fd7cea00640 1 -- 172.21.15.102:0/279213979 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c803b050 con 0x5595ed444e00 2024-04-25T02:03:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.413+0000 7fd7cea00640 1 -- 172.21.15.102:0/279213979 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7fd7c8003ad0 con 0x5595ed444e00 2024-04-25T02:03:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.413+0000 7fd7ea76e740 1 -- 172.21.15.102:0/279213979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7c8003ce0 con 0x5595ed444e00 2024-04-25T02:03:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.413+0000 7fd7cea00640 1 -- 172.21.15.102:0/279213979 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd7c8015040 con 0x5595ed444e00 2024-04-25T02:03:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.413+0000 7fd7ea76e740 10 client.16620 fetch_fsmap learned FSMap version 37 2024-04-25T02:03:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.413+0000 7fd7ea76e740 1 -- 172.21.15.102:0/279213979 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5595ed4d6660 con 0x5595ed444e00 2024-04-25T02:03:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.414+0000 7fd7cea00640 1 -- 172.21.15.102:0/279213979 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7c8003ce0 con 0x5595ed444e00 2024-04-25T02:03:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.414+0000 7fd7cea00640 1 -- 172.21.15.102:0/279213979 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 37) ==== 8358+0+0 (secure 0 0 0) 0x7fd7c8018050 con 0x5595ed444e00 2024-04-25T02:03:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.414+0000 7fd7ea76e740 10 client.16620 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T02:03:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.414+0000 7fd7ea76e740 10 client.16620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.414+0000 7fd7ea76e740 20 client.16620 populate_metadata read hostname 'smithi102' 2024-04-25T02:03:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.415+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595ed4d6c80 0x5595ed4d90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.415+0000 7fd7ea76e740 4 client.16620 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:03:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.415+0000 7fd7ea76e740 1 -- 172.21.15.102:0/279213979 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595ed3b0340 con 0x5595ed4d6c80 2024-04-25T02:03:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.416+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595ed4d6c80 0x5595ed4d90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:37.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.416+0000 7fd7d7e00640 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595ed4d6c80 0x5595ed4d90a0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4734097335e1e3e0 server_cookie=ef86a427a2ef71bf in_seq=0 out_seq=0 2024-04-25T02:03:37.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.417+0000 7fd7cea00640 10 client.16620 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.426+0000 7fd7cea00640 1 -- 172.21.15.102:0/279213979 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595ed3b0340 con 0x5595ed4d6c80 2024-04-25T02:03:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7cea00640 10 client.16620 handle_command_reply: tid=0 2024-04-25T02:03:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7ea76e740 1 client.16620 shutdown 2024-04-25T02:03:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7ea76e740 1 -- 172.21.15.102:0/279213979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed444e00 msgr2=0x5595ed452fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed444e00 0x5595ed452fa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8019420 tx=0x7fd7c8009f60 comp rx=0 tx=0).stop 2024-04-25T02:03:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7ea76e740 1 -- 172.21.15.102:0/279213979 shutdown_connections 2024-04-25T02:03:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5595ed4d6c80 0x5595ed4d90a0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5595ed44ffe0 0x5595ed453a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ed445720 0x5595ed4534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7cea00640 0 client.16620 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7ea76e740 1 --2- 172.21.15.102:0/279213979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5595ed444e00 0x5595ed452fa0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.427+0000 7fd7ea76e740 1 -- 172.21.15.102:0/279213979 >> 172.21.15.102:0/279213979 conn(0x5595ed441310 msgr2=0x5595ed447010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.428+0000 7fd7ea76e740 1 -- 172.21.15.102:0/279213979 shutdown_connections 2024-04-25T02:03:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.428+0000 7fd7ea76e740 1 -- 172.21.15.102:0/279213979 wait complete. 2024-04-25T02:03:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.428+0000 7fd7ea76e740 20 client.16620 trim_cache size 0 max 16384 2024-04-25T02:03:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:03:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:03:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:03:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:03:37.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 -- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7c40a2ea0 msgr2=0x7fd7c40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:37.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7c40a2ea0 0x7fd7c40a5360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8005fd0 tx=0x7fd7d80095f0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 -- 172.21.15.102:0/1227848125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0102110 msgr2=0x7fd7e0196320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:37.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0102110 0x7fd7e0196320 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd7d002ebb0 tx=0x7fd7d000f340 comp rx=0 tx=0).stop 2024-04-25T02:03:37.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 -- 172.21.15.102:0/1227848125 shutdown_connections 2024-04-25T02:03:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7c40a2ea0 0x7fd7c40a5360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7e0107760 0x7fd7e019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7e0102110 0x7fd7e0196320 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 --2- 172.21.15.102:0/1227848125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e0101750 0x7fd7e0195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 -- 172.21.15.102:0/1227848125 >> 172.21.15.102:0/1227848125 conn(0x7fd7e0078bd0 msgr2=0x7fd7e00fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 -- 172.21.15.102:0/1227848125 shutdown_connections 2024-04-25T02:03:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:37.429+0000 7fd7e7800640 1 -- 172.21.15.102:0/1227848125 wait complete. 2024-04-25T02:03:37.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:37 smithi102 ceph-mon[27796]: pgmap v257: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 10 MiB/s wr, 586 op/s 2024-04-25T02:03:37.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:37 smithi110 ceph-mon[32848]: pgmap v257: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 10 MiB/s wr, 586 op/s 2024-04-25T02:03:37.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:37 smithi193 ceph-mon[32981]: pgmap v257: 97 pgs: 97 active+clean; 4.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 10 MiB/s wr, 586 op/s 2024-04-25T02:03:38.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 465.0 s, 1506.4 tps, lat 66.449 ms stddev 127.737 2024-04-25T02:03:39.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:39 smithi102 ceph-mon[27796]: pgmap v258: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 MiB/s wr, 1.95k op/s 2024-04-25T02:03:39.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:39 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:03:39.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:39 smithi110 ceph-mon[32848]: pgmap v258: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 MiB/s wr, 1.95k op/s 2024-04-25T02:03:39.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:39 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:03:39.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:39 smithi193 ceph-mon[32981]: pgmap v258: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 MiB/s wr, 1.95k op/s 2024-04-25T02:03:39.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:39 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:03:41.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:41 smithi102 ceph-mon[27796]: pgmap v259: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 MiB/s wr, 1.63k op/s 2024-04-25T02:03:41.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:41 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:03:41.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:41 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:03:41.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:41 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:03:41.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:41 smithi110 ceph-mon[32848]: pgmap v259: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 MiB/s wr, 1.63k op/s 2024-04-25T02:03:41.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:41 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:03:41.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:41 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:03:41.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:41 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:03:41.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:41 smithi193 ceph-mon[32981]: pgmap v259: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 MiB/s wr, 1.63k op/s 2024-04-25T02:03:41.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:41 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:03:41.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:41 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:03:41.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:41 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:03:43.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 470.0 s, 1495.0 tps, lat 63.561 ms stddev 88.019 2024-04-25T02:03:43.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:43 smithi110 ceph-mon[32848]: pgmap v260: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 26 MiB/s wr, 1.80k op/s 2024-04-25T02:03:43.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:43 smithi193 ceph-mon[32981]: pgmap v260: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 26 MiB/s wr, 1.80k op/s 2024-04-25T02:03:43.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:43 smithi102 ceph-mon[27796]: pgmap v260: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 26 MiB/s wr, 1.80k op/s 2024-04-25T02:03:45.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:45 smithi110 ceph-mon[32848]: pgmap v261: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 MiB/s wr, 1.77k op/s 2024-04-25T02:03:45.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:45 smithi193 ceph-mon[32981]: pgmap v261: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 MiB/s wr, 1.77k op/s 2024-04-25T02:03:45.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:45 smithi102 ceph-mon[27796]: pgmap v261: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 26 MiB/s wr, 1.77k op/s 2024-04-25T02:03:47.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:47 smithi110 ceph-mon[32848]: pgmap v262: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 MiB/s wr, 1.66k op/s 2024-04-25T02:03:47.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:47 smithi193 ceph-mon[32981]: pgmap v262: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 MiB/s wr, 1.66k op/s 2024-04-25T02:03:47.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:47 smithi102 ceph-mon[27796]: pgmap v262: 97 pgs: 97 active+clean; 4.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 MiB/s wr, 1.66k op/s 2024-04-25T02:03:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 475.0 s, 2002.8 tps, lat 52.164 ms stddev 70.952 2024-04-25T02:03:48.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:48 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:03:48.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:48 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:03:48.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:48 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:03:49.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:49 smithi110 ceph-mon[32848]: pgmap v263: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 32 MiB/s wr, 2.39k op/s 2024-04-25T02:03:49.759 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:49 smithi193 ceph-mon[32981]: pgmap v263: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 32 MiB/s wr, 2.39k op/s 2024-04-25T02:03:49.828 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:49 smithi102 ceph-mon[27796]: pgmap v263: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 32 MiB/s wr, 2.39k op/s 2024-04-25T02:03:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.635+0000 7f6734200640 1 -- 172.21.15.102:0/1643113732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c101750 msgr2=0x7f672c101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:50.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.635+0000 7f6734200640 1 --2- 172.21.15.102:0/1643113732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c101750 0x7f672c101bd0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6718009860 tx=0x7f671802a360 comp rx=0 tx=0).stop 2024-04-25T02:03:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.636+0000 7f6734200640 1 -- 172.21.15.102:0/1643113732 shutdown_connections 2024-04-25T02:03:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.636+0000 7f6734200640 1 --2- 172.21.15.102:0/1643113732 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f672c102110 0x7f672c10e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.636+0000 7f6734200640 1 --2- 172.21.15.102:0/1643113732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c101750 0x7f672c101bd0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.636+0000 7f6734200640 1 --2- 172.21.15.102:0/1643113732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f672c107760 0x7f672c107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.636+0000 7f6734200640 1 -- 172.21.15.102:0/1643113732 >> 172.21.15.102:0/1643113732 conn(0x7f672c078bd0 msgr2=0x7f672c078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.637+0000 7f6734200640 1 -- 172.21.15.102:0/1643113732 shutdown_connections 2024-04-25T02:03:50.642 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.637+0000 7f6734200640 1 -- 172.21.15.102:0/1643113732 wait complete. 2024-04-25T02:03:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.638+0000 7f6734200640 1 Processor -- start 2024-04-25T02:03:50.643 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.638+0000 7f6734200640 1 -- start start 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.638+0000 7f6734200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f672c101750 0x7f672c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.638+0000 7f6734200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c102110 0x7f672c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.638+0000 7f6734200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f672c107760 0x7f672c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.638+0000 7f6734200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f672c111b30 con 0x7f672c101750 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.638+0000 7f6734200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1119b0 con 0x7f672c102110 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.638+0000 7f6734200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f672c111cb0 con 0x7f672c107760 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6730e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c102110 0x7f672c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6731800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f672c101750 0x7f672c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6730e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c102110 0x7f672c1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58874/0 (socket says 172.21.15.102:58874) 2024-04-25T02:03:50.645 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6730e00640 1 -- 172.21.15.102:0/2871854676 learned_addr learned my addr 172.21.15.102:0/2871854676 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6732200640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f672c107760 0x7f672c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6730e00640 1 -- 172.21.15.102:0/2871854676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f672c107760 msgr2=0x7f672c19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6730e00640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f672c107760 0x7f672c19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6730e00640 1 -- 172.21.15.102:0/2871854676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f672c101750 msgr2=0x7f672c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6730e00640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f672c101750 0x7f672c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6730e00640 1 -- 172.21.15.102:0/2871854676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672c19cf80 con 0x7f672c102110 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.639+0000 7f6730e00640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c102110 0x7f672c1984d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f67180057c0 tx=0x7f6718033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.640+0000 7f672a000640 1 -- 172.21.15.102:0/2871854676 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718037040 con 0x7f672c102110 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.640+0000 7f672a000640 1 -- 172.21.15.102:0/2871854676 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6718040070 con 0x7f672c102110 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.640+0000 7f672a000640 1 -- 172.21.15.102:0/2871854676 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67180039d0 con 0x7f672c102110 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.640+0000 7f6734200640 1 -- 172.21.15.102:0/2871854676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672c19d210 con 0x7f672c102110 2024-04-25T02:03:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.640+0000 7f6734200640 1 -- 172.21.15.102:0/2871854676 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f672c1a4b00 con 0x7f672c102110 2024-04-25T02:03:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.644+0000 7f672a000640 1 -- 172.21.15.102:0/2871854676 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6718003b70 con 0x7f672c102110 2024-04-25T02:03:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.644+0000 7f6737169740 1 Processor -- start 2024-04-25T02:03:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.644+0000 7f6737169740 1 -- start start 2024-04-25T02:03:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6737169740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b949e00 0x561a7b94a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6737169740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 0x561a7b954aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.654 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6737169740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9573d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.655 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6737169740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561a7b8adab0 con 0x561a7b949e00 2024-04-25T02:03:50.655 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6737169740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561a7b856640 con 0x561a7b94a720 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6737169740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561a7b8a9a70 con 0x561a7b954fe0 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6730e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6732200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b949e00 0x561a7b94a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6732200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b949e00 0x561a7b94a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39514/0 (socket says 172.21.15.102:39514) 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6730e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49714/0 (socket says 172.21.15.102:49714) 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.645+0000 7f6730e00640 1 -- 172.21.15.102:0/672797589 learned_addr learned my addr 172.21.15.102:0/672797589 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.646+0000 7f6730e00640 1 -- 172.21.15.102:0/672797589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 msgr2=0x561a7b954aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.646+0000 7f6730e00640 1 --2- 172.21.15.102:0/672797589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 0x561a7b954aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.646+0000 7f6730e00640 1 -- 172.21.15.102:0/672797589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b949e00 msgr2=0x561a7b94a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.646+0000 7f6730e00640 1 --2- 172.21.15.102:0/672797589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b949e00 0x561a7b94a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.646+0000 7f6730e00640 1 -- 172.21.15.102:0/672797589 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a7b5c2210 con 0x561a7b954fe0 2024-04-25T02:03:50.656 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.646+0000 7f6730e00640 1 --2- 172.21.15.102:0/672797589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9573d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f67180ec550 tx=0x7f67180f2e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=824dc6fb77508867 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.646+0000 7f672a000640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f67140a2e20 0x7f67140a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.647+0000 7f672a000640 1 -- 172.21.15.102:0/2871854676 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f67180eee10 con 0x7f672c102110 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.647+0000 7f671ea00640 1 -- 172.21.15.102:0/672797589 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67180b7680 con 0x561a7b954fe0 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.647+0000 7f671ea00640 1 -- 172.21.15.102:0/672797589 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67180fc070 con 0x561a7b954fe0 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.647+0000 7f671ea00640 1 -- 172.21.15.102:0/672797589 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67180bad40 con 0x561a7b954fe0 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.647+0000 7f6732200640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f67140a2e20 0x7f67140a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.648+0000 7f6737169740 1 -- 172.21.15.102:0/672797589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 msgr2=0x561a7b9573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.648+0000 7f6737169740 1 --2- 172.21.15.102:0/672797589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9573d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f67180ec550 tx=0x7f67180f2e20 comp rx=0 tx=0).stop 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.648+0000 7f6737169740 1 -- 172.21.15.102:0/672797589 shutdown_connections 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.648+0000 7f6737169740 1 --2- 172.21.15.102:0/672797589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9573d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.648+0000 7f6737169740 1 --2- 172.21.15.102:0/672797589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 0x561a7b954aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.648+0000 7f6737169740 1 --2- 172.21.15.102:0/672797589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b949e00 0x561a7b94a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.648+0000 7f6737169740 1 -- 172.21.15.102:0/672797589 >> 172.21.15.102:0/672797589 conn(0x561a7b946310 msgr2=0x561a7b9466f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:50.657 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.649+0000 7f6737169740 1 -- 172.21.15.102:0/672797589 shutdown_connections 2024-04-25T02:03:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.649+0000 7f6732200640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f67140a2e20 0x7f67140a52e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6724009a60 tx=0x7f6724010040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:50.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.649+0000 7f6737169740 1 -- 172.21.15.102:0/672797589 wait complete. 2024-04-25T02:03:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.650+0000 7f6737169740 1 -- start start 2024-04-25T02:03:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.650+0000 7f6737169740 1 Processor -- start 2024-04-25T02:03:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.652+0000 7f6737169740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b949e00 0x561a7b951f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.652+0000 7f6737169740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 0x561a7b952470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.652+0000 7f6737169740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9529b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.652+0000 7f6731800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b949e00 0x561a7b951f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.652+0000 7f6737169740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561a7b9579d0 con 0x561a7b94a720 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.652+0000 7f6731800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b949e00 0x561a7b951f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58894/0 (socket says 172.21.15.102:58894) 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.652+0000 7f6731800640 1 -- 172.21.15.102:0/3215160765 learned_addr learned my addr 172.21.15.102:0/3215160765 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.652+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x561a7b869990 con 0x561a7b949e00 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.652+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561a7b957b50 con 0x561a7b954fe0 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6732200640 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6730e00640 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 0x561a7b952470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6731800640 1 -- 172.21.15.102:0/3215160765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 msgr2=0x561a7b9529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6731800640 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6731800640 1 -- 172.21.15.102:0/3215160765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 msgr2=0x561a7b952470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6731800640 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 0x561a7b952470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6731800640 1 -- 172.21.15.102:0/3215160765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a7b68e620 con 0x561a7b949e00 2024-04-25T02:03:50.662 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6730e00640 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 0x561a7b952470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6732200640 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6737169740 10 client.0 _ll_register_callbacks cb 0x561a7b86b950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.653+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67180fc070 con 0x561a7b949e00 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.654+0000 7f6731800640 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b949e00 0x561a7b951f30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f672000be00 tx=0x7f67200107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cd455e5dd56dbe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.654+0000 7f6713400640 10 client.27245 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.654+0000 7f6713400640 1 -- 172.21.15.102:0/3215160765 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720011850 con 0x561a7b949e00 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.654+0000 7f6713400640 1 -- 172.21.15.102:0/3215160765 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f67200119f0 con 0x561a7b949e00 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.654+0000 7f6713400640 1 -- 172.21.15.102:0/3215160765 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6720011bf0 con 0x561a7b949e00 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.654+0000 7f6737169740 10 client.27245 fetch_fsmap learned FSMap version 38 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.654+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=38}) -- 0x561a7b9636d0 con 0x561a7b949e00 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.655+0000 7f6713400640 1 -- 172.21.15.102:0/3215160765 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f6720026420 con 0x561a7b949e00 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.655+0000 7f6737169740 10 client.27245 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.655+0000 7f6737169740 10 client.27245 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.655+0000 7f6737169740 20 client.27245 populate_metadata read hostname 'smithi102' 2024-04-25T02:03:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.655+0000 7f6737169740 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561a7b964a80 0x561a7b966f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.655+0000 7f6737169740 4 client.27245 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:03:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.655+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561a7b8baff0 con 0x561a7b964a80 2024-04-25T02:03:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.655+0000 7f6730e00640 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561a7b964a80 0x561a7b966f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.656+0000 7f6730e00640 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561a7b964a80 0x561a7b966f40 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2ba9b182750e175 server_cookie=f09717325c8f5505 in_seq=0 out_seq=0 2024-04-25T02:03:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.656+0000 7f6713400640 10 client.27245 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.656+0000 7f6713400640 1 -- 172.21.15.102:0/3215160765 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561a7b8baff0 con 0x561a7b964a80 2024-04-25T02:03:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.656+0000 7f6713400640 10 client.27245 handle_command_reply: tid=0 2024-04-25T02:03:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 client.27245 shutdown 2024-04-25T02:03:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b949e00 msgr2=0x561a7b951f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b949e00 0x561a7b951f30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f672000be00 tx=0x7f67200107a0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 shutdown_connections 2024-04-25T02:03:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561a7b964a80 0x561a7b966f40 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b954fe0 0x561a7b9529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 0x561a7b952470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 --2- 172.21.15.102:0/3215160765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b949e00 0x561a7b951f30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 >> 172.21.15.102:0/3215160765 conn(0x561a7b946310 msgr2=0x561a7b94c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:50.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6713400640 0 client.27245 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:50.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 shutdown_connections 2024-04-25T02:03:50.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.657+0000 7f6737169740 1 -- 172.21.15.102:0/3215160765 wait complete. 2024-04-25T02:03:50.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.658+0000 7f6737169740 20 client.27245 trim_cache size 0 max 16384 2024-04-25T02:03:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.700+0000 7f6737169740 1 Processor -- start 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.700+0000 7f6737169740 1 -- start start 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.701+0000 7f6737169740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b949e00 0x561a7b954700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.701+0000 7f6737169740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 0x561a7b945c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.701+0000 7f6737169740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b954fe0 0x561a7b94c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.701+0000 7f6737169740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1b1f20 con 0x561a7b94a720 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.701+0000 7f6737169740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1af730 con 0x561a7b954fe0 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.701+0000 7f6737169740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1ad5a0 con 0x561a7b949e00 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.701+0000 7f6731800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b949e00 0x561a7b954700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.701+0000 7f6731800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b949e00 0x561a7b954700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:49722/0 (socket says 172.21.15.102:49722) 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.701+0000 7f6731800640 1 -- 172.21.15.102:0/457462513 learned_addr learned my addr 172.21.15.102:0/457462513 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.702+0000 7f6731800640 1 -- 172.21.15.102:0/457462513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b954fe0 msgr2=0x561a7b94c810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.702+0000 7f6731800640 1 --2- 172.21.15.102:0/457462513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b954fe0 0x561a7b94c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.702+0000 7f6731800640 1 -- 172.21.15.102:0/457462513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 msgr2=0x561a7b945c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:03:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.702+0000 7f6731800640 1 --2- 172.21.15.102:0/457462513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 0x561a7b945c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.702+0000 7f6731800640 1 -- 172.21.15.102:0/457462513 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561a7b94cd50 con 0x561a7b949e00 2024-04-25T02:03:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.703+0000 7f6731800640 1 --2- 172.21.15.102:0/457462513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b949e00 0x561a7b954700 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f672000c000 tx=0x7f6720013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4d3d504ab3f65e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.704+0000 7f671e000640 1 -- 172.21.15.102:0/457462513 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720000e60 con 0x561a7b949e00 2024-04-25T02:03:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.704+0000 7f671e000640 1 -- 172.21.15.102:0/457462513 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67180fc070 con 0x561a7b949e00 2024-04-25T02:03:50.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.705+0000 7f6737169740 1 -- 172.21.15.102:0/457462513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b949e00 msgr2=0x561a7b954700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.705+0000 7f6737169740 1 --2- 172.21.15.102:0/457462513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b949e00 0x561a7b954700 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f672000c000 tx=0x7f6720013000 comp rx=0 tx=0).stop 2024-04-25T02:03:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.705+0000 7f6737169740 1 -- 172.21.15.102:0/457462513 shutdown_connections 2024-04-25T02:03:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.705+0000 7f6737169740 1 --2- 172.21.15.102:0/457462513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b954fe0 0x561a7b94c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.705+0000 7f6737169740 1 --2- 172.21.15.102:0/457462513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b94a720 0x561a7b945c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.705+0000 7f6737169740 1 --2- 172.21.15.102:0/457462513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b949e00 0x561a7b954700 secure :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f672000c000 tx=0x7f6720013000 comp rx=0 tx=0).stop 2024-04-25T02:03:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.705+0000 7f6737169740 1 -- 172.21.15.102:0/457462513 >> 172.21.15.102:0/457462513 conn(0x561a7b946310 msgr2=0x561a7b953ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.705+0000 7f6737169740 1 -- 172.21.15.102:0/457462513 shutdown_connections 2024-04-25T02:03:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.705+0000 7f6737169740 1 -- 172.21.15.102:0/457462513 wait complete. 2024-04-25T02:03:50.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.706+0000 7f6737169740 1 -- start start 2024-04-25T02:03:50.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.706+0000 7f6737169740 1 Processor -- start 2024-04-25T02:03:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.710+0000 7f6737169740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 0x561a7b9cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.710+0000 7f6737169740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b954fe0 0x561a7b95ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.710+0000 7f6731800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 0x561a7b9cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.710+0000 7f6731800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 0x561a7b9cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58914/0 (socket says 172.21.15.102:58914) 2024-04-25T02:03:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.711+0000 7f6737169740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b95cf50 0x561a7b94eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.711+0000 7f6731800640 1 -- 172.21.15.102:0/740161298 learned_addr learned my addr 172.21.15.102:0/740161298 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:03:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.711+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1adbf0 con 0x561a7b954fe0 2024-04-25T02:03:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.711+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1af060 con 0x561a7b94a720 2024-04-25T02:03:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.711+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f672c1ae2c0 con 0x561a7b95cf50 2024-04-25T02:03:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.711+0000 7f6737169740 10 client.0 _ll_register_callbacks cb 0x7f672000b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:03:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.711+0000 7f6730e00640 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b954fe0 0x561a7b95ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.711+0000 7f6732200640 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b95cf50 0x561a7b94eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.712+0000 7f6731800640 1 -- 172.21.15.102:0/740161298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b95cf50 msgr2=0x561a7b94eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.712+0000 7f6731800640 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b95cf50 0x561a7b94eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.712+0000 7f6731800640 1 -- 172.21.15.102:0/740161298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b954fe0 msgr2=0x561a7b95ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:03:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.712+0000 7f6731800640 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b954fe0 0x561a7b95ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.712+0000 7f6731800640 1 -- 172.21.15.102:0/740161298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561a7b94f420 con 0x561a7b94a720 2024-04-25T02:03:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.713+0000 7f6730e00640 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b954fe0 0x561a7b95ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:03:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.714+0000 7f6731800640 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 0x561a7b9cf310 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6720010fa0 tx=0x7f6720010fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5aeaf43b02e62de9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:03:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.714+0000 7f6713400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:03:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.714+0000 7f6713400640 1 -- 172.21.15.102:0/740161298 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720026420 con 0x561a7b94a720 2024-04-25T02:03:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.714+0000 7f6713400640 1 -- 172.21.15.102:0/740161298 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7f6720011650 con 0x561a7b94a720 2024-04-25T02:03:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.715+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67180fc070 con 0x561a7b94a720 2024-04-25T02:03:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.715+0000 7f6713400640 1 -- 172.21.15.102:0/740161298 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6720040050 con 0x561a7b94a720 2024-04-25T02:03:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.715+0000 7f6737169740 10 client.27248 fetch_fsmap learned FSMap version 38 2024-04-25T02:03:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.715+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561a7b9e5020 con 0x561a7b94a720 2024-04-25T02:03:50.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.715+0000 7f6713400640 1 -- 172.21.15.102:0/740161298 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67180fc070 con 0x561a7b94a720 2024-04-25T02:03:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.716+0000 7f6713400640 1 -- 172.21.15.102:0/740161298 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f6720004a20 con 0x561a7b94a720 2024-04-25T02:03:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.716+0000 7f6737169740 10 client.27248 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:03:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.716+0000 7f6737169740 10 client.27248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:03:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.716+0000 7f6737169740 20 client.27248 populate_metadata read hostname 'smithi102' 2024-04-25T02:03:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.716+0000 7f6737169740 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561a7b9e56d0 0x561a7b9e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:03:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.716+0000 7f6737169740 4 client.27248 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:03:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.716+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561a7b8b5340 con 0x561a7b9e56d0 2024-04-25T02:03:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.717+0000 7f6730e00640 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561a7b9e56d0 0x561a7b9e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:03:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.718+0000 7f6730e00640 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561a7b9e56d0 0x561a7b9e7ac0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=874b7352b7d5ea5 server_cookie=27edcc923a6f150 in_seq=0 out_seq=0 2024-04-25T02:03:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.718+0000 7f6713400640 10 client.27248 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:50.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:50 smithi110 ceph-mon[32848]: pgmap v264: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-25T02:03:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.741+0000 7f6713400640 1 -- 172.21.15.102:0/740161298 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561a7b8b5340 con 0x561a7b9e56d0 2024-04-25T02:03:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.741+0000 7f6713400640 10 client.27248 handle_command_reply: tid=0 2024-04-25T02:03:50.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.741+0000 7f6737169740 1 client.27248 shutdown 2024-04-25T02:03:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.742+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 msgr2=0x561a7b9cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.742+0000 7f6737169740 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 0x561a7b9cf310 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6720010fa0 tx=0x7f6720010fd0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.742+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 shutdown_connections 2024-04-25T02:03:50.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.742+0000 7f6737169740 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x561a7b9e56d0 0x561a7b9e7ac0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.742+0000 7f6737169740 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561a7b95cf50 0x561a7b94eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.742+0000 7f6737169740 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561a7b954fe0 0x561a7b95ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.742+0000 7f6737169740 1 --2- 172.21.15.102:0/740161298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561a7b94a720 0x561a7b9cf310 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.742+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 >> 172.21.15.102:0/740161298 conn(0x561a7b946310 msgr2=0x561a7b944bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.742+0000 7f6713400640 0 client.27248 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:03:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.743+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 shutdown_connections 2024-04-25T02:03:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.743+0000 7f6737169740 1 -- 172.21.15.102:0/740161298 wait complete. 2024-04-25T02:03:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.743+0000 7f6737169740 20 client.27248 trim_cache size 0 max 16384 2024-04-25T02:03:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:03:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:03:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:03:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.743+0000 7f6734200640 1 -- 172.21.15.102:0/2871854676 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f67140a2e20 msgr2=0x7f67140a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.743+0000 7f6734200640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f67140a2e20 0x7f67140a52e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6724009a60 tx=0x7f6724010040 comp rx=0 tx=0).stop 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 -- 172.21.15.102:0/2871854676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c102110 msgr2=0x7f672c1984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c102110 0x7f672c1984d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f67180057c0 tx=0x7f6718033d90 comp rx=0 tx=0).stop 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 -- 172.21.15.102:0/2871854676 shutdown_connections 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f67140a2e20 0x7f67140a52e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f672c107760 0x7f672c19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c102110 0x7f672c1984d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 --2- 172.21.15.102:0/2871854676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f672c101750 0x7f672c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 -- 172.21.15.102:0/2871854676 >> 172.21.15.102:0/2871854676 conn(0x7f672c078bd0 msgr2=0x7f672c0fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 -- 172.21.15.102:0/2871854676 shutdown_connections 2024-04-25T02:03:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:03:50.744+0000 7f6734200640 1 -- 172.21.15.102:0/2871854676 wait complete. 2024-04-25T02:03:50.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:50 smithi193 ceph-mon[32981]: pgmap v264: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-25T02:03:50.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:50 smithi102 ceph-mon[27796]: pgmap v264: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 12 MiB/s wr, 1.02k op/s 2024-04-25T02:03:53.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 480.0 s, 2073.6 tps, lat 48.475 ms stddev 59.813 2024-04-25T02:03:53.180 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:52 smithi102 ceph-mon[27796]: pgmap v265: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 15 MiB/s wr, 1.18k op/s 2024-04-25T02:03:53.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:52 smithi110 ceph-mon[32848]: pgmap v265: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 15 MiB/s wr, 1.18k op/s 2024-04-25T02:03:53.259 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:52 smithi193 ceph-mon[32981]: pgmap v265: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 15 MiB/s wr, 1.18k op/s 2024-04-25T02:03:55.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:54 smithi102 ceph-mon[27796]: pgmap v266: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 14 MiB/s wr, 1.13k op/s 2024-04-25T02:03:55.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:54 smithi110 ceph-mon[32848]: pgmap v266: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 14 MiB/s wr, 1.13k op/s 2024-04-25T02:03:55.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:54 smithi193 ceph-mon[32981]: pgmap v266: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 14 MiB/s wr, 1.13k op/s 2024-04-25T02:03:57.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:56 smithi102 ceph-mon[27796]: pgmap v267: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 12 MiB/s wr, 1.00k op/s 2024-04-25T02:03:57.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:56 smithi110 ceph-mon[32848]: pgmap v267: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 12 MiB/s wr, 1.00k op/s 2024-04-25T02:03:57.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:56 smithi193 ceph-mon[32981]: pgmap v267: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 12 MiB/s wr, 1.00k op/s 2024-04-25T02:03:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 485.0 s, 2029.5 tps, lat 49.107 ms stddev 65.220 2024-04-25T02:03:59.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:03:58 smithi102 ceph-mon[27796]: pgmap v268: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 16 MiB/s wr, 1.34k op/s 2024-04-25T02:03:59.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:03:58 smithi110 ceph-mon[32848]: pgmap v268: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 16 MiB/s wr, 1.34k op/s 2024-04-25T02:03:59.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:03:58 smithi193 ceph-mon[32981]: pgmap v268: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 16 MiB/s wr, 1.34k op/s 2024-04-25T02:04:01.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:00 smithi102 ceph-mon[27796]: pgmap v269: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.0 MiB/s wr, 613 op/s 2024-04-25T02:04:01.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:00 smithi110 ceph-mon[32848]: pgmap v269: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.0 MiB/s wr, 613 op/s 2024-04-25T02:04:01.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:00 smithi193 ceph-mon[32981]: pgmap v269: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 8.0 MiB/s wr, 613 op/s 2024-04-25T02:04:03.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 490.0 s, 1332.8 tps, lat 73.428 ms stddev 108.053 2024-04-25T02:04:03.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:02 smithi102 ceph-mon[27796]: pgmap v270: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 10 MiB/s wr, 784 op/s 2024-04-25T02:04:03.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:02 smithi110 ceph-mon[32848]: pgmap v270: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 10 MiB/s wr, 784 op/s 2024-04-25T02:04:03.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:02 smithi193 ceph-mon[32981]: pgmap v270: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 10 MiB/s wr, 784 op/s 2024-04-25T02:04:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.918+0000 7efdd4200640 1 -- 172.21.15.102:0/1808382253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 msgr2=0x7efdcc073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.918+0000 7efdd4200640 1 --2- 172.21.15.102:0/1808382253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 0x7efdcc073e20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009900 tx=0x7efdc002a480 comp rx=0 tx=0).stop 2024-04-25T02:04:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.918+0000 7efdd4200640 1 -- 172.21.15.102:0/1808382253 shutdown_connections 2024-04-25T02:04:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.918+0000 7efdd4200640 1 --2- 172.21.15.102:0/1808382253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdcc10e030 0x7efdcc110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.918+0000 7efdd4200640 1 --2- 172.21.15.102:0/1808382253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdcc074360 0x7efdcc10daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.918+0000 7efdd4200640 1 --2- 172.21.15.102:0/1808382253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 0x7efdcc073e20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.918+0000 7efdd4200640 1 -- 172.21.15.102:0/1808382253 >> 172.21.15.102:0/1808382253 conn(0x7efdcc078bd0 msgr2=0x7efdcc078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:03.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.919+0000 7efdd4200640 1 -- 172.21.15.102:0/1808382253 shutdown_connections 2024-04-25T02:04:03.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.919+0000 7efdd4200640 1 -- 172.21.15.102:0/1808382253 wait complete. 2024-04-25T02:04:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.919+0000 7efdd4200640 1 Processor -- start 2024-04-25T02:04:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd4200640 1 -- start start 2024-04-25T02:04:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd4200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 0x7efdcc19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd4200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdcc074360 0x7efdcc19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd4200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdcc10e030 0x7efdcc19e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd4200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc113db0 con 0x7efdcc074360 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd4200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc113c30 con 0x7efdcc10e030 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd4200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc113f30 con 0x7efdcc073a40 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 0x7efdcc19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdcc074360 0x7efdcc19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 0x7efdcc19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:47868/0 (socket says 172.21.15.102:47868) 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd1800640 1 -- 172.21.15.102:0/3426879802 learned_addr learned my addr 172.21.15.102:0/3426879802 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.920+0000 7efdd2200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdcc10e030 0x7efdcc19e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.921+0000 7efdd2200640 1 -- 172.21.15.102:0/3426879802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 msgr2=0x7efdcc19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.921+0000 7efdd2200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 0x7efdcc19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.921+0000 7efdd2200640 1 -- 172.21.15.102:0/3426879802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdcc074360 msgr2=0x7efdcc19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.921+0000 7efdd2200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdcc074360 0x7efdcc19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.921+0000 7efdd2200640 1 -- 172.21.15.102:0/3426879802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc19f110 con 0x7efdcc10e030 2024-04-25T02:04:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.921+0000 7efdd1800640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 0x7efdcc19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.921+0000 7efdd2200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdcc10e030 0x7efdcc19e990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7efdc400ebc0 tx=0x7efdc400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.922+0000 7efdca000640 1 -- 172.21.15.102:0/3426879802 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc4003b70 con 0x7efdcc10e030 2024-04-25T02:04:03.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.922+0000 7efdd4200640 1 -- 172.21.15.102:0/3426879802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc19f400 con 0x7efdcc10e030 2024-04-25T02:04:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.922+0000 7efdca000640 1 -- 172.21.15.102:0/3426879802 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc4003d10 con 0x7efdcc10e030 2024-04-25T02:04:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.922+0000 7efdca000640 1 -- 172.21.15.102:0/3426879802 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc4010720 con 0x7efdcc10e030 2024-04-25T02:04:03.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.922+0000 7efdd4200640 1 -- 172.21.15.102:0/3426879802 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc1a6c40 con 0x7efdcc10e030 2024-04-25T02:04:03.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.925+0000 7efdd6fb5740 1 Processor -- start 2024-04-25T02:04:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.925+0000 7efdca000640 1 -- 172.21.15.102:0/3426879802 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7efdc4004020 con 0x7efdcc10e030 2024-04-25T02:04:03.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.926+0000 7efdd6fb5740 1 -- start start 2024-04-25T02:04:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.926+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7feca41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.926+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca4720 0x55f7fecaeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.926+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7fecaefe0 0x55f7fecb13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.926+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f7fec07ab0 con 0x55f7feca4720 2024-04-25T02:04:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.926+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f7febb0640 con 0x55f7fecaefe0 2024-04-25T02:04:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.926+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f7fec03a70 con 0x55f7feca3e00 2024-04-25T02:04:03.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd1800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca4720 0x55f7fecaeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd0e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7fecaefe0 0x55f7fecb13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd0e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7fecaefe0 0x55f7fecb13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:35826/0 (socket says 172.21.15.102:35826) 2024-04-25T02:04:03.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd0e00640 1 -- 172.21.15.102:0/197654759 learned_addr learned my addr 172.21.15.102:0/197654759 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd1800640 1 -- 172.21.15.102:0/197654759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 msgr2=0x55f7feca41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd1800640 1 --2- 172.21.15.102:0/197654759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7feca41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd1800640 1 -- 172.21.15.102:0/197654759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7fecaefe0 msgr2=0x55f7fecb13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd1800640 1 --2- 172.21.15.102:0/197654759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7fecaefe0 0x55f7fecb13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd1800640 1 -- 172.21.15.102:0/197654759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7fe91c210 con 0x55f7feca4720 2024-04-25T02:04:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd0e00640 1 --2- 172.21.15.102:0/197654759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7fecaefe0 0x55f7fecb13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdca000640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efdb40a2e20 0x7efdb40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.927+0000 7efdd1800640 1 --2- 172.21.15.102:0/197654759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca4720 0x55f7fecaeaa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efdc0031550 tx=0x7efdc0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43eee968e4813dfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:03.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.928+0000 7efdca000640 1 -- 172.21.15.102:0/3426879802 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7efdc40cf950 con 0x7efdcc10e030 2024-04-25T02:04:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.928+0000 7efdd0e00640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efdb40a2e20 0x7efdb40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.928+0000 7efdbea00640 1 -- 172.21.15.102:0/197654759 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0040070 con 0x55f7feca4720 2024-04-25T02:04:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.928+0000 7efdbea00640 1 -- 172.21.15.102:0/197654759 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc00047a0 con 0x55f7feca4720 2024-04-25T02:04:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.928+0000 7efdbea00640 1 -- 172.21.15.102:0/197654759 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0035040 con 0x55f7feca4720 2024-04-25T02:04:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.929+0000 7efdd6fb5740 1 -- 172.21.15.102:0/197654759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca4720 msgr2=0x55f7fecaeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.929+0000 7efdd0e00640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efdb40a2e20 0x7efdb40a52e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55f7feca1730 tx=0x7efdb8005850 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.929+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/197654759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca4720 0x55f7fecaeaa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efdc0031550 tx=0x7efdc0005dd0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.930+0000 7efdd6fb5740 1 -- 172.21.15.102:0/197654759 shutdown_connections 2024-04-25T02:04:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.930+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/197654759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7fecaefe0 0x55f7fecb13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.930+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/197654759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca4720 0x55f7fecaeaa0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.930+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/197654759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7feca41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.930+0000 7efdd6fb5740 1 -- 172.21.15.102:0/197654759 >> 172.21.15.102:0/197654759 conn(0x55f7feca0310 msgr2=0x55f7feca06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.931+0000 7efdd6fb5740 1 -- 172.21.15.102:0/197654759 shutdown_connections 2024-04-25T02:04:03.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.931+0000 7efdd6fb5740 1 -- 172.21.15.102:0/197654759 wait complete. 2024-04-25T02:04:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.932+0000 7efdd6fb5740 1 -- start start 2024-04-25T02:04:03.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.932+0000 7efdd6fb5740 1 Processor -- start 2024-04-25T02:04:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.933+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecabf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.933+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.933+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 0x55f7fecaca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.933+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f7fecb1bb0 con 0x55f7fecaefe0 2024-04-25T02:04:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.933+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f7febc3990 con 0x55f7feca4720 2024-04-25T02:04:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.933+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f7fecb1d30 con 0x55f7feca3e00 2024-04-25T02:04:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.933+0000 7efdd6fb5740 10 client.0 _ll_register_callbacks cb 0x55f7febc5950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.934+0000 7efdd1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecabf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.934+0000 7efdd0e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.934+0000 7efdd1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecabf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:47880/0 (socket says 172.21.15.102:47880) 2024-04-25T02:04:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.934+0000 7efdd0e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:35830/0 (socket says 172.21.15.102:35830) 2024-04-25T02:04:03.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.934+0000 7efdd1800640 1 -- 172.21.15.102:0/818257541 learned_addr learned my addr 172.21.15.102:0/818257541 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.935+0000 7efdd1800640 1 -- 172.21.15.102:0/818257541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 msgr2=0x55f7fecac4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.935+0000 7efdd1800640 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.935+0000 7efdd1800640 1 -- 172.21.15.102:0/818257541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 msgr2=0x55f7fecaca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.935+0000 7efdd1800640 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 0x55f7fecaca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.935+0000 7efdd1800640 1 -- 172.21.15.102:0/818257541 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7fe9e8620 con 0x55f7feca3e00 2024-04-25T02:04:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.936+0000 7efdd1800640 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecabf80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efdc002aa70 tx=0x7efdc0003900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2c9f28e05818ba1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:03.943 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.936+0000 7efdb3400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:04:03.944 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.937+0000 7efdd0e00640 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:04:03.944 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.937+0000 7efdb3400640 1 -- 172.21.15.102:0/818257541 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0035070 con 0x55f7feca3e00 2024-04-25T02:04:03.944 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.937+0000 7efdd6fb5740 1 -- 172.21.15.102:0/818257541 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdc00047a0 con 0x55f7feca3e00 2024-04-25T02:04:03.944 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.938+0000 7efdb3400640 1 -- 172.21.15.102:0/818257541 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7efdc00315b0 con 0x55f7feca3e00 2024-04-25T02:04:03.944 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.938+0000 7efdb3400640 1 -- 172.21.15.102:0/818257541 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7efdc0040040 con 0x55f7feca3e00 2024-04-25T02:04:03.945 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.938+0000 7efdd6fb5740 10 client.27223 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:03.945 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.938+0000 7efdd6fb5740 1 -- 172.21.15.102:0/818257541 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f7fecb1bb0 con 0x55f7feca3e00 2024-04-25T02:04:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.939+0000 7efdb3400640 1 -- 172.21.15.102:0/818257541 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc00047a0 con 0x55f7feca3e00 2024-04-25T02:04:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.939+0000 7efdb3400640 1 -- 172.21.15.102:0/818257541 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7efdc003c8e0 con 0x55f7feca3e00 2024-04-25T02:04:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.941+0000 7efdd6fb5740 10 client.27223 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.941+0000 7efdd6fb5740 10 client.27223 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.941+0000 7efdd6fb5740 20 client.27223 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.941+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f7fecc7470 0x55f7fecc9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.941+0000 7efdd6fb5740 4 client.27223 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:04:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.941+0000 7efdd6fb5740 1 -- 172.21.15.102:0/818257541 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7fec14ff0 con 0x55f7fecc7470 2024-04-25T02:04:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.941+0000 7efdd0e00640 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f7fecc7470 0x55f7fecc9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.944+0000 7efdd0e00640 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f7fecc7470 0x55f7fecc9860 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=228cd80e8f46866e server_cookie=3fcdc3213c5550ee in_seq=0 out_seq=0 2024-04-25T02:04:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.944+0000 7efdb3400640 10 client.27223 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.945+0000 7efdb3400640 1 -- 172.21.15.102:0/818257541 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f7fec14ff0 con 0x55f7fecc7470 2024-04-25T02:04:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.945+0000 7efdb3400640 10 client.27223 handle_command_reply: tid=0 2024-04-25T02:04:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.945+0000 7efdd6fb5740 1 client.27223 shutdown 2024-04-25T02:04:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.946+0000 7efdd6fb5740 1 -- 172.21.15.102:0/818257541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 msgr2=0x55f7fecabf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.946+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecabf80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efdc002aa70 tx=0x7efdc0003900 comp rx=0 tx=0).stop 2024-04-25T02:04:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.946+0000 7efdd6fb5740 1 -- 172.21.15.102:0/818257541 shutdown_connections 2024-04-25T02:04:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.946+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f7fecc7470 0x55f7fecc9860 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.946+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 0x55f7fecaca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.946+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.946+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/818257541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecabf80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.946+0000 7efdb3400640 0 client.27223 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.946+0000 7efdd6fb5740 1 -- 172.21.15.102:0/818257541 >> 172.21.15.102:0/818257541 conn(0x55f7feca0310 msgr2=0x55f7feca6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:03.953 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.947+0000 7efdd6fb5740 1 -- 172.21.15.102:0/818257541 shutdown_connections 2024-04-25T02:04:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.947+0000 7efdd6fb5740 1 -- 172.21.15.102:0/818257541 wait complete. 2024-04-25T02:04:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.947+0000 7efdd6fb5740 20 client.27223 trim_cache size 0 max 16384 2024-04-25T02:04:03.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.978+0000 7efdd6fb5740 1 Processor -- start 2024-04-25T02:04:03.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.978+0000 7efdd6fb5740 1 -- start start 2024-04-25T02:04:03.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecbaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 0x55f7feccf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc1adb50 con 0x55f7fecaefe0 2024-04-25T02:04:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc1ab360 con 0x55f7feca4720 2024-04-25T02:04:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc1a91f0 con 0x55f7feca3e00 2024-04-25T02:04:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecbaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecbaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:47892/0 (socket says 172.21.15.102:47892) 2024-04-25T02:04:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd1800640 1 -- 172.21.15.102:0/2850454052 learned_addr learned my addr 172.21.15.102:0/2850454052 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.979+0000 7efdd0e00640 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdd1800640 1 -- 172.21.15.102:0/2850454052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 msgr2=0x55f7fecac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdd1800640 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdd1800640 1 -- 172.21.15.102:0/2850454052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 msgr2=0x55f7feccf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdd2200640 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 0x55f7feccf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdd1800640 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 0x55f7feccf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdd1800640 1 -- 172.21.15.102:0/2850454052 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7fecaaf70 con 0x55f7feca3e00 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdd0e00640 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdd2200640 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 0x55f7feccf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdd1800640 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecbaa70 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efdc003ccb0 tx=0x7efdc0005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fdf6914495ba8d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.980+0000 7efdbe000640 1 -- 172.21.15.102:0/2850454052 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0035070 con 0x55f7feca3e00 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.981+0000 7efdbe000640 1 -- 172.21.15.102:0/2850454052 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc003c920 con 0x55f7feca3e00 2024-04-25T02:04:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.981+0000 7efdbe000640 1 -- 172.21.15.102:0/2850454052 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc00315b0 con 0x55f7feca3e00 2024-04-25T02:04:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 -- 172.21.15.102:0/2850454052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 msgr2=0x55f7fecbaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecbaa70 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efdc003ccb0 tx=0x7efdc0005780 comp rx=0 tx=0).stop 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 -- 172.21.15.102:0/2850454052 shutdown_connections 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7fecaefe0 0x55f7feccf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7fecac820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/2850454052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7feca3e00 0x55f7fecbaa70 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 -- 172.21.15.102:0/2850454052 >> 172.21.15.102:0/2850454052 conn(0x55f7feca0310 msgr2=0x55f7fec9f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 -- 172.21.15.102:0/2850454052 shutdown_connections 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 -- 172.21.15.102:0/2850454052 wait complete. 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 -- start start 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.982+0000 7efdd6fb5740 1 Processor -- start 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca3e00 0x55f7feca8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7feca9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd6fb5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7fecaefe0 0x55f7feca96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc1a9840 con 0x55f7feca3e00 2024-04-25T02:04:03.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc1aac90 con 0x55f7feca4720 2024-04-25T02:04:03.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd6fb5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc1a9ef0 con 0x55f7fecaefe0 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd6fb5740 10 client.0 _ll_register_callbacks cb 0x7efdb80097f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd1800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca3e00 0x55f7feca8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd0e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7feca9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd0e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7feca9190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:35856/0 (socket says 172.21.15.102:35856) 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd0e00640 1 -- 172.21.15.102:0/684522052 learned_addr learned my addr 172.21.15.102:0/684522052 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.983+0000 7efdd2200640 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7fecaefe0 0x55f7feca96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdd0e00640 1 -- 172.21.15.102:0/684522052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7fecaefe0 msgr2=0x55f7feca96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdd0e00640 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7fecaefe0 0x55f7feca96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdd0e00640 1 -- 172.21.15.102:0/684522052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca3e00 msgr2=0x55f7feca8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdd0e00640 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca3e00 0x55f7feca8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdd0e00640 1 -- 172.21.15.102:0/684522052 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7fed2c910 con 0x55f7feca4720 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdd0e00640 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7feca9190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x55f7feca14a0 tx=0x7efdb8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b629da86143ade server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdb3400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdb3400640 1 -- 172.21.15.102:0/684522052 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8033070 con 0x55f7feca4720 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdd6fb5740 1 -- 172.21.15.102:0/684522052 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdc003c920 con 0x55f7feca4720 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.984+0000 7efdb3400640 1 -- 172.21.15.102:0/684522052 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8513+0+0 (secure 0 0 0) 0x7efdb802c480 con 0x55f7feca4720 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdb3400640 1 -- 172.21.15.102:0/684522052 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7efdb803f050 con 0x55f7feca4720 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdd6fb5740 10 client.27263 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdd6fb5740 1 -- 172.21.15.102:0/684522052 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55f7fed2db80 con 0x55f7feca4720 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdb3400640 1 -- 172.21.15.102:0/684522052 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc003c920 con 0x55f7feca4720 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdb3400640 1 -- 172.21.15.102:0/684522052 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7efdb80373b0 con 0x55f7feca4720 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdd6fb5740 10 client.27263 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdd6fb5740 10 client.27263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdd6fb5740 20 client.27263 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f7fed2e330 0x55f7fed30750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdd6fb5740 4 client.27263 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:04:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.985+0000 7efdd6fb5740 1 -- 172.21.15.102:0/684522052 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7fec0f340 con 0x55f7fed2e330 2024-04-25T02:04:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.986+0000 7efdd1800640 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f7fed2e330 0x55f7fed30750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.986+0000 7efdd1800640 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f7fed2e330 0x55f7fed30750 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc9167f8ea3f4797 server_cookie=229358ec65191efb in_seq=0 out_seq=0 2024-04-25T02:04:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:03.986+0000 7efdb3400640 10 client.27263 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdb3400640 1 -- 172.21.15.102:0/684522052 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7fec0f340 con 0x55f7fed2e330 2024-04-25T02:04:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdb3400640 10 client.27263 handle_command_reply: tid=0 2024-04-25T02:04:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdd6fb5740 1 client.27263 shutdown 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdd6fb5740 1 -- 172.21.15.102:0/684522052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 msgr2=0x55f7feca9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7feca9190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x55f7feca14a0 tx=0x7efdb8004730 comp rx=0 tx=0).stop 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdd6fb5740 1 -- 172.21.15.102:0/684522052 shutdown_connections 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f7fed2e330 0x55f7fed30750 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7fecaefe0 0x55f7feca96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f7feca4720 0x55f7feca9190 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdd6fb5740 1 --2- 172.21.15.102:0/684522052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f7feca3e00 0x55f7feca8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdb3400640 0 client.27263 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.009+0000 7efdd6fb5740 1 -- 172.21.15.102:0/684522052 >> 172.21.15.102:0/684522052 conn(0x55f7feca0310 msgr2=0x55f7feca5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.010+0000 7efdd6fb5740 1 -- 172.21.15.102:0/684522052 shutdown_connections 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.010+0000 7efdd6fb5740 1 -- 172.21.15.102:0/684522052 wait complete. 2024-04-25T02:04:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.010+0000 7efdd6fb5740 20 client.27263 trim_cache size 0 max 16384 2024-04-25T02:04:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:04:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:04:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:04:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.010+0000 7efdd4200640 1 -- 172.21.15.102:0/3426879802 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efdb40a2e20 msgr2=0x7efdb40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.010+0000 7efdd4200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efdb40a2e20 0x7efdb40a52e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55f7feca1730 tx=0x7efdb8005850 comp rx=0 tx=0).stop 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.010+0000 7efdd4200640 1 -- 172.21.15.102:0/3426879802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdcc10e030 msgr2=0x7efdcc19e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.010+0000 7efdd4200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdcc10e030 0x7efdcc19e990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7efdc400ebc0 tx=0x7efdc400c6a0 comp rx=0 tx=0).stop 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.011+0000 7efdd4200640 1 -- 172.21.15.102:0/3426879802 shutdown_connections 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.011+0000 7efdd4200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efdb40a2e20 0x7efdb40a52e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.011+0000 7efdd4200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdcc10e030 0x7efdcc19e990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.011+0000 7efdd4200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdcc074360 0x7efdcc19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.011+0000 7efdd4200640 1 --2- 172.21.15.102:0/3426879802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdcc073a40 0x7efdcc19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.011+0000 7efdd4200640 1 -- 172.21.15.102:0/3426879802 >> 172.21.15.102:0/3426879802 conn(0x7efdcc078bd0 msgr2=0x7efdcc0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.011+0000 7efdd4200640 1 -- 172.21.15.102:0/3426879802 shutdown_connections 2024-04-25T02:04:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:04.011+0000 7efdd4200640 1 -- 172.21.15.102:0/3426879802 wait complete. 2024-04-25T02:04:05.185 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:04 smithi102 ceph-mon[27796]: pgmap v271: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.8 MiB/s wr, 729 op/s 2024-04-25T02:04:05.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:04 smithi110 ceph-mon[32848]: pgmap v271: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.8 MiB/s wr, 729 op/s 2024-04-25T02:04:05.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:04 smithi193 ceph-mon[32981]: pgmap v271: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.8 MiB/s wr, 729 op/s 2024-04-25T02:04:07.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:06 smithi102 ceph-mon[27796]: pgmap v272: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 7.3 MiB/s wr, 602 op/s 2024-04-25T02:04:07.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:06 smithi110 ceph-mon[32848]: pgmap v272: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 7.3 MiB/s wr, 602 op/s 2024-04-25T02:04:07.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:06 smithi193 ceph-mon[32981]: pgmap v272: 97 pgs: 97 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 7.3 MiB/s wr, 602 op/s 2024-04-25T02:04:08.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 495.0 s, 655.0 tps, lat 154.731 ms stddev 207.186 2024-04-25T02:04:08.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:07 smithi110 ceph-mon[32848]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T02:04:08.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:07 smithi193 ceph-mon[32981]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T02:04:08.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:07 smithi102 ceph-mon[27796]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T02:04:09.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:08 smithi110 ceph-mon[32848]: pgmap v274: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:09.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:08 smithi193 ceph-mon[32981]: pgmap v274: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:09.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:08 smithi102 ceph-mon[27796]: pgmap v274: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:11.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:10 smithi110 ceph-mon[32848]: pgmap v275: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:11.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:10 smithi193 ceph-mon[32981]: pgmap v275: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:11.433 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:10 smithi102 ceph-mon[27796]: pgmap v275: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 4.7 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:13.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 500.0 s, 1911.2 tps, lat 52.580 ms stddev 64.070 2024-04-25T02:04:13.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:12 smithi110 ceph-mon[32848]: pgmap v276: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 19 MiB/s wr, 1.70k op/s 2024-04-25T02:04:13.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:12 smithi193 ceph-mon[32981]: pgmap v276: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 19 MiB/s wr, 1.70k op/s 2024-04-25T02:04:13.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:12 smithi102 ceph-mon[27796]: pgmap v276: 97 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 74 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 19 MiB/s wr, 1.70k op/s 2024-04-25T02:04:15.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:14 smithi110 ceph-mon[32848]: pgmap v277: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:15.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:14 smithi193 ceph-mon[32981]: pgmap v277: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:15.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:14 smithi102 ceph-mon[27796]: pgmap v277: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:17.203 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.195+0000 7f6ee0200640 1 -- 172.21.15.102:0/848354917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ed810cd40 msgr2=0x7f6ed8110d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:17.203 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.195+0000 7f6ee0200640 1 --2- 172.21.15.102:0/848354917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ed810cd40 0x7f6ed8110d80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009860 tx=0x7f6ec402a360 comp rx=0 tx=0).stop 2024-04-25T02:04:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.197+0000 7f6ee0200640 1 -- 172.21.15.102:0/848354917 shutdown_connections 2024-04-25T02:04:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.197+0000 7f6ee0200640 1 --2- 172.21.15.102:0/848354917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed8111460 0x7f6ed8119030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.197+0000 7f6ee0200640 1 --2- 172.21.15.102:0/848354917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ed810cd40 0x7f6ed8110d80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.197+0000 7f6ee0200640 1 --2- 172.21.15.102:0/848354917 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ed810c390 0x7f6ed810c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.197+0000 7f6ee0200640 1 -- 172.21.15.102:0/848354917 >> 172.21.15.102:0/848354917 conn(0x7f6ed806c9c0 msgr2=0x7f6ed810a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.197+0000 7f6ee0200640 1 -- 172.21.15.102:0/848354917 shutdown_connections 2024-04-25T02:04:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.197+0000 7f6ee0200640 1 -- 172.21.15.102:0/848354917 wait complete. 2024-04-25T02:04:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.198+0000 7f6ee0200640 1 Processor -- start 2024-04-25T02:04:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.198+0000 7f6ee0200640 1 -- start start 2024-04-25T02:04:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.199+0000 7f6ee0200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed810c390 0x7f6ed81a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.199+0000 7f6ee0200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ed8111460 0x7f6ed81a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.199+0000 7f6ee0200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ed81a2830 0x7f6ed81a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.199+0000 7f6ee0200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed811c550 con 0x7f6ed810c390 2024-04-25T02:04:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.199+0000 7f6ee0200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed811c3d0 con 0x7f6ed8111460 2024-04-25T02:04:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.199+0000 7f6ee0200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed811c6d0 con 0x7f6ed81a2830 2024-04-25T02:04:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.201+0000 7f6edce00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ed8111460 0x7f6ed81a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.201+0000 7f6edce00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ed8111460 0x7f6ed81a8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:43366/0 (socket says 172.21.15.102:43366) 2024-04-25T02:04:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.201+0000 7f6edce00640 1 -- 172.21.15.102:0/3902087664 learned_addr learned my addr 172.21.15.102:0/3902087664 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.202+0000 7f6edce00640 1 -- 172.21.15.102:0/3902087664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ed81a2830 msgr2=0x7f6ed81a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.202+0000 7f6edce00640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ed81a2830 0x7f6ed81a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.202+0000 7f6edce00640 1 -- 172.21.15.102:0/3902087664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed810c390 msgr2=0x7f6ed81a86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.202+0000 7f6edce00640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed810c390 0x7f6ed81a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.202+0000 7f6edce00640 1 -- 172.21.15.102:0/3902087664 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed81a33c0 con 0x7f6ed8111460 2024-04-25T02:04:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.202+0000 7f6edce00640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ed8111460 0x7f6ed81a8c00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4035040 tx=0x7f6ec40327b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.203+0000 7f6ed6000640 1 -- 172.21.15.102:0/3902087664 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4037040 con 0x7f6ed8111460 2024-04-25T02:04:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.203+0000 7f6ed6000640 1 -- 172.21.15.102:0/3902087664 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ec4040070 con 0x7f6ed8111460 2024-04-25T02:04:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.203+0000 7f6ee0200640 1 -- 172.21.15.102:0/3902087664 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed81a3650 con 0x7f6ed8111460 2024-04-25T02:04:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.203+0000 7f6ed6000640 1 -- 172.21.15.102:0/3902087664 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4005340 con 0x7f6ed8111460 2024-04-25T02:04:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.203+0000 7f6ee0200640 1 -- 172.21.15.102:0/3902087664 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed806d580 con 0x7f6ed8111460 2024-04-25T02:04:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.212+0000 7f6ee3005740 1 Processor -- start 2024-04-25T02:04:17.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.213+0000 7f6ee3005740 1 -- start start 2024-04-25T02:04:17.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.213+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352931e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.213+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635293720 0x55d63529daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.219 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.213+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d63529dfe0 0x55d6352a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.220 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.213+0000 7f6ee3005740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d6351f6ab0 con 0x55d63529dfe0 2024-04-25T02:04:17.220 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.213+0000 7f6ee3005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d63519f640 con 0x55d635292e00 2024-04-25T02:04:17.220 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.213+0000 7f6ee3005740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d6351f2a70 con 0x55d635293720 2024-04-25T02:04:17.220 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.213+0000 7f6ed6000640 1 -- 172.21.15.102:0/3902087664 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6ec4030520 con 0x7f6ed8111460 2024-04-25T02:04:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.213+0000 7f6ede200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352931e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.214+0000 7f6ede200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352931e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:43374/0 (socket says 172.21.15.102:43374) 2024-04-25T02:04:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.214+0000 7f6ede200640 1 -- 172.21.15.102:0/2727156647 learned_addr learned my addr 172.21.15.102:0/2727156647 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.214+0000 7f6edd800640 1 --2- 172.21.15.102:0/2727156647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635293720 0x55d63529daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.214+0000 7f6ede200640 1 -- 172.21.15.102:0/2727156647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635293720 msgr2=0x55d63529daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.214+0000 7f6ede200640 1 --2- 172.21.15.102:0/2727156647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635293720 0x55d63529daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.214+0000 7f6ede200640 1 -- 172.21.15.102:0/2727156647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d63529dfe0 msgr2=0x55d6352a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.214+0000 7f6ede200640 1 --2- 172.21.15.102:0/2727156647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d63529dfe0 0x55d6352a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.214+0000 7f6ede200640 1 -- 172.21.15.102:0/2727156647 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d634f0b210 con 0x55d635292e00 2024-04-25T02:04:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.214+0000 7f6ede200640 1 --2- 172.21.15.102:0/2727156647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352931e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009d00 tx=0x7f6ed000b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b14fb5dd0b1d6bf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.215+0000 7f6ecaa00640 1 -- 172.21.15.102:0/2727156647 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed000bd40 con 0x55d635292e00 2024-04-25T02:04:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.215+0000 7f6ecaa00640 1 -- 172.21.15.102:0/2727156647 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed0017070 con 0x55d635292e00 2024-04-25T02:04:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.215+0000 7f6ecaa00640 1 -- 172.21.15.102:0/2727156647 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00116c0 con 0x55d635292e00 2024-04-25T02:04:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.216+0000 7f6ed6000640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6ec00a2f00 0x7f6ec00a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.216+0000 7f6ee3005740 1 -- 172.21.15.102:0/2727156647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 msgr2=0x55d6352931e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.216+0000 7f6ee3005740 1 --2- 172.21.15.102:0/2727156647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352931e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009d00 tx=0x7f6ed000b420 comp rx=0 tx=0).stop 2024-04-25T02:04:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.216+0000 7f6ed6000640 1 -- 172.21.15.102:0/3902087664 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8557+0+0 (secure 0 0 0) 0x7f6ec40ef0c0 con 0x7f6ed8111460 2024-04-25T02:04:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.217+0000 7f6ee3005740 1 -- 172.21.15.102:0/2727156647 shutdown_connections 2024-04-25T02:04:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.217+0000 7f6ee3005740 1 --2- 172.21.15.102:0/2727156647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d63529dfe0 0x55d6352a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.217+0000 7f6ee3005740 1 --2- 172.21.15.102:0/2727156647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635293720 0x55d63529daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.217+0000 7f6ee3005740 1 --2- 172.21.15.102:0/2727156647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352931e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.217+0000 7f6ee3005740 1 -- 172.21.15.102:0/2727156647 >> 172.21.15.102:0/2727156647 conn(0x55d63528f310 msgr2=0x55d63528f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:17.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.217+0000 7f6ee3005740 1 -- 172.21.15.102:0/2727156647 shutdown_connections 2024-04-25T02:04:17.224 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.218+0000 7f6ee3005740 1 -- 172.21.15.102:0/2727156647 wait complete. 2024-04-25T02:04:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.218+0000 7f6ee3005740 1 -- start start 2024-04-25T02:04:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.218+0000 7f6ee3005740 1 Processor -- start 2024-04-25T02:04:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.219+0000 7f6edd800640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6ec00a2f00 0x7f6ec00a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.219+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 0x55d63529b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.219+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d63529b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.219+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d63529bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.219+0000 7f6ee3005740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d6352a0c10 con 0x55d635293720 2024-04-25T02:04:17.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.219+0000 7f6ee3005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d6351b2990 con 0x55d63529dfe0 2024-04-25T02:04:17.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.219+0000 7f6ee3005740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d6352a0d90 con 0x55d635292e00 2024-04-25T02:04:17.226 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.219+0000 7f6ee3005740 10 client.0 _ll_register_callbacks cb 0x55d6351b4950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6ede200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d63529bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6ede200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d63529bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:43376/0 (socket says 172.21.15.102:43376) 2024-04-25T02:04:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6ede200640 1 -- 172.21.15.102:0/234570883 learned_addr learned my addr 172.21.15.102:0/234570883 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6edce00640 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d63529b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6ede200640 1 -- 172.21.15.102:0/234570883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 msgr2=0x55d63529b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6edd800640 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 0x55d63529b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6ede200640 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 0x55d63529b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6ede200640 1 -- 172.21.15.102:0/234570883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 msgr2=0x55d63529b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6ede200640 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d63529b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.220+0000 7f6ede200640 1 -- 172.21.15.102:0/234570883 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d634fd7620 con 0x55d63529dfe0 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.221+0000 7f6edd800640 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 0x55d63529b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.221+0000 7f6ebf400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.221+0000 7f6ede200640 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d63529bb60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00099b0 tx=0x7f6ed00099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb1f32dd8e0fe040 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.221+0000 7f6edd800640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6ec00a2f00 0x7f6ec00a53c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc005fd0 tx=0x7f6ecc00deb0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.221+0000 7f6ebf400640 1 -- 172.21.15.102:0/234570883 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed001b800 con 0x55d63529dfe0 2024-04-25T02:04:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.221+0000 7f6ebf400640 1 -- 172.21.15.102:0/234570883 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8557+0+0 (secure 0 0 0) 0x7f6ed001b600 con 0x55d63529dfe0 2024-04-25T02:04:17.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.221+0000 7f6ee3005740 1 -- 172.21.15.102:0/234570883 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ed0017070 con 0x55d63529dfe0 2024-04-25T02:04:17.229 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.222+0000 7f6ebf400640 1 -- 172.21.15.102:0/234570883 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6ed001b800 con 0x55d63529dfe0 2024-04-25T02:04:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.224+0000 7f6ee3005740 10 client.27275 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.224+0000 7f6ee3005740 1 -- 172.21.15.102:0/234570883 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d6352aca00 con 0x55d63529dfe0 2024-04-25T02:04:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.225+0000 7f6ebf400640 1 -- 172.21.15.102:0/234570883 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed00253f0 con 0x55d63529dfe0 2024-04-25T02:04:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.225+0000 7f6ebf400640 1 -- 172.21.15.102:0/234570883 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f6ed0011070 con 0x55d63529dfe0 2024-04-25T02:04:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.225+0000 7f6ee3005740 10 client.27275 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.225+0000 7f6ee3005740 10 client.27275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.225+0000 7f6ee3005740 20 client.27275 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.225+0000 7f6ee3005740 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d6352ade10 0x55d6352b0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.225+0000 7f6ee3005740 4 client.27275 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:04:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.225+0000 7f6ee3005740 1 -- 172.21.15.102:0/234570883 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6ed0017070 con 0x55d6352ade10 2024-04-25T02:04:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.226+0000 7f6edd800640 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d6352ade10 0x55d6352b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.226+0000 7f6ebf400640 10 client.27275 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.227+0000 7f6edd800640 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d6352ade10 0x55d6352b0200 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87e8275b15414841 server_cookie=5a30bcee02c5c9e8 in_seq=0 out_seq=0 2024-04-25T02:04:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.228+0000 7f6ebf400640 1 -- 172.21.15.102:0/234570883 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f6ed0017070 con 0x55d6352ade10 2024-04-25T02:04:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.228+0000 7f6ebf400640 10 client.27275 handle_command_reply: tid=0 2024-04-25T02:04:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.229+0000 7f6ee3005740 1 client.27275 shutdown 2024-04-25T02:04:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.229+0000 7f6ee3005740 1 -- 172.21.15.102:0/234570883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 msgr2=0x55d63529bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.229+0000 7f6ee3005740 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d63529bb60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00099b0 tx=0x7f6ed00099e0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.231+0000 7f6ee3005740 1 -- 172.21.15.102:0/234570883 shutdown_connections 2024-04-25T02:04:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.231+0000 7f6ee3005740 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d6352ade10 0x55d6352b0200 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.231+0000 7f6ee3005740 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d63529bb60 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.231+0000 7f6ee3005740 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d63529b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.231+0000 7f6ee3005740 1 --2- 172.21.15.102:0/234570883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 0x55d63529b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.231+0000 7f6ee3005740 1 -- 172.21.15.102:0/234570883 >> 172.21.15.102:0/234570883 conn(0x55d63528f310 msgr2=0x55d63528e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.231+0000 7f6ebf400640 0 client.27275 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.232+0000 7f6ee3005740 1 -- 172.21.15.102:0/234570883 shutdown_connections 2024-04-25T02:04:17.240 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.232+0000 7f6ee3005740 1 -- 172.21.15.102:0/234570883 wait complete. 2024-04-25T02:04:17.240 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.234+0000 7f6ee3005740 20 client.27275 trim_cache size 0 max 16384 2024-04-25T02:04:17.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:16 smithi110 ceph-mon[32848]: pgmap v278: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:17.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:16 smithi193 ceph-mon[32981]: pgmap v278: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.274+0000 7f6ee3005740 1 Processor -- start 2024-04-25T02:04:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.274+0000 7f6ee3005740 1 -- start start 2024-04-25T02:04:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.274+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 0x55d6352a9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.274+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d6352957c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.274+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d6352b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.274+0000 7f6ee3005740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed81bef10 con 0x55d635293720 2024-04-25T02:04:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.274+0000 7f6ee3005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed81bc720 con 0x55d63529dfe0 2024-04-25T02:04:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.274+0000 7f6ee3005740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed81ba550 con 0x55d635292e00 2024-04-25T02:04:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.275+0000 7f6ede200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d6352b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.275+0000 7f6ede200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d6352b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:43392/0 (socket says 172.21.15.102:43392) 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.275+0000 7f6ede200640 1 -- 172.21.15.102:0/2251237040 learned_addr learned my addr 172.21.15.102:0/2251237040 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.275+0000 7f6edd800640 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 0x55d6352a9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.275+0000 7f6edce00640 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d6352957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.276+0000 7f6ede200640 1 -- 172.21.15.102:0/2251237040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 msgr2=0x55d6352a9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.276+0000 7f6ede200640 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 0x55d6352a9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.276+0000 7f6ede200640 1 -- 172.21.15.102:0/2251237040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 msgr2=0x55d6352957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.276+0000 7f6ede200640 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d6352957c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.276+0000 7f6ede200640 1 -- 172.21.15.102:0/2251237040 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6352b0660 con 0x55d63529dfe0 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.276+0000 7f6edce00640 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d6352957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:04:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.276+0000 7f6ede200640 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d6352b58f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009a50 tx=0x7f6ed000b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be035524218b1a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.277+0000 7f6eca000640 1 -- 172.21.15.102:0/2251237040 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed001b800 con 0x55d63529dfe0 2024-04-25T02:04:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.277+0000 7f6eca000640 1 -- 172.21.15.102:0/2251237040 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed0012e50 con 0x55d63529dfe0 2024-04-25T02:04:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.277+0000 7f6eca000640 1 -- 172.21.15.102:0/2251237040 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed002e2a0 con 0x55d63529dfe0 2024-04-25T02:04:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.278+0000 7f6ee3005740 1 -- 172.21.15.102:0/2251237040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 msgr2=0x55d6352b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.278+0000 7f6ee3005740 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d6352b58f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0009a50 tx=0x7f6ed000b420 comp rx=0 tx=0).stop 2024-04-25T02:04:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.278+0000 7f6ee3005740 1 -- 172.21.15.102:0/2251237040 shutdown_connections 2024-04-25T02:04:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.278+0000 7f6ee3005740 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d63529dfe0 0x55d6352b58f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.278+0000 7f6ee3005740 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d6352957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.278+0000 7f6ee3005740 1 --2- 172.21.15.102:0/2251237040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d635292e00 0x55d6352a9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.278+0000 7f6ee3005740 1 -- 172.21.15.102:0/2251237040 >> 172.21.15.102:0/2251237040 conn(0x55d63528f310 msgr2=0x55d63528e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.278+0000 7f6ee3005740 1 -- 172.21.15.102:0/2251237040 shutdown_connections 2024-04-25T02:04:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.279+0000 7f6ee3005740 1 -- 172.21.15.102:0/2251237040 wait complete. 2024-04-25T02:04:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.279+0000 7f6ee3005740 1 -- start start 2024-04-25T02:04:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.279+0000 7f6ee3005740 1 Processor -- start 2024-04-25T02:04:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.280+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352983f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d635298930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ee3005740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d63529dfe0 0x55d635313160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ee3005740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed81babe0 con 0x55d635293720 2024-04-25T02:04:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ee3005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed81bc050 con 0x55d635292e00 2024-04-25T02:04:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ee3005740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed81bb2b0 con 0x55d63529dfe0 2024-04-25T02:04:17.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ee3005740 10 client.0 _ll_register_callbacks cb 0x7f6ed0009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ede200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d63529dfe0 0x55d635313160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ede200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d63529dfe0 0x55d635313160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40486/0 (socket says 172.21.15.102:40486) 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ede200640 1 -- 172.21.15.102:0/1315361812 learned_addr learned my addr 172.21.15.102:0/1315361812 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6edd800640 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6edce00640 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d635298930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.281+0000 7f6ede200640 1 -- 172.21.15.102:0/1315361812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 msgr2=0x55d6352983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.282+0000 7f6ede200640 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.282+0000 7f6ede200640 1 -- 172.21.15.102:0/1315361812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 msgr2=0x55d635298930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.282+0000 7f6ede200640 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d635298930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.282+0000 7f6ede200640 1 -- 172.21.15.102:0/1315361812 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6353136a0 con 0x55d63529dfe0 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.282+0000 7f6edce00640 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d635298930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.282+0000 7f6ede200640 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d63529dfe0 0x55d635313160 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6ed002eb20 tx=0x7f6ed0004bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e85a8af57f64ad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.282+0000 7f6ebf400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:04:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.283+0000 7f6ebf400640 1 -- 172.21.15.102:0/1315361812 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0007000 con 0x55d63529dfe0 2024-04-25T02:04:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.283+0000 7f6ebf400640 1 -- 172.21.15.102:0/1315361812 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8557+0+0 (secure 0 0 0) 0x7f6ed0019d60 con 0x55d63529dfe0 2024-04-25T02:04:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.283+0000 7f6ee3005740 1 -- 172.21.15.102:0/1315361812 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ed0012e50 con 0x55d63529dfe0 2024-04-25T02:04:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ebf400640 1 -- 172.21.15.102:0/1315361812 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f6ed002e2a0 con 0x55d63529dfe0 2024-04-25T02:04:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ee3005740 10 client.27241 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ee3005740 1 -- 172.21.15.102:0/1315361812 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55d635314820 con 0x55d63529dfe0 2024-04-25T02:04:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ebf400640 1 -- 172.21.15.102:0/1315361812 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed0012e50 con 0x55d63529dfe0 2024-04-25T02:04:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ebf400640 1 -- 172.21.15.102:0/1315361812 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f6ed0011070 con 0x55d63529dfe0 2024-04-25T02:04:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ee3005740 10 client.27241 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ee3005740 10 client.27241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ee3005740 20 client.27241 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ee3005740 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d635314a50 0x55d635316e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ee3005740 4 client.27241 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:04:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.284+0000 7f6ee3005740 1 -- 172.21.15.102:0/1315361812 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d635203ff0 con 0x55d635314a50 2024-04-25T02:04:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.285+0000 7f6edd800640 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d635314a50 0x55d635316e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.285+0000 7f6edd800640 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d635314a50 0x55d635316e40 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bede4a98ec70ac33 server_cookie=c5f76096cff24a8e in_seq=0 out_seq=0 2024-04-25T02:04:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.286+0000 7f6ebf400640 10 client.27241 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.301+0000 7f6ebf400640 1 -- 172.21.15.102:0/1315361812 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d635203ff0 con 0x55d635314a50 2024-04-25T02:04:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.301+0000 7f6ebf400640 10 client.27241 handle_command_reply: tid=0 2024-04-25T02:04:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.301+0000 7f6ee3005740 1 client.27241 shutdown 2024-04-25T02:04:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ee3005740 1 -- 172.21.15.102:0/1315361812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d63529dfe0 msgr2=0x55d635313160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ee3005740 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d63529dfe0 0x55d635313160 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6ed002eb20 tx=0x7f6ed0004bc0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ee3005740 1 -- 172.21.15.102:0/1315361812 shutdown_connections 2024-04-25T02:04:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ee3005740 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d635314a50 0x55d635316e40 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ee3005740 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d63529dfe0 0x55d635313160 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ee3005740 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d635293720 0x55d635298930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ee3005740 1 --2- 172.21.15.102:0/1315361812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d635292e00 0x55d6352983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ebf400640 0 client.27241 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ee3005740 1 -- 172.21.15.102:0/1315361812 >> 172.21.15.102:0/1315361812 conn(0x55d63528f310 msgr2=0x55d63529b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.302+0000 7f6ee3005740 1 -- 172.21.15.102:0/1315361812 shutdown_connections 2024-04-25T02:04:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.303+0000 7f6ee3005740 1 -- 172.21.15.102:0/1315361812 wait complete. 2024-04-25T02:04:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.303+0000 7f6ee3005740 20 client.27241 trim_cache size 0 max 16384 2024-04-25T02:04:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:04:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:04:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:04:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:04:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.303+0000 7f6ee0200640 1 -- 172.21.15.102:0/3902087664 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6ec00a2f00 msgr2=0x7f6ec00a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.303+0000 7f6ee0200640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6ec00a2f00 0x7f6ec00a53c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc005fd0 tx=0x7f6ecc00deb0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.303+0000 7f6ee0200640 1 -- 172.21.15.102:0/3902087664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ed8111460 msgr2=0x7f6ed81a8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.303+0000 7f6ee0200640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ed8111460 0x7f6ed81a8c00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4035040 tx=0x7f6ec40327b0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.304+0000 7f6ee0200640 1 -- 172.21.15.102:0/3902087664 shutdown_connections 2024-04-25T02:04:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.304+0000 7f6ee0200640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f6ec00a2f00 0x7f6ec00a53c0 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc005fd0 tx=0x7f6ecc00deb0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.304+0000 7f6ee0200640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6ed81a2830 0x7f6ed81a2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.304+0000 7f6ee0200640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ed8111460 0x7f6ed81a8c00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.304+0000 7f6ee0200640 1 --2- 172.21.15.102:0/3902087664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed810c390 0x7f6ed81a86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.304+0000 7f6ee0200640 1 -- 172.21.15.102:0/3902087664 >> 172.21.15.102:0/3902087664 conn(0x7f6ed806c9c0 msgr2=0x7f6ed810f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.304+0000 7f6ee0200640 1 -- 172.21.15.102:0/3902087664 shutdown_connections 2024-04-25T02:04:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:17.305+0000 7f6ee0200640 1 -- 172.21.15.102:0/3902087664 wait complete. 2024-04-25T02:04:17.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:16 smithi102 ceph-mon[27796]: pgmap v278: 97 pgs: 97 active+clean; 4.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 20 MiB/s wr, 1.70k op/s 2024-04-25T02:04:18.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 505.0 s, 1936.6 tps, lat 51.028 ms stddev 64.816 2024-04-25T02:04:18.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:18 smithi102 ceph-mon[27796]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T02:04:18.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:18 smithi110 ceph-mon[32848]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T02:04:18.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:18 smithi193 ceph-mon[32981]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T02:04:19.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:19 smithi102 ceph-mon[27796]: pgmap v280: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 MiB/s wr, 630 op/s 2024-04-25T02:04:19.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:19 smithi110 ceph-mon[32848]: pgmap v280: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 MiB/s wr, 630 op/s 2024-04-25T02:04:19.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:19 smithi193 ceph-mon[32981]: pgmap v280: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 MiB/s wr, 630 op/s 2024-04-25T02:04:21.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:21 smithi102 ceph-mon[27796]: pgmap v281: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 MiB/s wr, 630 op/s 2024-04-25T02:04:21.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:21 smithi110 ceph-mon[32848]: pgmap v281: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 MiB/s wr, 630 op/s 2024-04-25T02:04:21.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:21 smithi193 ceph-mon[32981]: pgmap v281: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 MiB/s wr, 630 op/s 2024-04-25T02:04:23.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 510.0 s, 1453.2 tps, lat 63.794 ms stddev 119.090 2024-04-25T02:04:23.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:23 smithi102 ceph-mon[27796]: pgmap v282: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 579 op/s 2024-04-25T02:04:23.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:23 smithi110 ceph-mon[32848]: pgmap v282: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 579 op/s 2024-04-25T02:04:23.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:23 smithi193 ceph-mon[32981]: pgmap v282: 97 pgs: 97 active+clean; 4.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 MiB/s wr, 579 op/s 2024-04-25T02:04:25.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:25 smithi102 ceph-mon[27796]: pgmap v283: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 638 op/s 2024-04-25T02:04:25.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:25 smithi110 ceph-mon[32848]: pgmap v283: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 638 op/s 2024-04-25T02:04:25.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:25 smithi193 ceph-mon[32981]: pgmap v283: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 638 op/s 2024-04-25T02:04:27.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:27 smithi102 ceph-mon[27796]: pgmap v284: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 638 op/s 2024-04-25T02:04:27.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:27 smithi110 ceph-mon[32848]: pgmap v284: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 638 op/s 2024-04-25T02:04:27.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:27 smithi193 ceph-mon[32981]: pgmap v284: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.6 MiB/s wr, 638 op/s 2024-04-25T02:04:28.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 515.0 s, 1132.6 tps, lat 94.999 ms stddev 352.225 2024-04-25T02:04:29.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:29 smithi102 ceph-mon[27796]: pgmap v285: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 6.0 MiB/s wr, 548 op/s 2024-04-25T02:04:29.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:29 smithi110 ceph-mon[32848]: pgmap v285: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 6.0 MiB/s wr, 548 op/s 2024-04-25T02:04:29.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:29 smithi193 ceph-mon[32981]: pgmap v285: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 6.0 MiB/s wr, 548 op/s 2024-04-25T02:04:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.500+0000 7f3a59600640 1 -- 172.21.15.102:0/1373391073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54103020 msgr2=0x7f3a5410b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:30.506 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.500+0000 7f3a59600640 1 --2- 172.21.15.102:0/1373391073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54103020 0x7f3a5410b500 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009900 tx=0x7f3a4002a480 comp rx=0 tx=0).stop 2024-04-25T02:04:30.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.501+0000 7f3a59600640 1 -- 172.21.15.102:0/1373391073 shutdown_connections 2024-04-25T02:04:30.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.501+0000 7f3a59600640 1 --2- 172.21.15.102:0/1373391073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5410bb20 0x7f3a5410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.501+0000 7f3a59600640 1 --2- 172.21.15.102:0/1373391073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54103020 0x7f3a5410b500 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.501+0000 7f3a59600640 1 --2- 172.21.15.102:0/1373391073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a54102700 0x7f3a54102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.501+0000 7f3a59600640 1 -- 172.21.15.102:0/1373391073 >> 172.21.15.102:0/1373391073 conn(0x7f3a540fbf60 msgr2=0x7f3a540fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:30.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.501+0000 7f3a59600640 1 -- 172.21.15.102:0/1373391073 shutdown_connections 2024-04-25T02:04:30.510 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.502+0000 7f3a59600640 1 -- 172.21.15.102:0/1373391073 wait complete. 2024-04-25T02:04:30.510 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.503+0000 7f3a59600640 1 Processor -- start 2024-04-25T02:04:30.511 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.506+0000 7f3a59600640 1 -- start start 2024-04-25T02:04:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.507+0000 7f3a59600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54102700 0x7f3a5419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.507+0000 7f3a59600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a54103020 0x7f3a5419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.507+0000 7f3a59600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a5410bb20 0x7f3a5410a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.507+0000 7f3a53e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54102700 0x7f3a5419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.507+0000 7f3a53e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54102700 0x7f3a5419eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:45124/0 (socket says 172.21.15.102:45124) 2024-04-25T02:04:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541118e0 con 0x7f3a5410bb20 2024-04-25T02:04:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a53e00640 1 -- 172.21.15.102:0/1412816997 learned_addr learned my addr 172.21.15.102:0/1412816997 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:30.513 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54111760 con 0x7f3a54103020 2024-04-25T02:04:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a54111a60 con 0x7f3a54102700 2024-04-25T02:04:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a53400640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a54103020 0x7f3a5419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a53e00640 1 -- 172.21.15.102:0/1412816997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a54103020 msgr2=0x7f3a5419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a53e00640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a54103020 0x7f3a5419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a53e00640 1 -- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a5410bb20 msgr2=0x7f3a5410a9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T02:04:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a58c00640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a5410bb20 0x7f3a5410a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a53e00640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a5410bb20 0x7f3a5410a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.508+0000 7f3a53e00640 1 -- 172.21.15.102:0/1412816997 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5410b270 con 0x7f3a54102700 2024-04-25T02:04:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.509+0000 7f3a53e00640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54102700 0x7f3a5419eb40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400d950 tx=0x7f3a4400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.509+0000 7f3a50c00640 1 -- 172.21.15.102:0/1412816997 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44017030 con 0x7f3a54102700 2024-04-25T02:04:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.509+0000 7f3a50c00640 1 -- 172.21.15.102:0/1412816997 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a44004020 con 0x7f3a54102700 2024-04-25T02:04:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.509+0000 7f3a50c00640 1 -- 172.21.15.102:0/1412816997 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a440047a0 con 0x7f3a54102700 2024-04-25T02:04:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.512+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5410b560 con 0x7f3a54102700 2024-04-25T02:04:30.517 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.512+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a5419a7c0 con 0x7f3a54102700 2024-04-25T02:04:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.516+0000 7f3a50c00640 1 -- 172.21.15.102:0/1412816997 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3a440041d0 con 0x7f3a54102700 2024-04-25T02:04:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.520+0000 7f3a50c00640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3a380a2d70 0x7f3a380a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.521+0000 7f3a50c00640 1 -- 172.21.15.102:0/1412816997 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f3a440ce420 con 0x7f3a54102700 2024-04-25T02:04:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.521+0000 7f3a53400640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3a380a2d70 0x7f3a380a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.523+0000 7f3a5da6a740 1 Processor -- start 2024-04-25T02:04:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.523+0000 7f3a5da6a740 1 -- start start 2024-04-25T02:04:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.524+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682241390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.524+0000 7f3a53400640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3a380a2d70 0x7f3a380a5230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3a54197340 tx=0x7f3a40031400 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.524+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x56068224bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.524+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x56068224e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.524+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5606821a4ab0 con 0x560682240fb0 2024-04-25T02:04:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.524+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56068214d640 con 0x56068224c190 2024-04-25T02:04:30.530 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.524+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5606821a0a70 con 0x5606822418d0 2024-04-25T02:04:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a58c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x56068224e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a58c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x56068224e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45322/0 (socket says 172.21.15.102:45322) 2024-04-25T02:04:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a58c00640 1 -- 172.21.15.102:0/3697221120 learned_addr learned my addr 172.21.15.102:0/3697221120 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a53400640 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x56068224bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a53e00640 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682241390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a58c00640 1 -- 172.21.15.102:0/3697221120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 msgr2=0x56068224bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a58c00640 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x56068224bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a58c00640 1 -- 172.21.15.102:0/3697221120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 msgr2=0x560682241390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a58c00640 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682241390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a58c00640 1 -- 172.21.15.102:0/3697221120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560681eb9210 con 0x56068224c190 2024-04-25T02:04:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.525+0000 7f3a53e00640 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682241390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:04:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.526+0000 7f3a58c00640 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x56068224e580 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c009df0 tx=0x7f3a4c00e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d40d75722f06574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.526+0000 7f3a49200640 1 -- 172.21.15.102:0/3697221120 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c016070 con 0x56068224c190 2024-04-25T02:04:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.527+0000 7f3a49200640 1 -- 172.21.15.102:0/3697221120 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4c005020 con 0x56068224c190 2024-04-25T02:04:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.527+0000 7f3a49200640 1 -- 172.21.15.102:0/3697221120 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c0116d0 con 0x56068224c190 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.528+0000 7f3a5da6a740 1 -- 172.21.15.102:0/3697221120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 msgr2=0x56068224e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.528+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x56068224e580 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c009df0 tx=0x7f3a4c00e6c0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.528+0000 7f3a5da6a740 1 -- 172.21.15.102:0/3697221120 shutdown_connections 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.528+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x56068224e580 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.528+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x56068224bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.528+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/3697221120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682241390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.528+0000 7f3a5da6a740 1 -- 172.21.15.102:0/3697221120 >> 172.21.15.102:0/3697221120 conn(0x56068223d4c0 msgr2=0x56068223d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.528+0000 7f3a5da6a740 1 -- 172.21.15.102:0/3697221120 shutdown_connections 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.528+0000 7f3a5da6a740 1 -- 172.21.15.102:0/3697221120 wait complete. 2024-04-25T02:04:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.529+0000 7f3a5da6a740 1 -- start start 2024-04-25T02:04:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.529+0000 7f3a5da6a740 1 Processor -- start 2024-04-25T02:04:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x5606822490e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x560682249620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682249b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56068224edc0 con 0x560682240fb0 2024-04-25T02:04:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a58c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682249b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a58c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682249b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45336/0 (socket says 172.21.15.102:45336) 2024-04-25T02:04:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a58c00640 1 -- 172.21.15.102:0/2202044985 learned_addr learned my addr 172.21.15.102:0/2202044985 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:30.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560682160990 con 0x56068224c190 2024-04-25T02:04:30.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56068224ef40 con 0x5606822418d0 2024-04-25T02:04:30.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a53400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x560682249620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.530+0000 7f3a5da6a740 10 client.0 _ll_register_callbacks cb 0x560682162950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:30.537 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.531+0000 7f3a53e00640 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x5606822490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.531+0000 7f3a58c00640 1 -- 172.21.15.102:0/2202044985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 msgr2=0x560682249620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.531+0000 7f3a58c00640 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x560682249620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.531+0000 7f3a58c00640 1 -- 172.21.15.102:0/2202044985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 msgr2=0x5606822490e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.531+0000 7f3a58c00640 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x5606822490e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.531+0000 7f3a58c00640 1 -- 172.21.15.102:0/2202044985 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560681f85620 con 0x56068224c190 2024-04-25T02:04:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.531+0000 7f3a53400640 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x560682249620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.532+0000 7f3a58c00640 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682249b60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c004e60 tx=0x7f3a4c004e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8480ece1467def7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.532+0000 7f3a3d600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:04:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.533+0000 7f3a3d600640 1 -- 172.21.15.102:0/2202044985 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c028060 con 0x56068224c190 2024-04-25T02:04:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.533+0000 7f3a3d600640 1 -- 172.21.15.102:0/2202044985 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f3a4c0053e0 con 0x56068224c190 2024-04-25T02:04:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.533+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a4c005020 con 0x56068224c190 2024-04-25T02:04:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.533+0000 7f3a3d600640 1 -- 172.21.15.102:0/2202044985 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3a4c016040 con 0x56068224c190 2024-04-25T02:04:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.533+0000 7f3a5da6a740 10 client.27293 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.533+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56068225ac40 con 0x56068224c190 2024-04-25T02:04:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.533+0000 7f3a3d600640 1 -- 172.21.15.102:0/2202044985 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4c005020 con 0x56068224c190 2024-04-25T02:04:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.534+0000 7f3a3d600640 1 -- 172.21.15.102:0/2202044985 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f3a4c00f040 con 0x56068224c190 2024-04-25T02:04:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.535+0000 7f3a5da6a740 10 client.27293 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.535+0000 7f3a5da6a740 10 client.27293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.535+0000 7f3a5da6a740 20 client.27293 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.535+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56068225c040 0x56068225e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.535+0000 7f3a5da6a740 4 client.27293 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:04:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.535+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606821b1ff0 con 0x56068225c040 2024-04-25T02:04:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.535+0000 7f3a53e00640 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56068225c040 0x56068225e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.536+0000 7f3a53e00640 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56068225c040 0x56068225e430 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=174307dd8666f80a server_cookie=11a97550d204b02f in_seq=0 out_seq=0 2024-04-25T02:04:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.536+0000 7f3a3d600640 10 client.27293 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.537+0000 7f3a3d600640 1 -- 172.21.15.102:0/2202044985 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606821b1ff0 con 0x56068225c040 2024-04-25T02:04:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.537+0000 7f3a3d600640 10 client.27293 handle_command_reply: tid=0 2024-04-25T02:04:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.537+0000 7f3a5da6a740 1 client.27293 shutdown 2024-04-25T02:04:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.537+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 msgr2=0x560682249b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.537+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682249b60 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c004e60 tx=0x7f3a4c004e90 comp rx=0 tx=0).stop 2024-04-25T02:04:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.538+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 shutdown_connections 2024-04-25T02:04:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.538+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56068225c040 0x56068225e430 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.538+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682249b60 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.538+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x560682249620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.538+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2202044985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x5606822490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.538+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 >> 172.21.15.102:0/2202044985 conn(0x56068223d4c0 msgr2=0x560682243240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.538+0000 7f3a3d600640 0 client.27293 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.538+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 shutdown_connections 2024-04-25T02:04:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.539+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2202044985 wait complete. 2024-04-25T02:04:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.539+0000 7f3a5da6a740 20 client.27293 trim_cache size 0 max 16384 2024-04-25T02:04:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.565+0000 7f3a5da6a740 1 Processor -- start 2024-04-25T02:04:30.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.565+0000 7f3a5da6a740 1 -- start start 2024-04-25T02:04:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.565+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682243970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.565+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x5606822578a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.565+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682261450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.565+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541beb20 con 0x560682240fb0 2024-04-25T02:04:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.565+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541bc330 con 0x56068224c190 2024-04-25T02:04:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.565+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541ba1a0 con 0x5606822418d0 2024-04-25T02:04:30.571 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.566+0000 7f3a53400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x5606822578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.566+0000 7f3a53e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682243970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.566+0000 7f3a53400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x5606822578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:45168/0 (socket says 172.21.15.102:45168) 2024-04-25T02:04:30.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.566+0000 7f3a53400640 1 -- 172.21.15.102:0/2243695818 learned_addr learned my addr 172.21.15.102:0/2243695818 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:30.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.566+0000 7f3a58c00640 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682261450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.567+0000 7f3a58c00640 1 -- 172.21.15.102:0/2243695818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 msgr2=0x5606822578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:30.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.567+0000 7f3a58c00640 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x5606822578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.567+0000 7f3a58c00640 1 -- 172.21.15.102:0/2243695818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 msgr2=0x560682243970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:30.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.567+0000 7f3a58c00640 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682243970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.567+0000 7f3a58c00640 1 -- 172.21.15.102:0/2243695818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560682243eb0 con 0x56068224c190 2024-04-25T02:04:30.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.567+0000 7f3a53e00640 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682243970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.567+0000 7f3a53400640 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x5606822578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.568+0000 7f3a58c00640 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682261450 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c011f10 tx=0x7f3a4c0096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d1f3da487a0831 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.568+0000 7f3a3fe00640 1 -- 172.21.15.102:0/2243695818 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c028070 con 0x56068224c190 2024-04-25T02:04:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.569+0000 7f3a3fe00640 1 -- 172.21.15.102:0/2243695818 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4c004f00 con 0x56068224c190 2024-04-25T02:04:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.569+0000 7f3a3fe00640 1 -- 172.21.15.102:0/2243695818 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c01ab20 con 0x56068224c190 2024-04-25T02:04:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.570+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2243695818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 msgr2=0x560682261450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.570+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682261450 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c011f10 tx=0x7f3a4c0096c0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.570+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2243695818 shutdown_connections 2024-04-25T02:04:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.570+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x560682261450 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.570+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606822418d0 0x5606822578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.570+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/2243695818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560682240fb0 0x560682243970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.570+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2243695818 >> 172.21.15.102:0/2243695818 conn(0x56068223d4c0 msgr2=0x560682257190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.570+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2243695818 shutdown_connections 2024-04-25T02:04:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.571+0000 7f3a5da6a740 1 -- 172.21.15.102:0/2243695818 wait complete. 2024-04-25T02:04:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.571+0000 7f3a5da6a740 1 -- start start 2024-04-25T02:04:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.571+0000 7f3a5da6a740 1 Processor -- start 2024-04-25T02:04:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560682240fb0 0x56068225f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606822418d0 0x56068225fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a5da6a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x5606822c1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541ba7f0 con 0x5606822418d0 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541bbc60 con 0x56068224c190 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a5da6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a541baec0 con 0x560682240fb0 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a5da6a740 10 client.0 _ll_register_callbacks cb 0x7f3a40041070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a58c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x5606822c1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a58c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x5606822c1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:45362/0 (socket says 172.21.15.102:45362) 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a58c00640 1 -- 172.21.15.102:0/1909250668 learned_addr learned my addr 172.21.15.102:0/1909250668 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.572+0000 7f3a53400640 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606822418d0 0x56068225fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.573+0000 7f3a53e00640 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560682240fb0 0x56068225f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.573+0000 7f3a58c00640 1 -- 172.21.15.102:0/1909250668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560682240fb0 msgr2=0x56068225f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.573+0000 7f3a58c00640 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560682240fb0 0x56068225f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.573+0000 7f3a58c00640 1 -- 172.21.15.102:0/1909250668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606822418d0 msgr2=0x56068225fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.573+0000 7f3a58c00640 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606822418d0 0x56068225fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.573+0000 7f3a58c00640 1 -- 172.21.15.102:0/1909250668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606822c1760 con 0x56068224c190 2024-04-25T02:04:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.573+0000 7f3a53e00640 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560682240fb0 0x56068225f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:04:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.574+0000 7f3a58c00640 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x5606822c1220 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c01a470 tx=0x7f3a4c004e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22d54b5db739488e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.574+0000 7f3a3d600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:04:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.575+0000 7f3a3d600640 1 -- 172.21.15.102:0/1909250668 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c028070 con 0x56068224c190 2024-04-25T02:04:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.575+0000 7f3a3d600640 1 -- 172.21.15.102:0/1909250668 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f3a4c0053e0 con 0x56068224c190 2024-04-25T02:04:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.575+0000 7f3a5da6a740 1 -- 172.21.15.102:0/1909250668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a4c004f00 con 0x56068224c190 2024-04-25T02:04:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.575+0000 7f3a3d600640 1 -- 172.21.15.102:0/1909250668 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3a4c043050 con 0x56068224c190 2024-04-25T02:04:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.575+0000 7f3a5da6a740 10 client.27299 fetch_fsmap learned FSMap version 38 2024-04-25T02:04:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.575+0000 7f3a5da6a740 1 -- 172.21.15.102:0/1909250668 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5606822c2a20 con 0x56068224c190 2024-04-25T02:04:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.576+0000 7f3a3d600640 1 -- 172.21.15.102:0/1909250668 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4c004f00 con 0x56068224c190 2024-04-25T02:04:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.576+0000 7f3a3d600640 1 -- 172.21.15.102:0/1909250668 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f3a4c011950 con 0x56068224c190 2024-04-25T02:04:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.576+0000 7f3a5da6a740 10 client.27299 fetch_fsmap finished waiting for FSMap version 38 2024-04-25T02:04:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.576+0000 7f3a5da6a740 10 client.27299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.576+0000 7f3a5da6a740 20 client.27299 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.576+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5606822c3070 0x5606822c5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.576+0000 7f3a5da6a740 4 client.27299 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:04:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.576+0000 7f3a5da6a740 1 -- 172.21.15.102:0/1909250668 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5606821ac340 con 0x5606822c3070 2024-04-25T02:04:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.577+0000 7f3a53e00640 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5606822c3070 0x5606822c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.577+0000 7f3a53e00640 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5606822c3070 0x5606822c5490 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d5d1a0de13f744a server_cookie=ce12242b59c41af1 in_seq=0 out_seq=0 2024-04-25T02:04:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.577+0000 7f3a3d600640 10 client.27299 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a3d600640 1 -- 172.21.15.102:0/1909250668 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5606821ac340 con 0x5606822c3070 2024-04-25T02:04:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a3d600640 10 client.27299 handle_command_reply: tid=0 2024-04-25T02:04:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 client.27299 shutdown 2024-04-25T02:04:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 -- 172.21.15.102:0/1909250668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 msgr2=0x5606822c1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x5606822c1220 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c01a470 tx=0x7f3a4c004e90 comp rx=0 tx=0).stop 2024-04-25T02:04:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 -- 172.21.15.102:0/1909250668 shutdown_connections 2024-04-25T02:04:30.663 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5606822c3070 0x5606822c5490 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56068224c190 0x5606822c1220 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606822418d0 0x56068225fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a3d600640 0 client.27299 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 --2- 172.21.15.102:0/1909250668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560682240fb0 0x56068225f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 -- 172.21.15.102:0/1909250668 >> 172.21.15.102:0/1909250668 conn(0x56068223d4c0 msgr2=0x56068223c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.657+0000 7f3a5da6a740 1 -- 172.21.15.102:0/1909250668 shutdown_connections 2024-04-25T02:04:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.658+0000 7f3a5da6a740 1 -- 172.21.15.102:0/1909250668 wait complete. 2024-04-25T02:04:30.664 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.658+0000 7f3a5da6a740 20 client.27299 trim_cache size 0 max 16384 2024-04-25T02:04:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:04:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:04:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:04:30.665 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:04:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3a380a2d70 msgr2=0x7f3a380a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3a380a2d70 0x7f3a380a5230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3a54197340 tx=0x7f3a40031400 comp rx=0 tx=0).stop 2024-04-25T02:04:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54102700 msgr2=0x7f3a5419eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54102700 0x7f3a5419eb40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400d950 tx=0x7f3a4400de10 comp rx=0 tx=0).stop 2024-04-25T02:04:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 shutdown_connections 2024-04-25T02:04:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3a380a2d70 0x7f3a380a5230 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a5410bb20 0x7f3a5410a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a54103020 0x7f3a5419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 --2- 172.21.15.102:0/1412816997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a54102700 0x7f3a5419eb40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:30.667 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.659+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 >> 172.21.15.102:0/1412816997 conn(0x7f3a540fbf60 msgr2=0x7f3a54101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:30.667 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.660+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 shutdown_connections 2024-04-25T02:04:30.667 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:30.660+0000 7f3a59600640 1 -- 172.21.15.102:0/1412816997 wait complete. 2024-04-25T02:04:31.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:31 smithi102 ceph-mon[27796]: pgmap v286: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 5.3 MiB/s wr, 480 op/s 2024-04-25T02:04:31.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:31 smithi110 ceph-mon[32848]: pgmap v286: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 5.3 MiB/s wr, 480 op/s 2024-04-25T02:04:31.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:31 smithi193 ceph-mon[32981]: pgmap v286: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 5.3 MiB/s wr, 480 op/s 2024-04-25T02:04:33.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 520.0 s, 2118.1 tps, lat 47.589 ms stddev 57.559 2024-04-25T02:04:33.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:33 smithi102 ceph-mon[27796]: pgmap v287: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.1 MiB/s wr, 641 op/s 2024-04-25T02:04:33.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:33 smithi110 ceph-mon[32848]: pgmap v287: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.1 MiB/s wr, 641 op/s 2024-04-25T02:04:33.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:33 smithi193 ceph-mon[32981]: pgmap v287: 97 pgs: 97 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 7.1 MiB/s wr, 641 op/s 2024-04-25T02:04:35.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:35 smithi102 ceph-mon[27796]: pgmap v288: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 7.0 MiB/s wr, 647 op/s 2024-04-25T02:04:35.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:35 smithi110 ceph-mon[32848]: pgmap v288: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 7.0 MiB/s wr, 647 op/s 2024-04-25T02:04:35.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:35 smithi193 ceph-mon[32981]: pgmap v288: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 7.0 MiB/s wr, 647 op/s 2024-04-25T02:04:37.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:37 smithi102 ceph-mon[27796]: pgmap v289: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 5.3 MiB/s wr, 500 op/s 2024-04-25T02:04:37.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:37 smithi110 ceph-mon[32848]: pgmap v289: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 5.3 MiB/s wr, 500 op/s 2024-04-25T02:04:37.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:37 smithi193 ceph-mon[32981]: pgmap v289: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 5.3 MiB/s wr, 500 op/s 2024-04-25T02:04:38.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 525.0 s, 1201.5 tps, lat 82.614 ms stddev 156.494 2024-04-25T02:04:39.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:39 smithi102 ceph-mon[27796]: pgmap v290: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 12 MiB/s wr, 1.05k op/s 2024-04-25T02:04:39.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:39 smithi110 ceph-mon[32848]: pgmap v290: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 12 MiB/s wr, 1.05k op/s 2024-04-25T02:04:39.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:39 smithi193 ceph-mon[32981]: pgmap v290: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 12 MiB/s wr, 1.05k op/s 2024-04-25T02:04:41.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:41 smithi102 ceph-mon[27796]: pgmap v291: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.5 MiB/s wr, 840 op/s 2024-04-25T02:04:41.684 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:41 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:04:41.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:41 smithi110 ceph-mon[32848]: pgmap v291: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.5 MiB/s wr, 840 op/s 2024-04-25T02:04:41.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:41 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:04:41.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:41 smithi193 ceph-mon[32981]: pgmap v291: 97 pgs: 97 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.5 MiB/s wr, 840 op/s 2024-04-25T02:04:41.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:41 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:04:43.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 530.0 s, 1860.2 tps, lat 53.929 ms stddev 65.196 2024-04-25T02:04:43.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:43 smithi110 ceph-mon[32848]: pgmap v292: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 1.38k op/s 2024-04-25T02:04:43.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:43 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:04:43.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:43 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:04:43.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:43 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:04:43.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:43 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:04:43.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:43 smithi193 ceph-mon[32981]: pgmap v292: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 1.38k op/s 2024-04-25T02:04:43.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:43 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:04:43.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:43 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:04:43.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:43 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:04:43.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:43 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:04:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.835+0000 7f4103e00640 1 -- 172.21.15.102:0/1234486058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 msgr2=0x7f40fc1121f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.835+0000 7f4103e00640 1 --2- 172.21.15.102:0/1234486058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 0x7f40fc1121f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f40e8009900 tx=0x7f40e802a4b0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.835+0000 7f4103e00640 1 -- 172.21.15.102:0/1234486058 shutdown_connections 2024-04-25T02:04:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.835+0000 7f4103e00640 1 --2- 172.21.15.102:0/1234486058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40fc112730 0x7f40fc114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.835+0000 7f4103e00640 1 --2- 172.21.15.102:0/1234486058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 0x7f40fc1121f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.835+0000 7f4103e00640 1 --2- 172.21.15.102:0/1234486058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40fc073ae0 0x7f40fc073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.835+0000 7f4103e00640 1 -- 172.21.15.102:0/1234486058 >> 172.21.15.102:0/1234486058 conn(0x7f40fc0fd1f0 msgr2=0x7f40fc0ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.835+0000 7f4103e00640 1 -- 172.21.15.102:0/1234486058 shutdown_connections 2024-04-25T02:04:43.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.835+0000 7f4103e00640 1 -- 172.21.15.102:0/1234486058 wait complete. 2024-04-25T02:04:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.836+0000 7f4103e00640 1 Processor -- start 2024-04-25T02:04:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.836+0000 7f4103e00640 1 -- start start 2024-04-25T02:04:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4103e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40fc073ae0 0x7f40fc19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4103e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 0x7f40fc19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4103e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40fc112730 0x7f40fc1a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4100a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 0x7f40fc19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4101e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40fc112730 0x7f40fc1a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4101e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40fc112730 0x7f40fc1a2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42716/0 (socket says 172.21.15.102:42716) 2024-04-25T02:04:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4100a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 0x7f40fc19eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51000/0 (socket says 172.21.15.102:51000) 2024-04-25T02:04:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4101e00640 1 -- 172.21.15.102:0/2338944347 learned_addr learned my addr 172.21.15.102:0/2338944347 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc118150 con 0x7f40fc073ae0 2024-04-25T02:04:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc117fd0 con 0x7f40fc112730 2024-04-25T02:04:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc1182d0 con 0x7f40fc074400 2024-04-25T02:04:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.837+0000 7f4101400640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40fc073ae0 0x7f40fc19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.838+0000 7f4100a00640 1 -- 172.21.15.102:0/2338944347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40fc112730 msgr2=0x7f40fc1a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.838+0000 7f4100a00640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40fc112730 0x7f40fc1a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.838+0000 7f4100a00640 1 -- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40fc073ae0 msgr2=0x7f40fc19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.838+0000 7f4100a00640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40fc073ae0 0x7f40fc19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.838+0000 7f4100a00640 1 -- 172.21.15.102:0/2338944347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40fc1a35c0 con 0x7f40fc074400 2024-04-25T02:04:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.838+0000 7f4100a00640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 0x7f40fc19eb10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f40e8033040 tx=0x7f40e8005320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.838+0000 7f40fa000640 1 -- 172.21.15.102:0/2338944347 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8040030 con 0x7f40fc074400 2024-04-25T02:04:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.838+0000 7f40fa000640 1 -- 172.21.15.102:0/2338944347 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40e803c920 con 0x7f40fc074400 2024-04-25T02:04:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.839+0000 7f40fa000640 1 -- 172.21.15.102:0/2338944347 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8031ac0 con 0x7f40fc074400 2024-04-25T02:04:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.839+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40fc1a3850 con 0x7f40fc074400 2024-04-25T02:04:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.839+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40fc1a3cc0 con 0x7f40fc074400 2024-04-25T02:04:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.843+0000 7f40fa000640 1 -- 172.21.15.102:0/2338944347 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f40e8043030 con 0x7f40fc074400 2024-04-25T02:04:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.843+0000 7f4106d82740 1 Processor -- start 2024-04-25T02:04:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.843+0000 7f4106d82740 1 -- start start 2024-04-25T02:04:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.843+0000 7f4106d82740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8dc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.843+0000 7f4106d82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.843+0000 7f4106d82740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.843+0000 7f4106d82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557fcb83fab0 con 0x557fcb8dc720 2024-04-25T02:04:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.843+0000 7f4106d82740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557fcb7e8640 con 0x557fcb8dbe00 2024-04-25T02:04:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4100a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.843+0000 7f4106d82740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557fcb83ba70 con 0x557fcb8e6fe0 2024-04-25T02:04:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8dc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8dc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42724/0 (socket says 172.21.15.102:42724) 2024-04-25T02:04:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4100a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51002/0 (socket says 172.21.15.102:51002) 2024-04-25T02:04:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101e00640 1 -- 172.21.15.102:0/3721506760 learned_addr learned my addr 172.21.15.102:0/3721506760 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:43.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101400640 1 --2- 172.21.15.102:0/3721506760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101e00640 1 -- 172.21.15.102:0/3721506760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 msgr2=0x557fcb8e93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101e00640 1 --2- 172.21.15.102:0/3721506760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101e00640 1 -- 172.21.15.102:0/3721506760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 msgr2=0x557fcb8e6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101e00640 1 --2- 172.21.15.102:0/3721506760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101e00640 1 -- 172.21.15.102:0/3721506760 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fcb554210 con 0x557fcb8dbe00 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f4101e00640 1 --2- 172.21.15.102:0/3721506760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8dc1e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f40f40103d0 tx=0x7f40f4010890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fcb9d4e625b0675 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.844+0000 7f40fa000640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40e40a2e20 0x7f40e40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.845+0000 7f40fa000640 1 -- 172.21.15.102:0/2338944347 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f40e80efd40 con 0x7f40fc074400 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.845+0000 7f40ee600640 1 -- 172.21.15.102:0/3721506760 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4011700 con 0x557fcb8dbe00 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.845+0000 7f4101400640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40e40a2e20 0x7f40e40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.845+0000 7f40ee600640 1 -- 172.21.15.102:0/3721506760 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40f4011d80 con 0x557fcb8dbe00 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.845+0000 7f40ee600640 1 -- 172.21.15.102:0/3721506760 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f400e3c0 con 0x557fcb8dbe00 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.846+0000 7f4106d82740 1 -- 172.21.15.102:0/3721506760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 msgr2=0x557fcb8dc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.846+0000 7f4101400640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40e40a2e20 0x7f40e40a52e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x557fcb8dd3a0 tx=0x7f40f000b390 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.846+0000 7f4106d82740 1 --2- 172.21.15.102:0/3721506760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8dc1e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f40f40103d0 tx=0x7f40f4010890 comp rx=0 tx=0).stop 2024-04-25T02:04:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.846+0000 7f4106d82740 1 -- 172.21.15.102:0/3721506760 shutdown_connections 2024-04-25T02:04:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.846+0000 7f4106d82740 1 --2- 172.21.15.102:0/3721506760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.846+0000 7f4106d82740 1 --2- 172.21.15.102:0/3721506760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.846+0000 7f4106d82740 1 --2- 172.21.15.102:0/3721506760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8dc1e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.846+0000 7f4106d82740 1 -- 172.21.15.102:0/3721506760 >> 172.21.15.102:0/3721506760 conn(0x557fcb8d8310 msgr2=0x557fcb8d86f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.847+0000 7f4106d82740 1 -- 172.21.15.102:0/3721506760 shutdown_connections 2024-04-25T02:04:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.847+0000 7f4106d82740 1 -- 172.21.15.102:0/3721506760 wait complete. 2024-04-25T02:04:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.848+0000 7f4106d82740 1 -- start start 2024-04-25T02:04:43.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.848+0000 7f4106d82740 1 Processor -- start 2024-04-25T02:04:43.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.849+0000 7f4106d82740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8e3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.849+0000 7f4106d82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.849+0000 7f4106d82740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.849+0000 7f4106d82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x557fcb8e9c60 con 0x557fcb8dc720 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.849+0000 7f4106d82740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x557fcb7fb990 con 0x557fcb8dbe00 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.849+0000 7f4106d82740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557fcb8e9de0 con 0x557fcb8e6fe0 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.849+0000 7f4106d82740 10 client.0 _ll_register_callbacks cb 0x557fcb7fd950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4101400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8e3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4101e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4101e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51018/0 (socket says 172.21.15.102:51018) 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4101e00640 1 -- 172.21.15.102:0/2424800352 learned_addr learned my addr 172.21.15.102:0/2424800352 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:43.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4100a00640 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4101e00640 1 -- 172.21.15.102:0/2424800352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 msgr2=0x557fcb8e3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4101e00640 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8e3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4101e00640 1 -- 172.21.15.102:0/2424800352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 msgr2=0x557fcb8e4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4101e00640 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.850+0000 7f4101e00640 1 -- 172.21.15.102:0/2424800352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fcb620620 con 0x557fcb8e6fe0 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.851+0000 7f4101e00640 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e49a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009370 tx=0x7f40f401b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95d86a008ecf4ae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.851+0000 7f40e2a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.851+0000 7f40e2a00640 1 -- 172.21.15.102:0/2424800352 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4004e50 con 0x557fcb8e6fe0 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.851+0000 7f40e2a00640 1 -- 172.21.15.102:0/2424800352 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f40f4011500 con 0x557fcb8e6fe0 2024-04-25T02:04:43.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.851+0000 7f4106d82740 1 -- 172.21.15.102:0/2424800352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40f4011d80 con 0x557fcb8e6fe0 2024-04-25T02:04:43.859 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.852+0000 7f40e2a00640 1 -- 172.21.15.102:0/2424800352 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f40f4011700 con 0x557fcb8e6fe0 2024-04-25T02:04:43.859 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.852+0000 7f4106d82740 10 client.27271 fetch_fsmap learned FSMap version 39 2024-04-25T02:04:43.859 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.852+0000 7f4106d82740 1 -- 172.21.15.102:0/2424800352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557fcb906b80 con 0x557fcb8e6fe0 2024-04-25T02:04:43.859 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.852+0000 7f40e2a00640 1 -- 172.21.15.102:0/2424800352 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40f4011d80 con 0x557fcb8e6fe0 2024-04-25T02:04:43.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.852+0000 7f40e2a00640 1 -- 172.21.15.102:0/2424800352 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 39) ==== 8358+0+0 (secure 0 0 0) 0x7f40f402e030 con 0x557fcb8e6fe0 2024-04-25T02:04:43.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.853+0000 7f4106d82740 10 client.27271 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T02:04:43.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.853+0000 7f4106d82740 10 client.27271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:43.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.853+0000 7f4106d82740 20 client.27271 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:43.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.853+0000 7f4106d82740 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557fcb907f80 0x557fcb90a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.853+0000 7f4106d82740 4 client.27271 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:04:43.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.853+0000 7f4106d82740 1 -- 172.21.15.102:0/2424800352 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557fcb84cff0 con 0x557fcb907f80 2024-04-25T02:04:43.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.853+0000 7f4101400640 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557fcb907f80 0x557fcb90a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.854+0000 7f4101400640 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557fcb907f80 0x557fcb90a370 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3e949a84222f7e2 server_cookie=9a2739b7a1e240db in_seq=0 out_seq=0 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.854+0000 7f40e2a00640 10 client.27271 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f40e2a00640 1 -- 172.21.15.102:0/2424800352 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557fcb84cff0 con 0x557fcb907f80 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f40e2a00640 10 client.27271 handle_command_reply: tid=0 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 client.27271 shutdown 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 -- 172.21.15.102:0/2424800352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 msgr2=0x557fcb8e49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e49a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009370 tx=0x7f40f401b6e0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 -- 172.21.15.102:0/2424800352 shutdown_connections 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557fcb907f80 0x557fcb90a370 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e49a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 --2- 172.21.15.102:0/2424800352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8e3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 -- 172.21.15.102:0/2424800352 >> 172.21.15.102:0/2424800352 conn(0x557fcb8d8310 msgr2=0x557fcb8de090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f40e2a00640 0 client.27271 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.855+0000 7f4106d82740 1 -- 172.21.15.102:0/2424800352 shutdown_connections 2024-04-25T02:04:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.856+0000 7f4106d82740 1 -- 172.21.15.102:0/2424800352 wait complete. 2024-04-25T02:04:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.856+0000 7f4106d82740 20 client.27271 trim_cache size 0 max 16384 2024-04-25T02:04:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.873+0000 7f4106d82740 1 Processor -- start 2024-04-25T02:04:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.873+0000 7f4106d82740 1 -- start start 2024-04-25T02:04:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.873+0000 7f4106d82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dbe00 0x557fcb8d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.873+0000 7f4106d82740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8dc720 0x557fcb8e47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.873+0000 7f4106d82740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8e6fe0 0x557fcb90f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.873+0000 7f4106d82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc1adbc0 con 0x557fcb8dbe00 2024-04-25T02:04:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.873+0000 7f4106d82740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc1ab3d0 con 0x557fcb8e6fe0 2024-04-25T02:04:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.873+0000 7f4106d82740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc10c260 con 0x557fcb8dc720 2024-04-25T02:04:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4100a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8dc720 0x557fcb8e47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4101400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dbe00 0x557fcb8d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4100a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8dc720 0x557fcb8e47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51024/0 (socket says 172.21.15.102:51024) 2024-04-25T02:04:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4100a00640 1 -- 172.21.15.102:0/1450704151 learned_addr learned my addr 172.21.15.102:0/1450704151 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4101e00640 1 --2- 172.21.15.102:0/1450704151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8e6fe0 0x557fcb90f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4100a00640 1 -- 172.21.15.102:0/1450704151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8e6fe0 msgr2=0x557fcb90f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4100a00640 1 --2- 172.21.15.102:0/1450704151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8e6fe0 0x557fcb90f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4100a00640 1 -- 172.21.15.102:0/1450704151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dbe00 msgr2=0x557fcb8d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4100a00640 1 --2- 172.21.15.102:0/1450704151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dbe00 0x557fcb8d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.882 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.874+0000 7f4100a00640 1 -- 172.21.15.102:0/1450704151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557fcb8d8140 con 0x557fcb8dc720 2024-04-25T02:04:43.882 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.875+0000 7f4100a00640 1 --2- 172.21.15.102:0/1450704151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8dc720 0x557fcb8e47c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f40e80bc190 tx=0x7f40e80315b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee441c933a534207 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:43.882 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.875+0000 7f40edc00640 1 -- 172.21.15.102:0/1450704151 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e80bc900 con 0x557fcb8dc720 2024-04-25T02:04:43.882 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.875+0000 7f40edc00640 1 -- 172.21.15.102:0/1450704151 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40e80bc200 con 0x557fcb8dc720 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.875+0000 7f40edc00640 1 -- 172.21.15.102:0/1450704151 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8031c60 con 0x557fcb8dc720 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.876+0000 7f4106d82740 1 -- 172.21.15.102:0/1450704151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8dc720 msgr2=0x557fcb8e47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.876+0000 7f4106d82740 1 --2- 172.21.15.102:0/1450704151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8dc720 0x557fcb8e47c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f40e80bc190 tx=0x7f40e80315b0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.876+0000 7f4106d82740 1 -- 172.21.15.102:0/1450704151 shutdown_connections 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.876+0000 7f4106d82740 1 --2- 172.21.15.102:0/1450704151 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8e6fe0 0x557fcb90f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.876+0000 7f4106d82740 1 --2- 172.21.15.102:0/1450704151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8dc720 0x557fcb8e47c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.876+0000 7f4106d82740 1 --2- 172.21.15.102:0/1450704151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dbe00 0x557fcb8d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.876+0000 7f4106d82740 1 -- 172.21.15.102:0/1450704151 >> 172.21.15.102:0/1450704151 conn(0x557fcb8d8310 msgr2=0x557fcb8de090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.877+0000 7f4106d82740 1 -- 172.21.15.102:0/1450704151 shutdown_connections 2024-04-25T02:04:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.877+0000 7f4106d82740 1 -- 172.21.15.102:0/1450704151 wait complete. 2024-04-25T02:04:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.877+0000 7f4106d82740 1 -- start start 2024-04-25T02:04:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.878+0000 7f4106d82740 1 Processor -- start 2024-04-25T02:04:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.878+0000 7f4106d82740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8e0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.879+0000 7f4106d82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.879+0000 7f4106d82740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.879+0000 7f4101400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8e0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.879+0000 7f4101400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8e0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42762/0 (socket says 172.21.15.102:42762) 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.879+0000 7f4106d82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc10c8b0 con 0x557fcb8dc720 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.879+0000 7f4106d82740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc10dd20 con 0x557fcb8dbe00 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.879+0000 7f4106d82740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc10cf80 con 0x557fcb8e6fe0 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.880+0000 7f4106d82740 10 client.0 _ll_register_callbacks cb 0x7f40f4009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.880+0000 7f4101400640 1 -- 172.21.15.102:0/1916404104 learned_addr learned my addr 172.21.15.102:0/1916404104 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.880+0000 7f4101e00640 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.880+0000 7f4101e00640 1 -- 172.21.15.102:0/1916404104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 msgr2=0x557fcb8e0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.880+0000 7f4101e00640 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8e0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.880+0000 7f4101e00640 1 -- 172.21.15.102:0/1916404104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 msgr2=0x557fcb8e1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.880+0000 7f4101e00640 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.880+0000 7f4101e00640 1 -- 172.21.15.102:0/1916404104 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557fcb96d2e0 con 0x557fcb8e6fe0 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.881+0000 7f4101e00640 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e1750 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f40f4005910 tx=0x7f40f4009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbc0dc0cb65db4a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.881+0000 7f40e2a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.881+0000 7f40e2a00640 1 -- 172.21.15.102:0/1916404104 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f40268f0 con 0x557fcb8e6fe0 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.881+0000 7f40e2a00640 1 -- 172.21.15.102:0/1916404104 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f40f4026a90 con 0x557fcb8e6fe0 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.881+0000 7f4106d82740 1 -- 172.21.15.102:0/1916404104 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40e80bc200 con 0x557fcb8e6fe0 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.881+0000 7f40e2a00640 1 -- 172.21.15.102:0/1916404104 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f40f4045050 con 0x557fcb8e6fe0 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.881+0000 7f4106d82740 10 client.27277 fetch_fsmap learned FSMap version 39 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.881+0000 7f4106d82740 1 -- 172.21.15.102:0/1916404104 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557fcb96e5f0 con 0x557fcb8e6fe0 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.881+0000 7f40e2a00640 1 -- 172.21.15.102:0/1916404104 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40e80bc200 con 0x557fcb8e6fe0 2024-04-25T02:04:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.882+0000 7f40e2a00640 1 -- 172.21.15.102:0/1916404104 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 39) ==== 8358+0+0 (secure 0 0 0) 0x7f40f4040030 con 0x557fcb8e6fe0 2024-04-25T02:04:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.882+0000 7f4106d82740 10 client.27277 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T02:04:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.882+0000 7f4106d82740 10 client.27277 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.882+0000 7f4106d82740 20 client.27277 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.882+0000 7f4106d82740 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557fcb96edc0 0x557fcb9711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.882+0000 7f4106d82740 4 client.27277 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:04:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.882+0000 7f4106d82740 1 -- 172.21.15.102:0/1916404104 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557fcb847340 con 0x557fcb96edc0 2024-04-25T02:04:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.883+0000 7f4101400640 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557fcb96edc0 0x557fcb9711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.883+0000 7f4101400640 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557fcb96edc0 0x557fcb9711e0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e03102e4e2a97ba6 server_cookie=4f8d09792a9f002c in_seq=0 out_seq=0 2024-04-25T02:04:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.883+0000 7f40e2a00640 10 client.27277 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f40e2a00640 1 -- 172.21.15.102:0/1916404104 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557fcb847340 con 0x557fcb96edc0 2024-04-25T02:04:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f40e2a00640 10 client.27277 handle_command_reply: tid=0 2024-04-25T02:04:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f4106d82740 1 client.27277 shutdown 2024-04-25T02:04:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f4106d82740 1 -- 172.21.15.102:0/1916404104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 msgr2=0x557fcb8e1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f4106d82740 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e1750 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f40f4005910 tx=0x7f40f4009cb0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f4106d82740 1 -- 172.21.15.102:0/1916404104 shutdown_connections 2024-04-25T02:04:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f4106d82740 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x557fcb96edc0 0x557fcb9711e0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f4106d82740 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557fcb8e6fe0 0x557fcb8e1750 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f4106d82740 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557fcb8dc720 0x557fcb8e1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f4106d82740 1 --2- 172.21.15.102:0/1916404104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fcb8dbe00 0x557fcb8e0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f4106d82740 1 -- 172.21.15.102:0/1916404104 >> 172.21.15.102:0/1916404104 conn(0x557fcb8d8310 msgr2=0x557fcb8e5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.907+0000 7f40e2a00640 0 client.27277 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.908+0000 7f4106d82740 1 -- 172.21.15.102:0/1916404104 shutdown_connections 2024-04-25T02:04:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.908+0000 7f4106d82740 1 -- 172.21.15.102:0/1916404104 wait complete. 2024-04-25T02:04:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.908+0000 7f4106d82740 20 client.27277 trim_cache size 0 max 16384 2024-04-25T02:04:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:04:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:04:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:04:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:04:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40e40a2e20 msgr2=0x7f40e40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40e40a2e20 0x7f40e40a52e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x557fcb8dd3a0 tx=0x7f40f000b390 comp rx=0 tx=0).stop 2024-04-25T02:04:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 msgr2=0x7f40fc19eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 0x7f40fc19eb10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f40e8033040 tx=0x7f40e8005320 comp rx=0 tx=0).stop 2024-04-25T02:04:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 shutdown_connections 2024-04-25T02:04:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f40e40a2e20 0x7f40e40a52e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40fc112730 0x7f40fc1a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc074400 0x7f40fc19eb10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 --2- 172.21.15.102:0/2338944347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40fc073ae0 0x7f40fc19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 >> 172.21.15.102:0/2338944347 conn(0x7f40fc0fd1f0 msgr2=0x7f40fc0fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 shutdown_connections 2024-04-25T02:04:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:43.909+0000 7f4103e00640 1 -- 172.21.15.102:0/2338944347 wait complete. 2024-04-25T02:04:43.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:43 smithi102 ceph-mon[27796]: pgmap v292: 97 pgs: 97 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 1.38k op/s 2024-04-25T02:04:43.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:43 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:04:43.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:43 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:04:43.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:43 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:04:43.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:43 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:04:45.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:45 smithi110 ceph-mon[32848]: pgmap v293: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 MiB/s wr, 1.75k op/s 2024-04-25T02:04:45.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:45 smithi193 ceph-mon[32981]: pgmap v293: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 MiB/s wr, 1.75k op/s 2024-04-25T02:04:45.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:45 smithi102 ceph-mon[27796]: pgmap v293: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 MiB/s wr, 1.75k op/s 2024-04-25T02:04:47.746 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:47 smithi110 ceph-mon[32848]: pgmap v294: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 1.62k op/s 2024-04-25T02:04:47.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:47 smithi193 ceph-mon[32981]: pgmap v294: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 1.62k op/s 2024-04-25T02:04:47.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:47 smithi102 ceph-mon[27796]: pgmap v294: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 1.62k op/s 2024-04-25T02:04:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 535.0 s, 1614.8 tps, lat 62.179 ms stddev 69.288 2024-04-25T02:04:48.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:48 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:04:48.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:48 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:04:48.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:48 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:04:49.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:49 smithi110 ceph-mon[32848]: pgmap v295: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 23 MiB/s wr, 1.89k op/s 2024-04-25T02:04:49.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:49 smithi193 ceph-mon[32981]: pgmap v295: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 23 MiB/s wr, 1.89k op/s 2024-04-25T02:04:49.829 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:49 smithi102 ceph-mon[27796]: pgmap v295: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 23 MiB/s wr, 1.89k op/s 2024-04-25T02:04:50.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:50 smithi102 ceph-mon[27796]: pgmap v296: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 1.34k op/s 2024-04-25T02:04:50.992 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:50 smithi110 ceph-mon[32848]: pgmap v296: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 1.34k op/s 2024-04-25T02:04:51.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:50 smithi193 ceph-mon[32981]: pgmap v296: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 17 MiB/s wr, 1.34k op/s 2024-04-25T02:04:53.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 540.0 s, 2170.0 tps, lat 45.932 ms stddev 63.669 2024-04-25T02:04:53.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:52 smithi102 ceph-mon[27796]: pgmap v297: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 18 MiB/s wr, 1.46k op/s 2024-04-25T02:04:53.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:52 smithi110 ceph-mon[32848]: pgmap v297: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 18 MiB/s wr, 1.46k op/s 2024-04-25T02:04:53.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:52 smithi193 ceph-mon[32981]: pgmap v297: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 18 MiB/s wr, 1.46k op/s 2024-04-25T02:04:55.181 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:54 smithi102 ceph-mon[27796]: pgmap v298: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 13 MiB/s wr, 1.06k op/s 2024-04-25T02:04:55.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:54 smithi110 ceph-mon[32848]: pgmap v298: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 13 MiB/s wr, 1.06k op/s 2024-04-25T02:04:55.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:54 smithi193 ceph-mon[32981]: pgmap v298: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 13 MiB/s wr, 1.06k op/s 2024-04-25T02:04:57.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.094+0000 7fc9d0e00640 1 -- 172.21.15.102:0/4173673134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc102110 msgr2=0x7fc9cc10e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:57.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.094+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/4173673134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc102110 0x7fc9cc10e620 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009860 tx=0x7fc9bc02a340 comp rx=0 tx=0).stop 2024-04-25T02:04:57.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.094+0000 7fc9d0e00640 1 -- 172.21.15.102:0/4173673134 shutdown_connections 2024-04-25T02:04:57.101 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.094+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/4173673134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc102110 0x7fc9cc10e620 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.094+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/4173673134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9cc101750 0x7fc9cc101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.094+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/4173673134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9cc107760 0x7fc9cc107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.094+0000 7fc9d0e00640 1 -- 172.21.15.102:0/4173673134 >> 172.21.15.102:0/4173673134 conn(0x7fc9cc078bd0 msgr2=0x7fc9cc078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.094+0000 7fc9d0e00640 1 -- 172.21.15.102:0/4173673134 shutdown_connections 2024-04-25T02:04:57.102 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.094+0000 7fc9d0e00640 1 -- 172.21.15.102:0/4173673134 wait complete. 2024-04-25T02:04:57.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.095+0000 7fc9d0e00640 1 Processor -- start 2024-04-25T02:04:57.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.096+0000 7fc9d0e00640 1 -- start start 2024-04-25T02:04:57.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.096+0000 7fc9d0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9cc101750 0x7fc9cc19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.096+0000 7fc9d0e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc102110 0x7fc9cc19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.096+0000 7fc9d0e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9cc107760 0x7fc9cc19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9d0e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc074710 con 0x7fc9cc101750 2024-04-25T02:04:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9d0e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc074590 con 0x7fc9cc102110 2024-04-25T02:04:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9ca800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9cc107760 0x7fc9cc19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9ca800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9cc107760 0x7fc9cc19e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33532/0 (socket says 172.21.15.102:33532) 2024-04-25T02:04:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9ca800640 1 -- 172.21.15.102:0/2427521539 learned_addr learned my addr 172.21.15.102:0/2427521539 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9d0e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc9cc074890 con 0x7fc9cc107760 2024-04-25T02:04:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9c9400640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc102110 0x7fc9cc19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.104 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9ca800640 1 -- 172.21.15.102:0/2427521539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc102110 msgr2=0x7fc9cc19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9ca800640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc102110 0x7fc9cc19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9ca800640 1 -- 172.21.15.102:0/2427521539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9cc101750 msgr2=0x7fc9cc19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9ca800640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9cc101750 0x7fc9cc19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9ca800640 1 -- 172.21.15.102:0/2427521539 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9cc19f030 con 0x7fc9cc107760 2024-04-25T02:04:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.097+0000 7fc9ca800640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9cc107760 0x7fc9cc19e900 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc030460 tx=0x7fc9bc030490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.098+0000 7fc9c2a00640 1 -- 172.21.15.102:0/2427521539 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc037040 con 0x7fc9cc107760 2024-04-25T02:04:57.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.098+0000 7fc9d0e00640 1 -- 172.21.15.102:0/2427521539 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9cc19f2c0 con 0x7fc9cc107760 2024-04-25T02:04:57.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.098+0000 7fc9d0e00640 1 -- 172.21.15.102:0/2427521539 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9cc1a6bb0 con 0x7fc9cc107760 2024-04-25T02:04:57.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.098+0000 7fc9c2a00640 1 -- 172.21.15.102:0/2427521539 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9bc040070 con 0x7fc9cc107760 2024-04-25T02:04:57.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.098+0000 7fc9c2a00640 1 -- 172.21.15.102:0/2427521539 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc03b430 con 0x7fc9cc107760 2024-04-25T02:04:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.102+0000 7fc9c2a00640 1 -- 172.21.15.102:0/2427521539 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc9bc0035f0 con 0x7fc9cc107760 2024-04-25T02:04:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.103+0000 7fc9d3da6740 1 Processor -- start 2024-04-25T02:04:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.104+0000 7fc9d3da6740 1 -- start start 2024-04-25T02:04:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.104+0000 7fc9c2a00640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc9b00ab700 0x7fc9b00adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.104+0000 7fc9c2a00640 1 -- 172.21.15.102:0/2427521539 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fc9bc0eec50 con 0x7fc9cc107760 2024-04-25T02:04:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 0x55a91b2c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a91b21cab0 con 0x55a91b2b9720 2024-04-25T02:04:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a91b1c5640 con 0x55a91b2c3fe0 2024-04-25T02:04:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a91b218a70 con 0x55a91b2b8e00 2024-04-25T02:04:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9c9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9c9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2b91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33548/0 (socket says 172.21.15.102:33548) 2024-04-25T02:04:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9c9e00640 1 -- 172.21.15.102:0/2679660293 learned_addr learned my addr 172.21.15.102:0/2679660293 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9c9e00640 1 -- 172.21.15.102:0/2679660293 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 msgr2=0x55a91b2c63d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:04:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/2679660293 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c63d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9c9e00640 1 -- 172.21.15.102:0/2679660293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 msgr2=0x55a91b2c3aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:04:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/2679660293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 0x55a91b2c3aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.105+0000 7fc9c9e00640 1 -- 172.21.15.102:0/2679660293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a91af31210 con 0x55a91b2b8e00 2024-04-25T02:04:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.106+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/2679660293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2b91e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400e420 tx=0x7fc9c400e8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c414331683fea19f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:57.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.106+0000 7fc9c9400640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc9b00ab700 0x7fc9b00adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.106+0000 7fc9b7400640 1 -- 172.21.15.102:0/2679660293 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4004b20 con 0x55a91b2b8e00 2024-04-25T02:04:57.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.106+0000 7fc9b7400640 1 -- 172.21.15.102:0/2679660293 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c4003e50 con 0x55a91b2b8e00 2024-04-25T02:04:57.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.106+0000 7fc9b7400640 1 -- 172.21.15.102:0/2679660293 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400c6a0 con 0x55a91b2b8e00 2024-04-25T02:04:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.108+0000 7fc9d3da6740 1 -- 172.21.15.102:0/2679660293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 msgr2=0x55a91b2b91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.108+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/2679660293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2b91e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400e420 tx=0x7fc9c400e8e0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.108+0000 7fc9d3da6740 1 -- 172.21.15.102:0/2679660293 shutdown_connections 2024-04-25T02:04:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.108+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/2679660293 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.115 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.109+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/2679660293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 0x55a91b2c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.109+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/2679660293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2b91e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.109+0000 7fc9d3da6740 1 -- 172.21.15.102:0/2679660293 >> 172.21.15.102:0/2679660293 conn(0x55a91b2b5310 msgr2=0x55a91b2b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:57.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.110+0000 7fc9c9400640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc9b00ab700 0x7fc9b00adbc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55a91b2ba3a0 tx=0x7fc9b8005da0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:57.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.110+0000 7fc9d3da6740 1 -- 172.21.15.102:0/2679660293 shutdown_connections 2024-04-25T02:04:57.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.110+0000 7fc9d3da6740 1 -- 172.21.15.102:0/2679660293 wait complete. 2024-04-25T02:04:57.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.111+0000 7fc9d3da6740 1 -- start start 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.113+0000 7fc9d3da6740 1 Processor -- start 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b8e00 0x55a91b2c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b9720 0x55a91b2c1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a91b2c6c10 con 0x55a91b2b8e00 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a91b1d8990 con 0x55a91b2b9720 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a91b2c6d90 con 0x55a91b2c3fe0 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9d3da6740 10 client.0 _ll_register_callbacks cb 0x55a91b1da950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b8e00 0x55a91b2c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b8e00 0x55a91b2c0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:32916/0 (socket says 172.21.15.102:32916) 2024-04-25T02:04:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.114+0000 7fc9c9e00640 1 -- 172.21.15.102:0/1715165508 learned_addr learned my addr 172.21.15.102:0/1715165508 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:57.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.115+0000 7fc9ca800640 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.122 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.115+0000 7fc9c9400640 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b9720 0x55a91b2c1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.116+0000 7fc9c9e00640 1 -- 172.21.15.102:0/1715165508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2c3fe0 msgr2=0x55a91b2c1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.116+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.116+0000 7fc9c9e00640 1 -- 172.21.15.102:0/1715165508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b9720 msgr2=0x55a91b2c1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.116+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b9720 0x55a91b2c1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.116+0000 7fc9c9e00640 1 -- 172.21.15.102:0/1715165508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a91affd620 con 0x55a91b2b8e00 2024-04-25T02:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.116+0000 7fc9c9400640 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b9720 0x55a91b2c1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:04:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.116+0000 7fc9ca800640 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:04:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.117+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b8e00 0x55a91b2c0ee0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40044e0 tx=0x7fc9c40095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8c9a8ea9bda150 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.117+0000 7fc9abe00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.118+0000 7fc9abe00640 1 -- 172.21.15.102:0/1715165508 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4024020 con 0x55a91b2b8e00 2024-04-25T02:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.118+0000 7fc9abe00640 1 -- 172.21.15.102:0/1715165508 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fc9c4003a30 con 0x55a91b2b8e00 2024-04-25T02:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.118+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1715165508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9c4003e50 con 0x55a91b2b8e00 2024-04-25T02:04:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.119+0000 7fc9abe00640 1 -- 172.21.15.102:0/1715165508 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc9c401a040 con 0x55a91b2b8e00 2024-04-25T02:04:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.119+0000 7fc9d3da6740 10 client.16680 fetch_fsmap learned FSMap version 40 2024-04-25T02:04:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.119+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1715165508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a91b2e3ac0 con 0x55a91b2b8e00 2024-04-25T02:04:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.120+0000 7fc9abe00640 1 -- 172.21.15.102:0/1715165508 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c4003e50 con 0x55a91b2b8e00 2024-04-25T02:04:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.120+0000 7fc9abe00640 1 -- 172.21.15.102:0/1715165508 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fc9c400b040 con 0x55a91b2b8e00 2024-04-25T02:04:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.121+0000 7fc9d3da6740 10 client.16680 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:04:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.121+0000 7fc9d3da6740 10 client.16680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.121+0000 7fc9d3da6740 20 client.16680 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.121+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a91b2e4e70 0x55a91b2e7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.121+0000 7fc9d3da6740 4 client.16680 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:04:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.121+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1715165508 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a91b229ff0 con 0x55a91b2e4e70 2024-04-25T02:04:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.122+0000 7fc9c9400640 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a91b2e4e70 0x55a91b2e7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.123+0000 7fc9c9400640 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a91b2e4e70 0x55a91b2e7260 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b10969b8fdd4fd43 server_cookie=9e52908503d44363 in_seq=0 out_seq=0 2024-04-25T02:04:57.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.123+0000 7fc9abe00640 10 client.16680 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:57.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.124+0000 7fc9abe00640 1 -- 172.21.15.102:0/1715165508 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a91b229ff0 con 0x55a91b2e4e70 2024-04-25T02:04:57.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.124+0000 7fc9abe00640 10 client.16680 handle_command_reply: tid=0 2024-04-25T02:04:57.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.124+0000 7fc9d3da6740 1 client.16680 shutdown 2024-04-25T02:04:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.125+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1715165508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b8e00 msgr2=0x55a91b2c0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.125+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b8e00 0x55a91b2c0ee0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40044e0 tx=0x7fc9c40095a0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.126+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1715165508 shutdown_connections 2024-04-25T02:04:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.126+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a91b2e4e70 0x55a91b2e7260 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.126+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.126+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b9720 0x55a91b2c1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.126+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1715165508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b8e00 0x55a91b2c0ee0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.126+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1715165508 >> 172.21.15.102:0/1715165508 conn(0x55a91b2b5310 msgr2=0x55a91b2bb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.126+0000 7fc9abe00640 0 client.16680 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.127+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1715165508 shutdown_connections 2024-04-25T02:04:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.127+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1715165508 wait complete. 2024-04-25T02:04:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.127+0000 7fc9d3da6740 20 client.16680 trim_cache size 0 max 16384 2024-04-25T02:04:57.163 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.157+0000 7fc9d3da6740 1 Processor -- start 2024-04-25T02:04:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.157+0000 7fc9d3da6740 1 -- start start 2024-04-25T02:04:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.157+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.157+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 0x55a91b2b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.157+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.157+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac0111e0 con 0x55a91b2b9720 2024-04-25T02:04:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.157+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac00e9f0 con 0x55a91b2c3fe0 2024-04-25T02:04:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.157+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac00c860 con 0x55a91b2b8e00 2024-04-25T02:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.158+0000 7fc9c9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.158+0000 7fc9ca800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.158+0000 7fc9c9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:33562/0 (socket says 172.21.15.102:33562) 2024-04-25T02:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.158+0000 7fc9c9e00640 1 -- 172.21.15.102:0/3723939850 learned_addr learned my addr 172.21.15.102:0/3723939850 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.158+0000 7fc9c9400640 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 0x55a91b2b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.158+0000 7fc9c9e00640 1 -- 172.21.15.102:0/3723939850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 msgr2=0x55a91b2c1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.158+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.158+0000 7fc9c9e00640 1 -- 172.21.15.102:0/3723939850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 msgr2=0x55a91b2b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.159+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 0x55a91b2b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.159+0000 7fc9c9e00640 1 -- 172.21.15.102:0/3723939850 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a91b2ba1c0 con 0x55a91b2b8e00 2024-04-25T02:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.159+0000 7fc9ca800640 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.159+0000 7fc9c9400640 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 0x55a91b2b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.160+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2cfb00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4006010 tx=0x7fc9c400e8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fff784e1268fa661 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.160+0000 7fc9b6a00640 1 -- 172.21.15.102:0/3723939850 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4036070 con 0x55a91b2b8e00 2024-04-25T02:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.161+0000 7fc9b6a00640 1 -- 172.21.15.102:0/3723939850 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c4020470 con 0x55a91b2b8e00 2024-04-25T02:04:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.161+0000 7fc9b6a00640 1 -- 172.21.15.102:0/3723939850 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4024040 con 0x55a91b2b8e00 2024-04-25T02:04:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.162+0000 7fc9d3da6740 1 -- 172.21.15.102:0/3723939850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 msgr2=0x55a91b2cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.162+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2cfb00 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4006010 tx=0x7fc9c400e8e0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.163+0000 7fc9d3da6740 1 -- 172.21.15.102:0/3723939850 shutdown_connections 2024-04-25T02:04:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.163+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.163+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2b9720 0x55a91b2b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.163+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/3723939850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b8e00 0x55a91b2cfb00 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.163+0000 7fc9d3da6740 1 -- 172.21.15.102:0/3723939850 >> 172.21.15.102:0/3723939850 conn(0x55a91b2b5310 msgr2=0x55a91b2b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.163+0000 7fc9d3da6740 1 -- 172.21.15.102:0/3723939850 shutdown_connections 2024-04-25T02:04:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.163+0000 7fc9d3da6740 1 -- 172.21.15.102:0/3723939850 wait complete. 2024-04-25T02:04:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.164+0000 7fc9d3da6740 1 -- start start 2024-04-25T02:04:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.164+0000 7fc9d3da6740 1 Processor -- start 2024-04-25T02:04:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b8e00 0x55a91b2cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b9720 0x55a91b2cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9d3da6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac00ceb0 con 0x55a91b2c3fe0 2024-04-25T02:04:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac00e320 con 0x55a91b2b8e00 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9d3da6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc9ac00d580 con 0x55a91b2b9720 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9d3da6740 10 client.0 _ll_register_callbacks cb 0x7fc9c4009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9c9e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b8e00 0x55a91b2cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9c9e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b8e00 0x55a91b2cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:47948/0 (socket says 172.21.15.102:47948) 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9c9e00640 1 -- 172.21.15.102:0/1847037531 learned_addr learned my addr 172.21.15.102:0/1847037531 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.165+0000 7fc9c9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b9720 0x55a91b2cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.166+0000 7fc9ca800640 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.166+0000 7fc9c9e00640 1 -- 172.21.15.102:0/1847037531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b9720 msgr2=0x55a91b2cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.166+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b9720 0x55a91b2cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.166+0000 7fc9c9e00640 1 -- 172.21.15.102:0/1847037531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2c3fe0 msgr2=0x55a91b2c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:04:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.166+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.166+0000 7fc9c9e00640 1 -- 172.21.15.102:0/1847037531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a91b2c7c90 con 0x55a91b2b8e00 2024-04-25T02:04:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.167+0000 7fc9c9e00640 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b8e00 0x55a91b2cbf90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4003c60 tx=0x7fc9c4003c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55a94e4ae6026125 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:04:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.167+0000 7fc9abe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:04:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.168+0000 7fc9abe00640 1 -- 172.21.15.102:0/1847037531 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c403f060 con 0x55a91b2b8e00 2024-04-25T02:04:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.168+0000 7fc9abe00640 1 -- 172.21.15.102:0/1847037531 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7fc9c400c8a0 con 0x55a91b2b8e00 2024-04-25T02:04:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.168+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1847037531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9c4020470 con 0x55a91b2b8e00 2024-04-25T02:04:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.168+0000 7fc9abe00640 1 -- 172.21.15.102:0/1847037531 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc9c4036040 con 0x55a91b2b8e00 2024-04-25T02:04:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.168+0000 7fc9d3da6740 10 client.27332 fetch_fsmap learned FSMap version 40 2024-04-25T02:04:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.168+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1847037531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a91b34af00 con 0x55a91b2b8e00 2024-04-25T02:04:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.169+0000 7fc9abe00640 1 -- 172.21.15.102:0/1847037531 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c4020470 con 0x55a91b2b8e00 2024-04-25T02:04:57.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.169+0000 7fc9abe00640 1 -- 172.21.15.102:0/1847037531 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7fc9c4039070 con 0x55a91b2b8e00 2024-04-25T02:04:57.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.169+0000 7fc9d3da6740 10 client.27332 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:04:57.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.169+0000 7fc9d3da6740 10 client.27332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:04:57.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.170+0000 7fc9d3da6740 20 client.27332 populate_metadata read hostname 'smithi102' 2024-04-25T02:04:57.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.170+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a91b34b530 0x55a91b34d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:04:57.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.170+0000 7fc9d3da6740 4 client.27332 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:04:57.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.170+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1847037531 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a91b224340 con 0x55a91b34b530 2024-04-25T02:04:57.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.170+0000 7fc9c9400640 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a91b34b530 0x55a91b34d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:04:57.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.171+0000 7fc9c9400640 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a91b34b530 0x55a91b34d950 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=248a0bbeb3ea8958 server_cookie=e64cd5eae3b093a6 in_seq=0 out_seq=0 2024-04-25T02:04:57.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.171+0000 7fc9abe00640 10 client.27332 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:57.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:56 smithi102 ceph-mon[27796]: pgmap v299: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-04-25T02:04:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.188+0000 7fc9abe00640 1 -- 172.21.15.102:0/1847037531 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a91b224340 con 0x55a91b34b530 2024-04-25T02:04:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.188+0000 7fc9abe00640 10 client.27332 handle_command_reply: tid=0 2024-04-25T02:04:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9d3da6740 1 client.27332 shutdown 2024-04-25T02:04:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1847037531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b8e00 msgr2=0x55a91b2cbf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b8e00 0x55a91b2cbf90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4003c60 tx=0x7fc9c4003c90 comp rx=0 tx=0).stop 2024-04-25T02:04:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1847037531 shutdown_connections 2024-04-25T02:04:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a91b34b530 0x55a91b34d950 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a91b2c3fe0 0x55a91b2c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a91b2b9720 0x55a91b2cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9d3da6740 1 --2- 172.21.15.102:0/1847037531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a91b2b8e00 0x55a91b2cbf90 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1847037531 >> 172.21.15.102:0/1847037531 conn(0x55a91b2b5310 msgr2=0x55a91b2b4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.189+0000 7fc9abe00640 0 client.27332 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:04:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.190+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1847037531 shutdown_connections 2024-04-25T02:04:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.190+0000 7fc9d3da6740 1 -- 172.21.15.102:0/1847037531 wait complete. 2024-04-25T02:04:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.190+0000 7fc9d3da6740 20 client.27332 trim_cache size 0 max 16384 2024-04-25T02:04:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:04:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:04:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:04:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:04:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 -- 172.21.15.102:0/2427521539 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc9b00ab700 msgr2=0x7fc9b00adbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc9b00ab700 0x7fc9b00adbc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55a91b2ba3a0 tx=0x7fc9b8005da0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 -- 172.21.15.102:0/2427521539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9cc107760 msgr2=0x7fc9cc19e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:04:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9cc107760 0x7fc9cc19e900 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc030460 tx=0x7fc9bc030490 comp rx=0 tx=0).stop 2024-04-25T02:04:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 -- 172.21.15.102:0/2427521539 shutdown_connections 2024-04-25T02:04:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc9b00ab700 0x7fc9b00adbc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9cc107760 0x7fc9cc19e900 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9cc102110 0x7fc9cc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 --2- 172.21.15.102:0/2427521539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9cc101750 0x7fc9cc19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:04:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 -- 172.21.15.102:0/2427521539 >> 172.21.15.102:0/2427521539 conn(0x7fc9cc078bd0 msgr2=0x7fc9cc0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:04:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 -- 172.21.15.102:0/2427521539 shutdown_connections 2024-04-25T02:04:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:04:57.191+0000 7fc9d0e00640 1 -- 172.21.15.102:0/2427521539 wait complete. 2024-04-25T02:04:57.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:56 smithi110 ceph-mon[32848]: pgmap v299: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-04-25T02:04:57.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:56 smithi193 ceph-mon[32981]: pgmap v299: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 5.5 MiB/s wr, 527 op/s 2024-04-25T02:04:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 545.0 s, 2820.6 tps, lat 35.566 ms stddev 43.418 2024-04-25T02:04:59.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:04:58 smithi102 ceph-mon[27796]: pgmap v300: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 9.1 MiB/s wr, 853 op/s 2024-04-25T02:04:59.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:04:58 smithi110 ceph-mon[32848]: pgmap v300: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 9.1 MiB/s wr, 853 op/s 2024-04-25T02:04:59.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:04:58 smithi193 ceph-mon[32981]: pgmap v300: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 9.1 MiB/s wr, 853 op/s 2024-04-25T02:05:01.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:00 smithi110 ceph-mon[32848]: pgmap v301: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 6.3 MiB/s wr, 584 op/s 2024-04-25T02:05:01.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:00 smithi193 ceph-mon[32981]: pgmap v301: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 6.3 MiB/s wr, 584 op/s 2024-04-25T02:05:01.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:00 smithi102 ceph-mon[27796]: pgmap v301: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 6.3 MiB/s wr, 584 op/s 2024-04-25T02:05:03.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 550.0 s, 1060.2 tps, lat 86.478 ms stddev 207.537 2024-04-25T02:05:03.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:02 smithi110 ceph-mon[32848]: pgmap v302: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 6.7 MiB/s wr, 644 op/s 2024-04-25T02:05:03.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:02 smithi193 ceph-mon[32981]: pgmap v302: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 6.7 MiB/s wr, 644 op/s 2024-04-25T02:05:03.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:02 smithi102 ceph-mon[27796]: pgmap v302: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 6.7 MiB/s wr, 644 op/s 2024-04-25T02:05:05.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:04 smithi110 ceph-mon[32848]: pgmap v303: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 7.0 MiB/s wr, 684 op/s 2024-04-25T02:05:05.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:04 smithi193 ceph-mon[32981]: pgmap v303: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 7.0 MiB/s wr, 684 op/s 2024-04-25T02:05:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:04 smithi102 ceph-mon[27796]: pgmap v303: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 7.0 MiB/s wr, 684 op/s 2024-04-25T02:05:07.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:06 smithi110 ceph-mon[32848]: pgmap v304: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 5.6 MiB/s wr, 548 op/s 2024-04-25T02:05:07.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:06 smithi193 ceph-mon[32981]: pgmap v304: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 5.6 MiB/s wr, 548 op/s 2024-04-25T02:05:07.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:06 smithi102 ceph-mon[27796]: pgmap v304: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 5.6 MiB/s wr, 548 op/s 2024-04-25T02:05:08.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 555.0 s, 111.8 tps, lat 848.279 ms stddev 518.966 2024-04-25T02:05:09.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:08 smithi110 ceph-mon[32848]: pgmap v305: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 6.4 MiB/s wr, 658 op/s 2024-04-25T02:05:09.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:08 smithi193 ceph-mon[32981]: pgmap v305: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 6.4 MiB/s wr, 658 op/s 2024-04-25T02:05:09.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:08 smithi102 ceph-mon[27796]: pgmap v305: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 6.4 MiB/s wr, 658 op/s 2024-04-25T02:05:10.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.405+0000 7f18c1000640 1 -- 172.21.15.102:0/820316364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 msgr2=0x7f18bc1102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:10.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.405+0000 7f18c1000640 1 --2- 172.21.15.102:0/820316364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 0x7f18bc1102d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009be0 tx=0x7f18b402a6c0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.405+0000 7f18c1000640 1 -- 172.21.15.102:0/820316364 shutdown_connections 2024-04-25T02:05:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.405+0000 7f18c1000640 1 --2- 172.21.15.102:0/820316364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 0x7f18bc1102d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.405+0000 7f18c1000640 1 --2- 172.21.15.102:0/820316364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18bc100ce0 0x7f18bc10d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.405+0000 7f18c1000640 1 --2- 172.21.15.102:0/820316364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18bc100330 0x7f18bc100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.405+0000 7f18c1000640 1 -- 172.21.15.102:0/820316364 >> 172.21.15.102:0/820316364 conn(0x7f18bc0fbf60 msgr2=0x7f18bc0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.406+0000 7f18c1000640 1 -- 172.21.15.102:0/820316364 shutdown_connections 2024-04-25T02:05:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.406+0000 7f18c1000640 1 -- 172.21.15.102:0/820316364 wait complete. 2024-04-25T02:05:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.407+0000 7f18c1000640 1 Processor -- start 2024-04-25T02:05:10.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.408+0000 7f18c1000640 1 -- start start 2024-04-25T02:05:10.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.409+0000 7f18c1000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18bc100330 0x7f18bc10d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18c1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18bc100ce0 0x7f18bc1049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18c1000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 0x7f18bc104f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18c1000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc104210 con 0x7f18bc100ce0 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18c1000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc104090 con 0x7f18bc10dee0 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18c1000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc104390 con 0x7f18bc100330 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.409+0000 7f18bb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18bc100330 0x7f18bc10d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18bbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 0x7f18bc104f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18bbe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 0x7f18bc104f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:49734/0 (socket says 172.21.15.102:49734) 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18bbe00640 1 -- 172.21.15.102:0/2981650572 learned_addr learned my addr 172.21.15.102:0/2981650572 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18baa00640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18bc100ce0 0x7f18bc1049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.411+0000 7f18bbe00640 1 -- 172.21.15.102:0/2981650572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18bc100330 msgr2=0x7f18bc10d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.412+0000 7f18bbe00640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18bc100330 0x7f18bc10d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.412+0000 7f18bbe00640 1 -- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18bc100ce0 msgr2=0x7f18bc1049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.412+0000 7f18bbe00640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18bc100ce0 0x7f18bc1049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.412+0000 7f18bbe00640 1 -- 172.21.15.102:0/2981650572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18bc1057d0 con 0x7f18bc10dee0 2024-04-25T02:05:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.412+0000 7f18baa00640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18bc100ce0 0x7f18bc1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:05:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.412+0000 7f18bb400640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18bc100330 0x7f18bc10d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:05:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.412+0000 7f18bbe00640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 0x7f18bc104f10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f18b402abe0 tx=0x7f18b4030710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.413+0000 7f18b3e00640 1 -- 172.21.15.102:0/2981650572 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4041020 con 0x7f18bc10dee0 2024-04-25T02:05:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.413+0000 7f18b3e00640 1 -- 172.21.15.102:0/2981650572 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b4031920 con 0x7f18bc10dee0 2024-04-25T02:05:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.413+0000 7f18b3e00640 1 -- 172.21.15.102:0/2981650572 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4036040 con 0x7f18bc10dee0 2024-04-25T02:05:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.414+0000 7f18c1000640 1 -- 172.21.15.102:0/2981650572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18bc109110 con 0x7f18bc10dee0 2024-04-25T02:05:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.414+0000 7f18c1000640 1 -- 172.21.15.102:0/2981650572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18bc1095e0 con 0x7f18bc10dee0 2024-04-25T02:05:10.426 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.420+0000 7f18c556c740 1 Processor -- start 2024-04-25T02:05:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.421+0000 7f18b3e00640 1 -- 172.21.15.102:0/2981650572 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f18b4044020 con 0x7f18bc10dee0 2024-04-25T02:05:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.421+0000 7f18c556c740 1 -- start start 2024-04-25T02:05:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.422+0000 7f18c556c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20c9fb0 0x5587f20ca390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18c556c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20ca8d0 0x5587f20d4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18c556c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 0x5587f20d7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18c556c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5587f202dab0 con 0x5587f20ca8d0 2024-04-25T02:05:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18c556c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5587f1fd6640 con 0x5587f20d5190 2024-04-25T02:05:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18c556c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5587f2029a70 con 0x5587f20c9fb0 2024-04-25T02:05:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18baa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20c9fb0 0x5587f20ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18baa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20c9fb0 0x5587f20ca390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52322/0 (socket says 172.21.15.102:52322) 2024-04-25T02:05:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18baa00640 1 -- 172.21.15.102:0/691611257 learned_addr learned my addr 172.21.15.102:0/691611257 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18baa00640 1 --2- 172.21.15.102:0/691611257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 0x5587f20d7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18baa00640 1 -- 172.21.15.102:0/691611257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 msgr2=0x5587f20d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18baa00640 1 --2- 172.21.15.102:0/691611257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 0x5587f20d7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18baa00640 1 -- 172.21.15.102:0/691611257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20ca8d0 msgr2=0x5587f20d4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18baa00640 1 --2- 172.21.15.102:0/691611257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20ca8d0 0x5587f20d4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.423+0000 7f18baa00640 1 -- 172.21.15.102:0/691611257 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587f1d42210 con 0x5587f20c9fb0 2024-04-25T02:05:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.424+0000 7f18baa00640 1 --2- 172.21.15.102:0/691611257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20c9fb0 0x5587f20ca390 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009fd0 tx=0x7f18a8009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcdf297e4ac18f96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.424+0000 7f18b0c00640 1 -- 172.21.15.102:0/691611257 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a801b070 con 0x5587f20c9fb0 2024-04-25T02:05:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.425+0000 7f18b0c00640 1 -- 172.21.15.102:0/691611257 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a80139a0 con 0x5587f20c9fb0 2024-04-25T02:05:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.425+0000 7f18b0c00640 1 -- 172.21.15.102:0/691611257 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a8010880 con 0x5587f20c9fb0 2024-04-25T02:05:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.426+0000 7f18b3e00640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f18a00a2d70 0x7f18a00a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18b3e00640 1 -- 172.21.15.102:0/2981650572 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f18b40eff00 con 0x7f18bc10dee0 2024-04-25T02:05:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18c556c740 1 -- 172.21.15.102:0/691611257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20c9fb0 msgr2=0x5587f20ca390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18bb400640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f18a00a2d70 0x7f18a00a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18c556c740 1 --2- 172.21.15.102:0/691611257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20c9fb0 0x5587f20ca390 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009fd0 tx=0x7f18a8009610 comp rx=0 tx=0).stop 2024-04-25T02:05:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18c556c740 1 -- 172.21.15.102:0/691611257 shutdown_connections 2024-04-25T02:05:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18c556c740 1 --2- 172.21.15.102:0/691611257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 0x5587f20d7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18c556c740 1 --2- 172.21.15.102:0/691611257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20ca8d0 0x5587f20d4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18c556c740 1 --2- 172.21.15.102:0/691611257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20c9fb0 0x5587f20ca390 secure :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009fd0 tx=0x7f18a8009610 comp rx=0 tx=0).stop 2024-04-25T02:05:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18c556c740 1 -- 172.21.15.102:0/691611257 >> 172.21.15.102:0/691611257 conn(0x5587f20c64c0 msgr2=0x5587f20c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.427+0000 7f18c556c740 1 -- 172.21.15.102:0/691611257 shutdown_connections 2024-04-25T02:05:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.428+0000 7f18c556c740 1 -- 172.21.15.102:0/691611257 wait complete. 2024-04-25T02:05:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.428+0000 7f18bb400640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f18a00a2d70 0x7f18a00a5230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x5587f20cb550 tx=0x7f18ac008040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.428+0000 7f18c556c740 1 -- start start 2024-04-25T02:05:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.428+0000 7f18c556c740 1 Processor -- start 2024-04-25T02:05:10.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18c556c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18c556c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20d5190 0x5587f20d2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18c556c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20e1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18c556c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5587f20d7dc0 con 0x5587f20d5190 2024-04-25T02:05:10.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18c556c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5587f1fe9990 con 0x5587f20d2dd0 2024-04-25T02:05:10.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18c556c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5587f20d7f40 con 0x5587f20ca8d0 2024-04-25T02:05:10.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18c556c740 10 client.0 _ll_register_callbacks cb 0x5587f1feb950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:10.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18bb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18bb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52330/0 (socket says 172.21.15.102:52330) 2024-04-25T02:05:10.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.429+0000 7f18bb400640 1 -- 172.21.15.102:0/216299756 learned_addr learned my addr 172.21.15.102:0/216299756 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.430+0000 7f18bbe00640 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20e1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.430+0000 7f18bb400640 1 -- 172.21.15.102:0/216299756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 msgr2=0x5587f20e1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.430+0000 7f18bb400640 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20e1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.430+0000 7f18bb400640 1 -- 172.21.15.102:0/216299756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20d5190 msgr2=0x5587f20d2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.430+0000 7f18bb400640 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20d5190 0x5587f20d2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.430+0000 7f18bb400640 1 -- 172.21.15.102:0/216299756 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587f1e0e620 con 0x5587f20ca8d0 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.430+0000 7f18bbe00640 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.430+0000 7f18bb400640 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d2350 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00cde0 tx=0x7f18ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39f3c798ff9bbbf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.430+0000 7f18a5600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:05:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.431+0000 7f18a5600640 1 -- 172.21.15.102:0/216299756 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac023060 con 0x5587f20ca8d0 2024-04-25T02:05:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.431+0000 7f18a5600640 1 -- 172.21.15.102:0/216299756 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f18ac01e600 con 0x5587f20ca8d0 2024-04-25T02:05:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.431+0000 7f18c556c740 1 -- 172.21.15.102:0/216299756 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18a80139a0 con 0x5587f20ca8d0 2024-04-25T02:05:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.431+0000 7f18a5600640 1 -- 172.21.15.102:0/216299756 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f18ac01e800 con 0x5587f20ca8d0 2024-04-25T02:05:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.432+0000 7f18c556c740 10 client.27301 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.432+0000 7f18c556c740 1 -- 172.21.15.102:0/216299756 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5587f20e4340 con 0x5587f20ca8d0 2024-04-25T02:05:10.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.432+0000 7f18a5600640 1 -- 172.21.15.102:0/216299756 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a80139a0 con 0x5587f20ca8d0 2024-04-25T02:05:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.433+0000 7f18a5600640 1 -- 172.21.15.102:0/216299756 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f18ac011b90 con 0x5587f20ca8d0 2024-04-25T02:05:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.434+0000 7f18c556c740 10 client.27301 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.434+0000 7f18c556c740 10 client.27301 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.434+0000 7f18c556c740 20 client.27301 populate_metadata read hostname 'smithi102' 2024-04-25T02:05:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.434+0000 7f18c556c740 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5587f20e5800 0x5587f20e7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.434+0000 7f18c556c740 4 client.27301 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:05:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.434+0000 7f18c556c740 1 -- 172.21.15.102:0/216299756 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587f203aff0 con 0x5587f20e5800 2024-04-25T02:05:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.435+0000 7f18baa00640 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5587f20e5800 0x5587f20e7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.435+0000 7f18baa00640 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5587f20e5800 0x5587f20e7cc0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f57d1c86fbc721f8 server_cookie=f372b1b5901b0ad1 in_seq=0 out_seq=0 2024-04-25T02:05:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.435+0000 7f18a5600640 10 client.27301 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:10.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.436+0000 7f18a5600640 1 -- 172.21.15.102:0/216299756 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587f203aff0 con 0x5587f20e5800 2024-04-25T02:05:10.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.436+0000 7f18a5600640 10 client.27301 handle_command_reply: tid=0 2024-04-25T02:05:10.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.436+0000 7f18c556c740 1 client.27301 shutdown 2024-04-25T02:05:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.439+0000 7f18c556c740 1 -- 172.21.15.102:0/216299756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 msgr2=0x5587f20d2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.439+0000 7f18c556c740 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d2350 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00cde0 tx=0x7f18ac0046c0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.440+0000 7f18c556c740 1 -- 172.21.15.102:0/216299756 shutdown_connections 2024-04-25T02:05:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.440+0000 7f18c556c740 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5587f20e5800 0x5587f20e7cc0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.440+0000 7f18c556c740 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.440+0000 7f18c556c740 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20d5190 0x5587f20d2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.440+0000 7f18c556c740 1 --2- 172.21.15.102:0/216299756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d2350 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.440+0000 7f18c556c740 1 -- 172.21.15.102:0/216299756 >> 172.21.15.102:0/216299756 conn(0x5587f20c64c0 msgr2=0x5587f20cc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.440+0000 7f18a5600640 0 client.27301 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.440+0000 7f18c556c740 1 -- 172.21.15.102:0/216299756 shutdown_connections 2024-04-25T02:05:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.441+0000 7f18c556c740 1 -- 172.21.15.102:0/216299756 wait complete. 2024-04-25T02:05:10.448 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.441+0000 7f18c556c740 20 client.27301 trim_cache size 0 max 16384 2024-04-25T02:05:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.473+0000 7f18c556c740 1 Processor -- start 2024-04-25T02:05:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.474+0000 7f18c556c740 1 -- start start 2024-04-25T02:05:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.474+0000 7f18c556c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.474+0000 7f18c556c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20d2dd0 0x5587f20ed590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.475+0000 7f18bb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.475+0000 7f18bb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52344/0 (socket says 172.21.15.102:52344) 2024-04-25T02:05:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.476+0000 7f18c556c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 0x5587f20edad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.476+0000 7f18c556c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc1bea20 con 0x5587f20d2dd0 2024-04-25T02:05:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.476+0000 7f18c556c740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc1bc230 con 0x5587f20d5190 2024-04-25T02:05:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.476+0000 7f18c556c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc1ba0a0 con 0x5587f20ca8d0 2024-04-25T02:05:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.476+0000 7f18bb400640 1 -- 172.21.15.102:0/2196100901 learned_addr learned my addr 172.21.15.102:0/2196100901 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.477+0000 7f18bbe00640 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 0x5587f20edad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.478+0000 7f18baa00640 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20d2dd0 0x5587f20ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.478+0000 7f18bb400640 1 -- 172.21.15.102:0/2196100901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 msgr2=0x5587f20edad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.479+0000 7f18bb400640 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 0x5587f20edad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.479+0000 7f18bb400640 1 -- 172.21.15.102:0/2196100901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20d2dd0 msgr2=0x5587f20ed590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:10.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.479+0000 7f18bb400640 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20d2dd0 0x5587f20ed590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.479+0000 7f18bb400640 1 -- 172.21.15.102:0/2196100901 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587f20c19f0 con 0x5587f20ca8d0 2024-04-25T02:05:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.482+0000 7f18bbe00640 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 0x5587f20edad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:05:10.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.482+0000 7f18bb400640 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d1cf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f18ac008010 tx=0x7f18ac00c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0b9826d83618536 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.483+0000 7f18a7e00640 1 -- 172.21.15.102:0/2196100901 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac036070 con 0x5587f20ca8d0 2024-04-25T02:05:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.484+0000 7f18a7e00640 1 -- 172.21.15.102:0/2196100901 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18ac01e600 con 0x5587f20ca8d0 2024-04-25T02:05:10.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.484+0000 7f18a7e00640 1 -- 172.21.15.102:0/2196100901 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac02d080 con 0x5587f20ca8d0 2024-04-25T02:05:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.484+0000 7f18c556c740 1 -- 172.21.15.102:0/2196100901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 msgr2=0x5587f20d1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.484+0000 7f18c556c740 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d1cf0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f18ac008010 tx=0x7f18ac00c7b0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.485+0000 7f18c556c740 1 -- 172.21.15.102:0/2196100901 shutdown_connections 2024-04-25T02:05:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.485+0000 7f18c556c740 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d5190 0x5587f20edad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.485+0000 7f18c556c740 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20d2dd0 0x5587f20ed590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.485+0000 7f18c556c740 1 --2- 172.21.15.102:0/2196100901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20ca8d0 0x5587f20d1cf0 secure :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f18ac008010 tx=0x7f18ac00c7b0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.485+0000 7f18c556c740 1 -- 172.21.15.102:0/2196100901 >> 172.21.15.102:0/2196100901 conn(0x5587f20c64c0 msgr2=0x5587f20d3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.485+0000 7f18c556c740 1 -- 172.21.15.102:0/2196100901 shutdown_connections 2024-04-25T02:05:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.485+0000 7f18c556c740 1 -- 172.21.15.102:0/2196100901 wait complete. 2024-04-25T02:05:10.492 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.486+0000 7f18c556c740 1 -- start start 2024-04-25T02:05:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.486+0000 7f18c556c740 1 Processor -- start 2024-04-25T02:05:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.487+0000 7f18c556c740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.488+0000 7f18c556c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20d5190 0x5587f20cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.488+0000 7f18bb400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.488+0000 7f18bb400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20cef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:49778/0 (socket says 172.21.15.102:49778) 2024-04-25T02:05:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.488+0000 7f18bb400640 1 -- 172.21.15.102:0/3449055893 learned_addr learned my addr 172.21.15.102:0/3449055893 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.488+0000 7f18c556c740 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20cfa00 0x5587f214ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.489+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc1ba6f0 con 0x5587f20cfa00 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.489+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc1bbb60 con 0x5587f20d2dd0 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.489+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18bc1badc0 con 0x5587f20d5190 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.489+0000 7f18bbe00640 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20cfa00 0x5587f214ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.489+0000 7f18baa00640 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20d5190 0x5587f20cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.490+0000 7f18bb400640 1 -- 172.21.15.102:0/3449055893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20d5190 msgr2=0x5587f20cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.490+0000 7f18c556c740 10 client.0 _ll_register_callbacks cb 0x7f18ac00a4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.490+0000 7f18bb400640 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20d5190 0x5587f20cf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.490+0000 7f18bb400640 1 -- 172.21.15.102:0/3449055893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20cfa00 msgr2=0x5587f214ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.490+0000 7f18bb400640 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20cfa00 0x5587f214ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.490+0000 7f18bb400640 1 -- 172.21.15.102:0/3449055893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587f20dda60 con 0x5587f20d2dd0 2024-04-25T02:05:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.490+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18ac01e600 con 0x5587f20d2dd0 2024-04-25T02:05:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.490+0000 7f18bb400640 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20cef80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00c520 tx=0x7f18ac01ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16b5082933abffd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.490+0000 7f18a5600640 10 client.27344 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:05:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.491+0000 7f18a5600640 1 -- 172.21.15.102:0/3449055893 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac023030 con 0x5587f20d2dd0 2024-04-25T02:05:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.491+0000 7f18a5600640 1 -- 172.21.15.102:0/3449055893 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f18ac01e400 con 0x5587f20d2dd0 2024-04-25T02:05:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.491+0000 7f18a5600640 1 -- 172.21.15.102:0/3449055893 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f18ac036030 con 0x5587f20d2dd0 2024-04-25T02:05:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.491+0000 7f18c556c740 10 client.27344 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.491+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=40}) -- 0x5587f214c250 con 0x5587f20d2dd0 2024-04-25T02:05:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.492+0000 7f18a5600640 1 -- 172.21.15.102:0/3449055893 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f18ac031560 con 0x5587f20d2dd0 2024-04-25T02:05:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.492+0000 7f18c556c740 10 client.27344 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.492+0000 7f18c556c740 10 client.27344 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.492+0000 7f18c556c740 20 client.27344 populate_metadata read hostname 'smithi102' 2024-04-25T02:05:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.493+0000 7f18c556c740 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5587f214c770 0x5587f214eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.493+0000 7f18c556c740 4 client.27344 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:05:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.493+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5587f2035340 con 0x5587f214c770 2024-04-25T02:05:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.493+0000 7f18baa00640 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5587f214c770 0x5587f214eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.494+0000 7f18baa00640 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5587f214c770 0x5587f214eb90 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=649f3fd893b5ed8a server_cookie=5042a786ceffdaf3 in_seq=0 out_seq=0 2024-04-25T02:05:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.495+0000 7f18a5600640 10 client.27344 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18a5600640 1 -- 172.21.15.102:0/3449055893 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5587f2035340 con 0x5587f214c770 2024-04-25T02:05:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18a5600640 10 client.27344 handle_command_reply: tid=0 2024-04-25T02:05:10.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 client.27344 shutdown 2024-04-25T02:05:10.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 msgr2=0x5587f20cef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:10.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20cef80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00c520 tx=0x7f18ac01ebd0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 shutdown_connections 2024-04-25T02:05:10.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5587f214c770 0x5587f214eb90 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f20cfa00 0x5587f214ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587f20d5190 0x5587f20cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 --2- 172.21.15.102:0/3449055893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5587f20d2dd0 0x5587f20cef80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18a5600640 0 client.27344 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:10.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 >> 172.21.15.102:0/3449055893 conn(0x5587f20c64c0 msgr2=0x5587f20d4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:10.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 shutdown_connections 2024-04-25T02:05:10.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 1 -- 172.21.15.102:0/3449055893 wait complete. 2024-04-25T02:05:10.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.519+0000 7f18c556c740 20 client.27344 trim_cache size 0 max 16384 2024-04-25T02:05:10.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:05:10.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:05:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:05:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:05:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 -- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f18a00a2d70 msgr2=0x7f18a00a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f18a00a2d70 0x7f18a00a5230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x5587f20cb550 tx=0x7f18ac008040 comp rx=0 tx=0).stop 2024-04-25T02:05:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 -- 172.21.15.102:0/2981650572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 msgr2=0x7f18bc104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 0x7f18bc104f10 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f18b402abe0 tx=0x7f18b4030710 comp rx=0 tx=0).stop 2024-04-25T02:05:10.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 -- 172.21.15.102:0/2981650572 shutdown_connections 2024-04-25T02:05:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f18a00a2d70 0x7f18a00a5230 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18bc10dee0 0x7f18bc104f10 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18bc100ce0 0x7f18bc1049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 --2- 172.21.15.102:0/2981650572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18bc100330 0x7f18bc10d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 -- 172.21.15.102:0/2981650572 >> 172.21.15.102:0/2981650572 conn(0x7f18bc0fbf60 msgr2=0x7f18bc10f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.520+0000 7f18c1000640 1 -- 172.21.15.102:0/2981650572 shutdown_connections 2024-04-25T02:05:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:10.521+0000 7f18c1000640 1 -- 172.21.15.102:0/2981650572 wait complete. 2024-04-25T02:05:11.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:10 smithi110 ceph-mon[32848]: pgmap v306: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-04-25T02:05:11.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:10 smithi193 ceph-mon[32981]: pgmap v306: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-04-25T02:05:11.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:10 smithi102 ceph-mon[27796]: pgmap v306: 97 pgs: 97 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-04-25T02:05:13.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 560.0 s, 1040.6 tps, lat 107.892 ms stddev 275.550 2024-04-25T02:05:13.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:12 smithi110 ceph-mon[32848]: pgmap v307: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 12 MiB/s wr, 745 op/s 2024-04-25T02:05:13.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:12 smithi193 ceph-mon[32981]: pgmap v307: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 12 MiB/s wr, 745 op/s 2024-04-25T02:05:13.431 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:12 smithi102 ceph-mon[27796]: pgmap v307: 97 pgs: 97 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 12 MiB/s wr, 745 op/s 2024-04-25T02:05:15.242 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:14 smithi110 ceph-mon[32848]: pgmap v308: 97 pgs: 97 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 29 MiB/s wr, 1.48k op/s 2024-04-25T02:05:15.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:14 smithi193 ceph-mon[32981]: pgmap v308: 97 pgs: 97 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 29 MiB/s wr, 1.48k op/s 2024-04-25T02:05:15.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:14 smithi102 ceph-mon[27796]: pgmap v308: 97 pgs: 97 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 29 MiB/s wr, 1.48k op/s 2024-04-25T02:05:17.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:16 smithi110 ceph-mon[32848]: pgmap v309: 97 pgs: 97 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 27 MiB/s wr, 1.31k op/s 2024-04-25T02:05:17.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:16 smithi193 ceph-mon[32981]: pgmap v309: 97 pgs: 97 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 27 MiB/s wr, 1.31k op/s 2024-04-25T02:05:17.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:16 smithi102 ceph-mon[27796]: pgmap v309: 97 pgs: 97 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 27 MiB/s wr, 1.31k op/s 2024-04-25T02:05:18.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 565.0 s, 1482.6 tps, lat 68.560 ms stddev 85.654 2024-04-25T02:05:19.252 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:18 smithi102 ceph-mon[27796]: pgmap v310: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 30 MiB/s wr, 1.57k op/s 2024-04-25T02:05:19.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:18 smithi193 ceph-mon[32981]: pgmap v310: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 30 MiB/s wr, 1.57k op/s 2024-04-25T02:05:19.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:18 smithi110 ceph-mon[32848]: pgmap v310: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 30 MiB/s wr, 1.57k op/s 2024-04-25T02:05:21.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:20 smithi193 ceph-mon[32981]: pgmap v311: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 29 MiB/s wr, 1.46k op/s 2024-04-25T02:05:21.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:20 smithi102 ceph-mon[27796]: pgmap v311: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 29 MiB/s wr, 1.46k op/s 2024-04-25T02:05:21.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:20 smithi110 ceph-mon[32848]: pgmap v311: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 29 MiB/s wr, 1.46k op/s 2024-04-25T02:05:23.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 570.0 s, 2523.8 tps, lat 39.314 ms stddev 46.963 2024-04-25T02:05:23.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:22 smithi193 ceph-mon[32981]: pgmap v312: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 31 MiB/s wr, 1.61k op/s 2024-04-25T02:05:23.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:22 smithi102 ceph-mon[27796]: pgmap v312: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 31 MiB/s wr, 1.61k op/s 2024-04-25T02:05:23.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:22 smithi110 ceph-mon[32848]: pgmap v312: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 31 MiB/s wr, 1.61k op/s 2024-04-25T02:05:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.707+0000 7f03ed400640 1 -- 172.21.15.102:0/986374393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e8108740 msgr2=0x7f03e8108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.707+0000 7f03ed400640 1 --2- 172.21.15.102:0/986374393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e8108740 0x7f03e8108b20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009930 tx=0x7f03d002a4d0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.708+0000 7f03ed400640 1 -- 172.21.15.102:0/986374393 shutdown_connections 2024-04-25T02:05:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.708+0000 7f03ed400640 1 --2- 172.21.15.102:0/986374393 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e8114890 0x7f03e8116c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.708+0000 7f03ed400640 1 --2- 172.21.15.102:0/986374393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03e8109060 0x7f03e8114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.708+0000 7f03ed400640 1 --2- 172.21.15.102:0/986374393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e8108740 0x7f03e8108b20 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.708+0000 7f03ed400640 1 -- 172.21.15.102:0/986374393 >> 172.21.15.102:0/986374393 conn(0x7f03e806c810 msgr2=0x7f03e806cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.709+0000 7f03ed400640 1 -- 172.21.15.102:0/986374393 shutdown_connections 2024-04-25T02:05:23.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.709+0000 7f03ed400640 1 -- 172.21.15.102:0/986374393 wait complete. 2024-04-25T02:05:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.710+0000 7f03ed400640 1 Processor -- start 2024-04-25T02:05:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.713+0000 7f03ed400640 1 -- start start 2024-04-25T02:05:23.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.713+0000 7f03ed400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03e8108740 0x7f03e81a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.713+0000 7f03ed400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e8109060 0x7f03e81a8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.713+0000 7f03ed400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e8114890 0x7f03e81a2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.713+0000 7f03ed400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03e811a3a0 con 0x7f03e8109060 2024-04-25T02:05:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.713+0000 7f03ed400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03e811a220 con 0x7f03e8114890 2024-04-25T02:05:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.713+0000 7f03ed400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03e811a520 con 0x7f03e8108740 2024-04-25T02:05:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.714+0000 7f03e6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03e8108740 0x7f03e81a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.714+0000 7f03e6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03e8108740 0x7f03e81a86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41878/0 (socket says 172.21.15.102:41878) 2024-04-25T02:05:23.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.714+0000 7f03e6800640 1 -- 172.21.15.102:0/536159499 learned_addr learned my addr 172.21.15.102:0/536159499 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:23.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.714+0000 7f03e5e00640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e8109060 0x7f03e81a8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.714+0000 7f03e7200640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e8114890 0x7f03e81a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.714+0000 7f03e7200640 1 -- 172.21.15.102:0/536159499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03e8108740 msgr2=0x7f03e81a86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:23.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.714+0000 7f03e7200640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03e8108740 0x7f03e81a86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.714+0000 7f03e7200640 1 -- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e8109060 msgr2=0x7f03e81a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:23.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.715+0000 7f03e7200640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e8109060 0x7f03e81a8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.715+0000 7f03e7200640 1 -- 172.21.15.102:0/536159499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e81a2fe0 con 0x7f03e8114890 2024-04-25T02:05:23.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.716+0000 7f03e6800640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03e8108740 0x7f03e81a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:23.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.716+0000 7f03e5e00640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e8109060 0x7f03e81a8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:23.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.716+0000 7f03e7200640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e8114890 0x7f03e81a2860 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f03e000be20 tx=0x7f03e000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:23.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.716+0000 7f03df400640 1 -- 172.21.15.102:0/536159499 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e0019070 con 0x7f03e8114890 2024-04-25T02:05:23.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.716+0000 7f03df400640 1 -- 172.21.15.102:0/536159499 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03e0014920 con 0x7f03e8114890 2024-04-25T02:05:23.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.716+0000 7f03df400640 1 -- 172.21.15.102:0/536159499 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e0012910 con 0x7f03e8114890 2024-04-25T02:05:23.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.716+0000 7f03ed400640 1 -- 172.21.15.102:0/536159499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e81a32d0 con 0x7f03e8114890 2024-04-25T02:05:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.717+0000 7f03ed400640 1 -- 172.21.15.102:0/536159499 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e81af570 con 0x7f03e8114890 2024-04-25T02:05:23.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.723+0000 7f03f01a7740 1 Processor -- start 2024-04-25T02:05:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.725+0000 7f03df400640 1 -- 172.21.15.102:0/536159499 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f03e001c070 con 0x7f03e8114890 2024-04-25T02:05:23.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.725+0000 7f03f01a7740 1 -- start start 2024-04-25T02:05:23.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.726+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a8559e00 0x55a5a855a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.726+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a855a720 0x55a5a8564aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.726+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8564fe0 0x55a5a85673d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.726+0000 7f03f01a7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a5a84bdab0 con 0x55a5a8559e00 2024-04-25T02:05:23.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.726+0000 7f03f01a7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a5a8466640 con 0x55a5a855a720 2024-04-25T02:05:23.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.726+0000 7f03f01a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a5a84b9a70 con 0x55a5a8564fe0 2024-04-25T02:05:23.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.727+0000 7f03e6800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a855a720 0x55a5a8564aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.727+0000 7f03e6800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a855a720 0x55a5a8564aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54480/0 (socket says 172.21.15.102:54480) 2024-04-25T02:05:23.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.727+0000 7f03e6800640 1 -- 172.21.15.102:0/824773891 learned_addr learned my addr 172.21.15.102:0/824773891 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:23.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.727+0000 7f03e6800640 1 -- 172.21.15.102:0/824773891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8564fe0 msgr2=0x55a5a85673d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:05:23.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.727+0000 7f03e6800640 1 --2- 172.21.15.102:0/824773891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8564fe0 0x55a5a85673d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.727+0000 7f03e6800640 1 -- 172.21.15.102:0/824773891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a8559e00 msgr2=0x55a5a855a1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:05:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.727+0000 7f03e6800640 1 --2- 172.21.15.102:0/824773891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a8559e00 0x55a5a855a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.727+0000 7f03e6800640 1 -- 172.21.15.102:0/824773891 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a81d2210 con 0x55a5a855a720 2024-04-25T02:05:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.728+0000 7f03df400640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03cc0a2f00 0x7f03cc0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.728+0000 7f03e6800640 1 --2- 172.21.15.102:0/824773891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a855a720 0x55a5a8564aa0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f03d002a9f0 tx=0x7f03d0004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d83ec2559f999d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.728+0000 7f03df400640 1 -- 172.21.15.102:0/536159499 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f03e00ceb90 con 0x7f03e8114890 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.728+0000 7f03d7400640 1 -- 172.21.15.102:0/824773891 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0040070 con 0x55a5a855a720 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.728+0000 7f03d7400640 1 -- 172.21.15.102:0/824773891 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d0003830 con 0x55a5a855a720 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.728+0000 7f03d7400640 1 -- 172.21.15.102:0/824773891 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0035040 con 0x55a5a855a720 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.728+0000 7f03e5e00640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03cc0a2f00 0x7f03cc0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.729+0000 7f03f01a7740 1 -- 172.21.15.102:0/824773891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a855a720 msgr2=0x55a5a8564aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.729+0000 7f03f01a7740 1 --2- 172.21.15.102:0/824773891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a855a720 0x55a5a8564aa0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f03d002a9f0 tx=0x7f03d0004610 comp rx=0 tx=0).stop 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.729+0000 7f03f01a7740 1 -- 172.21.15.102:0/824773891 shutdown_connections 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.729+0000 7f03f01a7740 1 --2- 172.21.15.102:0/824773891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8564fe0 0x55a5a85673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.729+0000 7f03f01a7740 1 --2- 172.21.15.102:0/824773891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a855a720 0x55a5a8564aa0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.729+0000 7f03f01a7740 1 --2- 172.21.15.102:0/824773891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a8559e00 0x55a5a855a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.729+0000 7f03f01a7740 1 -- 172.21.15.102:0/824773891 >> 172.21.15.102:0/824773891 conn(0x55a5a8556310 msgr2=0x55a5a85566f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.730+0000 7f03f01a7740 1 -- 172.21.15.102:0/824773891 shutdown_connections 2024-04-25T02:05:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.730+0000 7f03e5e00640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03cc0a2f00 0x7f03cc0a53c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f03d800a520 tx=0x7f03d800a4b0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.730+0000 7f03f01a7740 1 -- 172.21.15.102:0/824773891 wait complete. 2024-04-25T02:05:23.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.731+0000 7f03f01a7740 1 -- start start 2024-04-25T02:05:23.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.734+0000 7f03f01a7740 1 Processor -- start 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.735+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8561f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.735+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a85624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.735+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.735+0000 7f03f01a7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55a5a8567bb0 con 0x55a5a855a720 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.735+0000 7f03f01a7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55a5a8479990 con 0x55a5a8564fe0 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.735+0000 7f03f01a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a5a8567d30 con 0x55a5a8559e00 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.735+0000 7f03f01a7740 10 client.0 _ll_register_callbacks cb 0x55a5a847b950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.736+0000 7f03e6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8561f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.736+0000 7f03e6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8561f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41892/0 (socket says 172.21.15.102:41892) 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.736+0000 7f03e6800640 1 -- 172.21.15.102:0/2343018469 learned_addr learned my addr 172.21.15.102:0/2343018469 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.736+0000 7f03e7200640 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.736+0000 7f03e5e00640 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a85624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.736+0000 7f03e6800640 1 -- 172.21.15.102:0/2343018469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 msgr2=0x55a5a8562a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.736+0000 7f03e6800640 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.736+0000 7f03e6800640 1 -- 172.21.15.102:0/2343018469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 msgr2=0x55a5a85624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.737+0000 7f03e6800640 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a85624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.737+0000 7f03e6800640 1 -- 172.21.15.102:0/2343018469 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a829e620 con 0x55a5a8559e00 2024-04-25T02:05:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.737+0000 7f03e7200640 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.737+0000 7f03e6800640 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8561f80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009900 tx=0x7f03d00047a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cca1b239e515eb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.737+0000 7f03cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.737+0000 7f03cbe00640 1 -- 172.21.15.102:0/2343018469 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0035070 con 0x55a5a8559e00 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.737+0000 7f03cbe00640 1 -- 172.21.15.102:0/2343018469 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f03d00319c0 con 0x55a5a8559e00 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.737+0000 7f03f01a7740 1 -- 172.21.15.102:0/2343018469 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03d0003830 con 0x55a5a8559e00 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03cbe00640 1 -- 172.21.15.102:0/2343018469 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f03d0040040 con 0x55a5a8559e00 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03f01a7740 10 client.27313 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03f01a7740 1 -- 172.21.15.102:0/2343018469 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a5a8567bb0 con 0x55a5a8559e00 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03cbe00640 1 -- 172.21.15.102:0/2343018469 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d0003830 con 0x55a5a8559e00 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03cbe00640 1 -- 172.21.15.102:0/2343018469 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f03d0044410 con 0x55a5a8559e00 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03f01a7740 10 client.27313 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03f01a7740 10 client.27313 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03f01a7740 20 client.27313 populate_metadata read hostname 'smithi102' 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a5a857d4a0 0x55a5a857f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03f01a7740 4 client.27313 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:05:23.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.738+0000 7f03f01a7740 1 -- 172.21.15.102:0/2343018469 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5a84caff0 con 0x55a5a857d4a0 2024-04-25T02:05:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.739+0000 7f03e5e00640 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a5a857d4a0 0x55a5a857f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.740+0000 7f03e5e00640 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a5a857d4a0 0x55a5a857f890 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88e64fbf6d6463d3 server_cookie=70a43872185c62bd in_seq=0 out_seq=0 2024-04-25T02:05:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.740+0000 7f03cbe00640 10 client.27313 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.740+0000 7f03cbe00640 1 -- 172.21.15.102:0/2343018469 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5a84caff0 con 0x55a5a857d4a0 2024-04-25T02:05:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03cbe00640 10 client.27313 handle_command_reply: tid=0 2024-04-25T02:05:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 client.27313 shutdown 2024-04-25T02:05:23.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 -- 172.21.15.102:0/2343018469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 msgr2=0x55a5a8561f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8561f80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009900 tx=0x7f03d00047a0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 -- 172.21.15.102:0/2343018469 shutdown_connections 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a5a857d4a0 0x55a5a857f890 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a85624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2343018469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8561f80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 -- 172.21.15.102:0/2343018469 >> 172.21.15.102:0/2343018469 conn(0x55a5a8556310 msgr2=0x55a5a855c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03cbe00640 0 client.27313 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 -- 172.21.15.102:0/2343018469 shutdown_connections 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.741+0000 7f03f01a7740 1 -- 172.21.15.102:0/2343018469 wait complete. 2024-04-25T02:05:23.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.742+0000 7f03f01a7740 20 client.27313 trim_cache size 0 max 16384 2024-04-25T02:05:23.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.765+0000 7f03f01a7740 1 Processor -- start 2024-04-25T02:05:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.765+0000 7f03f01a7740 1 -- start start 2024-04-25T02:05:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.765+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a857f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.765+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a855c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.765+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.765+0000 7f03f01a7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81b62d0 con 0x55a5a855a720 2024-04-25T02:05:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.765+0000 7f03f01a7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81b3ae0 con 0x55a5a8564fe0 2024-04-25T02:05:23.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.765+0000 7f03f01a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81b1990 con 0x55a5a8559e00 2024-04-25T02:05:23.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.766+0000 7f03e6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a857f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.766+0000 7f03e5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a855c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.766+0000 7f03e5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a855c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40236/0 (socket says 172.21.15.102:40236) 2024-04-25T02:05:23.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.766+0000 7f03e6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a857f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41908/0 (socket says 172.21.15.102:41908) 2024-04-25T02:05:23.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.766+0000 7f03e5e00640 1 -- 172.21.15.102:0/4055353692 learned_addr learned my addr 172.21.15.102:0/4055353692 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:23.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.766+0000 7f03e7200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.767+0000 7f03e5e00640 1 -- 172.21.15.102:0/4055353692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 msgr2=0x55a5a857f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:23.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.767+0000 7f03e5e00640 1 --2- 172.21.15.102:0/4055353692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a857f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.767+0000 7f03e5e00640 1 -- 172.21.15.102:0/4055353692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 msgr2=0x55a5a8562820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:23.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.767+0000 7f03e5e00640 1 --2- 172.21.15.102:0/4055353692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.767+0000 7f03e5e00640 1 -- 172.21.15.102:0/4055353692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5a8554890 con 0x55a5a855a720 2024-04-25T02:05:23.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.767+0000 7f03e7200640 1 --2- 172.21.15.102:0/4055353692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:05:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.767+0000 7f03e5e00640 1 --2- 172.21.15.102:0/4055353692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a855c7c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f03d80058c0 tx=0x7f03d8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5afb2c54d4839ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.767+0000 7f03d6a00640 1 -- 172.21.15.102:0/4055353692 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d802e070 con 0x55a5a855a720 2024-04-25T02:05:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.768+0000 7f03d6a00640 1 -- 172.21.15.102:0/4055353692 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d8028710 con 0x55a5a855a720 2024-04-25T02:05:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.768+0000 7f03d6a00640 1 -- 172.21.15.102:0/4055353692 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d8016070 con 0x55a5a855a720 2024-04-25T02:05:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.768+0000 7f03f01a7740 1 -- 172.21.15.102:0/4055353692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 msgr2=0x55a5a855c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.768+0000 7f03f01a7740 1 --2- 172.21.15.102:0/4055353692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a855c7c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f03d80058c0 tx=0x7f03d8012000 comp rx=0 tx=0).stop 2024-04-25T02:05:23.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.768+0000 7f03f01a7740 1 -- 172.21.15.102:0/4055353692 shutdown_connections 2024-04-25T02:05:23.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.768+0000 7f03f01a7740 1 --2- 172.21.15.102:0/4055353692 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a8562820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.768+0000 7f03f01a7740 1 --2- 172.21.15.102:0/4055353692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a855c7c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.768+0000 7f03f01a7740 1 --2- 172.21.15.102:0/4055353692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a857f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.768+0000 7f03f01a7740 1 -- 172.21.15.102:0/4055353692 >> 172.21.15.102:0/4055353692 conn(0x55a5a8556310 msgr2=0x55a5a8555480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:23.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.769+0000 7f03f01a7740 1 -- 172.21.15.102:0/4055353692 shutdown_connections 2024-04-25T02:05:23.775 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.769+0000 7f03f01a7740 1 -- 172.21.15.102:0/4055353692 wait complete. 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.769+0000 7f03f01a7740 1 -- start start 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.769+0000 7f03f01a7740 1 Processor -- start 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8575af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a8570e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03f01a7740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a85713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03f01a7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81b1fe0 con 0x55a5a855a720 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03f01a7740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81b3410 con 0x55a5a8564fe0 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03f01a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03e81b2670 con 0x55a5a8559e00 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03f01a7740 10 client.0 _ll_register_callbacks cb 0x7f03d00095e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03e6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8575af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03e7200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a85713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03e6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8575af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41910/0 (socket says 172.21.15.102:41910) 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03e7200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a85713a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54510/0 (socket says 172.21.15.102:54510) 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03e6800640 1 -- 172.21.15.102:0/2128615162 learned_addr learned my addr 172.21.15.102:0/2128615162 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.770+0000 7f03e5e00640 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a8570e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03e7200640 1 -- 172.21.15.102:0/2128615162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 msgr2=0x55a5a8575af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03e7200640 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8575af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03e7200640 1 -- 172.21.15.102:0/2128615162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 msgr2=0x55a5a8570e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03e7200640 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a8570e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03e7200640 1 -- 172.21.15.102:0/2128615162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5a8571a90 con 0x55a5a8564fe0 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03e7200640 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a85713a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f03e00cc390 tx=0x7f03e0014e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce35d8bc297c3e5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03cbe00640 1 -- 172.21.15.102:0/2128615162 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e00b3070 con 0x55a5a8564fe0 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03cbe00640 1 -- 172.21.15.102:0/2128615162 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f03e009f160 con 0x55a5a8564fe0 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03f01a7740 1 -- 172.21.15.102:0/2128615162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03d8028710 con 0x55a5a8564fe0 2024-04-25T02:05:23.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03cbe00640 1 -- 172.21.15.102:0/2128615162 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f03e00a3dd0 con 0x55a5a8564fe0 2024-04-25T02:05:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03f01a7740 10 client.27362 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.771+0000 7f03f01a7740 1 -- 172.21.15.102:0/2128615162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a5a85e3840 con 0x55a5a8564fe0 2024-04-25T02:05:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.772+0000 7f03cbe00640 1 -- 172.21.15.102:0/2128615162 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03d8028710 con 0x55a5a8564fe0 2024-04-25T02:05:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.772+0000 7f03cbe00640 1 -- 172.21.15.102:0/2128615162 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f03e00a3a10 con 0x55a5a8564fe0 2024-04-25T02:05:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.772+0000 7f03f01a7740 10 client.27362 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.772+0000 7f03f01a7740 10 client.27362 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.772+0000 7f03f01a7740 20 client.27362 populate_metadata read hostname 'smithi102' 2024-04-25T02:05:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.772+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a5a85e3f70 0x55a5a85e6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.772+0000 7f03f01a7740 4 client.27362 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:05:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.772+0000 7f03f01a7740 1 -- 172.21.15.102:0/2128615162 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a5a84c5340 con 0x55a5a85e3f70 2024-04-25T02:05:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.773+0000 7f03e6800640 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a5a85e3f70 0x55a5a85e6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.773+0000 7f03cbe00640 10 client.27362 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:23.781 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.774+0000 7f03e6800640 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a5a85e3f70 0x55a5a85e6390 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73898a1bc76ffeda server_cookie=31866dfb9b24334c in_seq=0 out_seq=0 2024-04-25T02:05:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03cbe00640 1 -- 172.21.15.102:0/2128615162 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a5a84c5340 con 0x55a5a85e3f70 2024-04-25T02:05:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03cbe00640 10 client.27362 handle_command_reply: tid=0 2024-04-25T02:05:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03f01a7740 1 client.27362 shutdown 2024-04-25T02:05:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03f01a7740 1 -- 172.21.15.102:0/2128615162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 msgr2=0x55a5a85713a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a85713a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f03e00cc390 tx=0x7f03e0014e60 comp rx=0 tx=0).stop 2024-04-25T02:05:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03f01a7740 1 -- 172.21.15.102:0/2128615162 shutdown_connections 2024-04-25T02:05:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55a5a85e3f70 0x55a5a85e6390 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a5a8564fe0 0x55a5a85713a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a5a855a720 0x55a5a8570e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03f01a7740 1 --2- 172.21.15.102:0/2128615162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a5a8559e00 0x55a5a8575af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.790+0000 7f03f01a7740 1 -- 172.21.15.102:0/2128615162 >> 172.21.15.102:0/2128615162 conn(0x55a5a8556310 msgr2=0x55a5a85637b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.791+0000 7f03cbe00640 0 client.27362 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.791+0000 7f03f01a7740 1 -- 172.21.15.102:0/2128615162 shutdown_connections 2024-04-25T02:05:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.791+0000 7f03f01a7740 1 -- 172.21.15.102:0/2128615162 wait complete. 2024-04-25T02:05:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.791+0000 7f03f01a7740 20 client.27362 trim_cache size 0 max 16384 2024-04-25T02:05:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:05:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:05:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:05:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 -- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03cc0a2f00 msgr2=0x7f03cc0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03cc0a2f00 0x7f03cc0a53c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f03d800a520 tx=0x7f03d800a4b0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 -- 172.21.15.102:0/536159499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e8114890 msgr2=0x7f03e81a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e8114890 0x7f03e81a2860 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f03e000be20 tx=0x7f03e000bf10 comp rx=0 tx=0).stop 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 -- 172.21.15.102:0/536159499 shutdown_connections 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f03cc0a2f00 0x7f03cc0a53c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e8114890 0x7f03e81a2860 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03e8109060 0x7f03e81a8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 --2- 172.21.15.102:0/536159499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03e8108740 0x7f03e81a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 -- 172.21.15.102:0/536159499 >> 172.21.15.102:0/536159499 conn(0x7f03e806c810 msgr2=0x7f03e810a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 -- 172.21.15.102:0/536159499 shutdown_connections 2024-04-25T02:05:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:23.793+0000 7f03ed400640 1 -- 172.21.15.102:0/536159499 wait complete. 2024-04-25T02:05:25.260 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:25 smithi193 ceph-mon[32981]: pgmap v313: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 MiB/s wr, 1.33k op/s 2024-04-25T02:05:25.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:25 smithi102 ceph-mon[27796]: pgmap v313: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 MiB/s wr, 1.33k op/s 2024-04-25T02:05:25.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:25 smithi110 ceph-mon[32848]: pgmap v313: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 MiB/s wr, 1.33k op/s 2024-04-25T02:05:27.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:27 smithi102 ceph-mon[27796]: pgmap v314: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-04-25T02:05:27.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:27 smithi110 ceph-mon[32848]: pgmap v314: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-04-25T02:05:27.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:27 smithi193 ceph-mon[32981]: pgmap v314: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.2 MiB/s wr, 535 op/s 2024-04-25T02:05:28.074 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 575.0 s, 1541.2 tps, lat 64.793 ms stddev 76.999 2024-04-25T02:05:29.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:29 smithi102 ceph-mon[27796]: pgmap v315: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.1 MiB/s wr, 839 op/s 2024-04-25T02:05:29.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:29 smithi110 ceph-mon[32848]: pgmap v315: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.1 MiB/s wr, 839 op/s 2024-04-25T02:05:29.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:29 smithi193 ceph-mon[32981]: pgmap v315: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 8.1 MiB/s wr, 839 op/s 2024-04-25T02:05:31.179 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:31 smithi102 ceph-mon[27796]: pgmap v316: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.7 MiB/s wr, 584 op/s 2024-04-25T02:05:31.313 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:31 smithi193 ceph-mon[32981]: pgmap v316: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.7 MiB/s wr, 584 op/s 2024-04-25T02:05:31.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:31 smithi110 ceph-mon[32848]: pgmap v316: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.7 MiB/s wr, 584 op/s 2024-04-25T02:05:33.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 580.0 s, 1908.6 tps, lat 52.511 ms stddev 62.904 2024-04-25T02:05:33.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:33 smithi102 ceph-mon[27796]: pgmap v317: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.5 MiB/s wr, 676 op/s 2024-04-25T02:05:33.492 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:33 smithi110 ceph-mon[32848]: pgmap v317: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.5 MiB/s wr, 676 op/s 2024-04-25T02:05:33.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:33 smithi193 ceph-mon[32981]: pgmap v317: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.5 MiB/s wr, 676 op/s 2024-04-25T02:05:35.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:35 smithi102 ceph-mon[27796]: pgmap v318: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 697 op/s 2024-04-25T02:05:35.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:35 smithi110 ceph-mon[32848]: pgmap v318: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 697 op/s 2024-04-25T02:05:35.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:35 smithi193 ceph-mon[32981]: pgmap v318: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 6.4 MiB/s wr, 697 op/s 2024-04-25T02:05:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.971+0000 7f34d6200640 1 -- 172.21.15.102:0/1583831686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34d010c3d0 msgr2=0x7f34d010c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.971+0000 7f34d6200640 1 --2- 172.21.15.102:0/1583831686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34d010c3d0 0x7f34d010c7b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009900 tx=0x7f34c002a3f0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.971+0000 7f34d6200640 1 -- 172.21.15.102:0/1583831686 shutdown_connections 2024-04-25T02:05:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.971+0000 7f34d6200640 1 --2- 172.21.15.102:0/1583831686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d0111450 0x7f34d0119020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.971+0000 7f34d6200640 1 --2- 172.21.15.102:0/1583831686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34d010cd80 0x7f34d0110d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.971+0000 7f34d6200640 1 --2- 172.21.15.102:0/1583831686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34d010c3d0 0x7f34d010c7b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.971+0000 7f34d6200640 1 -- 172.21.15.102:0/1583831686 >> 172.21.15.102:0/1583831686 conn(0x7f34d006c9e0 msgr2=0x7f34d010a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.971+0000 7f34d6200640 1 -- 172.21.15.102:0/1583831686 shutdown_connections 2024-04-25T02:05:36.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.971+0000 7f34d6200640 1 -- 172.21.15.102:0/1583831686 wait complete. 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.973+0000 7f34d6200640 1 Processor -- start 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.973+0000 7f34d6200640 1 -- start start 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.973+0000 7f34d6200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d010c3d0 0x7f34d01a8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.973+0000 7f34d6200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34d0111450 0x7f34d01a8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.973+0000 7f34d6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34d01a2750 0x7f34d01a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.973+0000 7f34d6200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f34d011c530 con 0x7f34d01a2750 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.973+0000 7f34d6200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f34d011c3b0 con 0x7f34d0111450 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.973+0000 7f34d6200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f34d011c6b0 con 0x7f34d010c3d0 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d010c3d0 0x7f34d01a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d010c3d0 0x7f34d01a8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:43272/0 (socket says 172.21.15.102:43272) 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cf400640 1 -- 172.21.15.102:0/3965713990 learned_addr learned my addr 172.21.15.102:0/3965713990 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:36.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cea00640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34d0111450 0x7f34d01a8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cfe00640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34d01a2750 0x7f34d01a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cf400640 1 -- 172.21.15.102:0/3965713990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34d0111450 msgr2=0x7f34d01a8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cf400640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34d0111450 0x7f34d01a8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cf400640 1 -- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34d01a2750 msgr2=0x7f34d01a2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cf400640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34d01a2750 0x7f34d01a2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cf400640 1 -- 172.21.15.102:0/3965713990 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d01a3350 con 0x7f34d010c3d0 2024-04-25T02:05:36.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cea00640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34d0111450 0x7f34d01a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.974+0000 7f34cf400640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d010c3d0 0x7f34d01a8690 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f34c0004770 tx=0x7f34c0003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.975+0000 7f34c7e00640 1 -- 172.21.15.102:0/3965713990 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c003f070 con 0x7f34d010c3d0 2024-04-25T02:05:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.975+0000 7f34d6200640 1 -- 172.21.15.102:0/3965713990 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d01a35e0 con 0x7f34d010c3d0 2024-04-25T02:05:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.975+0000 7f34d6200640 1 -- 172.21.15.102:0/3965713990 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d01af770 con 0x7f34d010c3d0 2024-04-25T02:05:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.975+0000 7f34c7e00640 1 -- 172.21.15.102:0/3965713990 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34c0031ca0 con 0x7f34d010c3d0 2024-04-25T02:05:36.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.975+0000 7f34c7e00640 1 -- 172.21.15.102:0/3965713990 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0035030 con 0x7f34d010c3d0 2024-04-25T02:05:36.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.980+0000 7f34c7e00640 1 -- 172.21.15.102:0/3965713990 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f34c0042070 con 0x7f34d010c3d0 2024-04-25T02:05:36.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.983+0000 7f34c7e00640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f34b40a2ea0 0x7f34b40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.983+0000 7f34d9005740 1 Processor -- start 2024-04-25T02:05:36.989 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.983+0000 7f34c7e00640 1 -- 172.21.15.102:0/3965713990 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f34c00ef920 con 0x7f34d010c3d0 2024-04-25T02:05:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34d9005740 1 -- start start 2024-04-25T02:05:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34cea00640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f34b40a2ea0 0x7f34b40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34d9005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34d9005740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f7aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34d9005740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f7d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34d9005740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560863ed3ab0 con 0x560863f7afe0 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34d9005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560863e7c640 con 0x560863f6fe00 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34d9005740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560863ecfa70 con 0x560863f70720 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34cf400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34cf400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:47608/0 (socket says 172.21.15.102:47608) 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34cf400640 1 -- 172.21.15.102:0/3089712557 learned_addr learned my addr 172.21.15.102:0/3089712557 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34cfe00640 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f7d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.984+0000 7f34cea00640 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f7aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34cf400640 1 -- 172.21.15.102:0/3089712557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 msgr2=0x560863f7aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34cf400640 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f7aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34cf400640 1 -- 172.21.15.102:0/3089712557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 msgr2=0x560863f7d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34cf400640 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f7d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34cf400640 1 -- 172.21.15.102:0/3089712557 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560863be8210 con 0x560863f6fe00 2024-04-25T02:05:36.992 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34cea00640 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f7aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:05:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34cea00640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f34b40a2ea0 0x7f34b40a5360 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f34d01a3fa0 tx=0x7f34bc011860 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34cf400640 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f701e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f34c00c45e0 tx=0x7f34c00bbb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd962011dfd1eefc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34bbe00640 1 -- 172.21.15.102:0/3089712557 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c00d9070 con 0x560863f6fe00 2024-04-25T02:05:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34bbe00640 1 -- 172.21.15.102:0/3089712557 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34c00c0250 con 0x560863f6fe00 2024-04-25T02:05:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.985+0000 7f34bbe00640 1 -- 172.21.15.102:0/3089712557 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0049070 con 0x560863f6fe00 2024-04-25T02:05:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 -- 172.21.15.102:0/3089712557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 msgr2=0x560863f701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:36.993 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f701e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f34c00c45e0 tx=0x7f34c00bbb50 comp rx=0 tx=0).stop 2024-04-25T02:05:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 -- 172.21.15.102:0/3089712557 shutdown_connections 2024-04-25T02:05:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f7d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f7aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 --2- 172.21.15.102:0/3089712557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f701e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 -- 172.21.15.102:0/3089712557 >> 172.21.15.102:0/3089712557 conn(0x560863f6c310 msgr2=0x560863f6c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 -- 172.21.15.102:0/3089712557 shutdown_connections 2024-04-25T02:05:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 -- 172.21.15.102:0/3089712557 wait complete. 2024-04-25T02:05:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.986+0000 7f34d9005740 1 -- start start 2024-04-25T02:05:36.995 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.987+0000 7f34d9005740 1 Processor -- start 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.987+0000 7f34d9005740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f6fe00 0x560863f77f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.987+0000 7f34d9005740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f70720 0x560863f784a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.987+0000 7f34d9005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f7afe0 0x560863f789e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.987+0000 7f34d9005740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560863f7d9d0 con 0x560863f70720 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.987+0000 7f34d9005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560863e8f990 con 0x560863f7afe0 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.987+0000 7f34d9005740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560863f7db50 con 0x560863f6fe00 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.987+0000 7f34d9005740 10 client.0 _ll_register_callbacks cb 0x560863e91950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f7afe0 0x560863f789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f7afe0 0x560863f789e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:47614/0 (socket says 172.21.15.102:47614) 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cfe00640 1 -- 172.21.15.102:0/2361380928 learned_addr learned my addr 172.21.15.102:0/2361380928 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cea00640 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f70720 0x560863f784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cf400640 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f6fe00 0x560863f77f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cfe00640 1 -- 172.21.15.102:0/2361380928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f6fe00 msgr2=0x560863f77f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cfe00640 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f6fe00 0x560863f77f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.996 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cfe00640 1 -- 172.21.15.102:0/2361380928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f70720 msgr2=0x560863f784a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cfe00640 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f70720 0x560863f784a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.988+0000 7f34cfe00640 1 -- 172.21.15.102:0/2361380928 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560863cb4620 con 0x560863f7afe0 2024-04-25T02:05:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.989+0000 7f34cfe00640 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f7afe0 0x560863f789e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f34c80129e0 tx=0x7f34c8012ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6f38d1c83a02f2f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.989+0000 7f34b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:05:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.990+0000 7f34b8c00640 1 -- 172.21.15.102:0/2361380928 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8003970 con 0x560863f7afe0 2024-04-25T02:05:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.990+0000 7f34b8c00640 1 -- 172.21.15.102:0/2361380928 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f34c8003b10 con 0x560863f7afe0 2024-04-25T02:05:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.990+0000 7f34d9005740 1 -- 172.21.15.102:0/2361380928 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34c00c0250 con 0x560863f7afe0 2024-04-25T02:05:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.991+0000 7f34b8c00640 1 -- 172.21.15.102:0/2361380928 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f34c8003d10 con 0x560863f7afe0 2024-04-25T02:05:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.991+0000 7f34d9005740 10 client.27374 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.991+0000 7f34d9005740 1 -- 172.21.15.102:0/2361380928 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560863f89830 con 0x560863f7afe0 2024-04-25T02:05:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.991+0000 7f34b8c00640 1 -- 172.21.15.102:0/2361380928 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34c00c0250 con 0x560863f7afe0 2024-04-25T02:05:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.991+0000 7f34b8c00640 1 -- 172.21.15.102:0/2361380928 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f34c8004630 con 0x560863f7afe0 2024-04-25T02:05:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.992+0000 7f34d9005740 10 client.27374 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.992+0000 7f34d9005740 10 client.27374 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.992+0000 7f34d9005740 20 client.27374 populate_metadata read hostname 'smithi102' 2024-04-25T02:05:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.992+0000 7f34d9005740 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560863f8ad10 0x560863f8d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.992+0000 7f34d9005740 4 client.27374 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:05:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.992+0000 7f34d9005740 1 -- 172.21.15.102:0/2361380928 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560863ee0ff0 con 0x560863f8ad10 2024-04-25T02:05:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.993+0000 7f34cf400640 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560863f8ad10 0x560863f8d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.994+0000 7f34cf400640 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560863f8ad10 0x560863f8d170 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e715033d0cbee5cd server_cookie=41c44495f1d59dc5 in_seq=0 out_seq=0 2024-04-25T02:05:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.994+0000 7f34b8c00640 10 client.27374 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.994+0000 7f34b8c00640 1 -- 172.21.15.102:0/2361380928 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560863ee0ff0 con 0x560863f8ad10 2024-04-25T02:05:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.994+0000 7f34b8c00640 10 client.27374 handle_command_reply: tid=0 2024-04-25T02:05:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.994+0000 7f34d9005740 1 client.27374 shutdown 2024-04-25T02:05:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 -- 172.21.15.102:0/2361380928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f7afe0 msgr2=0x560863f789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:37.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f7afe0 0x560863f789e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f34c80129e0 tx=0x7f34c8012ea0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 -- 172.21.15.102:0/2361380928 shutdown_connections 2024-04-25T02:05:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560863f8ad10 0x560863f8d170 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f7afe0 0x560863f789e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f70720 0x560863f784a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 --2- 172.21.15.102:0/2361380928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f6fe00 0x560863f77f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 -- 172.21.15.102:0/2361380928 >> 172.21.15.102:0/2361380928 conn(0x560863f6c310 msgr2=0x560863f72040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34b8c00640 0 client.27374 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 -- 172.21.15.102:0/2361380928 shutdown_connections 2024-04-25T02:05:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 1 -- 172.21.15.102:0/2361380928 wait complete. 2024-04-25T02:05:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:36.995+0000 7f34d9005740 20 client.27374 trim_cache size 0 max 16384 2024-04-25T02:05:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.020+0000 7f34d9005740 1 Processor -- start 2024-04-25T02:05:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.024+0000 7f34d9005740 1 -- start start 2024-04-25T02:05:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.024+0000 7f34d9005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f86800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.024+0000 7f34d9005740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f727c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.024+0000 7f34d9005740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f6b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.024+0000 7f34d9005740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f34b00111a0 con 0x560863f7afe0 2024-04-25T02:05:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.024+0000 7f34d9005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f34b000e9b0 con 0x560863f6fe00 2024-04-25T02:05:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.024+0000 7f34d9005740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f34b000c820 con 0x560863f70720 2024-04-25T02:05:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.025+0000 7f34cea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.025+0000 7f34cea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f727c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:43286/0 (socket says 172.21.15.102:43286) 2024-04-25T02:05:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.025+0000 7f34cea00640 1 -- 172.21.15.102:0/3214430658 learned_addr learned my addr 172.21.15.102:0/3214430658 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.025+0000 7f34cea00640 1 -- 172.21.15.102:0/3214430658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 msgr2=0x560863f86800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:05:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.025+0000 7f34cea00640 1 --2- 172.21.15.102:0/3214430658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f86800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.025+0000 7f34cea00640 1 -- 172.21.15.102:0/3214430658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 msgr2=0x560863f6b300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:05:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.025+0000 7f34cea00640 1 --2- 172.21.15.102:0/3214430658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f6b300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.025+0000 7f34cea00640 1 -- 172.21.15.102:0/3214430658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560863f76f30 con 0x560863f70720 2024-04-25T02:05:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.026+0000 7f34cea00640 1 --2- 172.21.15.102:0/3214430658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f727c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0044c0 tx=0x7f34bc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8670293f41897ea7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.026+0000 7f34bb400640 1 -- 172.21.15.102:0/3214430658 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc017040 con 0x560863f70720 2024-04-25T02:05:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.028+0000 7f34bb400640 1 -- 172.21.15.102:0/3214430658 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc019050 con 0x560863f70720 2024-04-25T02:05:37.034 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.028+0000 7f34bb400640 1 -- 172.21.15.102:0/3214430658 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc01eb00 con 0x560863f70720 2024-04-25T02:05:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.032+0000 7f34d9005740 1 -- 172.21.15.102:0/3214430658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 msgr2=0x560863f727c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.032+0000 7f34d9005740 1 --2- 172.21.15.102:0/3214430658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f727c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0044c0 tx=0x7f34bc005f90 comp rx=0 tx=0).stop 2024-04-25T02:05:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.032+0000 7f34d9005740 1 -- 172.21.15.102:0/3214430658 shutdown_connections 2024-04-25T02:05:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.032+0000 7f34d9005740 1 --2- 172.21.15.102:0/3214430658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f6b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.032+0000 7f34d9005740 1 --2- 172.21.15.102:0/3214430658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f727c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.032+0000 7f34d9005740 1 --2- 172.21.15.102:0/3214430658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f86800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.032+0000 7f34d9005740 1 -- 172.21.15.102:0/3214430658 >> 172.21.15.102:0/3214430658 conn(0x560863f6c310 msgr2=0x560863f6abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.032+0000 7f34d9005740 1 -- 172.21.15.102:0/3214430658 shutdown_connections 2024-04-25T02:05:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.032+0000 7f34d9005740 1 -- 172.21.15.102:0/3214430658 wait complete. 2024-04-25T02:05:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.033+0000 7f34d9005740 1 -- start start 2024-04-25T02:05:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.033+0000 7f34d9005740 1 Processor -- start 2024-04-25T02:05:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.034+0000 7f34d9005740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f74b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.034+0000 7f34d9005740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f75070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.034+0000 7f34d9005740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f755b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.034+0000 7f34d9005740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f34b000ce70 con 0x560863f7afe0 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.034+0000 7f34d9005740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f34b000e2e0 con 0x560863f6fe00 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.034+0000 7f34d9005740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f34b000d540 con 0x560863f70720 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.034+0000 7f34d9005740 10 client.0 _ll_register_callbacks cb 0x7f34c8009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.034+0000 7f34cea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f75070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.034+0000 7f34cea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f75070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:43294/0 (socket says 172.21.15.102:43294) 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.035+0000 7f34cea00640 1 -- 172.21.15.102:0/1979974580 learned_addr learned my addr 172.21.15.102:0/1979974580 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.035+0000 7f34cf400640 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f74b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.035+0000 7f34cea00640 1 -- 172.21.15.102:0/1979974580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 msgr2=0x560863f74b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.035+0000 7f34cea00640 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f74b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.035+0000 7f34cea00640 1 -- 172.21.15.102:0/1979974580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 msgr2=0x560863f755b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.035+0000 7f34cea00640 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f755b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.035+0000 7f34cea00640 1 -- 172.21.15.102:0/1979974580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560863f70550 con 0x560863f70720 2024-04-25T02:05:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.035+0000 7f34cea00640 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f75070 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f34bc004490 tx=0x7f34bc005650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c199047b4432d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.036+0000 7f34b8c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:05:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.036+0000 7f34b8c00640 1 -- 172.21.15.102:0/1979974580 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc017040 con 0x560863f70720 2024-04-25T02:05:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.036+0000 7f34b8c00640 1 -- 172.21.15.102:0/1979974580 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f34bc0040e0 con 0x560863f70720 2024-04-25T02:05:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.037+0000 7f34d9005740 1 -- 172.21.15.102:0/1979974580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34bc019050 con 0x560863f70720 2024-04-25T02:05:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.037+0000 7f34b8c00640 1 -- 172.21.15.102:0/1979974580 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f34bc01e780 con 0x560863f70720 2024-04-25T02:05:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.037+0000 7f34d9005740 10 client.27334 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.037+0000 7f34d9005740 1 -- 172.21.15.102:0/1979974580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x560863ff12d0 con 0x560863f70720 2024-04-25T02:05:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.038+0000 7f34b8c00640 1 -- 172.21.15.102:0/1979974580 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc019050 con 0x560863f70720 2024-04-25T02:05:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.038+0000 7f34b8c00640 1 -- 172.21.15.102:0/1979974580 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f34bc026070 con 0x560863f70720 2024-04-25T02:05:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.038+0000 7f34d9005740 10 client.27334 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.038+0000 7f34d9005740 10 client.27334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.038+0000 7f34d9005740 20 client.27334 populate_metadata read hostname 'smithi102' 2024-04-25T02:05:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.038+0000 7f34d9005740 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560863ff19b0 0x560863ff3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.038+0000 7f34d9005740 4 client.27334 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:05:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.038+0000 7f34d9005740 1 -- 172.21.15.102:0/1979974580 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560863edb340 con 0x560863ff19b0 2024-04-25T02:05:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.039+0000 7f34cf400640 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560863ff19b0 0x560863ff3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.040+0000 7f34b8c00640 10 client.27334 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.040+0000 7f34cf400640 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560863ff19b0 0x560863ff3dd0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b959259a3205b979 server_cookie=f4bc71442c7256d4 in_seq=0 out_seq=0 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.125+0000 7f34b8c00640 1 -- 172.21.15.102:0/1979974580 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560863edb340 con 0x560863ff19b0 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.125+0000 7f34b8c00640 10 client.27334 handle_command_reply: tid=0 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.125+0000 7f34d9005740 1 client.27334 shutdown 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 -- 172.21.15.102:0/1979974580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 msgr2=0x560863f75070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f75070 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f34bc004490 tx=0x7f34bc005650 comp rx=0 tx=0).stop 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 -- 172.21.15.102:0/1979974580 shutdown_connections 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560863ff19b0 0x560863ff3dd0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560863f7afe0 0x560863f755b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34b8c00640 0 client.27334 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560863f70720 0x560863f75070 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 --2- 172.21.15.102:0/1979974580 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560863f6fe00 0x560863f74b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 -- 172.21.15.102:0/1979974580 >> 172.21.15.102:0/1979974580 conn(0x560863f6c310 msgr2=0x560863f6b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 -- 172.21.15.102:0/1979974580 shutdown_connections 2024-04-25T02:05:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 1 -- 172.21.15.102:0/1979974580 wait complete. 2024-04-25T02:05:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.126+0000 7f34d9005740 20 client.27334 trim_cache size 0 max 16384 2024-04-25T02:05:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:05:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:05:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:05:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:05:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 -- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f34b40a2ea0 msgr2=0x7f34b40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f34b40a2ea0 0x7f34b40a5360 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f34d01a3fa0 tx=0x7f34bc011860 comp rx=0 tx=0).stop 2024-04-25T02:05:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 -- 172.21.15.102:0/3965713990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d010c3d0 msgr2=0x7f34d01a8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d010c3d0 0x7f34d01a8690 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f34c0004770 tx=0x7f34c0003c10 comp rx=0 tx=0).stop 2024-04-25T02:05:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 -- 172.21.15.102:0/3965713990 shutdown_connections 2024-04-25T02:05:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f34b40a2ea0 0x7f34b40a5360 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34d01a2750 0x7f34d01a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34d0111450 0x7f34d01a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 --2- 172.21.15.102:0/3965713990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d010c3d0 0x7f34d01a8690 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 -- 172.21.15.102:0/3965713990 >> 172.21.15.102:0/3965713990 conn(0x7f34d006c9e0 msgr2=0x7f34d010f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 -- 172.21.15.102:0/3965713990 shutdown_connections 2024-04-25T02:05:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:37.127+0000 7f34d6200640 1 -- 172.21.15.102:0/3965713990 wait complete. 2024-04-25T02:05:37.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:37 smithi102 ceph-mon[27796]: pgmap v319: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.3 MiB/s wr, 565 op/s 2024-04-25T02:05:37.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:37 smithi110 ceph-mon[32848]: pgmap v319: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.3 MiB/s wr, 565 op/s 2024-04-25T02:05:37.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:37 smithi193 ceph-mon[32981]: pgmap v319: 97 pgs: 97 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.3 MiB/s wr, 565 op/s 2024-04-25T02:05:38.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 585.0 s, 1555.0 tps, lat 64.032 ms stddev 74.687 2024-04-25T02:05:39.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:39 smithi102 ceph-mon[27796]: pgmap v320: 97 pgs: 97 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 8.2 MiB/s wr, 871 op/s 2024-04-25T02:05:39.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:39 smithi110 ceph-mon[32848]: pgmap v320: 97 pgs: 97 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 8.2 MiB/s wr, 871 op/s 2024-04-25T02:05:39.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:39 smithi193 ceph-mon[32981]: pgmap v320: 97 pgs: 97 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 8.2 MiB/s wr, 871 op/s 2024-04-25T02:05:41.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:41 smithi102 ceph-mon[27796]: pgmap v321: 97 pgs: 97 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 5.3 MiB/s wr, 567 op/s 2024-04-25T02:05:41.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:41 smithi110 ceph-mon[32848]: pgmap v321: 97 pgs: 97 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 5.3 MiB/s wr, 567 op/s 2024-04-25T02:05:41.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:41 smithi193 ceph-mon[32981]: pgmap v321: 97 pgs: 97 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 5.3 MiB/s wr, 567 op/s 2024-04-25T02:05:43.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 590.0 s, 1689.2 tps, lat 58.895 ms stddev 70.433 2024-04-25T02:05:43.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:43 smithi102 ceph-mon[27796]: pgmap v322: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 5.9 MiB/s wr, 642 op/s 2024-04-25T02:05:43.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:43 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:05:43.742 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:43 smithi110 ceph-mon[32848]: pgmap v322: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 5.9 MiB/s wr, 642 op/s 2024-04-25T02:05:43.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:43 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:05:43.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:43 smithi193 ceph-mon[32981]: pgmap v322: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 5.9 MiB/s wr, 642 op/s 2024-04-25T02:05:43.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:43 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:05:44.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:44 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:44 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:44 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:44 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:44 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:44 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.715 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:44 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.715 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:44 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.715 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:44 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.715 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:44 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.715 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:44 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.715 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:44 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:44 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:44 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:44 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:44 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:44 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:44.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:44 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:45.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:45 smithi102 ceph-mon[27796]: pgmap v323: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 6.8 MiB/s wr, 735 op/s 2024-04-25T02:05:45.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:45 smithi110 ceph-mon[32848]: pgmap v323: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 6.8 MiB/s wr, 735 op/s 2024-04-25T02:05:45.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:45 smithi193 ceph-mon[32981]: pgmap v323: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 6.8 MiB/s wr, 735 op/s 2024-04-25T02:05:46.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:46.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:46.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:46.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: pgmap v324: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.2 MiB/s wr, 565 op/s 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: Unable to set osd_memory_target on smithi110 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T02:05:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:46 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:46.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:46.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:46.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:46.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: pgmap v324: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.2 MiB/s wr, 565 op/s 2024-04-25T02:05:46.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: Unable to set osd_memory_target on smithi110 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:46 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: pgmap v324: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 5.2 MiB/s wr, 565 op/s 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi110 to 176.2M 2024-04-25T02:05:47.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: Unable to set osd_memory_target on smithi110 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-04-25T02:05:47.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:46 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:47 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:47 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:47 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-25T02:05:47.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:47 smithi102 ceph-mon[27796]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-25T02:05:47.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:47 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:47 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:47.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:47 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-25T02:05:47.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:47 smithi110 ceph-mon[32848]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-25T02:05:48.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:47 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:48.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:47 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:48.010 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:47 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-04-25T02:05:48.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:47 smithi193 ceph-mon[32981]: Unable to set osd_memory_target on smithi193 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-04-25T02:05:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 595.0 s, 1799.0 tps, lat 55.823 ms stddev 67.463 2024-04-25T02:05:49.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:49 smithi102 ceph-mon[27796]: pgmap v325: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 7.8 MiB/s wr, 838 op/s 2024-04-25T02:05:49.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:49 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:49.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:49 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:49.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:49 smithi110 ceph-mon[32848]: pgmap v325: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 7.8 MiB/s wr, 838 op/s 2024-04-25T02:05:49.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:49 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:49.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:49 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:49.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:49 smithi193 ceph-mon[32981]: pgmap v325: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 7.8 MiB/s wr, 838 op/s 2024-04-25T02:05:49.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:49 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:49.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:49 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:50.315 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.308+0000 7f38d6000640 1 -- 172.21.15.102:0/4009178344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d00ff7a0 msgr2=0x7f38d00ffc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:50.315 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.308+0000 7f38d6000640 1 --2- 172.21.15.102:0/4009178344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d00ff7a0 0x7f38d00ffc20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009900 tx=0x7f38c402a410 comp rx=0 tx=0).stop 2024-04-25T02:05:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.308+0000 7f38d6000640 1 -- 172.21.15.102:0/4009178344 shutdown_connections 2024-04-25T02:05:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.308+0000 7f38d6000640 1 --2- 172.21.15.102:0/4009178344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38d0068110 0x7f38d0100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.308+0000 7f38d6000640 1 --2- 172.21.15.102:0/4009178344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d00ff7a0 0x7f38d00ffc20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.308+0000 7f38d6000640 1 --2- 172.21.15.102:0/4009178344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d01057b0 0x7f38d0105b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.308+0000 7f38d6000640 1 -- 172.21.15.102:0/4009178344 >> 172.21.15.102:0/4009178344 conn(0x7f38d00765a0 msgr2=0x7f38d00fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.309+0000 7f38d6000640 1 -- 172.21.15.102:0/4009178344 shutdown_connections 2024-04-25T02:05:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.309+0000 7f38d6000640 1 -- 172.21.15.102:0/4009178344 wait complete. 2024-04-25T02:05:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.310+0000 7f38d6000640 1 Processor -- start 2024-04-25T02:05:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.311+0000 7f38d6000640 1 -- start start 2024-04-25T02:05:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.311+0000 7f38d6000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38d0068110 0x7f38d019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.313+0000 7f38d6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d00ff7a0 0x7f38d019acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.313+0000 7f38d6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d01057b0 0x7f38d0109090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.313+0000 7f38d6000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f38d01119c0 con 0x7f38d00ff7a0 2024-04-25T02:05:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.313+0000 7f38d6000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f38d0111840 con 0x7f38d0068110 2024-04-25T02:05:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.313+0000 7f38d6000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f38d0111b40 con 0x7f38d01057b0 2024-04-25T02:05:50.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.313+0000 7f38d4c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38d0068110 0x7f38d019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.313+0000 7f38d4c00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38d0068110 0x7f38d019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:44800/0 (socket says 172.21.15.102:44800) 2024-04-25T02:05:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.313+0000 7f38d4c00640 1 -- 172.21.15.102:0/2718251743 learned_addr learned my addr 172.21.15.102:0/2718251743 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.313+0000 7f38cfe00640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d00ff7a0 0x7f38d019acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.314+0000 7f38d4c00640 1 -- 172.21.15.102:0/2718251743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d01057b0 msgr2=0x7f38d0109090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.314+0000 7f38d4c00640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d01057b0 0x7f38d0109090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.314+0000 7f38d4c00640 1 -- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d00ff7a0 msgr2=0x7f38d019acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.316+0000 7f38d4c00640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d00ff7a0 0x7f38d019acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.316+0000 7f38d4c00640 1 -- 172.21.15.102:0/2718251743 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d0109950 con 0x7f38d0068110 2024-04-25T02:05:50.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.316+0000 7f38cfe00640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d00ff7a0 0x7f38d019acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:05:50.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.319+0000 7f38d4c00640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38d0068110 0x7f38d019a790 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00e9f0 tx=0x7f38bc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.321+0000 7f38cd600640 1 -- 172.21.15.102:0/2718251743 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc00cd80 con 0x7f38d0068110 2024-04-25T02:05:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.321+0000 7f38d6000640 1 -- 172.21.15.102:0/2718251743 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d0109c40 con 0x7f38d0068110 2024-04-25T02:05:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.321+0000 7f38d6000640 1 -- 172.21.15.102:0/2718251743 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d01a2690 con 0x7f38d0068110 2024-04-25T02:05:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.325+0000 7f38cd600640 1 -- 172.21.15.102:0/2718251743 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38bc005500 con 0x7f38d0068110 2024-04-25T02:05:50.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.325+0000 7f38cd600640 1 -- 172.21.15.102:0/2718251743 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc010690 con 0x7f38d0068110 2024-04-25T02:05:50.344 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.337+0000 7f38cd600640 1 -- 172.21.15.102:0/2718251743 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f38bc010830 con 0x7f38d0068110 2024-04-25T02:05:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.338+0000 7f38d9ffd740 1 Processor -- start 2024-04-25T02:05:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.338+0000 7f38d9ffd740 1 -- start start 2024-04-25T02:05:50.345 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.338+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c57fb0 0x55b866c58390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.338+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c588d0 0x55b866c62c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.338+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c63190 0x55b866c65580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.338+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b866bbbab0 con 0x55b866c57fb0 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.339+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b866b64640 con 0x55b866c63190 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.339+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b866bb7a70 con 0x55b866c588d0 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.339+0000 7f38d4c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c588d0 0x55b866c62c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.339+0000 7f38d4c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c588d0 0x55b866c62c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52618/0 (socket says 172.21.15.102:52618) 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.339+0000 7f38d4c00640 1 -- 172.21.15.102:0/2294248451 learned_addr learned my addr 172.21.15.102:0/2294248451 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.339+0000 7f38cfe00640 1 --2- 172.21.15.102:0/2294248451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c63190 0x55b866c65580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.347 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.340+0000 7f38d4c00640 1 -- 172.21.15.102:0/2294248451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c63190 msgr2=0x55b866c65580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.340+0000 7f38d4c00640 1 --2- 172.21.15.102:0/2294248451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c63190 0x55b866c65580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.340+0000 7f38d4c00640 1 -- 172.21.15.102:0/2294248451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c57fb0 msgr2=0x55b866c58390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:05:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.340+0000 7f38d4c00640 1 --2- 172.21.15.102:0/2294248451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c57fb0 0x55b866c58390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.348 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.340+0000 7f38d4c00640 1 -- 172.21.15.102:0/2294248451 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8668d0210 con 0x55b866c588d0 2024-04-25T02:05:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.340+0000 7f38d4c00640 1 --2- 172.21.15.102:0/2294248451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c588d0 0x55b866c62c50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0cd390 tx=0x7f38bc0d6900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfb7ea63081d7d1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.341+0000 7f38c1c00640 1 -- 172.21.15.102:0/2294248451 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc0e3070 con 0x55b866c588d0 2024-04-25T02:05:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.341+0000 7f38cd600640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38b80a2d70 0x7f38b80a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.349 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.342+0000 7f38d5600640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38b80a2d70 0x7f38b80a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.342+0000 7f38cd600640 1 -- 172.21.15.102:0/2718251743 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f38bc0cfca0 con 0x7f38d0068110 2024-04-25T02:05:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.343+0000 7f38c1c00640 1 -- 172.21.15.102:0/2294248451 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38bc0dca30 con 0x55b866c588d0 2024-04-25T02:05:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.343+0000 7f38c1c00640 1 -- 172.21.15.102:0/2294248451 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc0d3070 con 0x55b866c588d0 2024-04-25T02:05:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.343+0000 7f38d9ffd740 1 -- 172.21.15.102:0/2294248451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c588d0 msgr2=0x55b866c62c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:50.350 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.343+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/2294248451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c588d0 0x55b866c62c50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0cd390 tx=0x7f38bc0d6900 comp rx=0 tx=0).stop 2024-04-25T02:05:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.344+0000 7f38d9ffd740 1 -- 172.21.15.102:0/2294248451 shutdown_connections 2024-04-25T02:05:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.344+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/2294248451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c63190 0x55b866c65580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.344+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/2294248451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c588d0 0x55b866c62c50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.344+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/2294248451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c57fb0 0x55b866c58390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.344+0000 7f38d9ffd740 1 -- 172.21.15.102:0/2294248451 >> 172.21.15.102:0/2294248451 conn(0x55b866c544c0 msgr2=0x55b866c548a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.344+0000 7f38d9ffd740 1 -- 172.21.15.102:0/2294248451 shutdown_connections 2024-04-25T02:05:50.351 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.344+0000 7f38d9ffd740 1 -- 172.21.15.102:0/2294248451 wait complete. 2024-04-25T02:05:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.345+0000 7f38d9ffd740 1 -- start start 2024-04-25T02:05:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.345+0000 7f38d5600640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38b80a2d70 0x7f38b80a5230 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f38d010fad0 tx=0x7f38c8009440 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:50.352 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.345+0000 7f38d9ffd740 1 Processor -- start 2024-04-25T02:05:50.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c57fb0 0x55b866c60110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c588d0 0x55b866c60650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c63190 0x55b866c60b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b866c65dc0 con 0x55b866c63190 2024-04-25T02:05:50.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b866b77990 con 0x55b866c588d0 2024-04-25T02:05:50.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b866c65f40 con 0x55b866c57fb0 2024-04-25T02:05:50.353 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d9ffd740 10 client.0 _ll_register_callbacks cb 0x55b866b79950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d4c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c57fb0 0x55b866c60110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d4c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c57fb0 0x55b866c60110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52628/0 (socket says 172.21.15.102:52628) 2024-04-25T02:05:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.346+0000 7f38d4c00640 1 -- 172.21.15.102:0/1603085670 learned_addr learned my addr 172.21.15.102:0/1603085670 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.347+0000 7f38cfe00640 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c588d0 0x55b866c60650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.347+0000 7f38d4c00640 1 -- 172.21.15.102:0/1603085670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c588d0 msgr2=0x55b866c60650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:50.354 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.347+0000 7f38d4c00640 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c588d0 0x55b866c60650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.347+0000 7f38d4c00640 1 -- 172.21.15.102:0/1603085670 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c63190 msgr2=0x55b866c60b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:50.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.347+0000 7f38d4c00640 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c63190 0x55b866c60b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.347+0000 7f38d4c00640 1 -- 172.21.15.102:0/1603085670 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b86699c620 con 0x55b866c57fb0 2024-04-25T02:05:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.347+0000 7f38d4c00640 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c57fb0 0x55b866c60110 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f38bc099220 tx=0x7f38bc009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fb2c36fa3b03723 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.348+0000 7f38b6000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:05:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.348+0000 7f38b6000640 1 -- 172.21.15.102:0/1603085670 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc0d3070 con 0x55b866c57fb0 2024-04-25T02:05:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.348+0000 7f38d9ffd740 1 -- 172.21.15.102:0/1603085670 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38bc0dca30 con 0x55b866c57fb0 2024-04-25T02:05:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.348+0000 7f38b6000640 1 -- 172.21.15.102:0/1603085670 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f38bc09d020 con 0x55b866c57fb0 2024-04-25T02:05:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.348+0000 7f38b6000640 1 -- 172.21.15.102:0/1603085670 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f38bc0d9040 con 0x55b866c57fb0 2024-04-25T02:05:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.348+0000 7f38d9ffd740 10 client.27343 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.348+0000 7f38d9ffd740 1 -- 172.21.15.102:0/1603085670 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b866c65dc0 con 0x55b866c57fb0 2024-04-25T02:05:50.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.349+0000 7f38b6000640 1 -- 172.21.15.102:0/1603085670 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38bc0dca30 con 0x55b866c57fb0 2024-04-25T02:05:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.351+0000 7f38b6000640 1 -- 172.21.15.102:0/1603085670 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f38bc0dc030 con 0x55b866c57fb0 2024-04-25T02:05:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.351+0000 7f38d9ffd740 10 client.27343 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.351+0000 7f38d9ffd740 10 client.27343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.351+0000 7f38d9ffd740 20 client.27343 populate_metadata read hostname 'smithi102' 2024-04-25T02:05:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.351+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b866c72c20 0x55b866c75010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.351+0000 7f38d9ffd740 4 client.27343 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:05:50.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.351+0000 7f38d9ffd740 1 -- 172.21.15.102:0/1603085670 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b866bc8ff0 con 0x55b866c72c20 2024-04-25T02:05:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.352+0000 7f38cfe00640 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b866c72c20 0x55b866c75010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.353+0000 7f38cfe00640 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b866c72c20 0x55b866c75010 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27d7e5befef9d261 server_cookie=14506e2b87a07190 in_seq=0 out_seq=0 2024-04-25T02:05:50.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.354+0000 7f38b6000640 10 client.27343 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.356+0000 7f38b6000640 1 -- 172.21.15.102:0/1603085670 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b866bc8ff0 con 0x55b866c72c20 2024-04-25T02:05:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.356+0000 7f38b6000640 10 client.27343 handle_command_reply: tid=0 2024-04-25T02:05:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.356+0000 7f38d9ffd740 1 client.27343 shutdown 2024-04-25T02:05:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.358+0000 7f38d9ffd740 1 -- 172.21.15.102:0/1603085670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c57fb0 msgr2=0x55b866c60110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.358+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c57fb0 0x55b866c60110 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f38bc099220 tx=0x7f38bc009310 comp rx=0 tx=0).stop 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.358+0000 7f38d9ffd740 1 -- 172.21.15.102:0/1603085670 shutdown_connections 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.358+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b866c72c20 0x55b866c75010 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.358+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c63190 0x55b866c60b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.359+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c588d0 0x55b866c60650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.359+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/1603085670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c57fb0 0x55b866c60110 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.359+0000 7f38d9ffd740 1 -- 172.21.15.102:0/1603085670 >> 172.21.15.102:0/1603085670 conn(0x55b866c544c0 msgr2=0x55b866c5a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.359+0000 7f38b6000640 0 client.27343 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.359+0000 7f38d9ffd740 1 -- 172.21.15.102:0/1603085670 shutdown_connections 2024-04-25T02:05:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.359+0000 7f38d9ffd740 1 -- 172.21.15.102:0/1603085670 wait complete. 2024-04-25T02:05:50.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.359+0000 7f38d9ffd740 20 client.27343 trim_cache size 0 max 16384 2024-04-25T02:05:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.393+0000 7f38d9ffd740 1 Processor -- start 2024-04-25T02:05:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.393+0000 7f38d9ffd740 1 -- start start 2024-04-25T02:05:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.393+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.393+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c609b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.393+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866c7a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.393+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f38b0011090 con 0x55b866c588d0 2024-04-25T02:05:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.393+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f38b000e8a0 con 0x55b866c57fb0 2024-04-25T02:05:50.400 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.393+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f38b000c710 con 0x55b866c63190 2024-04-25T02:05:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.394+0000 7f38d5600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866c7a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.394+0000 7f38d5600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866c7a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52640/0 (socket says 172.21.15.102:52640) 2024-04-25T02:05:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.394+0000 7f38d5600640 1 -- 172.21.15.102:0/877384728 learned_addr learned my addr 172.21.15.102:0/877384728 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.394+0000 7f38cfe00640 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.395+0000 7f38d5600640 1 -- 172.21.15.102:0/877384728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 msgr2=0x55b866c5a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.395+0000 7f38d4c00640 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.395+0000 7f38d5600640 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.395+0000 7f38d5600640 1 -- 172.21.15.102:0/877384728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 msgr2=0x55b866c609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.395+0000 7f38d5600640 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c609b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.395+0000 7f38d5600640 1 -- 172.21.15.102:0/877384728 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b866c54e00 con 0x55b866c63190 2024-04-25T02:05:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.395+0000 7f38cfe00640 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:05:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.396+0000 7f38d4c00640 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:05:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.397+0000 7f38d5600640 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866c7a730 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f38c800df60 tx=0x7f38c8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d36a3136f3e4c83f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.397+0000 7f38c1200640 1 -- 172.21.15.102:0/877384728 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c8021020 con 0x55b866c63190 2024-04-25T02:05:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.397+0000 7f38c1200640 1 -- 172.21.15.102:0/877384728 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c8016030 con 0x55b866c63190 2024-04-25T02:05:50.404 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.397+0000 7f38c1200640 1 -- 172.21.15.102:0/877384728 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c800f9b0 con 0x55b866c63190 2024-04-25T02:05:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.398+0000 7f38d9ffd740 1 -- 172.21.15.102:0/877384728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 msgr2=0x55b866c7a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.398+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866c7a730 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f38c800df60 tx=0x7f38c8004a40 comp rx=0 tx=0).stop 2024-04-25T02:05:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.398+0000 7f38d9ffd740 1 -- 172.21.15.102:0/877384728 shutdown_connections 2024-04-25T02:05:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.398+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866c7a730 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.398+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.398+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/877384728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.398+0000 7f38d9ffd740 1 -- 172.21.15.102:0/877384728 >> 172.21.15.102:0/877384728 conn(0x55b866c544c0 msgr2=0x55b866c53620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.399+0000 7f38d9ffd740 1 -- 172.21.15.102:0/877384728 shutdown_connections 2024-04-25T02:05:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.399+0000 7f38d9ffd740 1 -- 172.21.15.102:0/877384728 wait complete. 2024-04-25T02:05:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.399+0000 7f38d9ffd740 1 -- start start 2024-04-25T02:05:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.399+0000 7f38d9ffd740 1 Processor -- start 2024-04-25T02:05:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.400+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.400+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c5da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.400+0000 7f38d9ffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866cd7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.400+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f38b000cd60 con 0x55b866c588d0 2024-04-25T02:05:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.400+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f38b000e1d0 con 0x55b866c57fb0 2024-04-25T02:05:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.400+0000 7f38d9ffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f38b000d430 con 0x55b866c63190 2024-04-25T02:05:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.400+0000 7f38d9ffd740 10 client.0 _ll_register_callbacks cb 0x7f38bc0e1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:05:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.401+0000 7f38d5600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866cd7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.401+0000 7f38d5600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866cd7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:52644/0 (socket says 172.21.15.102:52644) 2024-04-25T02:05:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.401+0000 7f38d5600640 1 -- 172.21.15.102:0/184438368 learned_addr learned my addr 172.21.15.102:0/184438368 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:05:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.401+0000 7f38d4c00640 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.402+0000 7f38cfe00640 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c5da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.402+0000 7f38d4c00640 1 -- 172.21.15.102:0/184438368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 msgr2=0x55b866cd7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.402+0000 7f38d4c00640 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866cd7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.402+0000 7f38d4c00640 1 -- 172.21.15.102:0/184438368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 msgr2=0x55b866c5da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:05:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.402+0000 7f38d4c00640 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c5da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.402+0000 7f38d4c00640 1 -- 172.21.15.102:0/184438368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b866cd84e0 con 0x55b866c57fb0 2024-04-25T02:05:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.402+0000 7f38d5600640 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866cd7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:05:50.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.402+0000 7f38cfe00640 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c5da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:05:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.403+0000 7f38d4c00640 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5d510 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0cd4a0 tx=0x7f38bc005ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fb7ffc353115b33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:05:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.403+0000 7f38b6000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:05:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.403+0000 7f38b6000640 1 -- 172.21.15.102:0/184438368 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc0d3030 con 0x55b866c57fb0 2024-04-25T02:05:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.403+0000 7f38b6000640 1 -- 172.21.15.102:0/184438368 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f38bc0a55a0 con 0x55b866c57fb0 2024-04-25T02:05:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.403+0000 7f38d9ffd740 1 -- 172.21.15.102:0/184438368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38c8016030 con 0x55b866c57fb0 2024-04-25T02:05:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.403+0000 7f38b6000640 1 -- 172.21.15.102:0/184438368 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f38bc099060 con 0x55b866c57fb0 2024-04-25T02:05:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.403+0000 7f38d9ffd740 10 client.27392 fetch_fsmap learned FSMap version 40 2024-04-25T02:05:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.403+0000 7f38d9ffd740 1 -- 172.21.15.102:0/184438368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b866cd84e0 con 0x55b866c57fb0 2024-04-25T02:05:50.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.404+0000 7f38b6000640 1 -- 172.21.15.102:0/184438368 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c8016030 con 0x55b866c57fb0 2024-04-25T02:05:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.404+0000 7f38b6000640 1 -- 172.21.15.102:0/184438368 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f38bc0dc070 con 0x55b866c57fb0 2024-04-25T02:05:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.404+0000 7f38d9ffd740 10 client.27392 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:05:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.404+0000 7f38d9ffd740 10 client.27392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:05:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.404+0000 7f38d9ffd740 20 client.27392 populate_metadata read hostname 'smithi102' 2024-04-25T02:05:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.404+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b866cd9ad0 0x55b866cdbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:05:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.404+0000 7f38d9ffd740 4 client.27392 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:05:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.404+0000 7f38d9ffd740 1 -- 172.21.15.102:0/184438368 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b866bc3340 con 0x55b866cd9ad0 2024-04-25T02:05:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.404+0000 7f38cfe00640 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b866cd9ad0 0x55b866cdbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:05:50.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.405+0000 7f38cfe00640 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b866cd9ad0 0x55b866cdbef0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eb9d8871b865f40 server_cookie=20d804c808e45ecd in_seq=0 out_seq=0 2024-04-25T02:05:50.413 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.405+0000 7f38b6000640 10 client.27392 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:50.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:05:50.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:05:50.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:50 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.429+0000 7f38b6000640 1 -- 172.21.15.102:0/184438368 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b866bc3340 con 0x55b866cd9ad0 2024-04-25T02:05:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.429+0000 7f38b6000640 10 client.27392 handle_command_reply: tid=0 2024-04-25T02:05:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.429+0000 7f38d9ffd740 1 client.27392 shutdown 2024-04-25T02:05:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.430+0000 7f38d9ffd740 1 -- 172.21.15.102:0/184438368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 msgr2=0x55b866c5d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.430+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5d510 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0cd4a0 tx=0x7f38bc005ab0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.430+0000 7f38d9ffd740 1 -- 172.21.15.102:0/184438368 shutdown_connections 2024-04-25T02:05:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.431+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b866cd9ad0 0x55b866cdbef0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.431+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b866c63190 0x55b866cd7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.431+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b866c588d0 0x55b866c5da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.431+0000 7f38d9ffd740 1 --2- 172.21.15.102:0/184438368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b866c57fb0 0x55b866c5d510 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.431+0000 7f38d9ffd740 1 -- 172.21.15.102:0/184438368 >> 172.21.15.102:0/184438368 conn(0x55b866c544c0 msgr2=0x55b866c61940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.431+0000 7f38b6000640 0 client.27392 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:05:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.431+0000 7f38d9ffd740 1 -- 172.21.15.102:0/184438368 shutdown_connections 2024-04-25T02:05:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.431+0000 7f38d9ffd740 1 -- 172.21.15.102:0/184438368 wait complete. 2024-04-25T02:05:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.432+0000 7f38d9ffd740 20 client.27392 trim_cache size 0 max 16384 2024-04-25T02:05:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:05:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:05:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:05:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:05:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.432+0000 7f38d6000640 1 -- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38b80a2d70 msgr2=0x7f38b80a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.432+0000 7f38d6000640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38b80a2d70 0x7f38b80a5230 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f38d010fad0 tx=0x7f38c8009440 comp rx=0 tx=0).stop 2024-04-25T02:05:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.432+0000 7f38d6000640 1 -- 172.21.15.102:0/2718251743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38d0068110 msgr2=0x7f38d019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:05:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.432+0000 7f38d6000640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38d0068110 0x7f38d019a790 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f38bc00e9f0 tx=0x7f38bc00edf0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.432+0000 7f38d6000640 1 -- 172.21.15.102:0/2718251743 shutdown_connections 2024-04-25T02:05:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.432+0000 7f38d6000640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f38b80a2d70 0x7f38b80a5230 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.432+0000 7f38d6000640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d01057b0 0x7f38d0109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.432+0000 7f38d6000640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38d00ff7a0 0x7f38d019acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.433+0000 7f38d6000640 1 --2- 172.21.15.102:0/2718251743 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38d0068110 0x7f38d019a790 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:05:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.433+0000 7f38d6000640 1 -- 172.21.15.102:0/2718251743 >> 172.21.15.102:0/2718251743 conn(0x7f38d00765a0 msgr2=0x7f38d010b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:05:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.433+0000 7f38d6000640 1 -- 172.21.15.102:0/2718251743 shutdown_connections 2024-04-25T02:05:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:05:50.433+0000 7f38d6000640 1 -- 172.21.15.102:0/2718251743 wait complete. 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:05:50.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:50 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:50.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:05:50.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:50 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:05:51.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:51 smithi102 ceph-mon[27796]: Adjusting osd_memory_target on smithi102 to 176.4M 2024-04-25T02:05:51.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:51 smithi102 ceph-mon[27796]: Unable to set osd_memory_target on smithi102 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-25T02:05:51.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:51 smithi102 ceph-mon[27796]: pgmap v326: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 4.9 MiB/s wr, 533 op/s 2024-04-25T02:05:51.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:51 smithi110 ceph-mon[32848]: Adjusting osd_memory_target on smithi102 to 176.4M 2024-04-25T02:05:51.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:51 smithi110 ceph-mon[32848]: Unable to set osd_memory_target on smithi102 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-25T02:05:51.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:51 smithi110 ceph-mon[32848]: pgmap v326: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 4.9 MiB/s wr, 533 op/s 2024-04-25T02:05:51.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:51 smithi193 ceph-mon[32981]: Adjusting osd_memory_target on smithi102 to 176.4M 2024-04-25T02:05:51.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:51 smithi193 ceph-mon[32981]: Unable to set osd_memory_target on smithi102 to 185061990: error parsing value: Value '185061990' is below minimum 939524096 2024-04-25T02:05:51.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:51 smithi193 ceph-mon[32981]: pgmap v326: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 4.9 MiB/s wr, 533 op/s 2024-04-25T02:05:53.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 600.0 s, 1765.2 tps, lat 56.747 ms stddev 65.381 2024-04-25T02:05:53.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:53 smithi102 ceph-mon[27796]: pgmap v327: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-04-25T02:05:53.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:53 smithi110 ceph-mon[32848]: pgmap v327: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-04-25T02:05:53.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:53 smithi193 ceph-mon[32981]: pgmap v327: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-04-25T02:05:55.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:55 smithi102 ceph-mon[27796]: pgmap v328: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 7.0 MiB/s wr, 726 op/s 2024-04-25T02:05:55.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:55 smithi110 ceph-mon[32848]: pgmap v328: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 7.0 MiB/s wr, 726 op/s 2024-04-25T02:05:55.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:55 smithi193 ceph-mon[32981]: pgmap v328: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 7.0 MiB/s wr, 726 op/s 2024-04-25T02:05:57.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:57 smithi102 ceph-mon[27796]: pgmap v329: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 5.3 MiB/s wr, 541 op/s 2024-04-25T02:05:57.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:57 smithi110 ceph-mon[32848]: pgmap v329: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 5.3 MiB/s wr, 541 op/s 2024-04-25T02:05:57.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:57 smithi193 ceph-mon[32981]: pgmap v329: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 5.3 MiB/s wr, 541 op/s 2024-04-25T02:05:58.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 605.0 s, 2447.4 tps, lat 40.960 ms stddev 52.754 2024-04-25T02:05:59.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:05:59 smithi102 ceph-mon[27796]: pgmap v330: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 8.1 MiB/s wr, 824 op/s 2024-04-25T02:05:59.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:05:59 smithi110 ceph-mon[32848]: pgmap v330: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 8.1 MiB/s wr, 824 op/s 2024-04-25T02:05:59.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:05:59 smithi193 ceph-mon[32981]: pgmap v330: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 8.1 MiB/s wr, 824 op/s 2024-04-25T02:06:01.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:01 smithi102 ceph-mon[27796]: pgmap v331: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 5.5 MiB/s wr, 551 op/s 2024-04-25T02:06:01.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:01 smithi110 ceph-mon[32848]: pgmap v331: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 5.5 MiB/s wr, 551 op/s 2024-04-25T02:06:01.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:01 smithi193 ceph-mon[32981]: pgmap v331: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 5.5 MiB/s wr, 551 op/s 2024-04-25T02:06:03.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 610.0 s, 897.0 tps, lat 102.962 ms stddev 230.424 2024-04-25T02:06:03.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:03 smithi102 ceph-mon[27796]: pgmap v332: 97 pgs: 97 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 640 op/s 2024-04-25T02:06:03.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:03 smithi110 ceph-mon[32848]: pgmap v332: 97 pgs: 97 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 640 op/s 2024-04-25T02:06:03.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:03 smithi193 ceph-mon[32981]: pgmap v332: 97 pgs: 97 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 640 op/s 2024-04-25T02:06:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.662+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3781108236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ca0075c70 msgr2=0x7f7ca0076050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.662+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3781108236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ca0075c70 0x7f7ca0076050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009930 tx=0x7f7c9802a440 comp rx=0 tx=0).stop 2024-04-25T02:06:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.664+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3781108236 shutdown_connections 2024-04-25T02:06:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.664+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3781108236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ca0112410 0x7f7ca0114800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.664+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3781108236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca0076590 0x7f7ca0111ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.664+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3781108236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ca0075c70 0x7f7ca0076050 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.664+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3781108236 >> 172.21.15.102:0/3781108236 conn(0x7f7ca00ff220 msgr2=0x7f7ca0101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.665+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3781108236 shutdown_connections 2024-04-25T02:06:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.665+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3781108236 wait complete. 2024-04-25T02:06:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.668+0000 7f7c9fe00640 1 Processor -- start 2024-04-25T02:06:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.669+0000 7f7c9fe00640 1 -- start start 2024-04-25T02:06:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.671+0000 7f7c9fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ca0075c70 0x7f7ca0072f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.677 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.671+0000 7f7c9fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca0076590 0x7f7ca006c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.671+0000 7f7c9fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ca0112410 0x7f7ca006c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.671+0000 7f7c9fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca0118040 con 0x7f7ca0112410 2024-04-25T02:06:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.671+0000 7f7c9fe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca0117ec0 con 0x7f7ca0076590 2024-04-25T02:06:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.671+0000 7f7c9fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca01181c0 con 0x7f7ca0075c70 2024-04-25T02:06:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.672+0000 7f7c9f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ca0112410 0x7f7ca006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.672+0000 7f7c9ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ca0075c70 0x7f7ca0072f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.673+0000 7f7c9f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ca0112410 0x7f7ca006c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48298/0 (socket says 172.21.15.102:48298) 2024-04-25T02:06:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.673+0000 7f7c9f400640 1 -- 172.21.15.102:0/3399453522 learned_addr learned my addr 172.21.15.102:0/3399453522 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:03.679 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.673+0000 7f7c9e000640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca0076590 0x7f7ca006c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.674+0000 7f7c9ea00640 1 -- 172.21.15.102:0/3399453522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca0076590 msgr2=0x7f7ca006c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.674+0000 7f7c9ea00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca0076590 0x7f7ca006c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.674+0000 7f7c9ea00640 1 -- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ca0112410 msgr2=0x7f7ca006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.674+0000 7f7c9ea00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ca0112410 0x7f7ca006c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.674+0000 7f7c9ea00640 1 -- 172.21.15.102:0/3399453522 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca006cb60 con 0x7f7ca0075c70 2024-04-25T02:06:03.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.675+0000 7f7c9f400640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ca0112410 0x7f7ca006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:06:03.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.675+0000 7f7c9ea00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ca0075c70 0x7f7ca0072f60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7c9802a960 tx=0x7f7c98003c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:03.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.675+0000 7f7c9e000640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca0076590 0x7f7ca006c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:06:03.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.675+0000 7f7c97400640 1 -- 172.21.15.102:0/3399453522 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9803f070 con 0x7f7ca0075c70 2024-04-25T02:06:03.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.676+0000 7f7c97400640 1 -- 172.21.15.102:0/3399453522 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c9803b460 con 0x7f7ca0075c70 2024-04-25T02:06:03.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.676+0000 7f7c97400640 1 -- 172.21.15.102:0/3399453522 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98035030 con 0x7f7ca0075c70 2024-04-25T02:06:03.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.679+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3399453522 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca006cdf0 con 0x7f7ca0075c70 2024-04-25T02:06:03.685 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.679+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3399453522 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca0070d20 con 0x7f7ca0075c70 2024-04-25T02:06:03.692 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.686+0000 7f7ca8b86740 1 Processor -- start 2024-04-25T02:06:03.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.688+0000 7f7ca8b86740 1 -- start start 2024-04-25T02:06:03.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.690+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541fa2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.693+0000 7f7c9f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541fa2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.693+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 0x55d541facc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.693+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fad190 0x55d541faf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.693+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d541f05ab0 con 0x55d541fad190 2024-04-25T02:06:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.693+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d541eae640 con 0x55d541fa1fb0 2024-04-25T02:06:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.693+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d541f01a70 con 0x55d541fa28d0 2024-04-25T02:06:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.693+0000 7f7c9f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541fa2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34120/0 (socket says 172.21.15.102:34120) 2024-04-25T02:06:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.693+0000 7f7c9f400640 1 -- 172.21.15.102:0/3684751146 learned_addr learned my addr 172.21.15.102:0/3684751146 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.694+0000 7f7c9f400640 1 -- 172.21.15.102:0/3684751146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 msgr2=0x55d541facc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:06:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.694+0000 7f7c9f400640 1 --2- 172.21.15.102:0/3684751146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 0x55d541facc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.694+0000 7f7c9f400640 1 -- 172.21.15.102:0/3684751146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fad190 msgr2=0x55d541faf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.694+0000 7f7c9f400640 1 --2- 172.21.15.102:0/3684751146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fad190 0x55d541faf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.694+0000 7f7c9f400640 1 -- 172.21.15.102:0/3684751146 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d541c1a210 con 0x55d541fa1fb0 2024-04-25T02:06:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.694+0000 7f7c97400640 1 -- 172.21.15.102:0/3399453522 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7c98042070 con 0x7f7ca0075c70 2024-04-25T02:06:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.700+0000 7f7c97400640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7c840a2ec0 0x7f7c840a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.700+0000 7f7c97400640 1 -- 172.21.15.102:0/3399453522 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f7c980efb70 con 0x7f7ca0075c70 2024-04-25T02:06:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.704+0000 7f7c9f400640 1 --2- 172.21.15.102:0/3684751146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541fa2390 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800acc0 tx=0x7f7c8800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20fac6ccdc38c1bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.704+0000 7f7c8fe00640 1 -- 172.21.15.102:0/3684751146 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c88004220 con 0x55d541fa1fb0 2024-04-25T02:06:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.705+0000 7f7c8fe00640 1 -- 172.21.15.102:0/3684751146 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c880043c0 con 0x55d541fa1fb0 2024-04-25T02:06:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.705+0000 7f7c8fe00640 1 -- 172.21.15.102:0/3684751146 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c880035f0 con 0x55d541fa1fb0 2024-04-25T02:06:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.705+0000 7f7c9ea00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7c840a2ec0 0x7f7c840a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.706+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3684751146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 msgr2=0x55d541fa2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.706+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3684751146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541fa2390 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800acc0 tx=0x7f7c8800e720 comp rx=0 tx=0).stop 2024-04-25T02:06:03.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.707+0000 7f7c9ea00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7c840a2ec0 0x7f7c840a5380 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55d541fa3550 tx=0x7f7c98005be0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.708+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3684751146 shutdown_connections 2024-04-25T02:06:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.708+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3684751146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fad190 0x55d541faf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.708+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3684751146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 0x55d541facc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.709+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3684751146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541fa2390 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.709+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3684751146 >> 172.21.15.102:0/3684751146 conn(0x55d541f9e4c0 msgr2=0x55d541f9e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.709+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3684751146 shutdown_connections 2024-04-25T02:06:03.717 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.711+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3684751146 wait complete. 2024-04-25T02:06:03.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.713+0000 7f7ca8b86740 1 -- start start 2024-04-25T02:06:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.716+0000 7f7ca8b86740 1 Processor -- start 2024-04-25T02:06:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.717+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541faa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.717+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa28d0 0x55d541faa630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.717+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fad190 0x55d541faab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.717+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d541fafdc0 con 0x55d541fa28d0 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.717+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d541ec1990 con 0x55d541fa1fb0 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.717+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d541faff40 con 0x55d541fad190 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.717+0000 7f7ca8b86740 10 client.0 _ll_register_callbacks cb 0x55d541ec3950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.717+0000 7f7c9f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541faa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.718+0000 7f7c9f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541faa0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34126/0 (socket says 172.21.15.102:34126) 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.718+0000 7f7c9f400640 1 -- 172.21.15.102:0/3896789595 learned_addr learned my addr 172.21.15.102:0/3896789595 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.718+0000 7f7c9f400640 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fad190 0x55d541faab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.718+0000 7f7c9f400640 1 -- 172.21.15.102:0/3896789595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fad190 msgr2=0x55d541faab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.718+0000 7f7c9f400640 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fad190 0x55d541faab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.718+0000 7f7c9f400640 1 -- 172.21.15.102:0/3896789595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa28d0 msgr2=0x55d541faa630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:06:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.718+0000 7f7c9f400640 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa28d0 0x55d541faa630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.718+0000 7f7c9f400640 1 -- 172.21.15.102:0/3896789595 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d541ce6620 con 0x55d541fa1fb0 2024-04-25T02:06:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.719+0000 7f7c9f400640 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541faa0f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800aa50 tx=0x7f7c88004b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf057a2c051ce77f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.720+0000 7f7c8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:06:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.720+0000 7f7c8cc00640 1 -- 172.21.15.102:0/3896789595 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8801e020 con 0x55d541fa1fb0 2024-04-25T02:06:03.727 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.721+0000 7f7c8cc00640 1 -- 172.21.15.102:0/3896789595 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f7c88005560 con 0x55d541fa1fb0 2024-04-25T02:06:03.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.722+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3896789595 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c880043c0 con 0x55d541fa1fb0 2024-04-25T02:06:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.725+0000 7f7c8cc00640 1 -- 172.21.15.102:0/3896789595 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7c8802f050 con 0x55d541fa1fb0 2024-04-25T02:06:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.725+0000 7f7ca8b86740 10 client.27401 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.725+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3896789595 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55d541fbbce0 con 0x55d541fa1fb0 2024-04-25T02:06:03.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.725+0000 7f7c8cc00640 1 -- 172.21.15.102:0/3896789595 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c880043c0 con 0x55d541fa1fb0 2024-04-25T02:06:03.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.726+0000 7f7c8cc00640 1 -- 172.21.15.102:0/3896789595 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f7c8802a540 con 0x55d541fa1fb0 2024-04-25T02:06:03.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.730+0000 7f7ca8b86740 10 client.27401 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:03.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.730+0000 7f7ca8b86740 10 client.27401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:03.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.730+0000 7f7ca8b86740 20 client.27401 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:03.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.730+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d541fbd140 0x55d541fbf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.730+0000 7f7ca8b86740 4 client.27401 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:06:03.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.730+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3896789595 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d541f12ff0 con 0x55d541fbd140 2024-04-25T02:06:03.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.731+0000 7f7c9e000640 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d541fbd140 0x55d541fbf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.732+0000 7f7c9e000640 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d541fbd140 0x55d541fbf530 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3880c21b388cebb5 server_cookie=ded33f2202621ae1 in_seq=0 out_seq=0 2024-04-25T02:06:03.738 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.732+0000 7f7c8cc00640 10 client.27401 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:03.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.733+0000 7f7c8cc00640 1 -- 172.21.15.102:0/3896789595 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d541f12ff0 con 0x55d541fbd140 2024-04-25T02:06:03.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.734+0000 7f7c8cc00640 10 client.27401 handle_command_reply: tid=0 2024-04-25T02:06:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.734+0000 7f7ca8b86740 1 client.27401 shutdown 2024-04-25T02:06:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.734+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3896789595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 msgr2=0x55d541faa0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:03.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.734+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541faa0f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800aa50 tx=0x7f7c88004b90 comp rx=0 tx=0).stop 2024-04-25T02:06:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.735+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3896789595 shutdown_connections 2024-04-25T02:06:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.735+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d541fbd140 0x55d541fbf530 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.735+0000 7f7c8cc00640 0 client.27401 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.735+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fad190 0x55d541faab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.735+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa28d0 0x55d541faa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.735+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/3896789595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fa1fb0 0x55d541faa0f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.735+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3896789595 >> 172.21.15.102:0/3896789595 conn(0x55d541f9e4c0 msgr2=0x55d541fa41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:03.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.736+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3896789595 shutdown_connections 2024-04-25T02:06:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.736+0000 7f7ca8b86740 1 -- 172.21.15.102:0/3896789595 wait complete. 2024-04-25T02:06:03.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.736+0000 7f7ca8b86740 20 client.27401 trim_cache size 0 max 16384 2024-04-25T02:06:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.802+0000 7f7ca8b86740 1 Processor -- start 2024-04-25T02:06:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.810+0000 7f7ca8b86740 1 -- start start 2024-04-25T02:06:03.816 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.810+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa1fb0 0x55d541fa41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.811+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa28d0 0x55d541faa990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.811+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 0x55d541f9d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.811+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca01bebd0 con 0x55d541fa28d0 2024-04-25T02:06:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.811+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca01bc3e0 con 0x55d541fad190 2024-04-25T02:06:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.811+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca01ba250 con 0x55d541fa1fb0 2024-04-25T02:06:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.811+0000 7f7c9f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa1fb0 0x55d541fa41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.811+0000 7f7c9f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 0x55d541f9d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.812+0000 7f7c9f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa1fb0 0x55d541fa41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44466/0 (socket says 172.21.15.102:44466) 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.812+0000 7f7c9f400640 1 -- 172.21.15.102:0/2409794940 learned_addr learned my addr 172.21.15.102:0/2409794940 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.812+0000 7f7c9e000640 1 --2- 172.21.15.102:0/2409794940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa28d0 0x55d541faa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.812+0000 7f7c9f400640 1 -- 172.21.15.102:0/2409794940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa1fb0 msgr2=0x55d541fa41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.812+0000 7f7c9f400640 1 --2- 172.21.15.102:0/2409794940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa1fb0 0x55d541fa41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.812+0000 7f7c9f400640 1 -- 172.21.15.102:0/2409794940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa28d0 msgr2=0x55d541faa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.812+0000 7f7c9f400640 1 --2- 172.21.15.102:0/2409794940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa28d0 0x55d541faa990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.812+0000 7f7c9f400640 1 -- 172.21.15.102:0/2409794940 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d541f999f0 con 0x55d541fad190 2024-04-25T02:06:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.813+0000 7f7c9f400640 1 --2- 172.21.15.102:0/2409794940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 0x55d541f9d4d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800a4d0 tx=0x7f7c88004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b4b305c9d7d5899 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.814+0000 7f7c8f400640 1 -- 172.21.15.102:0/2409794940 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8801e070 con 0x55d541fad190 2024-04-25T02:06:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.815+0000 7f7c8f400640 1 -- 172.21.15.102:0/2409794940 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c8802a640 con 0x55d541fad190 2024-04-25T02:06:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.815+0000 7f7c8f400640 1 -- 172.21.15.102:0/2409794940 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c88024040 con 0x55d541fad190 2024-04-25T02:06:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.819+0000 7f7ca8b86740 1 -- 172.21.15.102:0/2409794940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 msgr2=0x55d541f9d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.819+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/2409794940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 0x55d541f9d4d0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7c8800a4d0 tx=0x7f7c88004af0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.819+0000 7f7ca8b86740 1 -- 172.21.15.102:0/2409794940 shutdown_connections 2024-04-25T02:06:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.819+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/2409794940 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 0x55d541f9d4d0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.819+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/2409794940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa28d0 0x55d541faa990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.819+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/2409794940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa1fb0 0x55d541fa41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.819+0000 7f7ca8b86740 1 -- 172.21.15.102:0/2409794940 >> 172.21.15.102:0/2409794940 conn(0x55d541f9e4c0 msgr2=0x55d541f9cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.819+0000 7f7ca8b86740 1 -- 172.21.15.102:0/2409794940 shutdown_connections 2024-04-25T02:06:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.820+0000 7f7ca8b86740 1 -- 172.21.15.102:0/2409794940 wait complete. 2024-04-25T02:06:03.826 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.820+0000 7f7ca8b86740 1 -- start start 2024-04-25T02:06:03.827 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.821+0000 7f7ca8b86740 1 Processor -- start 2024-04-25T02:06:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.822+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa1fb0 0x55d541fb1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.822+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 0x55d541fb17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.828 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.822+0000 7f7ca8b86740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 0x55d542022610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.822+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca01ba8a0 con 0x55d541fa1fb0 2024-04-25T02:06:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.822+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca01bbd10 con 0x55d541fad190 2024-04-25T02:06:03.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.822+0000 7f7ca8b86740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca01baf70 con 0x55d541fa28d0 2024-04-25T02:06:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.822+0000 7f7ca8b86740 10 client.0 _ll_register_callbacks cb 0x7f7c88014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.822+0000 7f7c9e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 0x55d541fb17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.823+0000 7f7c9e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 0x55d541fb17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:44468/0 (socket says 172.21.15.102:44468) 2024-04-25T02:06:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.823+0000 7f7c9e000640 1 -- 172.21.15.102:0/4127711653 learned_addr learned my addr 172.21.15.102:0/4127711653 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.823+0000 7f7c9e000640 1 -- 172.21.15.102:0/4127711653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 msgr2=0x55d542022610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:06:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.823+0000 7f7c9e000640 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 0x55d542022610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.823+0000 7f7c9e000640 1 -- 172.21.15.102:0/4127711653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa1fb0 msgr2=0x55d541fb1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.823+0000 7f7c9e000640 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa1fb0 0x55d541fb1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.823+0000 7f7c9e000640 1 -- 172.21.15.102:0/4127711653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d542022b50 con 0x55d541fa28d0 2024-04-25T02:06:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.824+0000 7f7c9e000640 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 0x55d541fb17c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7c90004aa0 tx=0x7f7c9000b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=785270b4f782f6fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.824+0000 7f7c8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:06:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.824+0000 7f7c8cc00640 1 -- 172.21.15.102:0/4127711653 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90020670 con 0x55d541fa28d0 2024-04-25T02:06:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.825+0000 7f7c8cc00640 1 -- 172.21.15.102:0/4127711653 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8513+0+0 (secure 0 0 0) 0x7f7c9001f3d0 con 0x55d541fa28d0 2024-04-25T02:06:03.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.826+0000 7f7ca8b86740 1 -- 172.21.15.102:0/4127711653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c8802a640 con 0x55d541fa28d0 2024-04-25T02:06:03.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.828+0000 7f7c8cc00640 1 -- 172.21.15.102:0/4127711653 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f7c90020810 con 0x55d541fa28d0 2024-04-25T02:06:03.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.828+0000 7f7ca8b86740 10 client.27364 fetch_fsmap learned FSMap version 40 2024-04-25T02:06:03.834 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.828+0000 7f7ca8b86740 1 -- 172.21.15.102:0/4127711653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55d542023df0 con 0x55d541fa28d0 2024-04-25T02:06:03.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.829+0000 7f7c8cc00640 1 -- 172.21.15.102:0/4127711653 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c8802a640 con 0x55d541fa28d0 2024-04-25T02:06:03.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.831+0000 7f7c8cc00640 1 -- 172.21.15.102:0/4127711653 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f7c90016070 con 0x55d541fa28d0 2024-04-25T02:06:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.834+0000 7f7ca8b86740 10 client.27364 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T02:06:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.834+0000 7f7ca8b86740 10 client.27364 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.834+0000 7f7ca8b86740 20 client.27364 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.834+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d542024550 0x55d5420269b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.834+0000 7f7ca8b86740 4 client.27364 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:06:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.834+0000 7f7ca8b86740 1 -- 172.21.15.102:0/4127711653 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d541f0d340 con 0x55d542024550 2024-04-25T02:06:03.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.834+0000 7f7c9f400640 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d542024550 0x55d5420269b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:03.841 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.836+0000 7f7c9f400640 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d542024550 0x55d5420269b0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48a9fb367cb0d263 server_cookie=5298b8ab86a67f65 in_seq=0 out_seq=0 2024-04-25T02:06:03.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:03.836+0000 7f7c8cc00640 10 client.27364 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.500+0000 7f7c8cc00640 1 -- 172.21.15.102:0/4127711653 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d541f0d340 con 0x55d542024550 2024-04-25T02:06:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.500+0000 7f7c8cc00640 10 client.27364 handle_command_reply: tid=0 2024-04-25T02:06:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.500+0000 7f7ca8b86740 1 client.27364 shutdown 2024-04-25T02:06:04.506 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.500+0000 7f7ca8b86740 1 -- 172.21.15.102:0/4127711653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 msgr2=0x55d541fb17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.501+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 0x55d541fb17c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7c90004aa0 tx=0x7f7c9000b7e0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.501+0000 7f7ca8b86740 1 -- 172.21.15.102:0/4127711653 shutdown_connections 2024-04-25T02:06:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.501+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d542024550 0x55d5420269b0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.501+0000 7f7c8cc00640 0 client.27364 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.501+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d541fad190 0x55d542022610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.501+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d541fa28d0 0x55d541fb17c0 secure :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7c90004aa0 tx=0x7f7c9000b7e0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.507 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.501+0000 7f7ca8b86740 1 --2- 172.21.15.102:0/4127711653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d541fa1fb0 0x55d541fb1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.501+0000 7f7ca8b86740 1 -- 172.21.15.102:0/4127711653 >> 172.21.15.102:0/4127711653 conn(0x55d541f9e4c0 msgr2=0x55d541fa4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.502+0000 7f7ca8b86740 1 -- 172.21.15.102:0/4127711653 shutdown_connections 2024-04-25T02:06:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.502+0000 7f7ca8b86740 1 -- 172.21.15.102:0/4127711653 wait complete. 2024-04-25T02:06:04.508 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.503+0000 7f7ca8b86740 20 client.27364 trim_cache size 0 max 16384 2024-04-25T02:06:04.509 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:06:04.509 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:06:04.509 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:06:04.509 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:06:04.509 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7c840a2ec0 msgr2=0x7f7c840a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:04.509 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7c840a2ec0 0x7f7c840a5380 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55d541fa3550 tx=0x7f7c98005be0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.510 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3399453522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ca0075c70 msgr2=0x7f7ca0072f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:04.510 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ca0075c70 0x7f7ca0072f60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7c9802a960 tx=0x7f7c98003c00 comp rx=0 tx=0).stop 2024-04-25T02:06:04.510 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3399453522 shutdown_connections 2024-04-25T02:06:04.510 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7c840a2ec0 0x7f7c840a5380 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.510 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ca0112410 0x7f7ca006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca0076590 0x7f7ca006c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 --2- 172.21.15.102:0/3399453522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ca0075c70 0x7f7ca0072f60 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.504+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3399453522 >> 172.21.15.102:0/3399453522 conn(0x7f7ca00ff220 msgr2=0x7f7ca0113080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.505+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3399453522 shutdown_connections 2024-04-25T02:06:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:04.505+0000 7f7c9fe00640 1 -- 172.21.15.102:0/3399453522 wait complete. 2024-04-25T02:06:05.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:05 smithi102 ceph-mon[27796]: pgmap v333: 97 pgs: 97 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.0 MiB/s wr, 721 op/s 2024-04-25T02:06:05.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:05 smithi110 ceph-mon[32848]: pgmap v333: 97 pgs: 97 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.0 MiB/s wr, 721 op/s 2024-04-25T02:06:05.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:05 smithi193 ceph-mon[32981]: pgmap v333: 97 pgs: 97 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.0 MiB/s wr, 721 op/s 2024-04-25T02:06:07.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:07 smithi102 ceph-mon[27796]: pgmap v334: 97 pgs: 97 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 4.7 MiB/s wr, 510 op/s 2024-04-25T02:06:07.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:07 smithi110 ceph-mon[32848]: pgmap v334: 97 pgs: 97 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 4.7 MiB/s wr, 510 op/s 2024-04-25T02:06:07.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:07 smithi193 ceph-mon[32981]: pgmap v334: 97 pgs: 97 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 4.7 MiB/s wr, 510 op/s 2024-04-25T02:06:08.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 615.0 s, 1025.6 tps, lat 105.009 ms stddev 257.958 2024-04-25T02:06:08.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:08 smithi102 ceph-mon[27796]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T02:06:08.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:08 smithi110 ceph-mon[32848]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T02:06:08.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:08 smithi193 ceph-mon[32981]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T02:06:09.394 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:09 smithi110 ceph-mon[32848]: pgmap v336: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 29 MiB/s wr, 1.87k op/s 2024-04-25T02:06:09.394 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:09 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:06:09.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:09 smithi102 ceph-mon[27796]: pgmap v336: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 29 MiB/s wr, 1.87k op/s 2024-04-25T02:06:09.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:09 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:06:09.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:09 smithi193 ceph-mon[32981]: pgmap v336: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 29 MiB/s wr, 1.87k op/s 2024-04-25T02:06:09.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:09 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:06:11.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:11 smithi102 ceph-mon[27796]: pgmap v337: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 29 MiB/s wr, 1.87k op/s 2024-04-25T02:06:11.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:11 smithi110 ceph-mon[32848]: pgmap v337: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 29 MiB/s wr, 1.87k op/s 2024-04-25T02:06:11.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:11 smithi193 ceph-mon[32981]: pgmap v337: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 29 MiB/s wr, 1.87k op/s 2024-04-25T02:06:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 620.0 s, 2051.0 tps, lat 48.867 ms stddev 58.390 2024-04-25T02:06:13.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:13 smithi102 ceph-mon[27796]: pgmap v338: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.91k op/s 2024-04-25T02:06:13.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:13 smithi110 ceph-mon[32848]: pgmap v338: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.91k op/s 2024-04-25T02:06:13.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:13 smithi193 ceph-mon[32981]: pgmap v338: 97 pgs: 3 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 90 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.91k op/s 2024-04-25T02:06:15.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:15 smithi102 ceph-mon[27796]: pgmap v339: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.93k op/s 2024-04-25T02:06:15.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:15 smithi110 ceph-mon[32848]: pgmap v339: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.93k op/s 2024-04-25T02:06:15.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:15 smithi193 ceph-mon[32981]: pgmap v339: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.93k op/s 2024-04-25T02:06:17.428 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:17 smithi102 ceph-mon[27796]: pgmap v340: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.93k op/s 2024-04-25T02:06:17.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:17 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:06:17.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:17 smithi110 ceph-mon[32848]: pgmap v340: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.93k op/s 2024-04-25T02:06:17.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:17 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:06:17.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:17 smithi193 ceph-mon[32981]: pgmap v340: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 30 MiB/s wr, 1.93k op/s 2024-04-25T02:06:17.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:17 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:06:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.687+0000 7f7ed1a00640 1 -- 172.21.15.102:0/981689315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc0ff810 msgr2=0x7f7ecc0ffc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.687+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/981689315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc0ff810 0x7f7ecc0ffc90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009890 tx=0x7f7eb802a380 comp rx=0 tx=0).stop 2024-04-25T02:06:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.687+0000 7f7ed1a00640 1 -- 172.21.15.102:0/981689315 shutdown_connections 2024-04-25T02:06:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.687+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/981689315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc1001d0 0x7f7ecc10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.687+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/981689315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc0ff810 0x7f7ecc0ffc90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.687+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/981689315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ecc106860 0x7f7ecc106c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.687+0000 7f7ed1a00640 1 -- 172.21.15.102:0/981689315 >> 172.21.15.102:0/981689315 conn(0x7f7ecc076740 msgr2=0x7f7ecc0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.687+0000 7f7ed1a00640 1 -- 172.21.15.102:0/981689315 shutdown_connections 2024-04-25T02:06:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.688+0000 7f7ed1a00640 1 -- 172.21.15.102:0/981689315 wait complete. 2024-04-25T02:06:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7ed1a00640 1 Processor -- start 2024-04-25T02:06:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7ed1a00640 1 -- start start 2024-04-25T02:06:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7ed1a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc0ff810 0x7f7ecc19b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7ed1a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc1001d0 0x7f7ecc19be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7ed1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ecc106860 0x7f7ecc195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7ed1a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc10f920 con 0x7f7ecc106860 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7ed1a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc10f7a0 con 0x7f7ecc0ff810 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7ed1a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc10faa0 con 0x7f7ecc1001d0 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7eca000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc1001d0 0x7f7ecc19be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7eca000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc1001d0 0x7f7ecc19be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:37796/0 (socket says 172.21.15.102:37796) 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7eca000640 1 -- 172.21.15.102:0/4008834880 learned_addr learned my addr 172.21.15.102:0/4008834880 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.689+0000 7f7ecb400640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ecc106860 0x7f7ecc195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc0ff810 0x7f7ecc19b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7eca000640 1 -- 172.21.15.102:0/4008834880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc0ff810 msgr2=0x7f7ecc19b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7eca000640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc0ff810 0x7f7ecc19b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7eca000640 1 -- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ecc106860 msgr2=0x7f7ecc195b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7eca000640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ecc106860 0x7f7ecc195b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7eca000640 1 -- 172.21.15.102:0/4008834880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ecc1962a0 con 0x7f7ecc1001d0 2024-04-25T02:06:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc0ff810 0x7f7ecc19b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:06:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7ecb400640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ecc106860 0x7f7ecc195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:06:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7eca000640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc1001d0 0x7f7ecc19be60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8035040 tx=0x7f7eb8033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7ec3400640 1 -- 172.21.15.102:0/4008834880 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8037040 con 0x7f7ecc1001d0 2024-04-25T02:06:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7ec3400640 1 -- 172.21.15.102:0/4008834880 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7eb8040070 con 0x7f7ecc1001d0 2024-04-25T02:06:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7ec3400640 1 -- 172.21.15.102:0/4008834880 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8003a50 con 0x7f7ecc1001d0 2024-04-25T02:06:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7ed1a00640 1 -- 172.21.15.102:0/4008834880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ecc196530 con 0x7f7ecc1001d0 2024-04-25T02:06:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.690+0000 7f7ed1a00640 1 -- 172.21.15.102:0/4008834880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ecc1a27e0 con 0x7f7ecc1001d0 2024-04-25T02:06:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.694+0000 7f7ed4780740 1 Processor -- start 2024-04-25T02:06:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.694+0000 7f7ed4780740 1 -- start start 2024-04-25T02:06:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.695+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786cdce0 0x5599786ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.695+0000 7f7ec3400640 1 -- 172.21.15.102:0/4008834880 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7eb8003bf0 con 0x7f7ecc1001d0 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.695+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.695+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786e9ff0 0x5599786ec3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.695+0000 7f7ed4780740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559978631ab0 con 0x5599786cdce0 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.695+0000 7f7ed4780740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5599785da640 con 0x5599786e9ff0 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.695+0000 7f7ed4780740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55997862da70 con 0x5599786ce600 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.696+0000 7f7ecb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786cdce0 0x5599786ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.696+0000 7f7ecaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.696+0000 7f7ecaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:37808/0 (socket says 172.21.15.102:37808) 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.696+0000 7f7ecaa00640 1 -- 172.21.15.102:0/1883693821 learned_addr learned my addr 172.21.15.102:0/1883693821 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.696+0000 7f7ecaa00640 1 -- 172.21.15.102:0/1883693821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786e9ff0 msgr2=0x5599786ec3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.696+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/1883693821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786e9ff0 0x5599786ec3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.696+0000 7f7ecaa00640 1 -- 172.21.15.102:0/1883693821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786cdce0 msgr2=0x5599786ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.696+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/1883693821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786cdce0 0x5599786ce0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.696+0000 7f7ecaa00640 1 -- 172.21.15.102:0/1883693821 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559978346210 con 0x5599786ce600 2024-04-25T02:06:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.697+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/1883693821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e9ab0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009cb0 tx=0x7f7ebc0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f49d4aea32d17456 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.697+0000 7f7ec3400640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7eb00a2e20 0x7f7eb00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.697+0000 7f7ec3400640 1 -- 172.21.15.102:0/4008834880 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f7eb80ef570 con 0x7f7ecc1001d0 2024-04-25T02:06:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.697+0000 7f7eb7e00640 1 -- 172.21.15.102:0/1883693821 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc005020 con 0x5599786ce600 2024-04-25T02:06:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.698+0000 7f7eb7e00640 1 -- 172.21.15.102:0/1883693821 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ebc0047a0 con 0x5599786ce600 2024-04-25T02:06:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.698+0000 7f7eb7e00640 1 -- 172.21.15.102:0/1883693821 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc012890 con 0x5599786ce600 2024-04-25T02:06:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.698+0000 7f7ecb400640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7eb00a2e20 0x7f7eb00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.699+0000 7f7ed4780740 1 -- 172.21.15.102:0/1883693821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 msgr2=0x5599786e9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.699+0000 7f7ed4780740 1 --2- 172.21.15.102:0/1883693821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e9ab0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009cb0 tx=0x7f7ebc0117b0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.699+0000 7f7ed4780740 1 -- 172.21.15.102:0/1883693821 shutdown_connections 2024-04-25T02:06:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.699+0000 7f7ed4780740 1 --2- 172.21.15.102:0/1883693821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786e9ff0 0x5599786ec3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.699+0000 7f7ed4780740 1 --2- 172.21.15.102:0/1883693821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e9ab0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.699+0000 7f7ed4780740 1 --2- 172.21.15.102:0/1883693821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786cdce0 0x5599786ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.699+0000 7f7ed4780740 1 -- 172.21.15.102:0/1883693821 >> 172.21.15.102:0/1883693821 conn(0x5599786ca310 msgr2=0x5599786ca6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.700+0000 7f7ed4780740 1 -- 172.21.15.102:0/1883693821 shutdown_connections 2024-04-25T02:06:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.700+0000 7f7ecb400640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7eb00a2e20 0x7f7eb00a52e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4005fd0 tx=0x7f7ec400d040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.700+0000 7f7ed4780740 1 -- 172.21.15.102:0/1883693821 wait complete. 2024-04-25T02:06:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.700+0000 7f7ed4780740 1 -- start start 2024-04-25T02:06:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.701+0000 7f7ed4780740 1 Processor -- start 2024-04-25T02:06:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.701+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.701+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786ce600 0x5599786e7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.701+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786e9ff0 0x5599786e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.701+0000 7f7ed4780740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5599786ecc20 con 0x5599786ce600 2024-04-25T02:06:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.701+0000 7f7ed4780740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5599785ed990 con 0x5599786cdce0 2024-04-25T02:06:17.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.701+0000 7f7ed4780740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5599786ecda0 con 0x5599786e9ff0 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.701+0000 7f7ed4780740 10 client.0 _ll_register_callbacks cb 0x5599785ef950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.701+0000 7f7ecb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786e9ff0 0x5599786e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7ecaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7eca000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786ce600 0x5599786e7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7ecb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786e9ff0 0x5599786e7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:37812/0 (socket says 172.21.15.102:37812) 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7ecb400640 1 -- 172.21.15.102:0/428247785 learned_addr learned my addr 172.21.15.102:0/428247785 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7ecaa00640 1 -- 172.21.15.102:0/428247785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786e9ff0 msgr2=0x5599786e7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786e9ff0 0x5599786e7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7ecaa00640 1 -- 172.21.15.102:0/428247785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786ce600 msgr2=0x5599786e7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786ce600 0x5599786e7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7ecaa00640 1 -- 172.21.15.102:0/428247785 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559978412620 con 0x5599786cdce0 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e7040 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000c00 tx=0x7f7ebc009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e26d5dfd6afccf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:17.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.702+0000 7f7eb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:06:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.703+0000 7f7eb4c00640 1 -- 172.21.15.102:0/428247785 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc028020 con 0x5599786cdce0 2024-04-25T02:06:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.703+0000 7f7eb4c00640 1 -- 172.21.15.102:0/428247785 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f7ebc005200 con 0x5599786cdce0 2024-04-25T02:06:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.703+0000 7f7ed4780740 1 -- 172.21.15.102:0/428247785 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ebc0047a0 con 0x5599786cdce0 2024-04-25T02:06:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.703+0000 7f7eb4c00640 1 -- 172.21.15.102:0/428247785 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7ebc01f040 con 0x5599786cdce0 2024-04-25T02:06:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.703+0000 7f7ed4780740 10 client.27413 fetch_fsmap learned FSMap version 42 2024-04-25T02:06:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.703+0000 7f7ed4780740 1 -- 172.21.15.102:0/428247785 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5599786f8c30 con 0x5599786cdce0 2024-04-25T02:06:17.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.703+0000 7f7eb4c00640 1 -- 172.21.15.102:0/428247785 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ebc0047a0 con 0x5599786cdce0 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.703+0000 7f7eb4c00640 1 -- 172.21.15.102:0/428247785 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7f7ebc024410 con 0x5599786cdce0 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.704+0000 7f7ed4780740 10 client.27413 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.704+0000 7f7ed4780740 10 client.27413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.704+0000 7f7ed4780740 20 client.27413 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.704+0000 7f7ed4780740 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5599786fa090 0x5599786fc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.704+0000 7f7ed4780740 4 client.27413 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.704+0000 7f7ed4780740 1 -- 172.21.15.102:0/428247785 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55997863eff0 con 0x5599786fa090 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.704+0000 7f7eca000640 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5599786fa090 0x5599786fc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.704+0000 7f7eca000640 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5599786fa090 0x5599786fc480 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a687082abc38d3ca server_cookie=b1f8ad7032a5361e in_seq=0 out_seq=0 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.704+0000 7f7eb4c00640 10 client.27413 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7eb4c00640 1 -- 172.21.15.102:0/428247785 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55997863eff0 con 0x5599786fa090 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7eb4c00640 10 client.27413 handle_command_reply: tid=0 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7ed4780740 1 client.27413 shutdown 2024-04-25T02:06:17.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7ed4780740 1 -- 172.21.15.102:0/428247785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 msgr2=0x5599786e7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7ed4780740 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e7040 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000c00 tx=0x7f7ebc009fd0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7ed4780740 1 -- 172.21.15.102:0/428247785 shutdown_connections 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7ed4780740 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5599786fa090 0x5599786fc480 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7ed4780740 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786e9ff0 0x5599786e7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7eb4c00640 0 client.27413 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7ed4780740 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786ce600 0x5599786e7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7ed4780740 1 --2- 172.21.15.102:0/428247785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e7040 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.705+0000 7f7ed4780740 1 -- 172.21.15.102:0/428247785 >> 172.21.15.102:0/428247785 conn(0x5599786ca310 msgr2=0x5599786d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.706+0000 7f7ed4780740 1 -- 172.21.15.102:0/428247785 shutdown_connections 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.706+0000 7f7ed4780740 1 -- 172.21.15.102:0/428247785 wait complete. 2024-04-25T02:06:17.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.706+0000 7f7ed4780740 20 client.27413 trim_cache size 0 max 16384 2024-04-25T02:06:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7ed4780740 1 Processor -- start 2024-04-25T02:06:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7ed4780740 1 -- start start 2024-04-25T02:06:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 0x5599786cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7ed4780740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc1a5260 con 0x5599786e9ff0 2024-04-25T02:06:17.739 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7ed4780740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc1a2af0 con 0x5599786cdce0 2024-04-25T02:06:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7ed4780740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc196fc0 con 0x5599786ce600 2024-04-25T02:06:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7ecaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.732+0000 7f7eca000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7ecaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786f5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54806/0 (socket says 172.21.15.102:54806) 2024-04-25T02:06:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7ecaa00640 1 -- 172.21.15.102:0/2261351125 learned_addr learned my addr 172.21.15.102:0/2261351125 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7ecb400640 1 --2- 172.21.15.102:0/2261351125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 0x5599786cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7ecaa00640 1 -- 172.21.15.102:0/2261351125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 msgr2=0x5599786c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/2261351125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786c9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.740 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7ecaa00640 1 -- 172.21.15.102:0/2261351125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 msgr2=0x5599786cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/2261351125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 0x5599786cff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7ecaa00640 1 -- 172.21.15.102:0/2261351125 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599786ca140 con 0x5599786cdce0 2024-04-25T02:06:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7eca000640 1 --2- 172.21.15.102:0/2261351125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/2261351125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786f5b20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc001eb0 tx=0x7f7ebc0092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8118a7fd6f195618 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.733+0000 7f7eb7400640 1 -- 172.21.15.102:0/2261351125 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc028070 con 0x5599786cdce0 2024-04-25T02:06:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7eb7400640 1 -- 172.21.15.102:0/2261351125 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ebc011cb0 con 0x5599786cdce0 2024-04-25T02:06:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7eb7400640 1 -- 172.21.15.102:0/2261351125 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc033030 con 0x5599786cdce0 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7ed4780740 1 -- 172.21.15.102:0/2261351125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 msgr2=0x5599786f5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2261351125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786f5b20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc001eb0 tx=0x7f7ebc0092e0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7ed4780740 1 -- 172.21.15.102:0/2261351125 shutdown_connections 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2261351125 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 0x5599786cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2261351125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2261351125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786f5b20 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7ed4780740 1 -- 172.21.15.102:0/2261351125 >> 172.21.15.102:0/2261351125 conn(0x5599786ca310 msgr2=0x5599786cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7ed4780740 1 -- 172.21.15.102:0/2261351125 shutdown_connections 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.734+0000 7f7ed4780740 1 -- 172.21.15.102:0/2261351125 wait complete. 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.735+0000 7f7ed4780740 1 -- start start 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.735+0000 7f7ed4780740 1 Processor -- start 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.735+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.735+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.735+0000 7f7ed4780740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 0x5599786e46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.736+0000 7f7ed4780740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc071fc0 con 0x5599786e9ff0 2024-04-25T02:06:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.736+0000 7f7ed4780740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc073430 con 0x5599786cdce0 2024-04-25T02:06:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.736+0000 7f7ed4780740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc072690 con 0x5599786ce600 2024-04-25T02:06:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.736+0000 7f7ed4780740 10 client.0 _ll_register_callbacks cb 0x7f7ebc009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.736+0000 7f7ecaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.736+0000 7f7ecaa00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54814/0 (socket says 172.21.15.102:54814) 2024-04-25T02:06:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.736+0000 7f7ecaa00640 1 -- 172.21.15.102:0/2389051096 learned_addr learned my addr 172.21.15.102:0/2389051096 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.736+0000 7f7eca000640 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.736+0000 7f7ecb400640 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 0x5599786e46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.737+0000 7f7ecaa00640 1 -- 172.21.15.102:0/2389051096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 msgr2=0x5599786e4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.737+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.737+0000 7f7ecaa00640 1 -- 172.21.15.102:0/2389051096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 msgr2=0x5599786e46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.737+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 0x5599786e46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.737+0000 7f7ecaa00640 1 -- 172.21.15.102:0/2389051096 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55997875f380 con 0x5599786cdce0 2024-04-25T02:06:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.737+0000 7f7ecb400640 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 0x5599786e46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.738+0000 7f7ecaa00640 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e3c30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc024b30 tx=0x7f7ebc005490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff74c608991bb1c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.739+0000 7f7eb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:06:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.740+0000 7f7eb4c00640 1 -- 172.21.15.102:0/2389051096 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc028070 con 0x5599786cdce0 2024-04-25T02:06:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.740+0000 7f7eb4c00640 1 -- 172.21.15.102:0/2389051096 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f7ebc024bd0 con 0x5599786cdce0 2024-04-25T02:06:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.740+0000 7f7ed4780740 1 -- 172.21.15.102:0/2389051096 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ebc011cb0 con 0x5599786cdce0 2024-04-25T02:06:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.740+0000 7f7eb4c00640 1 -- 172.21.15.102:0/2389051096 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f7ebc033040 con 0x5599786cdce0 2024-04-25T02:06:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.741+0000 7f7ed4780740 10 client.27419 fetch_fsmap learned FSMap version 42 2024-04-25T02:06:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.741+0000 7f7ed4780740 1 -- 172.21.15.102:0/2389051096 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5599787605c0 con 0x5599786cdce0 2024-04-25T02:06:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.741+0000 7f7eb4c00640 1 -- 172.21.15.102:0/2389051096 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ebc011cb0 con 0x5599786cdce0 2024-04-25T02:06:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.741+0000 7f7eb4c00640 1 -- 172.21.15.102:0/2389051096 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7f7ebc03f030 con 0x5599786cdce0 2024-04-25T02:06:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.742+0000 7f7ed4780740 10 client.27419 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T02:06:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.742+0000 7f7ed4780740 10 client.27419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.742+0000 7f7ed4780740 20 client.27419 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.742+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559978760ca0 0x5599787630c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.742+0000 7f7ed4780740 4 client.27419 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:06:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.742+0000 7f7ed4780740 1 -- 172.21.15.102:0/2389051096 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559978639340 con 0x559978760ca0 2024-04-25T02:06:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.742+0000 7f7eca000640 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559978760ca0 0x5599787630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:17.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.743+0000 7f7eca000640 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559978760ca0 0x5599787630c0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b41a48fc82f5146b server_cookie=464b3d64d22b7b40 in_seq=0 out_seq=0 2024-04-25T02:06:17.749 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.743+0000 7f7eb4c00640 10 client.27419 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.760+0000 7f7eb4c00640 1 -- 172.21.15.102:0/2389051096 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559978639340 con 0x559978760ca0 2024-04-25T02:06:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.760+0000 7f7eb4c00640 10 client.27419 handle_command_reply: tid=0 2024-04-25T02:06:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 client.27419 shutdown 2024-04-25T02:06:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 -- 172.21.15.102:0/2389051096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 msgr2=0x5599786e3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e3c30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc024b30 tx=0x7f7ebc005490 comp rx=0 tx=0).stop 2024-04-25T02:06:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 -- 172.21.15.102:0/2389051096 shutdown_connections 2024-04-25T02:06:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559978760ca0 0x5599787630c0 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5599786e9ff0 0x5599786e46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599786ce600 0x5599786e4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 --2- 172.21.15.102:0/2389051096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5599786cdce0 0x5599786e3c30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 -- 172.21.15.102:0/2389051096 >> 172.21.15.102:0/2389051096 conn(0x5599786ca310 msgr2=0x5599786d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7eb4c00640 0 client.27419 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 -- 172.21.15.102:0/2389051096 shutdown_connections 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 1 -- 172.21.15.102:0/2389051096 wait complete. 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.761+0000 7f7ed4780740 20 client.27419 trim_cache size 0 max 16384 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:06:17.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:06:17.769 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:06:17.769 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 -- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7eb00a2e20 msgr2=0x7f7eb00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7eb00a2e20 0x7f7eb00a52e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4005fd0 tx=0x7f7ec400d040 comp rx=0 tx=0).stop 2024-04-25T02:06:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 -- 172.21.15.102:0/4008834880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc1001d0 msgr2=0x7f7ecc19be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc1001d0 0x7f7ecc19be60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8035040 tx=0x7f7eb8033d90 comp rx=0 tx=0).stop 2024-04-25T02:06:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 -- 172.21.15.102:0/4008834880 shutdown_connections 2024-04-25T02:06:17.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f7eb00a2e20 0x7f7eb00a52e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ecc106860 0x7f7ecc195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7ecc1001d0 0x7f7ecc19be60 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 --2- 172.21.15.102:0/4008834880 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ecc0ff810 0x7f7ecc19b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.764+0000 7f7ed1a00640 1 -- 172.21.15.102:0/4008834880 >> 172.21.15.102:0/4008834880 conn(0x7f7ecc076740 msgr2=0x7f7ecc0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:17.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.765+0000 7f7ed1a00640 1 -- 172.21.15.102:0/4008834880 shutdown_connections 2024-04-25T02:06:17.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:17.766+0000 7f7ed1a00640 1 -- 172.21.15.102:0/4008834880 wait complete. 2024-04-25T02:06:18.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 625.0 s, 1641.8 tps, lat 60.190 ms stddev 71.689 2024-04-25T02:06:18.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:18 smithi102 ceph-mon[27796]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T02:06:18.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:18 smithi110 ceph-mon[32848]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T02:06:18.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:18 smithi193 ceph-mon[32981]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T02:06:19.681 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:19 smithi102 ceph-mon[27796]: pgmap v342: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.1 MiB/s wr, 636 op/s 2024-04-25T02:06:19.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:19 smithi110 ceph-mon[32848]: pgmap v342: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.1 MiB/s wr, 636 op/s 2024-04-25T02:06:19.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:19 smithi193 ceph-mon[32981]: pgmap v342: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.1 MiB/s wr, 636 op/s 2024-04-25T02:06:21.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:21 smithi110 ceph-mon[32848]: pgmap v343: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.1 MiB/s wr, 636 op/s 2024-04-25T02:06:21.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:21 smithi102 ceph-mon[27796]: pgmap v343: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.1 MiB/s wr, 636 op/s 2024-04-25T02:06:21.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:21 smithi193 ceph-mon[32981]: pgmap v343: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.1 MiB/s wr, 636 op/s 2024-04-25T02:06:23.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 630.0 s, 1413.2 tps, lat 70.991 ms stddev 82.377 2024-04-25T02:06:23.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:23 smithi102 ceph-mon[27796]: pgmap v344: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.3 MiB/s wr, 595 op/s 2024-04-25T02:06:23.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:23 smithi110 ceph-mon[32848]: pgmap v344: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.3 MiB/s wr, 595 op/s 2024-04-25T02:06:23.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:23 smithi193 ceph-mon[32981]: pgmap v344: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.3 MiB/s wr, 595 op/s 2024-04-25T02:06:24.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:24 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:06:24.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:24 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:06:24.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:24 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:06:25.678 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:25 smithi102 ceph-mon[27796]: pgmap v345: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.2 MiB/s wr, 650 op/s 2024-04-25T02:06:25.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:25 smithi110 ceph-mon[32848]: pgmap v345: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.2 MiB/s wr, 650 op/s 2024-04-25T02:06:25.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:25 smithi193 ceph-mon[32981]: pgmap v345: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.2 MiB/s wr, 650 op/s 2024-04-25T02:06:27.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:27 smithi102 ceph-mon[27796]: pgmap v346: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.2 MiB/s wr, 650 op/s 2024-04-25T02:06:27.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:27 smithi110 ceph-mon[32848]: pgmap v346: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.2 MiB/s wr, 650 op/s 2024-04-25T02:06:27.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:27 smithi193 ceph-mon[32981]: pgmap v346: 97 pgs: 97 active+clean; 4.9 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.2 MiB/s wr, 650 op/s 2024-04-25T02:06:28.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 635.0 s, 1952.0 tps, lat 51.701 ms stddev 64.523 2024-04-25T02:06:29.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:29 smithi110 ceph-mon[32848]: pgmap v347: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 7.6 MiB/s wr, 788 op/s 2024-04-25T02:06:29.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:29 smithi193 ceph-mon[32981]: pgmap v347: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 7.6 MiB/s wr, 788 op/s 2024-04-25T02:06:29.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:29 smithi102 ceph-mon[27796]: pgmap v347: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 7.6 MiB/s wr, 788 op/s 2024-04-25T02:06:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.945+0000 7f3dfc000640 1 -- 172.21.15.102:0/3961669847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3df4109930 msgr2=0x7f3df4109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.945+0000 7f3dfc000640 1 --2- 172.21.15.102:0/3961669847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3df4109930 0x7f3df4109d10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009930 tx=0x7f3de802a420 comp rx=0 tx=0).stop 2024-04-25T02:06:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.945+0000 7f3dfc000640 1 -- 172.21.15.102:0/3961669847 shutdown_connections 2024-04-25T02:06:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.945+0000 7f3dfc000640 1 --2- 172.21.15.102:0/3961669847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df41042e0 0x7f3df41107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.945+0000 7f3dfc000640 1 --2- 172.21.15.102:0/3961669847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df4103920 0x7f3df4103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.945+0000 7f3dfc000640 1 --2- 172.21.15.102:0/3961669847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3df4109930 0x7f3df4109d10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.945+0000 7f3dfc000640 1 -- 172.21.15.102:0/3961669847 >> 172.21.15.102:0/3961669847 conn(0x7f3df40ff430 msgr2=0x7f3df4101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.945+0000 7f3dfc000640 1 -- 172.21.15.102:0/3961669847 shutdown_connections 2024-04-25T02:06:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.945+0000 7f3dfc000640 1 -- 172.21.15.102:0/3961669847 wait complete. 2024-04-25T02:06:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.947+0000 7f3dfc000640 1 Processor -- start 2024-04-25T02:06:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3dfc000640 1 -- start start 2024-04-25T02:06:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3dfc000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df4103920 0x7f3df4075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3dfc000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3df41042e0 0x7f3df4076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3dfc000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df4109930 0x7f3df40766d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3dfc000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3df4113d60 con 0x7f3df4109930 2024-04-25T02:06:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3dfc000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3df4113be0 con 0x7f3df4103920 2024-04-25T02:06:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3dfc000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3df4113ee0 con 0x7f3df41042e0 2024-04-25T02:06:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3df9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df4103920 0x7f3df4075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3df9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df4103920 0x7f3df4075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51204/0 (socket says 172.21.15.102:51204) 2024-04-25T02:06:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.948+0000 7f3df9600640 1 -- 172.21.15.102:0/2374084253 learned_addr learned my addr 172.21.15.102:0/2374084253 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.949+0000 7f3df8c00640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3df41042e0 0x7f3df4076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.956 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.949+0000 7f3df9600640 1 -- 172.21.15.102:0/2374084253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3df41042e0 msgr2=0x7f3df4076190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:30.956 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.949+0000 7f3df9600640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3df41042e0 0x7f3df4076190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.956 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.949+0000 7f3df9600640 1 -- 172.21.15.102:0/2374084253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df4109930 msgr2=0x7f3df40766d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:30.956 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.949+0000 7f3df9600640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df4109930 0x7f3df40766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.956 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.949+0000 7f3df9600640 1 -- 172.21.15.102:0/2374084253 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df4199fa0 con 0x7f3df4103920 2024-04-25T02:06:30.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.949+0000 7f3df9600640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df4103920 0x7f3df4075c50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3de8003b90 tx=0x7f3de8003bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:30.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.949+0000 7f3df2000640 1 -- 172.21.15.102:0/2374084253 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de803f070 con 0x7f3df4103920 2024-04-25T02:06:30.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.950+0000 7f3df2000640 1 -- 172.21.15.102:0/2374084253 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3de8031c90 con 0x7f3df4103920 2024-04-25T02:06:30.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.950+0000 7f3df2000640 1 -- 172.21.15.102:0/2374084253 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8035030 con 0x7f3df4103920 2024-04-25T02:06:30.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.950+0000 7f3dfc000640 1 -- 172.21.15.102:0/2374084253 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df419a230 con 0x7f3df4103920 2024-04-25T02:06:30.957 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.950+0000 7f3dfc000640 1 -- 172.21.15.102:0/2374084253 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df419a6d0 con 0x7f3df4103920 2024-04-25T02:06:30.964 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3dfef70740 1 Processor -- start 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3dfef70740 1 -- start start 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3df2000640 1 -- 172.21.15.102:0/2374084253 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3de8042070 con 0x7f3df4103920 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ade00 0x55be131ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ae720 0x55be131b8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131bb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3dfef70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55be13111ab0 con 0x55be131ade00 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3dfef70740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55be130ba640 con 0x55be131ae720 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3dfef70740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55be1310da70 con 0x55be131b8fe0 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3df9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ae720 0x55be131b8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.958+0000 7f3dfa000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ade00 0x55be131ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.959+0000 7f3df9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ae720 0x55be131b8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51212/0 (socket says 172.21.15.102:51212) 2024-04-25T02:06:30.965 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.959+0000 7f3df9600640 1 -- 172.21.15.102:0/4103401025 learned_addr learned my addr 172.21.15.102:0/4103401025 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.959+0000 7f3df8c00640 1 --2- 172.21.15.102:0/4103401025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131bb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.959+0000 7f3df9600640 1 -- 172.21.15.102:0/4103401025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 msgr2=0x55be131bb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.959+0000 7f3df9600640 1 --2- 172.21.15.102:0/4103401025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131bb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.959+0000 7f3df9600640 1 -- 172.21.15.102:0/4103401025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ade00 msgr2=0x55be131ae1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.959+0000 7f3df9600640 1 --2- 172.21.15.102:0/4103401025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ade00 0x55be131ae1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.959+0000 7f3df9600640 1 -- 172.21.15.102:0/4103401025 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be12e26210 con 0x55be131ae720 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.959+0000 7f3df8c00640 1 --2- 172.21.15.102:0/4103401025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131bb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.960+0000 7f3df9600640 1 --2- 172.21.15.102:0/4103401025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ae720 0x55be131b8aa0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3de80ed570 tx=0x7f3de80f8220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7548a8a4154e6bf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.960+0000 7f3df2000640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3ddc0a2e20 0x7f3ddc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.960+0000 7f3df2000640 1 -- 172.21.15.102:0/2374084253 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f3de80efeb0 con 0x7f3df4103920 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.960+0000 7f3de6600640 1 -- 172.21.15.102:0/4103401025 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de80b8670 con 0x55be131ae720 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.961+0000 7f3de6600640 1 -- 172.21.15.102:0/4103401025 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3de80fd070 con 0x55be131ae720 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.961+0000 7f3de6600640 1 -- 172.21.15.102:0/4103401025 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de80bbd30 con 0x55be131ae720 2024-04-25T02:06:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.961+0000 7f3dfa000640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3ddc0a2e20 0x7f3ddc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfef70740 1 -- 172.21.15.102:0/4103401025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ae720 msgr2=0x55be131b8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfa000640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3ddc0a2e20 0x7f3ddc0a52e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3dec005c40 tx=0x7f3dec009460 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfef70740 1 --2- 172.21.15.102:0/4103401025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ae720 0x55be131b8aa0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3de80ed570 tx=0x7f3de80f8220 comp rx=0 tx=0).stop 2024-04-25T02:06:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfef70740 1 -- 172.21.15.102:0/4103401025 shutdown_connections 2024-04-25T02:06:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfef70740 1 --2- 172.21.15.102:0/4103401025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131bb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfef70740 1 --2- 172.21.15.102:0/4103401025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ae720 0x55be131b8aa0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfef70740 1 --2- 172.21.15.102:0/4103401025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ade00 0x55be131ae1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfef70740 1 -- 172.21.15.102:0/4103401025 >> 172.21.15.102:0/4103401025 conn(0x55be131aa310 msgr2=0x55be131aa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfef70740 1 -- 172.21.15.102:0/4103401025 shutdown_connections 2024-04-25T02:06:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.962+0000 7f3dfef70740 1 -- 172.21.15.102:0/4103401025 wait complete. 2024-04-25T02:06:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.963+0000 7f3dfef70740 1 -- start start 2024-04-25T02:06:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.963+0000 7f3dfef70740 1 Processor -- start 2024-04-25T02:06:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.963+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.963+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 0x55be131b64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.963+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131b6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.963+0000 7f3dfef70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55be131bb9d0 con 0x55be131ae720 2024-04-25T02:06:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.963+0000 7f3dfef70740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55be130cd990 con 0x55be131ade00 2024-04-25T02:06:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.963+0000 7f3dfef70740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55be131bbb50 con 0x55be131b8fe0 2024-04-25T02:06:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.964+0000 7f3dfef70740 10 client.0 _ll_register_callbacks cb 0x55be130cf950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:30.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.964+0000 7f3df9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.971 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.964+0000 7f3df9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131b5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51222/0 (socket says 172.21.15.102:51222) 2024-04-25T02:06:30.971 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.964+0000 7f3df9600640 1 -- 172.21.15.102:0/2698259017 learned_addr learned my addr 172.21.15.102:0/2698259017 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:30.971 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.964+0000 7f3df8c00640 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 0x55be131b64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.971 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.964+0000 7f3dfa000640 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131b6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.965+0000 7f3df9600640 1 -- 172.21.15.102:0/2698259017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 msgr2=0x55be131b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:30.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.965+0000 7f3df9600640 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131b6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.965+0000 7f3df9600640 1 -- 172.21.15.102:0/2698259017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 msgr2=0x55be131b64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:30.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.965+0000 7f3df9600640 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 0x55be131b64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.965+0000 7f3df9600640 1 -- 172.21.15.102:0/2698259017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be12ef2620 con 0x55be131ade00 2024-04-25T02:06:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.965+0000 7f3dfa000640 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.965+0000 7f3df9600640 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131b5fa0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f3de80ed6b0 tx=0x7f3de802a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=874729e485045fdb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.965+0000 7f3ddaa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:06:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.966+0000 7f3ddaa00640 1 -- 172.21.15.102:0/2698259017 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8049030 con 0x55be131ade00 2024-04-25T02:06:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.966+0000 7f3ddaa00640 1 -- 172.21.15.102:0/2698259017 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f3de80ed880 con 0x55be131ade00 2024-04-25T02:06:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.966+0000 7f3dfef70740 1 -- 172.21.15.102:0/2698259017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3de80fd070 con 0x55be131ade00 2024-04-25T02:06:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.966+0000 7f3ddaa00640 1 -- 172.21.15.102:0/2698259017 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3de80bbf10 con 0x55be131ade00 2024-04-25T02:06:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.966+0000 7f3dfef70740 10 client.27431 fetch_fsmap learned FSMap version 43 2024-04-25T02:06:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.966+0000 7f3dfef70740 1 -- 172.21.15.102:0/2698259017 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55be131bb9d0 con 0x55be131ade00 2024-04-25T02:06:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.967+0000 7f3ddaa00640 1 -- 172.21.15.102:0/2698259017 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3de80fd070 con 0x55be131ade00 2024-04-25T02:06:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.967+0000 7f3ddaa00640 1 -- 172.21.15.102:0/2698259017 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f3de80b8a40 con 0x55be131ade00 2024-04-25T02:06:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.967+0000 7f3dfef70740 10 client.27431 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:06:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.967+0000 7f3dfef70740 10 client.27431 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.967+0000 7f3dfef70740 20 client.27431 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.967+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55be131c8a00 0x55be131cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.967+0000 7f3dfef70740 4 client.27431 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:06:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.967+0000 7f3dfef70740 1 -- 172.21.15.102:0/2698259017 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be1311eff0 con 0x55be131c8a00 2024-04-25T02:06:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.967+0000 7f3df8c00640 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55be131c8a00 0x55be131cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.968+0000 7f3ddaa00640 10 client.27431 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.969+0000 7f3df8c00640 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55be131c8a00 0x55be131cae10 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc2db0376664b878 server_cookie=5b2b39bd790c0d64 in_seq=0 out_seq=0 2024-04-25T02:06:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.970+0000 7f3ddaa00640 1 -- 172.21.15.102:0/2698259017 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be1311eff0 con 0x55be131c8a00 2024-04-25T02:06:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.970+0000 7f3ddaa00640 10 client.27431 handle_command_reply: tid=0 2024-04-25T02:06:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.970+0000 7f3dfef70740 1 client.27431 shutdown 2024-04-25T02:06:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.970+0000 7f3dfef70740 1 -- 172.21.15.102:0/2698259017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 msgr2=0x55be131b5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.970+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131b5fa0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f3de80ed6b0 tx=0x7f3de802a420 comp rx=0 tx=0).stop 2024-04-25T02:06:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.971+0000 7f3dfef70740 1 -- 172.21.15.102:0/2698259017 shutdown_connections 2024-04-25T02:06:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.971+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55be131c8a00 0x55be131cae10 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.971+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131b8fe0 0x55be131b6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.971+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 0x55be131b64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.971+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2698259017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131b5fa0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.971+0000 7f3dfef70740 1 -- 172.21.15.102:0/2698259017 >> 172.21.15.102:0/2698259017 conn(0x55be131aa310 msgr2=0x55be131b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.971+0000 7f3ddaa00640 0 client.27431 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.971+0000 7f3dfef70740 1 -- 172.21.15.102:0/2698259017 shutdown_connections 2024-04-25T02:06:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.972+0000 7f3dfef70740 1 -- 172.21.15.102:0/2698259017 wait complete. 2024-04-25T02:06:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.972+0000 7f3dfef70740 20 client.27431 trim_cache size 0 max 16384 2024-04-25T02:06:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.997+0000 7f3dfef70740 1 Processor -- start 2024-04-25T02:06:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.997+0000 7f3dfef70740 1 -- start start 2024-04-25T02:06:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ade00 0x55be131c4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 0x55be131b6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131b8fe0 0x55be131d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3dfef70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3df41bebd0 con 0x55be131ae720 2024-04-25T02:06:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3dfef70740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3df41bc410 con 0x55be131b8fe0 2024-04-25T02:06:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3dfef70740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3df41a1220 con 0x55be131ade00 2024-04-25T02:06:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3dfa000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131b8fe0 0x55be131d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3dfa000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131b8fe0 0x55be131d0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51232/0 (socket says 172.21.15.102:51232) 2024-04-25T02:06:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3dfa000640 1 -- 172.21.15.102:0/2850707520 learned_addr learned my addr 172.21.15.102:0/2850707520 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.998+0000 7f3df9600640 1 --2- 172.21.15.102:0/2850707520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ade00 0x55be131c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.999+0000 7f3dfa000640 1 -- 172.21.15.102:0/2850707520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ade00 msgr2=0x55be131c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.999+0000 7f3dfa000640 1 --2- 172.21.15.102:0/2850707520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ade00 0x55be131c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.999+0000 7f3df8c00640 1 --2- 172.21.15.102:0/2850707520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 0x55be131b6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.999+0000 7f3dfa000640 1 -- 172.21.15.102:0/2850707520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 msgr2=0x55be131b6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.999+0000 7f3dfa000640 1 --2- 172.21.15.102:0/2850707520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 0x55be131b6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.999+0000 7f3dfa000640 1 -- 172.21.15.102:0/2850707520 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be131a8890 con 0x55be131b8fe0 2024-04-25T02:06:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:30.999+0000 7f3dfa000640 1 --2- 172.21.15.102:0/2850707520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131b8fe0 0x55be131d0720 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0125e0 tx=0x7f3dec013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81a4637c186f865e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.000+0000 7f3de5c00640 1 -- 172.21.15.102:0/2850707520 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec0117a0 con 0x55be131b8fe0 2024-04-25T02:06:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.000+0000 7f3de5c00640 1 -- 172.21.15.102:0/2850707520 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dec011e20 con 0x55be131b8fe0 2024-04-25T02:06:31.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.000+0000 7f3de5c00640 1 -- 172.21.15.102:0/2850707520 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec0117a0 con 0x55be131b8fe0 2024-04-25T02:06:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.001+0000 7f3dfef70740 1 -- 172.21.15.102:0/2850707520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131b8fe0 msgr2=0x55be131d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:31.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.001+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2850707520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131b8fe0 0x55be131d0720 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3dec0125e0 tx=0x7f3dec013000 comp rx=0 tx=0).stop 2024-04-25T02:06:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.001+0000 7f3dfef70740 1 -- 172.21.15.102:0/2850707520 shutdown_connections 2024-04-25T02:06:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.002+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2850707520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131b8fe0 0x55be131d0720 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.002+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2850707520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131ae720 0x55be131b6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.002+0000 7f3dfef70740 1 --2- 172.21.15.102:0/2850707520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ade00 0x55be131c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.002+0000 7f3dfef70740 1 -- 172.21.15.102:0/2850707520 >> 172.21.15.102:0/2850707520 conn(0x55be131aa310 msgr2=0x55be131b77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.002+0000 7f3dfef70740 1 -- 172.21.15.102:0/2850707520 shutdown_connections 2024-04-25T02:06:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.002+0000 7f3dfef70740 1 -- 172.21.15.102:0/2850707520 wait complete. 2024-04-25T02:06:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.002+0000 7f3dfef70740 1 -- start start 2024-04-25T02:06:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.002+0000 7f3dfef70740 1 Processor -- start 2024-04-25T02:06:31.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131bc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:31.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ae720 0x55be131bce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:31.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3dfef70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131b8fe0 0x55be131bd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:31.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3dfef70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3df41a1870 con 0x55be131b8fe0 2024-04-25T02:06:31.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3dfef70740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3df41bbd40 con 0x55be131ade00 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3dfef70740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3df41a1f40 con 0x55be131ae720 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3dfef70740 10 client.0 _ll_register_callbacks cb 0x7f3de80f3eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3df9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131bc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3df9600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131bc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:51238/0 (socket says 172.21.15.102:51238) 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3df9600640 1 -- 172.21.15.102:0/1716535852 learned_addr learned my addr 172.21.15.102:0/1716535852 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3dfa000640 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131b8fe0 0x55be131bd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.003+0000 7f3df8c00640 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ae720 0x55be131bce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.004+0000 7f3df9600640 1 -- 172.21.15.102:0/1716535852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ae720 msgr2=0x55be131bce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.004+0000 7f3df9600640 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ae720 0x55be131bce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.004+0000 7f3df9600640 1 -- 172.21.15.102:0/1716535852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131b8fe0 msgr2=0x55be131bd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.004+0000 7f3df9600640 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131b8fe0 0x55be131bd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.004+0000 7f3df9600640 1 -- 172.21.15.102:0/1716535852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be1322e1e0 con 0x55be131ade00 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.004+0000 7f3dfa000640 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131b8fe0 0x55be131bd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:06:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.004+0000 7f3df9600640 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131bc8f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3de8004010 tx=0x7f3de80c5090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81729c3491125500 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:31.012 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.004+0000 7f3ddaa00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.005+0000 7f3ddaa00640 1 -- 172.21.15.102:0/1716535852 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3de8049050 con 0x55be131ade00 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.005+0000 7f3ddaa00640 1 -- 172.21.15.102:0/1716535852 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7f3de80ed880 con 0x55be131ade00 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.005+0000 7f3dfef70740 1 -- 172.21.15.102:0/1716535852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3dec011e20 con 0x55be131ade00 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.005+0000 7f3ddaa00640 1 -- 172.21.15.102:0/1716535852 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3de80bc1a0 con 0x55be131ade00 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.005+0000 7f3dfef70740 10 client.27437 fetch_fsmap learned FSMap version 43 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.005+0000 7f3dfef70740 1 -- 172.21.15.102:0/1716535852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55be1322e1e0 con 0x55be131ade00 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.006+0000 7f3ddaa00640 1 -- 172.21.15.102:0/1716535852 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dec011e20 con 0x55be131ade00 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.006+0000 7f3ddaa00640 1 -- 172.21.15.102:0/1716535852 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f3de80b8a40 con 0x55be131ade00 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.006+0000 7f3dfef70740 10 client.27437 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:06:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.006+0000 7f3dfef70740 10 client.27437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.006+0000 7f3dfef70740 20 client.27437 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.006+0000 7f3dfef70740 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55be1322f800 0x55be13231c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.006+0000 7f3dfef70740 4 client.27437 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:06:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.006+0000 7f3dfef70740 1 -- 172.21.15.102:0/1716535852 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55be13119340 con 0x55be1322f800 2024-04-25T02:06:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.006+0000 7f3df8c00640 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55be1322f800 0x55be13231c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.007+0000 7f3df8c00640 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55be1322f800 0x55be13231c20 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57e521efa89b0022 server_cookie=ef385b857d238c9b in_seq=0 out_seq=0 2024-04-25T02:06:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.007+0000 7f3ddaa00640 10 client.27437 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3ddaa00640 1 -- 172.21.15.102:0/1716535852 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55be13119340 con 0x55be1322f800 2024-04-25T02:06:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3ddaa00640 10 client.27437 handle_command_reply: tid=0 2024-04-25T02:06:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3dfef70740 1 client.27437 shutdown 2024-04-25T02:06:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3dfef70740 1 -- 172.21.15.102:0/1716535852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 msgr2=0x55be131bc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3dfef70740 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131bc8f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3de8004010 tx=0x7f3de80c5090 comp rx=0 tx=0).stop 2024-04-25T02:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3dfef70740 1 -- 172.21.15.102:0/1716535852 shutdown_connections 2024-04-25T02:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3dfef70740 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55be1322f800 0x55be13231c20 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3dfef70740 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55be131b8fe0 0x55be131bd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3ddaa00640 0 client.27437 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3dfef70740 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be131ae720 0x55be131bce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3dfef70740 1 --2- 172.21.15.102:0/1716535852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55be131ade00 0x55be131bc8f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.119+0000 7f3dfef70740 1 -- 172.21.15.102:0/1716535852 >> 172.21.15.102:0/1716535852 conn(0x55be131aa310 msgr2=0x55be131b8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.120+0000 7f3dfef70740 1 -- 172.21.15.102:0/1716535852 shutdown_connections 2024-04-25T02:06:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.120+0000 7f3dfef70740 1 -- 172.21.15.102:0/1716535852 wait complete. 2024-04-25T02:06:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.120+0000 7f3dfef70740 20 client.27437 trim_cache size 0 max 16384 2024-04-25T02:06:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:06:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:06:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:06:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:06:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 -- 172.21.15.102:0/2374084253 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3ddc0a2e20 msgr2=0x7f3ddc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3ddc0a2e20 0x7f3ddc0a52e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3dec005c40 tx=0x7f3dec009460 comp rx=0 tx=0).stop 2024-04-25T02:06:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 -- 172.21.15.102:0/2374084253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df4103920 msgr2=0x7f3df4075c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df4103920 0x7f3df4075c50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3de8003b90 tx=0x7f3de8003bc0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.128 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 -- 172.21.15.102:0/2374084253 shutdown_connections 2024-04-25T02:06:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3ddc0a2e20 0x7f3ddc0a52e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df4109930 0x7f3df40766d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3df41042e0 0x7f3df4076190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 --2- 172.21.15.102:0/2374084253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3df4103920 0x7f3df4075c50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.121+0000 7f3dfc000640 1 -- 172.21.15.102:0/2374084253 >> 172.21.15.102:0/2374084253 conn(0x7f3df40ff430 msgr2=0x7f3df40ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.122+0000 7f3dfc000640 1 -- 172.21.15.102:0/2374084253 shutdown_connections 2024-04-25T02:06:31.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:31.122+0000 7f3dfc000640 1 -- 172.21.15.102:0/2374084253 wait complete. 2024-04-25T02:06:31.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:31 smithi110 ceph-mon[32848]: pgmap v348: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.1 MiB/s wr, 552 op/s 2024-04-25T02:06:31.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:31 smithi193 ceph-mon[32981]: pgmap v348: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.1 MiB/s wr, 552 op/s 2024-04-25T02:06:31.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:31 smithi102 ceph-mon[27796]: pgmap v348: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.1 MiB/s wr, 552 op/s 2024-04-25T02:06:33.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 640.0 s, 2074.4 tps, lat 47.795 ms stddev 58.402 2024-04-25T02:06:33.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:33 smithi110 ceph-mon[32848]: pgmap v349: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.4 MiB/s wr, 594 op/s 2024-04-25T02:06:33.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:33 smithi193 ceph-mon[32981]: pgmap v349: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.4 MiB/s wr, 594 op/s 2024-04-25T02:06:33.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:33 smithi102 ceph-mon[27796]: pgmap v349: 97 pgs: 97 active+clean; 5.0 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.4 MiB/s wr, 594 op/s 2024-04-25T02:06:35.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:35 smithi110 ceph-mon[32848]: pgmap v350: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 7.4 MiB/s wr, 765 op/s 2024-04-25T02:06:35.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:35 smithi193 ceph-mon[32981]: pgmap v350: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 7.4 MiB/s wr, 765 op/s 2024-04-25T02:06:35.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:35 smithi102 ceph-mon[27796]: pgmap v350: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 7.4 MiB/s wr, 765 op/s 2024-04-25T02:06:37.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:37 smithi110 ceph-mon[32848]: pgmap v351: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 5.3 MiB/s wr, 564 op/s 2024-04-25T02:06:37.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:37 smithi193 ceph-mon[32981]: pgmap v351: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 5.3 MiB/s wr, 564 op/s 2024-04-25T02:06:37.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:37 smithi102 ceph-mon[27796]: pgmap v351: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 5.3 MiB/s wr, 564 op/s 2024-04-25T02:06:38.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 645.0 s, 2800.7 tps, lat 34.377 ms stddev 43.692 2024-04-25T02:06:39.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:39 smithi110 ceph-mon[32848]: pgmap v352: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 8.2 MiB/s wr, 854 op/s 2024-04-25T02:06:39.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:39 smithi193 ceph-mon[32981]: pgmap v352: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 8.2 MiB/s wr, 854 op/s 2024-04-25T02:06:39.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:39 smithi102 ceph-mon[27796]: pgmap v352: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 8.2 MiB/s wr, 854 op/s 2024-04-25T02:06:41.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:41 smithi110 ceph-mon[32848]: pgmap v353: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 5.8 MiB/s wr, 594 op/s 2024-04-25T02:06:41.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:41 smithi193 ceph-mon[32981]: pgmap v353: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 5.8 MiB/s wr, 594 op/s 2024-04-25T02:06:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:41 smithi102 ceph-mon[27796]: pgmap v353: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 5.8 MiB/s wr, 594 op/s 2024-04-25T02:06:43.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 650.0 s, 1368.0 tps, lat 74.474 ms stddev 131.257 2024-04-25T02:06:43.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:43 smithi110 ceph-mon[32848]: pgmap v354: 97 pgs: 97 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 892 op/s 2024-04-25T02:06:43.760 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:43 smithi193 ceph-mon[32981]: pgmap v354: 97 pgs: 97 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 892 op/s 2024-04-25T02:06:43.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:43 smithi102 ceph-mon[27796]: pgmap v354: 97 pgs: 97 active+clean; 5.1 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 892 op/s 2024-04-25T02:06:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.301+0000 7ffafe800640 1 -- 172.21.15.102:0/108420854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 msgr2=0x7ffaf8110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.301+0000 7ffafe800640 1 --2- 172.21.15.102:0/108420854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 0x7ffaf8110280 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009be0 tx=0x7ffaf002a6c0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.302+0000 7ffafe800640 1 -- 172.21.15.102:0/108420854 shutdown_connections 2024-04-25T02:06:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.302+0000 7ffafe800640 1 --2- 172.21.15.102:0/108420854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 0x7ffaf8110280 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.302+0000 7ffafe800640 1 --2- 172.21.15.102:0/108420854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaf8100c90 0x7ffaf810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.302+0000 7ffafe800640 1 --2- 172.21.15.102:0/108420854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffaf81002e0 0x7ffaf81006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.302+0000 7ffafe800640 1 -- 172.21.15.102:0/108420854 >> 172.21.15.102:0/108420854 conn(0x7ffaf80fbf60 msgr2=0x7ffaf80fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.302+0000 7ffafe800640 1 -- 172.21.15.102:0/108420854 shutdown_connections 2024-04-25T02:06:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.302+0000 7ffafe800640 1 -- 172.21.15.102:0/108420854 wait complete. 2024-04-25T02:06:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.303+0000 7ffafe800640 1 Processor -- start 2024-04-25T02:06:44.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.303+0000 7ffafe800640 1 -- start start 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafe800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffaf81002e0 0x7ffaf8106ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafe800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaf8100c90 0x7ffaf8107410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafe800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 0x7ffaf81024f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafe800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf8113b40 con 0x7ffaf81002e0 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafe800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf81139c0 con 0x7ffaf810de90 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafe800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf8113cc0 con 0x7ffaf8100c90 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaf8100c90 0x7ffaf8107410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 0x7ffaf81024f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 0x7ffaf81024f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42058/0 (socket says 172.21.15.102:42058) 2024-04-25T02:06:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafde00640 1 -- 172.21.15.102:0/1735424523 learned_addr learned my addr 172.21.15.102:0/1735424523 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaf8100c90 0x7ffaf8107410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40924/0 (socket says 172.21.15.102:40924) 2024-04-25T02:06:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafd400640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffaf81002e0 0x7ffaf8106ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafde00640 1 -- 172.21.15.102:0/1735424523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaf8100c90 msgr2=0x7ffaf8107410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafde00640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaf8100c90 0x7ffaf8107410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.304+0000 7ffafde00640 1 -- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffaf81002e0 msgr2=0x7ffaf8106ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.305+0000 7ffafde00640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffaf81002e0 0x7ffaf8106ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.305+0000 7ffafde00640 1 -- 172.21.15.102:0/1735424523 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf8102c70 con 0x7ffaf810de90 2024-04-25T02:06:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.305+0000 7ffafde00640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 0x7ffaf81024f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ffaf002abe0 tx=0x7ffaf0030710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.305+0000 7ffaf6000640 1 -- 172.21.15.102:0/1735424523 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0041020 con 0x7ffaf810de90 2024-04-25T02:06:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.305+0000 7ffaf6000640 1 -- 172.21.15.102:0/1735424523 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaf0031920 con 0x7ffaf810de90 2024-04-25T02:06:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.305+0000 7ffaf6000640 1 -- 172.21.15.102:0/1735424523 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0036040 con 0x7ffaf810de90 2024-04-25T02:06:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.305+0000 7ffafe800640 1 -- 172.21.15.102:0/1735424523 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaf8102f00 con 0x7ffaf810de90 2024-04-25T02:06:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.305+0000 7ffafe800640 1 -- 172.21.15.102:0/1735424523 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaf810cde0 con 0x7ffaf810de90 2024-04-25T02:06:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.313+0000 7ffb02922740 1 Processor -- start 2024-04-25T02:06:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.314+0000 7ffaf6000640 1 -- 172.21.15.102:0/1735424523 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ffaf0044020 con 0x7ffaf810de90 2024-04-25T02:06:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.314+0000 7ffb02922740 1 -- start start 2024-04-25T02:06:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.314+0000 7ffb02922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392d93390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.314+0000 7ffb02922740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d9dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.314+0000 7ffb02922740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392da0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.314+0000 7ffb02922740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f392cf6ab0 con 0x55f392d938d0 2024-04-25T02:06:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.314+0000 7ffb02922740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f392c9f640 con 0x55f392d9e190 2024-04-25T02:06:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.314+0000 7ffb02922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f392cf2a70 con 0x55f392d92fb0 2024-04-25T02:06:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392d93390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d9dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392d93390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40938/0 (socket says 172.21.15.102:40938) 2024-04-25T02:06:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafca00640 1 -- 172.21.15.102:0/3997856454 learned_addr learned my addr 172.21.15.102:0/3997856454 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafca00640 1 --2- 172.21.15.102:0/3997856454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392da0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafd400640 1 -- 172.21.15.102:0/3997856454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 msgr2=0x55f392d93390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafd400640 1 --2- 172.21.15.102:0/3997856454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392d93390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafd400640 1 -- 172.21.15.102:0/3997856454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 msgr2=0x55f392da0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafd400640 1 --2- 172.21.15.102:0/3997856454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392da0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.315+0000 7ffafd400640 1 -- 172.21.15.102:0/3997856454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f392a0b210 con 0x55f392d938d0 2024-04-25T02:06:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.316+0000 7ffaf6000640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffae00a2d70 0x7ffae00a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.316+0000 7ffaf6000640 1 -- 172.21.15.102:0/1735424523 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7ffaf00efb70 con 0x7ffaf810de90 2024-04-25T02:06:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.316+0000 7ffafd400640 1 --2- 172.21.15.102:0/3997856454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d9dc50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00bbe0 tx=0x7ffaec00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9e3c856ab89bb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.317+0000 7ffafca00640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffae00a2d70 0x7ffae00a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.317+0000 7ffaea600640 1 -- 172.21.15.102:0/3997856454 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec015070 con 0x55f392d938d0 2024-04-25T02:06:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.317+0000 7ffaea600640 1 -- 172.21.15.102:0/3997856454 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaec0035f0 con 0x55f392d938d0 2024-04-25T02:06:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.317+0000 7ffaea600640 1 -- 172.21.15.102:0/3997856454 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec0116a0 con 0x55f392d938d0 2024-04-25T02:06:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffafca00640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffae00a2d70 0x7ffae00a5230 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x55f392d908e0 tx=0x7ffae40046c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffb02922740 1 -- 172.21.15.102:0/3997856454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 msgr2=0x55f392d9dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffb02922740 1 --2- 172.21.15.102:0/3997856454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d9dc50 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00bbe0 tx=0x7ffaec00f730 comp rx=0 tx=0).stop 2024-04-25T02:06:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffb02922740 1 -- 172.21.15.102:0/3997856454 shutdown_connections 2024-04-25T02:06:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffb02922740 1 --2- 172.21.15.102:0/3997856454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392da0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffb02922740 1 --2- 172.21.15.102:0/3997856454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d9dc50 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffb02922740 1 --2- 172.21.15.102:0/3997856454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392d93390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffb02922740 1 -- 172.21.15.102:0/3997856454 >> 172.21.15.102:0/3997856454 conn(0x55f392d8f4c0 msgr2=0x55f392d8f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffb02922740 1 -- 172.21.15.102:0/3997856454 shutdown_connections 2024-04-25T02:06:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.319+0000 7ffb02922740 1 -- 172.21.15.102:0/3997856454 wait complete. 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.320+0000 7ffb02922740 1 -- start start 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.320+0000 7ffb02922740 1 Processor -- start 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.320+0000 7ffb02922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392d9b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.320+0000 7ffb02922740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d9b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.320+0000 7ffb02922740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392d9bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.320+0000 7ffb02922740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55f392db1dd0 con 0x55f392d938d0 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.320+0000 7ffb02922740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55f392cb2990 con 0x55f392d9e190 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.320+0000 7ffb02922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f392db1f50 con 0x55f392d92fb0 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.320+0000 7ffb02922740 10 client.0 _ll_register_callbacks cb 0x55f392cb4950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d9b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392d9bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392d9bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42076/0 (socket says 172.21.15.102:42076) 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafde00640 1 -- 172.21.15.102:0/3704249905 learned_addr learned my addr 172.21.15.102:0/3704249905 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafd400640 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392d9b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafde00640 1 -- 172.21.15.102:0/3704249905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 msgr2=0x55f392d9b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafde00640 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392d9b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafde00640 1 -- 172.21.15.102:0/3704249905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 msgr2=0x55f392d9b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafde00640 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d9b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafde00640 1 -- 172.21.15.102:0/3704249905 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f392ad7620 con 0x55f392d9e190 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffafde00640 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392d9bcb0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00c47b0 tx=0x7ffaf00bbbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9041f409732dbb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffadea00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffadea00640 1 -- 172.21.15.102:0/3704249905 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf00bc070 con 0x55f392d9e190 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffadea00640 1 -- 172.21.15.102:0/3704249905 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7ffaf00b86e0 con 0x55f392d9e190 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.321+0000 7ffb02922740 1 -- 172.21.15.102:0/3704249905 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaec0035f0 con 0x55f392d9e190 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffadea00640 1 -- 172.21.15.102:0/3704249905 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7ffaf00c0420 con 0x55f392d9e190 2024-04-25T02:06:44.331 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffb02922740 10 client.27449 fetch_fsmap learned FSMap version 43 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffb02922740 1 -- 172.21.15.102:0/3704249905 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55f392db1dd0 con 0x55f392d9e190 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffadea00640 1 -- 172.21.15.102:0/3704249905 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaec0035f0 con 0x55f392d9e190 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffadea00640 1 -- 172.21.15.102:0/3704249905 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7ffaf00bc2d0 con 0x55f392d9e190 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffb02922740 10 client.27449 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffb02922740 10 client.27449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffb02922740 20 client.27449 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffb02922740 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f392dbf2b0 0x55f392dc1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffb02922740 4 client.27449 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffb02922740 1 -- 172.21.15.102:0/3704249905 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f392d03ff0 con 0x55f392dbf2b0 2024-04-25T02:06:44.332 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.322+0000 7ffafd400640 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f392dbf2b0 0x55f392dc1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.323+0000 7ffadea00640 10 client.27449 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.323+0000 7ffafd400640 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f392dbf2b0 0x55f392dc1710 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a941fe093214c468 server_cookie=696587156b7ac112 in_seq=0 out_seq=0 2024-04-25T02:06:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.324+0000 7ffadea00640 1 -- 172.21.15.102:0/3704249905 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f392d03ff0 con 0x55f392dbf2b0 2024-04-25T02:06:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.324+0000 7ffadea00640 10 client.27449 handle_command_reply: tid=0 2024-04-25T02:06:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.324+0000 7ffb02922740 1 client.27449 shutdown 2024-04-25T02:06:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.325+0000 7ffb02922740 1 -- 172.21.15.102:0/3704249905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 msgr2=0x55f392d9bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:44.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.325+0000 7ffb02922740 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392d9bcb0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00c47b0 tx=0x7ffaf00bbbb0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.326+0000 7ffb02922740 1 -- 172.21.15.102:0/3704249905 shutdown_connections 2024-04-25T02:06:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.326+0000 7ffb02922740 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f392dbf2b0 0x55f392dc1710 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.326+0000 7ffb02922740 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392d9bcb0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.326+0000 7ffb02922740 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d9b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.326+0000 7ffb02922740 1 --2- 172.21.15.102:0/3704249905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392d9b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.326+0000 7ffb02922740 1 -- 172.21.15.102:0/3704249905 >> 172.21.15.102:0/3704249905 conn(0x55f392d8f4c0 msgr2=0x55f392d95240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.326+0000 7ffadea00640 0 client.27449 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.327+0000 7ffb02922740 1 -- 172.21.15.102:0/3704249905 shutdown_connections 2024-04-25T02:06:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.327+0000 7ffb02922740 1 -- 172.21.15.102:0/3704249905 wait complete. 2024-04-25T02:06:44.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.327+0000 7ffb02922740 20 client.27449 trim_cache size 0 max 16384 2024-04-25T02:06:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.348+0000 7ffb02922740 1 Processor -- start 2024-04-25T02:06:44.355 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.348+0000 7ffb02922740 1 -- start start 2024-04-25T02:06:44.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffb02922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392dbac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffb02922740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d96510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.356 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffb02922740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392d95970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffb02922740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf81b6100 con 0x55f392d938d0 2024-04-25T02:06:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffb02922740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf8072ab0 con 0x55f392d9e190 2024-04-25T02:06:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffb02922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf8070920 con 0x55f392d92fb0 2024-04-25T02:06:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffafd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392dbac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffafca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d96510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffafca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d96510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43262/0 (socket says 172.21.15.102:43262) 2024-04-25T02:06:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffafca00640 1 -- 172.21.15.102:0/1485264680 learned_addr learned my addr 172.21.15.102:0/1485264680 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:44.357 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.349+0000 7ffafd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392dbac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40950/0 (socket says 172.21.15.102:40950) 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.350+0000 7ffafd400640 1 -- 172.21.15.102:0/1485264680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 msgr2=0x55f392d95970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.350+0000 7ffafd400640 1 --2- 172.21.15.102:0/1485264680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392d95970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.350+0000 7ffafd400640 1 -- 172.21.15.102:0/1485264680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 msgr2=0x55f392d96510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.350+0000 7ffafd400640 1 --2- 172.21.15.102:0/1485264680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d96510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.350+0000 7ffafd400640 1 -- 172.21.15.102:0/1485264680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f392d99780 con 0x55f392d92fb0 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.350+0000 7ffafca00640 1 --2- 172.21.15.102:0/1485264680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d96510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.350+0000 7ffafd400640 1 --2- 172.21.15.102:0/1485264680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392dbac90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00fc30 tx=0x7ffaec00b4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73430c1ca095d436 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.350+0000 7ffae9c00640 1 -- 172.21.15.102:0/1485264680 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec015050 con 0x55f392d92fb0 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.351+0000 7ffae9c00640 1 -- 172.21.15.102:0/1485264680 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaec005500 con 0x55f392d92fb0 2024-04-25T02:06:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.351+0000 7ffae9c00640 1 -- 172.21.15.102:0/1485264680 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec003850 con 0x55f392d92fb0 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.351+0000 7ffb02922740 1 -- 172.21.15.102:0/1485264680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 msgr2=0x55f392dbac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.351+0000 7ffb02922740 1 --2- 172.21.15.102:0/1485264680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392dbac90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ffaec00fc30 tx=0x7ffaec00b4d0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.351+0000 7ffb02922740 1 -- 172.21.15.102:0/1485264680 shutdown_connections 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.351+0000 7ffb02922740 1 --2- 172.21.15.102:0/1485264680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d9e190 0x55f392d95970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.351+0000 7ffb02922740 1 --2- 172.21.15.102:0/1485264680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d938d0 0x55f392d96510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.351+0000 7ffb02922740 1 --2- 172.21.15.102:0/1485264680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d92fb0 0x55f392dbac90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.351+0000 7ffb02922740 1 -- 172.21.15.102:0/1485264680 >> 172.21.15.102:0/1485264680 conn(0x55f392d8f4c0 msgr2=0x55f392d8e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.352+0000 7ffb02922740 1 -- 172.21.15.102:0/1485264680 shutdown_connections 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.352+0000 7ffb02922740 1 -- 172.21.15.102:0/1485264680 wait complete. 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.352+0000 7ffb02922740 1 -- start start 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.352+0000 7ffb02922740 1 Processor -- start 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffb02922740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d92fb0 0x55f392db2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffb02922740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d938d0 0x55f392db32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffb02922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d9e190 0x55f392db3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffb02922740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf8070f70 con 0x55f392d92fb0 2024-04-25T02:06:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffb02922740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf80723e0 con 0x55f392d938d0 2024-04-25T02:06:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffb02922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf8071640 con 0x55f392d9e190 2024-04-25T02:06:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffb02922740 10 client.0 _ll_register_callbacks cb 0x7ffaf0003b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffafde00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d9e190 0x55f392db3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffafde00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d9e190 0x55f392db3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:40960/0 (socket says 172.21.15.102:40960) 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffafde00640 1 -- 172.21.15.102:0/4146207286 learned_addr learned my addr 172.21.15.102:0/4146207286 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffafca00640 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d938d0 0x55f392db32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.353+0000 7ffafd400640 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d92fb0 0x55f392db2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.354+0000 7ffafde00640 1 -- 172.21.15.102:0/4146207286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d938d0 msgr2=0x55f392db32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.354+0000 7ffafde00640 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d938d0 0x55f392db32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.354+0000 7ffafde00640 1 -- 172.21.15.102:0/4146207286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d92fb0 msgr2=0x55f392db2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.354+0000 7ffafde00640 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d92fb0 0x55f392db2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.354+0000 7ffafde00640 1 -- 172.21.15.102:0/4146207286 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f392e24a90 con 0x55f392d9e190 2024-04-25T02:06:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.354+0000 7ffafca00640 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d938d0 0x55f392db32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.355+0000 7ffafde00640 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d9e190 0x55f392db3820 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffaf003c040 tx=0x7ffaf0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dddad7419d6c7323 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.355+0000 7ffadea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:06:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.356+0000 7ffadea00640 1 -- 172.21.15.102:0/4146207286 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf00b8310 con 0x55f392d9e190 2024-04-25T02:06:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.356+0000 7ffadea00640 1 -- 172.21.15.102:0/4146207286 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7ffaf0004b40 con 0x55f392d9e190 2024-04-25T02:06:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.356+0000 7ffb02922740 1 -- 172.21.15.102:0/4146207286 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaec005500 con 0x55f392d9e190 2024-04-25T02:06:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.358+0000 7ffadea00640 1 -- 172.21.15.102:0/4146207286 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7ffaf00bb9d0 con 0x55f392d9e190 2024-04-25T02:06:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.358+0000 7ffb02922740 10 client.27418 fetch_fsmap learned FSMap version 43 2024-04-25T02:06:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.358+0000 7ffb02922740 1 -- 172.21.15.102:0/4146207286 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55f392e25d80 con 0x55f392d9e190 2024-04-25T02:06:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.359+0000 7ffadea00640 1 -- 172.21.15.102:0/4146207286 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaec005500 con 0x55f392d9e190 2024-04-25T02:06:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.359+0000 7ffadea00640 1 -- 172.21.15.102:0/4146207286 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7ffaf00bb4c0 con 0x55f392d9e190 2024-04-25T02:06:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.359+0000 7ffb02922740 10 client.27418 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:06:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.359+0000 7ffb02922740 10 client.27418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.359+0000 7ffb02922740 20 client.27418 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.359+0000 7ffb02922740 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f392e26340 0x55f392e28760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.359+0000 7ffb02922740 4 client.27418 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:06:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.359+0000 7ffb02922740 1 -- 172.21.15.102:0/4146207286 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f392cfe340 con 0x55f392e26340 2024-04-25T02:06:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.360+0000 7ffafd400640 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f392e26340 0x55f392e28760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.360+0000 7ffafd400640 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f392e26340 0x55f392e28760 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2aaab504affa4fdd server_cookie=ccdfa1e6a88122c4 in_seq=0 out_seq=0 2024-04-25T02:06:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.360+0000 7ffadea00640 10 client.27418 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.373+0000 7ffadea00640 1 -- 172.21.15.102:0/4146207286 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f392cfe340 con 0x55f392e26340 2024-04-25T02:06:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.373+0000 7ffadea00640 10 client.27418 handle_command_reply: tid=0 2024-04-25T02:06:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.373+0000 7ffb02922740 1 client.27418 shutdown 2024-04-25T02:06:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.374+0000 7ffb02922740 1 -- 172.21.15.102:0/4146207286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d9e190 msgr2=0x55f392db3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.374+0000 7ffb02922740 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d9e190 0x55f392db3820 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffaf003c040 tx=0x7ffaf0004040 comp rx=0 tx=0).stop 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.374+0000 7ffb02922740 1 -- 172.21.15.102:0/4146207286 shutdown_connections 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.374+0000 7ffb02922740 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55f392e26340 0x55f392e28760 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.374+0000 7ffb02922740 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f392d9e190 0x55f392db3820 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.374+0000 7ffb02922740 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f392d938d0 0x55f392db32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.374+0000 7ffb02922740 1 --2- 172.21.15.102:0/4146207286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f392d92fb0 0x55f392db2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.374+0000 7ffb02922740 1 -- 172.21.15.102:0/4146207286 >> 172.21.15.102:0/4146207286 conn(0x55f392d8f4c0 msgr2=0x55f392d8dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.374+0000 7ffadea00640 0 client.27418 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.375+0000 7ffb02922740 1 -- 172.21.15.102:0/4146207286 shutdown_connections 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.375+0000 7ffb02922740 1 -- 172.21.15.102:0/4146207286 wait complete. 2024-04-25T02:06:44.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.375+0000 7ffb02922740 20 client.27418 trim_cache size 0 max 16384 2024-04-25T02:06:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:06:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:06:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:06:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 -- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffae00a2d70 msgr2=0x7ffae00a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffae00a2d70 0x7ffae00a5230 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x55f392d908e0 tx=0x7ffae40046c0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 -- 172.21.15.102:0/1735424523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 msgr2=0x7ffaf81024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 0x7ffaf81024f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ffaf002abe0 tx=0x7ffaf0030710 comp rx=0 tx=0).stop 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 -- 172.21.15.102:0/1735424523 shutdown_connections 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ffae00a2d70 0x7ffae00a5230 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaf810de90 0x7ffaf81024f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffaf8100c90 0x7ffaf8107410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 --2- 172.21.15.102:0/1735424523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffaf81002e0 0x7ffaf8106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 -- 172.21.15.102:0/1735424523 >> 172.21.15.102:0/1735424523 conn(0x7ffaf80fbf60 msgr2=0x7ffaf80fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 -- 172.21.15.102:0/1735424523 shutdown_connections 2024-04-25T02:06:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:44.376+0000 7ffafe800640 1 -- 172.21.15.102:0/1735424523 wait complete. 2024-04-25T02:06:45.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:45 smithi193 ceph-mon[32981]: pgmap v355: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 26 MiB/s wr, 1.93k op/s 2024-04-25T02:06:45.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:45 smithi102 ceph-mon[27796]: pgmap v355: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 26 MiB/s wr, 1.93k op/s 2024-04-25T02:06:45.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:45 smithi110 ceph-mon[32848]: pgmap v355: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 26 MiB/s wr, 1.93k op/s 2024-04-25T02:06:47.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:47 smithi193 ceph-mon[32981]: pgmap v356: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 24 MiB/s wr, 1.67k op/s 2024-04-25T02:06:47.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:47 smithi102 ceph-mon[27796]: pgmap v356: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 24 MiB/s wr, 1.67k op/s 2024-04-25T02:06:47.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:47 smithi110 ceph-mon[32848]: pgmap v356: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 24 MiB/s wr, 1.67k op/s 2024-04-25T02:06:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 655.0 s, 1338.9 tps, lat 75.750 ms stddev 91.829 2024-04-25T02:06:49.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:49 smithi193 ceph-mon[32981]: pgmap v357: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 26 MiB/s wr, 1.92k op/s 2024-04-25T02:06:49.928 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:49 smithi102 ceph-mon[27796]: pgmap v357: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 26 MiB/s wr, 1.92k op/s 2024-04-25T02:06:49.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:49 smithi110 ceph-mon[32848]: pgmap v357: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 26 MiB/s wr, 1.92k op/s 2024-04-25T02:06:50.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:50 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:06:50.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:50 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:06:51.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:50 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:06:51.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:51 smithi102 ceph-mon[27796]: pgmap v358: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 23 MiB/s wr, 1.63k op/s 2024-04-25T02:06:51.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:51 smithi110 ceph-mon[32848]: pgmap v358: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 23 MiB/s wr, 1.63k op/s 2024-04-25T02:06:52.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:51 smithi193 ceph-mon[32981]: pgmap v358: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 23 MiB/s wr, 1.63k op/s 2024-04-25T02:06:52.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:52 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:06:52.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:52 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:06:52.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:52 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:06:52.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:52 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:06:52.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:52 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:06:52.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:52 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:06:53.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:52 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:06:53.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:52 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:06:53.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:52 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:06:53.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 660.0 s, 1382.4 tps, lat 72.504 ms stddev 83.280 2024-04-25T02:06:53.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:53 smithi102 ceph-mon[27796]: pgmap v359: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 23 MiB/s wr, 1.67k op/s 2024-04-25T02:06:53.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:53 smithi110 ceph-mon[32848]: pgmap v359: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 23 MiB/s wr, 1.67k op/s 2024-04-25T02:06:54.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:53 smithi193 ceph-mon[32981]: pgmap v359: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 23 MiB/s wr, 1.67k op/s 2024-04-25T02:06:54.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:54 smithi102 ceph-mon[27796]: pgmap v360: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 21 MiB/s wr, 1.57k op/s 2024-04-25T02:06:54.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:54 smithi110 ceph-mon[32848]: pgmap v360: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 21 MiB/s wr, 1.57k op/s 2024-04-25T02:06:55.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:54 smithi193 ceph-mon[32981]: pgmap v360: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 21 MiB/s wr, 1.57k op/s 2024-04-25T02:06:57.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:56 smithi110 ceph-mon[32848]: pgmap v361: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 4.5 MiB/s wr, 496 op/s 2024-04-25T02:06:57.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:56 smithi193 ceph-mon[32981]: pgmap v361: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 4.5 MiB/s wr, 496 op/s 2024-04-25T02:06:57.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:56 smithi102 ceph-mon[27796]: pgmap v361: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 4.5 MiB/s wr, 496 op/s 2024-04-25T02:06:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.547+0000 7fa435000640 1 -- 172.21.15.102:0/1140352841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430109940 msgr2=0x7fa430109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.547+0000 7fa435000640 1 --2- 172.21.15.102:0/1140352841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430109940 0x7fa430109d20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa420009900 tx=0x7fa42002a3f0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.547+0000 7fa435000640 1 -- 172.21.15.102:0/1140352841 shutdown_connections 2024-04-25T02:06:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.547+0000 7fa435000640 1 --2- 172.21.15.102:0/1140352841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4301042f0 0x7fa430110800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.547+0000 7fa435000640 1 --2- 172.21.15.102:0/1140352841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa430103930 0x7fa430103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.548+0000 7fa435000640 1 --2- 172.21.15.102:0/1140352841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430109940 0x7fa430109d20 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.555 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.548+0000 7fa435000640 1 -- 172.21.15.102:0/1140352841 >> 172.21.15.102:0/1140352841 conn(0x7fa4300ff440 msgr2=0x7fa430101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:57.555 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.548+0000 7fa435000640 1 -- 172.21.15.102:0/1140352841 shutdown_connections 2024-04-25T02:06:57.555 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.548+0000 7fa435000640 1 -- 172.21.15.102:0/1140352841 wait complete. 2024-04-25T02:06:57.555 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.549+0000 7fa435000640 1 Processor -- start 2024-04-25T02:06:57.555 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.549+0000 7fa435000640 1 -- start start 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa435000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430103930 0x7fa43019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa435000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4301042f0 0x7fa43019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa435000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa430109940 0x7fa43019ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa435000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa430113d20 con 0x7fa4301042f0 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa435000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa430113ba0 con 0x7fa430103930 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa435000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa430113ea0 con 0x7fa430109940 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42de00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430103930 0x7fa43019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42de00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430103930 0x7fa43019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46324/0 (socket says 172.21.15.102:46324) 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42de00640 1 -- 172.21.15.102:0/705697176 learned_addr learned my addr 172.21.15.102:0/705697176 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42e800640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa430109940 0x7fa43019ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42d400640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4301042f0 0x7fa43019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42de00640 1 -- 172.21.15.102:0/705697176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa430109940 msgr2=0x7fa43019ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:57.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42de00640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa430109940 0x7fa43019ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42de00640 1 -- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4301042f0 msgr2=0x7fa43019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:57.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42de00640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4301042f0 0x7fa43019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.550+0000 7fa42de00640 1 -- 172.21.15.102:0/705697176 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa43019f170 con 0x7fa430103930 2024-04-25T02:06:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.551+0000 7fa42de00640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430103930 0x7fa43019a180 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa420000c00 tx=0x7fa420003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.552+0000 7fa426a00640 1 -- 172.21.15.102:0/705697176 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42003f070 con 0x7fa430103930 2024-04-25T02:06:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.552+0000 7fa435000640 1 -- 172.21.15.102:0/705697176 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa43019f400 con 0x7fa430103930 2024-04-25T02:06:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.553+0000 7fa435000640 1 -- 172.21.15.102:0/705697176 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4301a6cf0 con 0x7fa430103930 2024-04-25T02:06:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.552+0000 7fa426a00640 1 -- 172.21.15.102:0/705697176 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa420031c30 con 0x7fa430103930 2024-04-25T02:06:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.553+0000 7fa426a00640 1 -- 172.21.15.102:0/705697176 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42003f070 con 0x7fa430103930 2024-04-25T02:06:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.557+0000 7fa437faa740 1 Processor -- start 2024-04-25T02:06:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.557+0000 7fa437faa740 1 -- start start 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.557+0000 7fa437faa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad08721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.557+0000 7fa437faa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0872720 0x559ad087caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.557+0000 7fa437faa740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad087cfe0 0x559ad087f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.557+0000 7fa437faa740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559ad07d5ab0 con 0x559ad087cfe0 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.557+0000 7fa437faa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559ad077e640 con 0x559ad0871e00 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.557+0000 7fa437faa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559ad07d1a70 con 0x559ad0872720 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa426a00640 1 -- 172.21.15.102:0/705697176 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa420042070 con 0x7fa430103930 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad08721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad08721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46330/0 (socket says 172.21.15.102:46330) 2024-04-25T02:06:57.565 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42e800640 1 -- 172.21.15.102:0/602859066 learned_addr learned my addr 172.21.15.102:0/602859066 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42d400640 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad087cfe0 0x559ad087f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42de00640 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0872720 0x559ad087caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42e800640 1 -- 172.21.15.102:0/602859066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0872720 msgr2=0x559ad087caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42e800640 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0872720 0x559ad087caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42e800640 1 -- 172.21.15.102:0/602859066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad087cfe0 msgr2=0x559ad087f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42e800640 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad087cfe0 0x559ad087f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42e800640 1 -- 172.21.15.102:0/602859066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ad04ea210 con 0x559ad0871e00 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.558+0000 7fa42de00640 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0872720 0x559ad087caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.559+0000 7fa42e800640 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad08721e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa428009f90 tx=0x7fa428010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab170764bd734d3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.559+0000 7fa41b400640 1 -- 172.21.15.102:0/602859066 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428011700 con 0x559ad0871e00 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.559+0000 7fa41b400640 1 -- 172.21.15.102:0/602859066 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa428011d80 con 0x559ad0871e00 2024-04-25T02:06:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.559+0000 7fa41b400640 1 -- 172.21.15.102:0/602859066 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428018380 con 0x559ad0871e00 2024-04-25T02:06:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.559+0000 7fa426a00640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4140a2fa0 0x7fa4140a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.560+0000 7fa437faa740 1 -- 172.21.15.102:0/602859066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 msgr2=0x559ad08721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.560+0000 7fa437faa740 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad08721e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa428009f90 tx=0x7fa428010860 comp rx=0 tx=0).stop 2024-04-25T02:06:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.560+0000 7fa437faa740 1 -- 172.21.15.102:0/602859066 shutdown_connections 2024-04-25T02:06:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.560+0000 7fa437faa740 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad087cfe0 0x559ad087f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.560+0000 7fa437faa740 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0872720 0x559ad087caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.560+0000 7fa437faa740 1 --2- 172.21.15.102:0/602859066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad08721e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.560+0000 7fa437faa740 1 -- 172.21.15.102:0/602859066 >> 172.21.15.102:0/602859066 conn(0x559ad086e310 msgr2=0x559ad086e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.561+0000 7fa42de00640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4140a2fa0 0x7fa4140a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.562+0000 7fa426a00640 1 -- 172.21.15.102:0/705697176 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7fa4200ef6c0 con 0x7fa430103930 2024-04-25T02:06:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.562+0000 7fa437faa740 1 -- 172.21.15.102:0/602859066 shutdown_connections 2024-04-25T02:06:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.562+0000 7fa437faa740 1 -- 172.21.15.102:0/602859066 wait complete. 2024-04-25T02:06:57.569 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.562+0000 7fa42de00640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4140a2fa0 0x7fa4140a5460 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa4200ecec0 tx=0x7fa42004cf70 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.563+0000 7fa437faa740 1 -- start start 2024-04-25T02:06:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.564+0000 7fa437faa740 1 Processor -- start 2024-04-25T02:06:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.564+0000 7fa437faa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 0x559ad0879f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.564+0000 7fa437faa740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 0x559ad087a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.564+0000 7fa437faa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad087cfe0 0x559ad087a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.564+0000 7fa437faa740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559ad087fc10 con 0x559ad0872720 2024-04-25T02:06:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.564+0000 7fa437faa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559ad0791990 con 0x559ad087cfe0 2024-04-25T02:06:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.564+0000 7fa437faa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559ad087fd90 con 0x559ad0871e00 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.564+0000 7fa437faa740 10 client.0 _ll_register_callbacks cb 0x559ad0793950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad087cfe0 0x559ad087a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 0x559ad087a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad087cfe0 0x559ad087a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46336/0 (socket says 172.21.15.102:46336) 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 -- 172.21.15.102:0/2153236546 learned_addr learned my addr 172.21.15.102:0/2153236546 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 0x559ad0879f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 -- 172.21.15.102:0/2153236546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 msgr2=0x559ad0879f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 0x559ad0879f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 -- 172.21.15.102:0/2153236546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 msgr2=0x559ad087a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 0x559ad087a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 -- 172.21.15.102:0/2153236546 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ad05b6620 con 0x559ad087cfe0 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42d400640 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 0x559ad087a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa42e800640 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad087cfe0 0x559ad087a990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa428019930 tx=0x7fa428013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=272a76cd0d11e9bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.565+0000 7fa40fe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:06:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.566+0000 7fa40fe00640 1 -- 172.21.15.102:0/2153236546 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428011910 con 0x559ad087cfe0 2024-04-25T02:06:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.566+0000 7fa40fe00640 1 -- 172.21.15.102:0/2153236546 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7fa428011ab0 con 0x559ad087cfe0 2024-04-25T02:06:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.566+0000 7fa437faa740 1 -- 172.21.15.102:0/2153236546 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa428011d80 con 0x559ad087cfe0 2024-04-25T02:06:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.566+0000 7fa40fe00640 1 -- 172.21.15.102:0/2153236546 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa428003970 con 0x559ad087cfe0 2024-04-25T02:06:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.566+0000 7fa437faa740 10 client.27464 fetch_fsmap learned FSMap version 43 2024-04-25T02:06:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.566+0000 7fa437faa740 1 -- 172.21.15.102:0/2153236546 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559ad0894240 con 0x559ad087cfe0 2024-04-25T02:06:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.566+0000 7fa40fe00640 1 -- 172.21.15.102:0/2153236546 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa428011d80 con 0x559ad087cfe0 2024-04-25T02:06:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.566+0000 7fa40fe00640 1 -- 172.21.15.102:0/2153236546 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7fa428031490 con 0x559ad087cfe0 2024-04-25T02:06:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.567+0000 7fa437faa740 10 client.27464 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:06:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.567+0000 7fa437faa740 10 client.27464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.567+0000 7fa437faa740 20 client.27464 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.567+0000 7fa437faa740 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559ad0895650 0x559ad0897a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.567+0000 7fa437faa740 4 client.27464 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:06:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.567+0000 7fa437faa740 1 -- 172.21.15.102:0/2153236546 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ad07e2ff0 con 0x559ad0895650 2024-04-25T02:06:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.567+0000 7fa42d400640 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559ad0895650 0x559ad0897a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.568+0000 7fa42d400640 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559ad0895650 0x559ad0897a40 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f927725534f65d9 server_cookie=1f89736ff3ce1060 in_seq=0 out_seq=0 2024-04-25T02:06:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.568+0000 7fa40fe00640 10 client.27464 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.569+0000 7fa40fe00640 1 -- 172.21.15.102:0/2153236546 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ad07e2ff0 con 0x559ad0895650 2024-04-25T02:06:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.569+0000 7fa40fe00640 10 client.27464 handle_command_reply: tid=0 2024-04-25T02:06:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.570+0000 7fa437faa740 1 client.27464 shutdown 2024-04-25T02:06:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 -- 172.21.15.102:0/2153236546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad087cfe0 msgr2=0x559ad087a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad087cfe0 0x559ad087a990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa428019930 tx=0x7fa428013040 comp rx=0 tx=0).stop 2024-04-25T02:06:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 -- 172.21.15.102:0/2153236546 shutdown_connections 2024-04-25T02:06:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559ad0895650 0x559ad0897a40 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad087cfe0 0x559ad087a990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 0x559ad087a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 --2- 172.21.15.102:0/2153236546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 0x559ad0879f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 -- 172.21.15.102:0/2153236546 >> 172.21.15.102:0/2153236546 conn(0x559ad086e310 msgr2=0x559ad0874040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa40fe00640 0 client.27464 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 -- 172.21.15.102:0/2153236546 shutdown_connections 2024-04-25T02:06:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 1 -- 172.21.15.102:0/2153236546 wait complete. 2024-04-25T02:06:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.571+0000 7fa437faa740 20 client.27464 trim_cache size 0 max 16384 2024-04-25T02:06:57.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.612+0000 7fa437faa740 1 Processor -- start 2024-04-25T02:06:57.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.612+0000 7fa437faa740 1 -- start start 2024-04-25T02:06:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.613+0000 7fa437faa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad0891360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.613+0000 7fa437faa740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 0x559ad087a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.613+0000 7fa437faa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad087cfe0 0x559ad089d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.614+0000 7fa437faa740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa4301adb70 con 0x559ad0872720 2024-04-25T02:06:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.614+0000 7fa42e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad0891360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.614+0000 7fa42e800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad0891360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46350/0 (socket says 172.21.15.102:46350) 2024-04-25T02:06:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.614+0000 7fa42e800640 1 -- 172.21.15.102:0/3417161546 learned_addr learned my addr 172.21.15.102:0/3417161546 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.614+0000 7fa437faa740 1 -- 172.21.15.102:0/3417161546 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa4301ab3b0 con 0x559ad0871e00 2024-04-25T02:06:57.620 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.614+0000 7fa437faa740 1 -- 172.21.15.102:0/3417161546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa4301a9220 con 0x559ad087cfe0 2024-04-25T02:06:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.614+0000 7fa42e800640 1 --2- 172.21.15.102:0/3417161546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad087cfe0 0x559ad089d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.615+0000 7fa42e800640 1 -- 172.21.15.102:0/3417161546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad087cfe0 msgr2=0x559ad089d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:57.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.615+0000 7fa42e800640 1 --2- 172.21.15.102:0/3417161546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad087cfe0 0x559ad089d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.615+0000 7fa42e800640 1 -- 172.21.15.102:0/3417161546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 msgr2=0x559ad087a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.615+0000 7fa42e800640 1 --2- 172.21.15.102:0/3417161546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 0x559ad087a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.616+0000 7fa42e800640 1 -- 172.21.15.102:0/3417161546 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ad0872550 con 0x559ad0871e00 2024-04-25T02:06:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.616+0000 7fa42e800640 1 --2- 172.21.15.102:0/3417161546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad0891360 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa4280108a0 tx=0x7fa4280021d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f2bbb3c8374c8ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:57.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.617+0000 7fa41aa00640 1 -- 172.21.15.102:0/3417161546 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428019c70 con 0x559ad0871e00 2024-04-25T02:06:57.625 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.617+0000 7fa41aa00640 1 -- 172.21.15.102:0/3417161546 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa428038070 con 0x559ad0871e00 2024-04-25T02:06:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.617+0000 7fa41aa00640 1 -- 172.21.15.102:0/3417161546 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428019c70 con 0x559ad0871e00 2024-04-25T02:06:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.619+0000 7fa437faa740 1 -- 172.21.15.102:0/3417161546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 msgr2=0x559ad0891360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.619+0000 7fa437faa740 1 --2- 172.21.15.102:0/3417161546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad0891360 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa4280108a0 tx=0x7fa4280021d0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.619+0000 7fa437faa740 1 -- 172.21.15.102:0/3417161546 shutdown_connections 2024-04-25T02:06:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.619+0000 7fa437faa740 1 --2- 172.21.15.102:0/3417161546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad087cfe0 0x559ad089d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.619+0000 7fa437faa740 1 --2- 172.21.15.102:0/3417161546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad0872720 0x559ad087a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.619+0000 7fa437faa740 1 --2- 172.21.15.102:0/3417161546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0871e00 0x559ad0891360 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.620+0000 7fa437faa740 1 -- 172.21.15.102:0/3417161546 >> 172.21.15.102:0/3417161546 conn(0x559ad086e310 msgr2=0x559ad0879ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.625+0000 7fa437faa740 1 -- 172.21.15.102:0/3417161546 shutdown_connections 2024-04-25T02:06:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.625+0000 7fa437faa740 1 -- 172.21.15.102:0/3417161546 wait complete. 2024-04-25T02:06:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.625+0000 7fa437faa740 1 -- start start 2024-04-25T02:06:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.626+0000 7fa437faa740 1 Processor -- start 2024-04-25T02:06:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.626+0000 7fa437faa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 0x559ad088d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa437faa740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0872720 0x559ad088dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa42e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 0x559ad088d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa42e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 0x559ad088d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:36312/0 (socket says 172.21.15.102:36312) 2024-04-25T02:06:57.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa437faa740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad087cfe0 0x559ad0888ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa437faa740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa4301a9870 con 0x559ad087cfe0 2024-04-25T02:06:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa437faa740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa4301aace0 con 0x559ad0872720 2024-04-25T02:06:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa437faa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa4301a9f40 con 0x559ad0871e00 2024-04-25T02:06:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa437faa740 10 client.0 _ll_register_callbacks cb 0x7fa428009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:06:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa42d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0872720 0x559ad088dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa42d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0872720 0x559ad088dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46356/0 (socket says 172.21.15.102:46356) 2024-04-25T02:06:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.627+0000 7fa42d400640 1 -- 172.21.15.102:0/3535078731 learned_addr learned my addr 172.21.15.102:0/3535078731 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:06:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.628+0000 7fa42d400640 1 -- 172.21.15.102:0/3535078731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 msgr2=0x559ad088d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:57.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.628+0000 7fa42d400640 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 0x559ad088d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.628+0000 7fa42d400640 1 -- 172.21.15.102:0/3535078731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad087cfe0 msgr2=0x559ad0888ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:06:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.628+0000 7fa42d400640 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad087cfe0 0x559ad0888ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.628+0000 7fa42d400640 1 -- 172.21.15.102:0/3535078731 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ad08897f0 con 0x559ad0872720 2024-04-25T02:06:57.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.628+0000 7fa42d400640 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0872720 0x559ad088dde0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x559ad086f730 tx=0x7fa41c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b650a573fac4d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:06:57.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.631+0000 7fa40fe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:06:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.631+0000 7fa40fe00640 1 -- 172.21.15.102:0/3535078731 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c0222f0 con 0x559ad0872720 2024-04-25T02:06:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.631+0000 7fa40fe00640 1 -- 172.21.15.102:0/3535078731 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8513+0+0 (secure 0 0 0) 0x7fa41c022490 con 0x559ad0872720 2024-04-25T02:06:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.631+0000 7fa437faa740 1 -- 172.21.15.102:0/3535078731 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa428038070 con 0x559ad0872720 2024-04-25T02:06:57.637 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.631+0000 7fa40fe00640 1 -- 172.21.15.102:0/3535078731 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa41c032050 con 0x559ad0872720 2024-04-25T02:06:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.631+0000 7fa437faa740 10 client.27470 fetch_fsmap learned FSMap version 43 2024-04-25T02:06:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.632+0000 7fa437faa740 1 -- 172.21.15.102:0/3535078731 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559ad08fb6e0 con 0x559ad0872720 2024-04-25T02:06:57.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.632+0000 7fa40fe00640 1 -- 172.21.15.102:0/3535078731 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa428038070 con 0x559ad0872720 2024-04-25T02:06:57.639 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.633+0000 7fa40fe00640 1 -- 172.21.15.102:0/3535078731 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7fa41c010430 con 0x559ad0872720 2024-04-25T02:06:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.633+0000 7fa437faa740 10 client.27470 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:06:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.633+0000 7fa437faa740 10 client.27470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:06:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.633+0000 7fa437faa740 20 client.27470 populate_metadata read hostname 'smithi102' 2024-04-25T02:06:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.634+0000 7fa437faa740 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559ad08fbe40 0x559ad08fe260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:06:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.634+0000 7fa437faa740 4 client.27470 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:06:57.640 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.634+0000 7fa437faa740 1 -- 172.21.15.102:0/3535078731 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ad07dd340 con 0x559ad08fbe40 2024-04-25T02:06:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.634+0000 7fa42e800640 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559ad08fbe40 0x559ad08fe260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:06:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.637+0000 7fa42e800640 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559ad08fbe40 0x559ad08fe260 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de023a375072aec2 server_cookie=a252480506a61423 in_seq=0 out_seq=0 2024-04-25T02:06:57.643 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.637+0000 7fa40fe00640 10 client.27470 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:57.666 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.660+0000 7fa40fe00640 1 -- 172.21.15.102:0/3535078731 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559ad07dd340 con 0x559ad08fbe40 2024-04-25T02:06:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.661+0000 7fa40fe00640 10 client.27470 handle_command_reply: tid=0 2024-04-25T02:06:57.667 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.661+0000 7fa437faa740 1 client.27470 shutdown 2024-04-25T02:06:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.662+0000 7fa437faa740 1 -- 172.21.15.102:0/3535078731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0872720 msgr2=0x559ad088dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:57.668 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.662+0000 7fa437faa740 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0872720 0x559ad088dde0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x559ad086f730 tx=0x7fa41c009f90 comp rx=0 tx=0).stop 2024-04-25T02:06:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.663+0000 7fa437faa740 1 -- 172.21.15.102:0/3535078731 shutdown_connections 2024-04-25T02:06:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.663+0000 7fa437faa740 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559ad08fbe40 0x559ad08fe260 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.663+0000 7fa437faa740 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559ad087cfe0 0x559ad0888ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.663+0000 7fa437faa740 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559ad0872720 0x559ad088dde0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.663+0000 7fa437faa740 1 --2- 172.21.15.102:0/3535078731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ad0871e00 0x559ad088d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.663+0000 7fa437faa740 1 -- 172.21.15.102:0/3535078731 >> 172.21.15.102:0/3535078731 conn(0x559ad086e310 msgr2=0x559ad0874040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:57.669 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.663+0000 7fa40fe00640 0 client.27470 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:06:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.663+0000 7fa437faa740 1 -- 172.21.15.102:0/3535078731 shutdown_connections 2024-04-25T02:06:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.664+0000 7fa437faa740 1 -- 172.21.15.102:0/3535078731 wait complete. 2024-04-25T02:06:57.670 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.664+0000 7fa437faa740 20 client.27470 trim_cache size 0 max 16384 2024-04-25T02:06:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:06:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:06:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:06:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 -- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4140a2fa0 msgr2=0x7fa4140a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4140a2fa0 0x7fa4140a5460 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa4200ecec0 tx=0x7fa42004cf70 comp rx=0 tx=0).stop 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 -- 172.21.15.102:0/705697176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430103930 msgr2=0x7fa43019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430103930 0x7fa43019a180 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa420000c00 tx=0x7fa420003c10 comp rx=0 tx=0).stop 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 -- 172.21.15.102:0/705697176 shutdown_connections 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa4140a2fa0 0x7fa4140a5460 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa430109940 0x7fa43019ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4301042f0 0x7fa43019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 --2- 172.21.15.102:0/705697176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa430103930 0x7fa43019a180 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.665+0000 7fa435000640 1 -- 172.21.15.102:0/705697176 >> 172.21.15.102:0/705697176 conn(0x7fa4300ff440 msgr2=0x7fa4300ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.666+0000 7fa435000640 1 -- 172.21.15.102:0/705697176 shutdown_connections 2024-04-25T02:06:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:06:57.666+0000 7fa435000640 1 -- 172.21.15.102:0/705697176 wait complete. 2024-04-25T02:06:58.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 665.0 s, 1453.2 tps, lat 67.560 ms stddev 76.099 2024-04-25T02:06:59.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:06:58 smithi110 ceph-mon[32848]: pgmap v362: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 7.0 MiB/s wr, 690 op/s 2024-04-25T02:06:59.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:06:58 smithi193 ceph-mon[32981]: pgmap v362: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 7.0 MiB/s wr, 690 op/s 2024-04-25T02:06:59.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:06:58 smithi102 ceph-mon[27796]: pgmap v362: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 7.0 MiB/s wr, 690 op/s 2024-04-25T02:07:01.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:00 smithi110 ceph-mon[32848]: pgmap v363: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 513 op/s 2024-04-25T02:07:01.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:00 smithi193 ceph-mon[32981]: pgmap v363: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 513 op/s 2024-04-25T02:07:01.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:00 smithi102 ceph-mon[27796]: pgmap v363: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.4 MiB/s wr, 513 op/s 2024-04-25T02:07:03.077 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 670.0 s, 813.1 tps, lat 117.243 ms stddev 117.118 2024-04-25T02:07:03.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:02 smithi110 ceph-mon[32848]: pgmap v364: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 557 op/s 2024-04-25T02:07:03.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:02 smithi193 ceph-mon[32981]: pgmap v364: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 557 op/s 2024-04-25T02:07:03.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:02 smithi102 ceph-mon[27796]: pgmap v364: 97 pgs: 97 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 557 op/s 2024-04-25T02:07:05.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:04 smithi110 ceph-mon[32848]: pgmap v365: 97 pgs: 97 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 8.9 MiB/s wr, 901 op/s 2024-04-25T02:07:05.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:04 smithi193 ceph-mon[32981]: pgmap v365: 97 pgs: 97 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 8.9 MiB/s wr, 901 op/s 2024-04-25T02:07:05.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:04 smithi102 ceph-mon[27796]: pgmap v365: 97 pgs: 97 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 8.9 MiB/s wr, 901 op/s 2024-04-25T02:07:07.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:06 smithi193 ceph-mon[32981]: pgmap v366: 97 pgs: 97 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.0 MiB/s wr, 697 op/s 2024-04-25T02:07:07.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:06 smithi102 ceph-mon[27796]: pgmap v366: 97 pgs: 97 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.0 MiB/s wr, 697 op/s 2024-04-25T02:07:07.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:06 smithi110 ceph-mon[32848]: pgmap v366: 97 pgs: 97 active+clean; 5.3 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 7.0 MiB/s wr, 697 op/s 2024-04-25T02:07:08.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 675.0 s, 1080.0 tps, lat 98.222 ms stddev 163.835 2024-04-25T02:07:08.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:08 smithi193 ceph-mon[32981]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T02:07:08.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:08 smithi102 ceph-mon[27796]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T02:07:08.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:08 smithi110 ceph-mon[32848]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T02:07:09.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:09 smithi193 ceph-mon[32981]: pgmap v368: 97 pgs: 8 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 78 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 14 MiB/s wr, 1.44k op/s 2024-04-25T02:07:09.394 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:09 smithi110 ceph-mon[32848]: pgmap v368: 97 pgs: 8 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 78 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 14 MiB/s wr, 1.44k op/s 2024-04-25T02:07:09.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:09 smithi102 ceph-mon[27796]: pgmap v368: 97 pgs: 8 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 78 active+clean; 5.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 14 MiB/s wr, 1.44k op/s 2024-04-25T02:07:10.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.838+0000 7f9973000640 1 -- 172.21.15.102:0/3498273654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c102700 msgr2=0x7f996c102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:10.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.838+0000 7f9973000640 1 --2- 172.21.15.102:0/3498273654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c102700 0x7f996c102ae0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9964009ca0 tx=0x7f996402a940 comp rx=0 tx=0).stop 2024-04-25T02:07:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.838+0000 7f9973000640 1 -- 172.21.15.102:0/3498273654 shutdown_connections 2024-04-25T02:07:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.838+0000 7f9973000640 1 --2- 172.21.15.102:0/3498273654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996c10bb20 0x7f996c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.838+0000 7f9973000640 1 --2- 172.21.15.102:0/3498273654 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f996c103020 0x7f996c10b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.838+0000 7f9973000640 1 --2- 172.21.15.102:0/3498273654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c102700 0x7f996c102ae0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.838+0000 7f9973000640 1 -- 172.21.15.102:0/3498273654 >> 172.21.15.102:0/3498273654 conn(0x7f996c0fbf60 msgr2=0x7f996c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.839+0000 7f9973000640 1 -- 172.21.15.102:0/3498273654 shutdown_connections 2024-04-25T02:07:10.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.839+0000 7f9973000640 1 -- 172.21.15.102:0/3498273654 wait complete. 2024-04-25T02:07:10.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.840+0000 7f9973000640 1 Processor -- start 2024-04-25T02:07:10.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.840+0000 7f9973000640 1 -- start start 2024-04-25T02:07:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9973000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f996c102700 0x7f996c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9973000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c103020 0x7f996c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9973000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996c10bb20 0x7f996c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9973000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f996c111950 con 0x7f996c10bb20 2024-04-25T02:07:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9973000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1117d0 con 0x7f996c102700 2024-04-25T02:07:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9973000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f996c111ad0 con 0x7f996c103020 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9971200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c103020 0x7f996c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9971200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c103020 0x7f996c1983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35394/0 (socket says 172.21.15.102:35394) 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9971200640 1 -- 172.21.15.102:0/777058722 learned_addr learned my addr 172.21.15.102:0/777058722 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9971c00640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f996c102700 0x7f996c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.841+0000 7f9972600640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996c10bb20 0x7f996c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f9971c00640 1 -- 172.21.15.102:0/777058722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c103020 msgr2=0x7f996c1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f9971c00640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c103020 0x7f996c1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f9971c00640 1 -- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996c10bb20 msgr2=0x7f996c19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f9971c00640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996c10bb20 0x7f996c19c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f9971c00640 1 -- 172.21.15.102:0/777058722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f996c19ce50 con 0x7f996c102700 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f9971200640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c103020 0x7f996c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f9971c00640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f996c102700 0x7f996c197eb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9964005e30 tx=0x7f996402f620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:10.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f996a000640 1 -- 172.21.15.102:0/777058722 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964034040 con 0x7f996c102700 2024-04-25T02:07:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f996a000640 1 -- 172.21.15.102:0/777058722 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996402fe10 con 0x7f996c102700 2024-04-25T02:07:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f996a000640 1 -- 172.21.15.102:0/777058722 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964004600 con 0x7f996c102700 2024-04-25T02:07:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f9973000640 1 -- 172.21.15.102:0/777058722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f996c19d0e0 con 0x7f996c102700 2024-04-25T02:07:10.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.842+0000 7f9973000640 1 -- 172.21.15.102:0/777058722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f996c100280 con 0x7f996c102700 2024-04-25T02:07:10.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.848+0000 7f996a000640 1 -- 172.21.15.102:0/777058722 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9964037020 con 0x7f996c102700 2024-04-25T02:07:10.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.848+0000 7f997741d740 1 Processor -- start 2024-04-25T02:07:10.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.848+0000 7f997741d740 1 -- start start 2024-04-25T02:07:10.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f997741d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfaa1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f997741d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaac460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f997741d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfaaed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f997741d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c7bf9fcab0 con 0x55c7bfaa17c0 2024-04-25T02:07:10.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f997741d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c7bf9a5640 con 0x55c7bfaa20e0 2024-04-25T02:07:10.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f997741d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c7bf9f8a70 con 0x55c7bfaac9a0 2024-04-25T02:07:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f9972600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfaa1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f9971200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfaaed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f9972600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfaa1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51352/0 (socket says 172.21.15.102:51352) 2024-04-25T02:07:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f9972600640 1 -- 172.21.15.102:0/782581615 learned_addr learned my addr 172.21.15.102:0/782581615 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.849+0000 7f9971c00640 1 --2- 172.21.15.102:0/782581615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f9971200640 1 -- 172.21.15.102:0/782581615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 msgr2=0x55c7bfaac460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f9971200640 1 --2- 172.21.15.102:0/782581615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaac460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f9971200640 1 -- 172.21.15.102:0/782581615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 msgr2=0x55c7bfaa1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f9971200640 1 --2- 172.21.15.102:0/782581615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfaa1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f9971200640 1 -- 172.21.15.102:0/782581615 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7bf711210 con 0x55c7bfaac9a0 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f996a000640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99540a2d70 0x7f99540a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f9971c00640 1 --2- 172.21.15.102:0/782581615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f996a000640 1 -- 172.21.15.102:0/777058722 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8557+0+0 (secure 0 0 0) 0x7f99640f0120 con 0x7f996c102700 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f9971200640 1 --2- 172.21.15.102:0/782581615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfaaed90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f996c0fe5f0 tx=0x7f99600119a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51c1d65f52bc3091 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f9972600640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99540a2d70 0x7f99540a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f995aa00640 1 -- 172.21.15.102:0/782581615 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996001e070 con 0x55c7bfaac9a0 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f995aa00640 1 -- 172.21.15.102:0/782581615 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996000fa40 con 0x55c7bfaac9a0 2024-04-25T02:07:10.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.850+0000 7f995aa00640 1 -- 172.21.15.102:0/782581615 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9960019bf0 con 0x55c7bfaac9a0 2024-04-25T02:07:10.859 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.851+0000 7f997741d740 1 -- 172.21.15.102:0/782581615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 msgr2=0x55c7bfaaed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:10.859 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.851+0000 7f997741d740 1 --2- 172.21.15.102:0/782581615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfaaed90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f996c0fe5f0 tx=0x7f99600119a0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.859 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.851+0000 7f9972600640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99540a2d70 0x7f99540a5230 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f995c009e40 tx=0x7f995c005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:10.859 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.851+0000 7f997741d740 1 -- 172.21.15.102:0/782581615 shutdown_connections 2024-04-25T02:07:10.859 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.851+0000 7f997741d740 1 --2- 172.21.15.102:0/782581615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfaaed90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.851+0000 7f997741d740 1 --2- 172.21.15.102:0/782581615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.851+0000 7f997741d740 1 --2- 172.21.15.102:0/782581615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfaa1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.851+0000 7f997741d740 1 -- 172.21.15.102:0/782581615 >> 172.21.15.102:0/782581615 conn(0x55c7bfa9dcd0 msgr2=0x55c7bfa9e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.851+0000 7f997741d740 1 -- 172.21.15.102:0/782581615 shutdown_connections 2024-04-25T02:07:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.852+0000 7f997741d740 1 -- 172.21.15.102:0/782581615 wait complete. 2024-04-25T02:07:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.852+0000 7f997741d740 1 -- start start 2024-04-25T02:07:10.860 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.853+0000 7f997741d740 1 Processor -- start 2024-04-25T02:07:10.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f997741d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfaa99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f997741d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaa9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f997741d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaac9a0 0x55c7bfaaa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f997741d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55c7bfaaf5d0 con 0x55c7bfaa17c0 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f997741d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55c7bf9b8990 con 0x55c7bfaac9a0 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f997741d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c7bfaaf750 con 0x55c7bfaa20e0 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f997741d740 10 client.0 _ll_register_callbacks cb 0x55c7bf9ba950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9971200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaa9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9972600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaac9a0 0x55c7bfaaa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9971200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaa9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35400/0 (socket says 172.21.15.102:35400) 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9971200640 1 -- 172.21.15.102:0/2221197974 learned_addr learned my addr 172.21.15.102:0/2221197974 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9971c00640 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfaa99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9971200640 1 -- 172.21.15.102:0/2221197974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaac9a0 msgr2=0x55c7bfaaa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9971200640 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaac9a0 0x55c7bfaaa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9971200640 1 -- 172.21.15.102:0/2221197974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 msgr2=0x55c7bfaa99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9971200640 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfaa99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.854+0000 7f9971200640 1 -- 172.21.15.102:0/2221197974 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7bf7dd620 con 0x55c7bfaa20e0 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.855+0000 7f9971200640 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaa9f00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9960019770 tx=0x7f996000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d407f9b5df19ba97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.855+0000 7f994f400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.855+0000 7f994f400640 1 -- 172.21.15.102:0/2221197974 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996001e040 con 0x55c7bfaa20e0 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.855+0000 7f997741d740 1 -- 172.21.15.102:0/2221197974 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f996000fa40 con 0x55c7bfaa20e0 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.855+0000 7f994f400640 1 -- 172.21.15.102:0/2221197974 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8557+0+0 (secure 0 0 0) 0x7f9960019970 con 0x55c7bfaa20e0 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.855+0000 7f994f400640 1 -- 172.21.15.102:0/2221197974 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f9960032050 con 0x55c7bfaa20e0 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.855+0000 7f997741d740 10 client.27445 fetch_fsmap learned FSMap version 43 2024-04-25T02:07:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.855+0000 7f997741d740 1 -- 172.21.15.102:0/2221197974 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c7bfabb530 con 0x55c7bfaa20e0 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f994f400640 1 -- 172.21.15.102:0/2221197974 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996000fa40 con 0x55c7bfaa20e0 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f994f400640 1 -- 172.21.15.102:0/2221197974 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f9960021030 con 0x55c7bfaa20e0 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f997741d740 10 client.27445 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f997741d740 10 client.27445 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f997741d740 20 client.27445 populate_metadata read hostname 'smithi102' 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f997741d740 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c7bfabc8e0 0x55c7bfabecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f997741d740 4 client.27445 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f997741d740 1 -- 172.21.15.102:0/2221197974 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7bfa09ff0 con 0x55c7bfabc8e0 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f9971c00640 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c7bfabc8e0 0x55c7bfabecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f994f400640 10 client.27445 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.856+0000 7f9971c00640 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c7bfabc8e0 0x55c7bfabecd0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f596670ef452a20 server_cookie=793abe7a1a7541dc in_seq=0 out_seq=0 2024-04-25T02:07:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.857+0000 7f994f400640 1 -- 172.21.15.102:0/2221197974 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7bfa09ff0 con 0x55c7bfabc8e0 2024-04-25T02:07:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.857+0000 7f994f400640 10 client.27445 handle_command_reply: tid=0 2024-04-25T02:07:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.857+0000 7f997741d740 1 client.27445 shutdown 2024-04-25T02:07:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.857+0000 7f997741d740 1 -- 172.21.15.102:0/2221197974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaa20e0 msgr2=0x55c7bfaa9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.857+0000 7f997741d740 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaa9f00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9960019770 tx=0x7f996000f790 comp rx=0 tx=0).stop 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f997741d740 1 -- 172.21.15.102:0/2221197974 shutdown_connections 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f997741d740 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c7bfabc8e0 0x55c7bfabecd0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f997741d740 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaac9a0 0x55c7bfaaa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f997741d740 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaa20e0 0x55c7bfaa9f00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f997741d740 1 --2- 172.21.15.102:0/2221197974 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfaa99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f994f400640 0 client.27445 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f997741d740 1 -- 172.21.15.102:0/2221197974 >> 172.21.15.102:0/2221197974 conn(0x55c7bfa9dcd0 msgr2=0x55c7bfa94620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f997741d740 1 -- 172.21.15.102:0/2221197974 shutdown_connections 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f997741d740 1 -- 172.21.15.102:0/2221197974 wait complete. 2024-04-25T02:07:10.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.858+0000 7f997741d740 20 client.27445 trim_cache size 0 max 16384 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.881+0000 7f997741d740 1 Processor -- start 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.881+0000 7f997741d740 1 -- start start 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f997741d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfab8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f997741d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfa944a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f997741d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfac44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f997741d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1adac0 con 0x55c7bfaa17c0 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f997741d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1ab300 con 0x55c7bfaa20e0 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f997741d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1a9170 con 0x55c7bfaac9a0 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9971200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfa944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9971200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfa944a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:38802/0 (socket says 172.21.15.102:38802) 2024-04-25T02:07:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9971200640 1 -- 172.21.15.102:0/2678281081 learned_addr learned my addr 172.21.15.102:0/2678281081 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9971c00640 1 --2- 172.21.15.102:0/2678281081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfab8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9972600640 1 --2- 172.21.15.102:0/2678281081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfac44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9971200640 1 -- 172.21.15.102:0/2678281081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 msgr2=0x55c7bfac44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9971200640 1 --2- 172.21.15.102:0/2678281081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfac44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9971200640 1 -- 172.21.15.102:0/2678281081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 msgr2=0x55c7bfab8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9971200640 1 --2- 172.21.15.102:0/2678281081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfab8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.882+0000 7f9971200640 1 -- 172.21.15.102:0/2678281081 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7bfaaf4b0 con 0x55c7bfaa20e0 2024-04-25T02:07:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.883+0000 7f9971200640 1 --2- 172.21.15.102:0/2678281081 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfa944a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9960030040 tx=0x7f9960019620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=954824bf91ab0286 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.883+0000 7f995a000640 1 -- 172.21.15.102:0/2678281081 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996001e040 con 0x55c7bfaa20e0 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.883+0000 7f995a000640 1 -- 172.21.15.102:0/2678281081 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9960016070 con 0x55c7bfaa20e0 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.883+0000 7f995a000640 1 -- 172.21.15.102:0/2678281081 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99600246a0 con 0x55c7bfaa20e0 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.884+0000 7f997741d740 1 -- 172.21.15.102:0/2678281081 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 msgr2=0x55c7bfa944a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.884+0000 7f997741d740 1 --2- 172.21.15.102:0/2678281081 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfa944a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9960030040 tx=0x7f9960019620 comp rx=0 tx=0).stop 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.884+0000 7f997741d740 1 -- 172.21.15.102:0/2678281081 shutdown_connections 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.884+0000 7f997741d740 1 --2- 172.21.15.102:0/2678281081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfac44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.884+0000 7f997741d740 1 --2- 172.21.15.102:0/2678281081 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfa944a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.884+0000 7f997741d740 1 --2- 172.21.15.102:0/2678281081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfab8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.884+0000 7f997741d740 1 -- 172.21.15.102:0/2678281081 >> 172.21.15.102:0/2678281081 conn(0x55c7bfa9dcd0 msgr2=0x55c7bfa93d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.884+0000 7f997741d740 1 -- 172.21.15.102:0/2678281081 shutdown_connections 2024-04-25T02:07:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.884+0000 7f997741d740 1 -- 172.21.15.102:0/2678281081 wait complete. 2024-04-25T02:07:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.885+0000 7f997741d740 1 -- start start 2024-04-25T02:07:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.885+0000 7f997741d740 1 Processor -- start 2024-04-25T02:07:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f997741d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfab4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f997741d740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfab1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f997741d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfab1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f997741d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1a97c0 con 0x55c7bfaa17c0 2024-04-25T02:07:10.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f997741d740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1aac30 con 0x55c7bfaa20e0 2024-04-25T02:07:10.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f997741d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f996c1a9e90 con 0x55c7bfaac9a0 2024-04-25T02:07:10.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f997741d740 10 client.0 _ll_register_callbacks cb 0x7f996001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f9971200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfab1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f9971200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfab1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:38810/0 (socket says 172.21.15.102:38810) 2024-04-25T02:07:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f9971c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfab4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f9971200640 1 -- 172.21.15.102:0/4018051259 learned_addr learned my addr 172.21.15.102:0/4018051259 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.886+0000 7f9972600640 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfab1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.887+0000 7f9971200640 1 -- 172.21.15.102:0/4018051259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 msgr2=0x55c7bfab1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.887+0000 7f9971200640 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfab1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.887+0000 7f9971200640 1 -- 172.21.15.102:0/4018051259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 msgr2=0x55c7bfab4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.887+0000 7f9971200640 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfab4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.887+0000 7f9971200640 1 -- 172.21.15.102:0/4018051259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7bfab2420 con 0x55c7bfaa20e0 2024-04-25T02:07:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.887+0000 7f9971c00640 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfab4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:07:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.887+0000 7f9971200640 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfab1970 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f99600119e0 tx=0x7f9960004890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20ec8ceb907de3a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.887+0000 7f994f400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:07:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.888+0000 7f994f400640 1 -- 172.21.15.102:0/4018051259 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996001e040 con 0x55c7bfaa20e0 2024-04-25T02:07:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.888+0000 7f994f400640 1 -- 172.21.15.102:0/4018051259 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8557+0+0 (secure 0 0 0) 0x7f9960019710 con 0x55c7bfaa20e0 2024-04-25T02:07:10.895 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.888+0000 7f997741d740 1 -- 172.21.15.102:0/4018051259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9960016070 con 0x55c7bfaa20e0 2024-04-25T02:07:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.888+0000 7f994f400640 1 -- 172.21.15.102:0/4018051259 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f9960019910 con 0x55c7bfaa20e0 2024-04-25T02:07:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.888+0000 7f997741d740 10 client.27488 fetch_fsmap learned FSMap version 43 2024-04-25T02:07:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.888+0000 7f997741d740 1 -- 172.21.15.102:0/4018051259 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c7bfb23ab0 con 0x55c7bfaa20e0 2024-04-25T02:07:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.888+0000 7f994f400640 1 -- 172.21.15.102:0/4018051259 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9960016070 con 0x55c7bfaa20e0 2024-04-25T02:07:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.889+0000 7f994f400640 1 -- 172.21.15.102:0/4018051259 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f9960021070 con 0x55c7bfaa20e0 2024-04-25T02:07:10.896 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.889+0000 7f997741d740 10 client.27488 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:07:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.889+0000 7f997741d740 10 client.27488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.889+0000 7f997741d740 20 client.27488 populate_metadata read hostname 'smithi102' 2024-04-25T02:07:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.889+0000 7f997741d740 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c7bfb24140 0x55c7bfb26560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.889+0000 7f997741d740 4 client.27488 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:07:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.889+0000 7f997741d740 1 -- 172.21.15.102:0/4018051259 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c7bfa04340 con 0x55c7bfb24140 2024-04-25T02:07:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.889+0000 7f9971c00640 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c7bfb24140 0x55c7bfb26560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.890+0000 7f9971c00640 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c7bfb24140 0x55c7bfb26560 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d368590d81c2e88 server_cookie=3160a21638e0d24e in_seq=0 out_seq=0 2024-04-25T02:07:10.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.890+0000 7f994f400640 10 client.27488 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f994f400640 1 -- 172.21.15.102:0/4018051259 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c7bfa04340 con 0x55c7bfb24140 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f994f400640 10 client.27488 handle_command_reply: tid=0 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 client.27488 shutdown 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 -- 172.21.15.102:0/4018051259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 msgr2=0x55c7bfab1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfab1970 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f99600119e0 tx=0x7f9960004890 comp rx=0 tx=0).stop 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 -- 172.21.15.102:0/4018051259 shutdown_connections 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55c7bfb24140 0x55c7bfb26560 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c7bfaac9a0 0x55c7bfab1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c7bfaa20e0 0x55c7bfab1970 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f994f400640 0 client.27488 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 --2- 172.21.15.102:0/4018051259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c7bfaa17c0 0x55c7bfab4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 -- 172.21.15.102:0/4018051259 >> 172.21.15.102:0/4018051259 conn(0x55c7bfa9dcd0 msgr2=0x55c7bfaa3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 -- 172.21.15.102:0/4018051259 shutdown_connections 2024-04-25T02:07:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 1 -- 172.21.15.102:0/4018051259 wait complete. 2024-04-25T02:07:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.909+0000 7f997741d740 20 client.27488 trim_cache size 0 max 16384 2024-04-25T02:07:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:07:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:07:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:07:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 -- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99540a2d70 msgr2=0x7f99540a5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99540a2d70 0x7f99540a5230 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f995c009e40 tx=0x7f995c005f90 comp rx=0 tx=0).stop 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 -- 172.21.15.102:0/777058722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f996c102700 msgr2=0x7f996c197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f996c102700 0x7f996c197eb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9964005e30 tx=0x7f996402f620 comp rx=0 tx=0).stop 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 -- 172.21.15.102:0/777058722 shutdown_connections 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99540a2d70 0x7f99540a5230 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f996c10bb20 0x7f996c19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996c103020 0x7f996c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 --2- 172.21.15.102:0/777058722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f996c102700 0x7f996c197eb0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 -- 172.21.15.102:0/777058722 >> 172.21.15.102:0/777058722 conn(0x7f996c0fbf60 msgr2=0x7f996c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 -- 172.21.15.102:0/777058722 shutdown_connections 2024-04-25T02:07:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:10.910+0000 7f9973000640 1 -- 172.21.15.102:0/777058722 wait complete. 2024-04-25T02:07:11.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:11 smithi102 ceph-mon[27796]: pgmap v369: 97 pgs: 8 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 78 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-25T02:07:11.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:11 smithi110 ceph-mon[32848]: pgmap v369: 97 pgs: 8 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 78 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-25T02:07:11.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:11 smithi193 ceph-mon[32981]: pgmap v369: 97 pgs: 8 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 78 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-25T02:07:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 680.0 s, 1477.5 tps, lat 67.754 ms stddev 85.181 2024-04-25T02:07:13.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:13 smithi102 ceph-mon[27796]: pgmap v370: 97 pgs: 8 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 78 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-25T02:07:13.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:13 smithi110 ceph-mon[32848]: pgmap v370: 97 pgs: 8 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 78 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-25T02:07:13.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:13 smithi193 ceph-mon[32981]: pgmap v370: 97 pgs: 8 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 78 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 15 MiB/s wr, 1.58k op/s 2024-04-25T02:07:15.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:15 smithi102 ceph-mon[27796]: pgmap v371: 97 pgs: 97 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 14 MiB/s wr, 1.39k op/s 2024-04-25T02:07:15.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:15 smithi110 ceph-mon[32848]: pgmap v371: 97 pgs: 97 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 14 MiB/s wr, 1.39k op/s 2024-04-25T02:07:15.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:15 smithi193 ceph-mon[32981]: pgmap v371: 97 pgs: 97 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 14 MiB/s wr, 1.39k op/s 2024-04-25T02:07:17.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:17 smithi102 ceph-mon[27796]: pgmap v372: 97 pgs: 97 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 14 MiB/s wr, 1.39k op/s 2024-04-25T02:07:17.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:17 smithi110 ceph-mon[32848]: pgmap v372: 97 pgs: 97 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 14 MiB/s wr, 1.39k op/s 2024-04-25T02:07:17.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:17 smithi193 ceph-mon[32981]: pgmap v372: 97 pgs: 97 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 14 MiB/s wr, 1.39k op/s 2024-04-25T02:07:18.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 685.0 s, 2278.7 tps, lat 43.855 ms stddev 60.916 2024-04-25T02:07:18.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:18 smithi102 ceph-mon[27796]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T02:07:18.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:18 smithi110 ceph-mon[32848]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T02:07:18.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:18 smithi193 ceph-mon[32981]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T02:07:19.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:19 smithi102 ceph-mon[27796]: pgmap v374: 97 pgs: 97 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 6.9 MiB/s wr, 746 op/s 2024-04-25T02:07:19.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:19 smithi110 ceph-mon[32848]: pgmap v374: 97 pgs: 97 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 6.9 MiB/s wr, 746 op/s 2024-04-25T02:07:19.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:19 smithi193 ceph-mon[32981]: pgmap v374: 97 pgs: 97 active+clean; 4.9 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 6.9 MiB/s wr, 746 op/s 2024-04-25T02:07:21.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:21 smithi102 ceph-mon[27796]: pgmap v375: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.8 MiB/s wr, 637 op/s 2024-04-25T02:07:21.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:21 smithi110 ceph-mon[32848]: pgmap v375: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.8 MiB/s wr, 637 op/s 2024-04-25T02:07:21.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:21 smithi193 ceph-mon[32981]: pgmap v375: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.8 MiB/s wr, 637 op/s 2024-04-25T02:07:23.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 690.0 s, 2247.2 tps, lat 44.687 ms stddev 53.144 2024-04-25T02:07:23.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:23 smithi102 ceph-mon[27796]: pgmap v376: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.5 MiB/s wr, 604 op/s 2024-04-25T02:07:23.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:23 smithi110 ceph-mon[32848]: pgmap v376: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.5 MiB/s wr, 604 op/s 2024-04-25T02:07:23.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:23 smithi193 ceph-mon[32981]: pgmap v376: 97 pgs: 97 active+clean; 5.0 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 5.5 MiB/s wr, 604 op/s 2024-04-25T02:07:24.079 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.071+0000 7fd819e00640 1 -- 172.21.15.102:0/2038665045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814107760 msgr2=0x7fd814107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.071+0000 7fd819e00640 1 --2- 172.21.15.102:0/2038665045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814107760 0x7fd814107b40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fd804009900 tx=0x7fd80402a400 comp rx=0 tx=0).stop 2024-04-25T02:07:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.071+0000 7fd819e00640 1 -- 172.21.15.102:0/2038665045 shutdown_connections 2024-04-25T02:07:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.071+0000 7fd819e00640 1 --2- 172.21.15.102:0/2038665045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd814102110 0x7fd81410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.071+0000 7fd819e00640 1 --2- 172.21.15.102:0/2038665045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd814101750 0x7fd814101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.071+0000 7fd819e00640 1 --2- 172.21.15.102:0/2038665045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814107760 0x7fd814107b40 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.071+0000 7fd819e00640 1 -- 172.21.15.102:0/2038665045 >> 172.21.15.102:0/2038665045 conn(0x7fd814078bd0 msgr2=0x7fd814078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.072+0000 7fd819e00640 1 -- 172.21.15.102:0/2038665045 shutdown_connections 2024-04-25T02:07:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.072+0000 7fd819e00640 1 -- 172.21.15.102:0/2038665045 wait complete. 2024-04-25T02:07:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.073+0000 7fd819e00640 1 Processor -- start 2024-04-25T02:07:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.073+0000 7fd819e00640 1 -- start start 2024-04-25T02:07:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.073+0000 7fd819e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd814101750 0x7fd81419e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.073+0000 7fd819e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814102110 0x7fd81419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.073+0000 7fd819e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd814107760 0x7fd8141a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.073+0000 7fd819e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd81410f9c0 con 0x7fd814101750 2024-04-25T02:07:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.074+0000 7fd819e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd81410f840 con 0x7fd814102110 2024-04-25T02:07:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.074+0000 7fd819e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd81410fb40 con 0x7fd814107760 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.074+0000 7fd812000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814102110 0x7fd81419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.074+0000 7fd812000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814102110 0x7fd81419eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34902/0 (socket says 172.21.15.102:34902) 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.074+0000 7fd812000640 1 -- 172.21.15.102:0/3524761510 learned_addr learned my addr 172.21.15.102:0/3524761510 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.074+0000 7fd813400640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd814107760 0x7fd8141a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.074+0000 7fd812a00640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd814101750 0x7fd81419e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd812000640 1 -- 172.21.15.102:0/3524761510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd814107760 msgr2=0x7fd8141a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd812000640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd814107760 0x7fd8141a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd812000640 1 -- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd814101750 msgr2=0x7fd81419e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd812000640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd814101750 0x7fd81419e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd812000640 1 -- 172.21.15.102:0/3524761510 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8141a3590 con 0x7fd814102110 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd813400640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd814107760 0x7fd8141a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd812a00640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd814101750 0x7fd81419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:07:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd812000640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814102110 0x7fd81419eb10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd80000db20 tx=0x7fd80000af60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd818a00640 1 -- 172.21.15.102:0/3524761510 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80001a070 con 0x7fd814102110 2024-04-25T02:07:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd818a00640 1 -- 172.21.15.102:0/3524761510 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd80000cc70 con 0x7fd814102110 2024-04-25T02:07:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd819e00640 1 -- 172.21.15.102:0/3524761510 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8141a3880 con 0x7fd814102110 2024-04-25T02:07:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd818a00640 1 -- 172.21.15.102:0/3524761510 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800014730 con 0x7fd814102110 2024-04-25T02:07:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.075+0000 7fd819e00640 1 -- 172.21.15.102:0/3524761510 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd814195b10 con 0x7fd814102110 2024-04-25T02:07:24.088 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.081+0000 7fd818a00640 1 -- 172.21.15.102:0/3524761510 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd80001d070 con 0x7fd814102110 2024-04-25T02:07:24.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.081+0000 7fd81cbc0740 1 Processor -- start 2024-04-25T02:07:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.082+0000 7fd81cbc0740 1 -- start start 2024-04-25T02:07:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 0x559d7eb3e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb3e720 0x559d7eb48aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb48fe0 0x559d7eb4b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd81cbc0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559d7eaa1ab0 con 0x559d7eb3de00 2024-04-25T02:07:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd81cbc0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559d7ea4a640 con 0x559d7eb48fe0 2024-04-25T02:07:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd81cbc0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559d7ea9da70 con 0x559d7eb3e720 2024-04-25T02:07:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd813400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 0x559d7eb3e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd812a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb3e720 0x559d7eb48aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd812a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb3e720 0x559d7eb48aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:39412/0 (socket says 172.21.15.102:39412) 2024-04-25T02:07:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.083+0000 7fd812a00640 1 -- 172.21.15.102:0/381405290 learned_addr learned my addr 172.21.15.102:0/381405290 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.084+0000 7fd812000640 1 --2- 172.21.15.102:0/381405290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb48fe0 0x559d7eb4b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.084+0000 7fd818a00640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7f80a2ea0 0x7fd7f80a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.084+0000 7fd818a00640 1 -- 172.21.15.102:0/3524761510 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7fd8000ce950 con 0x7fd814102110 2024-04-25T02:07:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.084+0000 7fd813400640 1 -- 172.21.15.102:0/381405290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb3e720 msgr2=0x559d7eb48aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.084+0000 7fd813400640 1 --2- 172.21.15.102:0/381405290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb3e720 0x559d7eb48aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.084+0000 7fd813400640 1 -- 172.21.15.102:0/381405290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb48fe0 msgr2=0x559d7eb4b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.084+0000 7fd813400640 1 --2- 172.21.15.102:0/381405290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb48fe0 0x559d7eb4b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.084+0000 7fd813400640 1 -- 172.21.15.102:0/381405290 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d7e7b6210 con 0x559d7eb3de00 2024-04-25T02:07:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.084+0000 7fd812a00640 1 --2- 172.21.15.102:0/381405290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb3e720 0x559d7eb48aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.085+0000 7fd813400640 1 --2- 172.21.15.102:0/381405290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 0x559d7eb3e1e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd80c010710 tx=0x7fd80c010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382d956b90a3f07f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:24.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.085+0000 7fd813400640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7f80a2ea0 0x7fd7f80a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.086+0000 7fd7ffe00640 1 -- 172.21.15.102:0/381405290 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c00ecd0 con 0x559d7eb3de00 2024-04-25T02:07:24.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.086+0000 7fd7ffe00640 1 -- 172.21.15.102:0/381405290 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd80c012930 con 0x559d7eb3de00 2024-04-25T02:07:24.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.086+0000 7fd7ffe00640 1 -- 172.21.15.102:0/381405290 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c01cc10 con 0x559d7eb3de00 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd813400640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7f80a2ea0 0x7fd7f80a5360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd80c01a000 tx=0x7fd80c0117b0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd81cbc0740 1 -- 172.21.15.102:0/381405290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 msgr2=0x559d7eb3e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/381405290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 0x559d7eb3e1e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd80c010710 tx=0x7fd80c010bd0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd81cbc0740 1 -- 172.21.15.102:0/381405290 shutdown_connections 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/381405290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb48fe0 0x559d7eb4b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/381405290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb3e720 0x559d7eb48aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/381405290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 0x559d7eb3e1e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd81cbc0740 1 -- 172.21.15.102:0/381405290 >> 172.21.15.102:0/381405290 conn(0x559d7eb3a310 msgr2=0x559d7eb3a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd81cbc0740 1 -- 172.21.15.102:0/381405290 shutdown_connections 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.087+0000 7fd81cbc0740 1 -- 172.21.15.102:0/381405290 wait complete. 2024-04-25T02:07:24.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.088+0000 7fd81cbc0740 1 -- start start 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.088+0000 7fd81cbc0740 1 Processor -- start 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.088+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb45f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.088+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3e720 0x559d7eb46480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.088+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 0x559d7eb469c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.089+0000 7fd81cbc0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559d7eb4bc10 con 0x559d7eb3e720 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.089+0000 7fd81cbc0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559d7ea5d990 con 0x559d7eb3de00 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.089+0000 7fd81cbc0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559d7eb4bd90 con 0x559d7eb48fe0 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.089+0000 7fd81cbc0740 10 client.0 _ll_register_callbacks cb 0x559d7ea5f950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.089+0000 7fd812a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb45f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.089+0000 7fd812000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3e720 0x559d7eb46480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.089+0000 7fd812000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3e720 0x559d7eb46480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48666/0 (socket says 172.21.15.102:48666) 2024-04-25T02:07:24.097 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.089+0000 7fd812000640 1 -- 172.21.15.102:0/460572755 learned_addr learned my addr 172.21.15.102:0/460572755 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.092+0000 7fd812000640 1 -- 172.21.15.102:0/460572755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 msgr2=0x559d7eb469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.092+0000 7fd812000640 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 0x559d7eb469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.092+0000 7fd812000640 1 -- 172.21.15.102:0/460572755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 msgr2=0x559d7eb45f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.092+0000 7fd812000640 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb45f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.099 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.092+0000 7fd812000640 1 -- 172.21.15.102:0/460572755 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d7e882620 con 0x559d7eb3e720 2024-04-25T02:07:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.092+0000 7fd812000640 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3e720 0x559d7eb46480 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd80009abc0 tx=0x7fd80000ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a2787fb8a42526 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:24.100 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.092+0000 7fd7fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:07:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.093+0000 7fd7fcc00640 1 -- 172.21.15.102:0/460572755 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80009f1c0 con 0x559d7eb3e720 2024-04-25T02:07:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.093+0000 7fd7fcc00640 1 -- 172.21.15.102:0/460572755 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7fd80009a310 con 0x559d7eb3e720 2024-04-25T02:07:24.103 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.096+0000 7fd81cbc0740 1 -- 172.21.15.102:0/460572755 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd80c012930 con 0x559d7eb3e720 2024-04-25T02:07:24.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.098+0000 7fd7fcc00640 1 -- 172.21.15.102:0/460572755 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd8000a3a80 con 0x559d7eb3e720 2024-04-25T02:07:24.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.098+0000 7fd81cbc0740 10 client.16812 fetch_fsmap learned FSMap version 43 2024-04-25T02:07:24.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.098+0000 7fd81cbc0740 1 -- 172.21.15.102:0/460572755 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559d7eb57aa0 con 0x559d7eb3e720 2024-04-25T02:07:24.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.099+0000 7fd7fcc00640 1 -- 172.21.15.102:0/460572755 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8000b8030 con 0x559d7eb3e720 2024-04-25T02:07:24.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.099+0000 7fd7fcc00640 1 -- 172.21.15.102:0/460572755 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7fd800079ce0 con 0x559d7eb3e720 2024-04-25T02:07:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.101+0000 7fd81cbc0740 10 client.16812 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:07:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.101+0000 7fd81cbc0740 10 client.16812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.101+0000 7fd81cbc0740 20 client.16812 populate_metadata read hostname 'smithi102' 2024-04-25T02:07:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.101+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559d7eb58fb0 0x559d7eb5b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.101+0000 7fd81cbc0740 4 client.16812 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:07:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.101+0000 7fd81cbc0740 1 -- 172.21.15.102:0/460572755 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d7eaaeff0 con 0x559d7eb58fb0 2024-04-25T02:07:24.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.102+0000 7fd812a00640 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559d7eb58fb0 0x559d7eb5b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.103+0000 7fd812a00640 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559d7eb58fb0 0x559d7eb5b420 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12215376cb5859a6 server_cookie=6b912de26273e18e in_seq=0 out_seq=0 2024-04-25T02:07:24.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.105+0000 7fd7fcc00640 10 client.16812 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.106+0000 7fd7fcc00640 1 -- 172.21.15.102:0/460572755 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d7eaaeff0 con 0x559d7eb58fb0 2024-04-25T02:07:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.106+0000 7fd7fcc00640 10 client.16812 handle_command_reply: tid=0 2024-04-25T02:07:24.114 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.106+0000 7fd81cbc0740 1 client.16812 shutdown 2024-04-25T02:07:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.107+0000 7fd81cbc0740 1 -- 172.21.15.102:0/460572755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3e720 msgr2=0x559d7eb46480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.107+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3e720 0x559d7eb46480 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd80009abc0 tx=0x7fd80000ce20 comp rx=0 tx=0).stop 2024-04-25T02:07:24.116 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.107+0000 7fd81cbc0740 1 -- 172.21.15.102:0/460572755 shutdown_connections 2024-04-25T02:07:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.107+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559d7eb58fb0 0x559d7eb5b420 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.107+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 0x559d7eb469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.107+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3e720 0x559d7eb46480 secure :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd80009abc0 tx=0x7fd80000ce20 comp rx=0 tx=0).stop 2024-04-25T02:07:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.107+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/460572755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb45f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.107+0000 7fd81cbc0740 1 -- 172.21.15.102:0/460572755 >> 172.21.15.102:0/460572755 conn(0x559d7eb3a310 msgr2=0x559d7eb40040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.107+0000 7fd7fcc00640 0 client.16812 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:24.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.108+0000 7fd81cbc0740 1 -- 172.21.15.102:0/460572755 shutdown_connections 2024-04-25T02:07:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.108+0000 7fd81cbc0740 1 -- 172.21.15.102:0/460572755 wait complete. 2024-04-25T02:07:24.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.109+0000 7fd81cbc0740 20 client.16812 trim_cache size 0 max 16384 2024-04-25T02:07:24.165 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.157+0000 7fd81cbc0740 1 Processor -- start 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.157+0000 7fd81cbc0740 1 -- start start 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.157+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 0x559d7eb54ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.157+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 0x559d7eb60d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.157+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb35410 0x559d7eb61270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.158+0000 7fd81cbc0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd8141bebe0 con 0x559d7eb3de00 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.158+0000 7fd81cbc0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd8141bc420 con 0x559d7eb35410 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.158+0000 7fd81cbc0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd81419c450 con 0x559d7eb48fe0 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.158+0000 7fd812000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 0x559d7eb60d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.158+0000 7fd812000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 0x559d7eb60d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:39442/0 (socket says 172.21.15.102:39442) 2024-04-25T02:07:24.166 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.158+0000 7fd812000640 1 -- 172.21.15.102:0/243009071 learned_addr learned my addr 172.21.15.102:0/243009071 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:24.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.158+0000 7fd813400640 1 --2- 172.21.15.102:0/243009071 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb35410 0x559d7eb61270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.159+0000 7fd812000640 1 -- 172.21.15.102:0/243009071 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb35410 msgr2=0x559d7eb61270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:24.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.159+0000 7fd812000640 1 --2- 172.21.15.102:0/243009071 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb35410 0x559d7eb61270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.159+0000 7fd812000640 1 -- 172.21.15.102:0/243009071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 msgr2=0x559d7eb54ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:24.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.159+0000 7fd812000640 1 --2- 172.21.15.102:0/243009071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 0x559d7eb54ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.159+0000 7fd812000640 1 -- 172.21.15.102:0/243009071 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d7eb4b700 con 0x559d7eb48fe0 2024-04-25T02:07:24.167 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.159+0000 7fd812000640 1 --2- 172.21.15.102:0/243009071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 0x559d7eb60d30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd80000afd0 tx=0x7fd8000cc120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4e9a4835abca5de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:24.169 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.162+0000 7fd7ff400640 1 -- 172.21.15.102:0/243009071 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80009f1c0 con 0x559d7eb48fe0 2024-04-25T02:07:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.162+0000 7fd7ff400640 1 -- 172.21.15.102:0/243009071 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8000b8070 con 0x559d7eb48fe0 2024-04-25T02:07:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.162+0000 7fd7ff400640 1 -- 172.21.15.102:0/243009071 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80009a570 con 0x559d7eb48fe0 2024-04-25T02:07:24.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.163+0000 7fd81cbc0740 1 -- 172.21.15.102:0/243009071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 msgr2=0x559d7eb60d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:24.171 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.163+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/243009071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 0x559d7eb60d30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd80000afd0 tx=0x7fd8000cc120 comp rx=0 tx=0).stop 2024-04-25T02:07:24.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.166+0000 7fd81cbc0740 1 -- 172.21.15.102:0/243009071 shutdown_connections 2024-04-25T02:07:24.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.166+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/243009071 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb35410 0x559d7eb61270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.166+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/243009071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb48fe0 0x559d7eb60d30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.166+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/243009071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb3de00 0x559d7eb54ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.166+0000 7fd81cbc0740 1 -- 172.21.15.102:0/243009071 >> 172.21.15.102:0/243009071 conn(0x559d7eb3a310 msgr2=0x559d7eb39480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.166+0000 7fd81cbc0740 1 -- 172.21.15.102:0/243009071 shutdown_connections 2024-04-25T02:07:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.166+0000 7fd81cbc0740 1 -- 172.21.15.102:0/243009071 wait complete. 2024-04-25T02:07:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.166+0000 7fd81cbc0740 1 -- start start 2024-04-25T02:07:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.167+0000 7fd81cbc0740 1 Processor -- start 2024-04-25T02:07:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.170+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb35410 0x559d7eb514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.170+0000 7fd81cbc0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb4d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.171+0000 7fd812a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb35410 0x559d7eb514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.171+0000 7fd812000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb4d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.171+0000 7fd812000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb4d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34916/0 (socket says 172.21.15.102:34916) 2024-04-25T02:07:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.171+0000 7fd812000640 1 -- 172.21.15.102:0/1789320930 learned_addr learned my addr 172.21.15.102:0/1789320930 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.171+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb48fe0 0x559d7eb4dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.172+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd81419caa0 con 0x559d7eb48fe0 2024-04-25T02:07:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.172+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd81419df10 con 0x559d7eb3de00 2024-04-25T02:07:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.172+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd81419d170 con 0x559d7eb35410 2024-04-25T02:07:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.173+0000 7fd81cbc0740 10 client.0 _ll_register_callbacks cb 0x7fd80009ac90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.173+0000 7fd812000640 1 -- 172.21.15.102:0/1789320930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb35410 msgr2=0x559d7eb514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.173+0000 7fd812000640 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb35410 0x559d7eb514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.173+0000 7fd812000640 1 -- 172.21.15.102:0/1789320930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb48fe0 msgr2=0x559d7eb4dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.173+0000 7fd812000640 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb48fe0 0x559d7eb4dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.173+0000 7fd812000640 1 -- 172.21.15.102:0/1789320930 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d7ebd05c0 con 0x559d7eb3de00 2024-04-25T02:07:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.174+0000 7fd812000640 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb4d860 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd8000a3550 tx=0x7fd800000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c87abc10c1cbde3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.174+0000 7fd7fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:07:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.174+0000 7fd7fcc00640 1 -- 172.21.15.102:0/1789320930 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80009f1c0 con 0x559d7eb3de00 2024-04-25T02:07:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.175+0000 7fd7fcc00640 1 -- 172.21.15.102:0/1789320930 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7fd800097530 con 0x559d7eb3de00 2024-04-25T02:07:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.175+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8000b8070 con 0x559d7eb3de00 2024-04-25T02:07:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.175+0000 7fd7fcc00640 1 -- 172.21.15.102:0/1789320930 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd800096c10 con 0x559d7eb3de00 2024-04-25T02:07:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.175+0000 7fd81cbc0740 10 client.27506 fetch_fsmap learned FSMap version 43 2024-04-25T02:07:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.175+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x559d7ebd05c0 con 0x559d7eb3de00 2024-04-25T02:07:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.176+0000 7fd7fcc00640 1 -- 172.21.15.102:0/1789320930 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8000b8070 con 0x559d7eb3de00 2024-04-25T02:07:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.176+0000 7fd7fcc00640 1 -- 172.21.15.102:0/1789320930 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7fd800079ce0 con 0x559d7eb3de00 2024-04-25T02:07:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.176+0000 7fd81cbc0740 10 client.27506 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T02:07:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.176+0000 7fd81cbc0740 10 client.27506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.176+0000 7fd81cbc0740 20 client.27506 populate_metadata read hostname 'smithi102' 2024-04-25T02:07:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.176+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559d7ebd19a0 0x559d7ebd3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.176+0000 7fd81cbc0740 4 client.27506 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:07:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.176+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559d7eaa9340 con 0x559d7ebd19a0 2024-04-25T02:07:24.184 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.177+0000 7fd812a00640 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559d7ebd19a0 0x559d7ebd3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.179+0000 7fd812a00640 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559d7ebd19a0 0x559d7ebd3dc0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a738d01a690e07e server_cookie=b8e0005510ee6265 in_seq=0 out_seq=0 2024-04-25T02:07:24.186 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.179+0000 7fd7fcc00640 10 client.27506 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd7fcc00640 1 -- 172.21.15.102:0/1789320930 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559d7eaa9340 con 0x559d7ebd19a0 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd7fcc00640 10 client.27506 handle_command_reply: tid=0 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 client.27506 shutdown 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 msgr2=0x559d7eb4d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb4d860 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd8000a3550 tx=0x7fd800000bc0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 shutdown_connections 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559d7ebd19a0 0x559d7ebd3dc0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d7eb48fe0 0x559d7eb4dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559d7eb3de00 0x559d7eb4d860 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 --2- 172.21.15.102:0/1789320930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d7eb35410 0x559d7eb514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 >> 172.21.15.102:0/1789320930 conn(0x559d7eb3a310 msgr2=0x559d7eb38bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd7fcc00640 0 client.27506 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.222+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 shutdown_connections 2024-04-25T02:07:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.223+0000 7fd81cbc0740 1 -- 172.21.15.102:0/1789320930 wait complete. 2024-04-25T02:07:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.223+0000 7fd81cbc0740 20 client.27506 trim_cache size 0 max 16384 2024-04-25T02:07:24.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:07:24.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:07:24.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:07:24.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:07:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 -- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7f80a2ea0 msgr2=0x7fd7f80a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7f80a2ea0 0x7fd7f80a5360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd80c01a000 tx=0x7fd80c0117b0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 -- 172.21.15.102:0/3524761510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814102110 msgr2=0x7fd81419eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814102110 0x7fd81419eb10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd80000db20 tx=0x7fd80000af60 comp rx=0 tx=0).stop 2024-04-25T02:07:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 -- 172.21.15.102:0/3524761510 shutdown_connections 2024-04-25T02:07:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd7f80a2ea0 0x7fd7f80a5360 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd814107760 0x7fd8141a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd814102110 0x7fd81419eb10 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 --2- 172.21.15.102:0/3524761510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd814101750 0x7fd81419e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 -- 172.21.15.102:0/3524761510 >> 172.21.15.102:0/3524761510 conn(0x7fd814078bd0 msgr2=0x7fd8140feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.224+0000 7fd819e00640 1 -- 172.21.15.102:0/3524761510 shutdown_connections 2024-04-25T02:07:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:24.225+0000 7fd819e00640 1 -- 172.21.15.102:0/3524761510 wait complete. 2024-04-25T02:07:25.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:25 smithi102 ceph-mon[27796]: pgmap v377: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.6 MiB/s wr, 715 op/s 2024-04-25T02:07:25.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:25 smithi110 ceph-mon[32848]: pgmap v377: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.6 MiB/s wr, 715 op/s 2024-04-25T02:07:25.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:25 smithi193 ceph-mon[32981]: pgmap v377: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.6 MiB/s wr, 715 op/s 2024-04-25T02:07:27.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:27 smithi102 ceph-mon[27796]: pgmap v378: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.6 MiB/s wr, 715 op/s 2024-04-25T02:07:27.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:27 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:07:27.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:27 smithi110 ceph-mon[32848]: pgmap v378: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.6 MiB/s wr, 715 op/s 2024-04-25T02:07:27.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:27 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:07:27.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:27 smithi193 ceph-mon[32981]: pgmap v378: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.6 MiB/s wr, 715 op/s 2024-04-25T02:07:27.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:27 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:07:28.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 695.0 s, 1528.6 tps, lat 65.284 ms stddev 77.136 2024-04-25T02:07:29.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:29 smithi110 ceph-mon[32848]: pgmap v379: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 6.2 MiB/s wr, 682 op/s 2024-04-25T02:07:29.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:29 smithi193 ceph-mon[32981]: pgmap v379: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 6.2 MiB/s wr, 682 op/s 2024-04-25T02:07:29.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:29 smithi102 ceph-mon[27796]: pgmap v379: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 6.2 MiB/s wr, 682 op/s 2024-04-25T02:07:31.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:31 smithi110 ceph-mon[32848]: pgmap v380: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.6 MiB/s wr, 700 op/s 2024-04-25T02:07:31.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:31 smithi193 ceph-mon[32981]: pgmap v380: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.6 MiB/s wr, 700 op/s 2024-04-25T02:07:31.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:31 smithi102 ceph-mon[27796]: pgmap v380: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.6 MiB/s wr, 700 op/s 2024-04-25T02:07:33.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 700.0 s, 1983.8 tps, lat 50.398 ms stddev 64.521 2024-04-25T02:07:33.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:33 smithi110 ceph-mon[32848]: pgmap v381: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.5 MiB/s wr, 595 op/s 2024-04-25T02:07:33.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:33 smithi193 ceph-mon[32981]: pgmap v381: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.5 MiB/s wr, 595 op/s 2024-04-25T02:07:33.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:33 smithi102 ceph-mon[27796]: pgmap v381: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.5 MiB/s wr, 595 op/s 2024-04-25T02:07:35.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:35 smithi110 ceph-mon[32848]: pgmap v382: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 8.3 MiB/s wr, 862 op/s 2024-04-25T02:07:35.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:35 smithi193 ceph-mon[32981]: pgmap v382: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 8.3 MiB/s wr, 862 op/s 2024-04-25T02:07:35.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:35 smithi102 ceph-mon[27796]: pgmap v382: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 8.3 MiB/s wr, 862 op/s 2024-04-25T02:07:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.383+0000 7f0117400640 1 -- 172.21.15.102:0/3082375427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0110100e70 msgr2=0x7f011010dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:37.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.383+0000 7f0117400640 1 --2- 172.21.15.102:0/3082375427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0110100e70 0x7f011010dbc0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f00fc009900 tx=0x7f00fc02a400 comp rx=0 tx=0).stop 2024-04-25T02:07:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.383+0000 7f0117400640 1 -- 172.21.15.102:0/3082375427 shutdown_connections 2024-04-25T02:07:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.383+0000 7f0117400640 1 --2- 172.21.15.102:0/3082375427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011010e100 0x7f01101104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.383+0000 7f0117400640 1 --2- 172.21.15.102:0/3082375427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0110100e70 0x7f011010dbc0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.383+0000 7f0117400640 1 --2- 172.21.15.102:0/3082375427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0110100550 0x7f0110100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.383+0000 7f0117400640 1 -- 172.21.15.102:0/3082375427 >> 172.21.15.102:0/3082375427 conn(0x7f01100777c0 msgr2=0x7f0110077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.383+0000 7f0117400640 1 -- 172.21.15.102:0/3082375427 shutdown_connections 2024-04-25T02:07:37.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.383+0000 7f0117400640 1 -- 172.21.15.102:0/3082375427 wait complete. 2024-04-25T02:07:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.384+0000 7f0117400640 1 Processor -- start 2024-04-25T02:07:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f0117400640 1 -- start start 2024-04-25T02:07:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f0117400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0110100550 0x7f011019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f0117400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0110100e70 0x7f011019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f0117400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f011010e100 0x7f011019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f0117400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0110113d50 con 0x7f0110100550 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f0117400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0110113bd0 con 0x7f011010e100 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f0117400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0110113ed0 con 0x7f0110100e70 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f0114a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0110100550 0x7f011019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f010fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0110100e70 0x7f011019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f010fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0110100e70 0x7f011019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:47674/0 (socket says 172.21.15.102:47674) 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f010fe00640 1 -- 172.21.15.102:0/2958313684 learned_addr learned my addr 172.21.15.102:0/2958313684 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.385+0000 7f0115400640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f011010e100 0x7f011019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.386+0000 7f0114a00640 1 -- 172.21.15.102:0/2958313684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0110100e70 msgr2=0x7f011019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.386+0000 7f0114a00640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0110100e70 0x7f011019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.394 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.386+0000 7f0114a00640 1 -- 172.21.15.102:0/2958313684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f011010e100 msgr2=0x7f011019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.386+0000 7f0114a00640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f011010e100 0x7f011019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.386+0000 7f0114a00640 1 -- 172.21.15.102:0/2958313684 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011019f1c0 con 0x7f0110100550 2024-04-25T02:07:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.386+0000 7f010fe00640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0110100e70 0x7f011019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.386+0000 7f0115400640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f011010e100 0x7f011019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.386+0000 7f0114a00640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0110100550 0x7f011019a1d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f010400e9f0 tx=0x7f010400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.387+0000 7f010d600640 1 -- 172.21.15.102:0/2958313684 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010400ccc0 con 0x7f0110100550 2024-04-25T02:07:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.388+0000 7f010d600640 1 -- 172.21.15.102:0/2958313684 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0104010910 con 0x7f0110100550 2024-04-25T02:07:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.388+0000 7f010d600640 1 -- 172.21.15.102:0/2958313684 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104018960 con 0x7f0110100550 2024-04-25T02:07:37.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.388+0000 7f0117400640 1 -- 172.21.15.102:0/2958313684 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011019f4b0 con 0x7f0110100550 2024-04-25T02:07:37.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.388+0000 7f0117400640 1 -- 172.21.15.102:0/2958313684 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01101a6d40 con 0x7f0110100550 2024-04-25T02:07:37.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.398+0000 7f010d600640 1 -- 172.21.15.102:0/2958313684 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f010401f020 con 0x7f0110100550 2024-04-25T02:07:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.399+0000 7f011a3b8740 1 Processor -- start 2024-04-25T02:07:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.401+0000 7f011a3b8740 1 -- start start 2024-04-25T02:07:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.401+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec77508d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.401+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7750e10 0x55aec774bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.401+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 0x55aec7766650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.401+0000 7f011a3b8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55aec76abab0 con 0x55aec7750e10 2024-04-25T02:07:37.409 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.401+0000 7f011a3b8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aec7654640 con 0x55aec77504f0 2024-04-25T02:07:37.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.401+0000 7f011a3b8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55aec76a7a70 con 0x55aec7764260 2024-04-25T02:07:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.403+0000 7f0115400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec77508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.403+0000 7f0115400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec77508d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36390/0 (socket says 172.21.15.102:36390) 2024-04-25T02:07:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.403+0000 7f0115400640 1 -- 172.21.15.102:0/2557202709 learned_addr learned my addr 172.21.15.102:0/2557202709 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.404+0000 7f0115400640 1 -- 172.21.15.102:0/2557202709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 msgr2=0x55aec7766650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.404+0000 7f0115400640 1 --2- 172.21.15.102:0/2557202709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 0x55aec7766650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.404+0000 7f0115400640 1 -- 172.21.15.102:0/2557202709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7750e10 msgr2=0x55aec774bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.404+0000 7f0115400640 1 --2- 172.21.15.102:0/2557202709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7750e10 0x55aec774bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.414 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.404+0000 7f0115400640 1 -- 172.21.15.102:0/2557202709 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aec73c0210 con 0x55aec77504f0 2024-04-25T02:07:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.404+0000 7f0115400640 1 --2- 172.21.15.102:0/2557202709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec77508d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0108011790 tx=0x7f0108011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ff0ed7b853268d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.404+0000 7f010d600640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00f80a2f80 0x7f00f80a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.405+0000 7f010d600640 1 -- 172.21.15.102:0/2958313684 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7f01040cf660 con 0x7f0110100550 2024-04-25T02:07:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.406+0000 7f0101c00640 1 -- 172.21.15.102:0/2557202709 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0108004d20 con 0x55aec77504f0 2024-04-25T02:07:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.406+0000 7f0101c00640 1 -- 172.21.15.102:0/2557202709 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f010800f9a0 con 0x55aec77504f0 2024-04-25T02:07:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.406+0000 7f0101c00640 1 -- 172.21.15.102:0/2557202709 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0108004550 con 0x55aec77504f0 2024-04-25T02:07:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.407+0000 7f011a3b8740 1 -- 172.21.15.102:0/2557202709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 msgr2=0x55aec77508d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.407+0000 7f011a3b8740 1 --2- 172.21.15.102:0/2557202709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec77508d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0108011790 tx=0x7f0108011c50 comp rx=0 tx=0).stop 2024-04-25T02:07:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.407+0000 7f011a3b8740 1 -- 172.21.15.102:0/2557202709 shutdown_connections 2024-04-25T02:07:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.407+0000 7f011a3b8740 1 --2- 172.21.15.102:0/2557202709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 0x55aec7766650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.407+0000 7f011a3b8740 1 --2- 172.21.15.102:0/2557202709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7750e10 0x55aec774bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.407+0000 7f011a3b8740 1 --2- 172.21.15.102:0/2557202709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec77508d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.407+0000 7f011a3b8740 1 -- 172.21.15.102:0/2557202709 >> 172.21.15.102:0/2557202709 conn(0x55aec774cb20 msgr2=0x55aec774cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.407+0000 7f0114a00640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00f80a2f80 0x7f00f80a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.410+0000 7f011a3b8740 1 -- 172.21.15.102:0/2557202709 shutdown_connections 2024-04-25T02:07:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.410+0000 7f0114a00640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00f80a2f80 0x7f00f80a5440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55aec774de10 tx=0x7f0104009700 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.410+0000 7f011a3b8740 1 -- 172.21.15.102:0/2557202709 wait complete. 2024-04-25T02:07:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.410+0000 7f011a3b8740 1 -- start start 2024-04-25T02:07:37.418 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.411+0000 7f011a3b8740 1 Processor -- start 2024-04-25T02:07:37.419 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.412+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec7749360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.412+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 0x55aec77498a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.412+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 0x55aec7770100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.412+0000 7f011a3b8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55aec7766c50 con 0x55aec7749de0 2024-04-25T02:07:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.412+0000 7f011a3b8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55aec7667990 con 0x55aec77504f0 2024-04-25T02:07:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.412+0000 7f011a3b8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55aec7766dd0 con 0x55aec7764260 2024-04-25T02:07:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.412+0000 7f011a3b8740 10 client.0 _ll_register_callbacks cb 0x55aec7669950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.412+0000 7f0115400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec7749360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.420 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.413+0000 7f0115400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec7749360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36396/0 (socket says 172.21.15.102:36396) 2024-04-25T02:07:37.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.413+0000 7f0115400640 1 -- 172.21.15.102:0/731746787 learned_addr learned my addr 172.21.15.102:0/731746787 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:37.421 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.414+0000 7f010fe00640 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 0x55aec77498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.416+0000 7f0115400640 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 0x55aec7770100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.423 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.416+0000 7f010fe00640 1 -- 172.21.15.102:0/731746787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 msgr2=0x55aec7749360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.416+0000 7f010fe00640 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec7749360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.416+0000 7f010fe00640 1 -- 172.21.15.102:0/731746787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 msgr2=0x55aec7770100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.416+0000 7f010fe00640 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 0x55aec7770100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.416+0000 7f010fe00640 1 -- 172.21.15.102:0/731746787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aec748c620 con 0x55aec7764260 2024-04-25T02:07:37.424 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.417+0000 7f0115400640 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 0x55aec7770100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:07:37.428 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.421+0000 7f010fe00640 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 0x55aec77498a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f00fc031550 tx=0x7f00fc005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b5a8fd73fca3f19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:37.428 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.421+0000 7f00f6000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:07:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.423+0000 7f00f6000640 1 -- 172.21.15.102:0/731746787 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc035040 con 0x55aec7764260 2024-04-25T02:07:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.423+0000 7f00f6000640 1 -- 172.21.15.102:0/731746787 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7f00fc005220 con 0x55aec7764260 2024-04-25T02:07:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.424+0000 7f011a3b8740 1 -- 172.21.15.102:0/731746787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f010800f9a0 con 0x55aec7764260 2024-04-25T02:07:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.427+0000 7f00f6000640 1 -- 172.21.15.102:0/731746787 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f00fc005420 con 0x55aec7764260 2024-04-25T02:07:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.427+0000 7f011a3b8740 10 client.27475 fetch_fsmap learned FSMap version 44 2024-04-25T02:07:37.434 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.427+0000 7f011a3b8740 1 -- 172.21.15.102:0/731746787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55aec7773100 con 0x55aec7764260 2024-04-25T02:07:37.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.427+0000 7f00f6000640 1 -- 172.21.15.102:0/731746787 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f010800f9a0 con 0x55aec7764260 2024-04-25T02:07:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.429+0000 7f00f6000640 1 -- 172.21.15.102:0/731746787 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f00fc004ca0 con 0x55aec7764260 2024-04-25T02:07:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.429+0000 7f011a3b8740 10 client.27475 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T02:07:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.429+0000 7f011a3b8740 10 client.27475 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:37.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.429+0000 7f011a3b8740 20 client.27475 populate_metadata read hostname 'smithi102' 2024-04-25T02:07:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.429+0000 7f011a3b8740 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aec7774b20 0x55aec7776f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.429+0000 7f011a3b8740 4 client.27475 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:07:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.429+0000 7f011a3b8740 1 -- 172.21.15.102:0/731746787 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aec76b8ff0 con 0x55aec7774b20 2024-04-25T02:07:37.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.429+0000 7f0115400640 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aec7774b20 0x55aec7776f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.433+0000 7f00f6000640 10 client.27475 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:37.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.433+0000 7f0115400640 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aec7774b20 0x55aec7776f60 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f989a994139592e1 server_cookie=59879b2b459b7f82 in_seq=0 out_seq=0 2024-04-25T02:07:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.434+0000 7f00f6000640 1 -- 172.21.15.102:0/731746787 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aec76b8ff0 con 0x55aec7774b20 2024-04-25T02:07:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.434+0000 7f00f6000640 10 client.27475 handle_command_reply: tid=0 2024-04-25T02:07:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.434+0000 7f011a3b8740 1 client.27475 shutdown 2024-04-25T02:07:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 -- 172.21.15.102:0/731746787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 msgr2=0x55aec77498a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 0x55aec77498a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f00fc031550 tx=0x7f00fc005d50 comp rx=0 tx=0).stop 2024-04-25T02:07:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 -- 172.21.15.102:0/731746787 shutdown_connections 2024-04-25T02:07:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aec7774b20 0x55aec7776f60 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 0x55aec7770100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7764260 0x55aec77498a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 --2- 172.21.15.102:0/731746787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec7749360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f00f6000640 0 client.27475 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 -- 172.21.15.102:0/731746787 >> 172.21.15.102:0/731746787 conn(0x55aec774cb20 msgr2=0x55aec7752930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 -- 172.21.15.102:0/731746787 shutdown_connections 2024-04-25T02:07:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.435+0000 7f011a3b8740 1 -- 172.21.15.102:0/731746787 wait complete. 2024-04-25T02:07:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.436+0000 7f011a3b8740 20 client.27475 trim_cache size 0 max 16384 2024-04-25T02:07:37.474 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.467+0000 7f011a3b8740 1 Processor -- start 2024-04-25T02:07:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.469+0000 7f011a3b8740 1 -- start start 2024-04-25T02:07:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.469+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7749de0 0x55aec7743480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.469+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec777c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.469+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7764260 0x55aec777cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.469+0000 7f011a3b8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f01101add30 con 0x55aec7764260 2024-04-25T02:07:37.477 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.469+0000 7f011a3b8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f01101ab5a0 con 0x55aec77504f0 2024-04-25T02:07:37.478 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.469+0000 7f011a3b8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f01101a93d0 con 0x55aec7749de0 2024-04-25T02:07:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.470+0000 7f0115400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7749de0 0x55aec7743480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.470+0000 7f010fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec777c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.471+0000 7f010fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec777c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36410/0 (socket says 172.21.15.102:36410) 2024-04-25T02:07:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.471+0000 7f010fe00640 1 -- 172.21.15.102:0/2149008429 learned_addr learned my addr 172.21.15.102:0/2149008429 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:37.480 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.471+0000 7f0115400640 1 --2- 172.21.15.102:0/2149008429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7764260 0x55aec777cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.471+0000 7f010fe00640 1 -- 172.21.15.102:0/2149008429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7749de0 msgr2=0x55aec7743480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.472+0000 7f010fe00640 1 --2- 172.21.15.102:0/2149008429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7749de0 0x55aec7743480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.472+0000 7f010fe00640 1 -- 172.21.15.102:0/2149008429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7764260 msgr2=0x55aec777cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.472+0000 7f010fe00640 1 --2- 172.21.15.102:0/2149008429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7764260 0x55aec777cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.472+0000 7f010fe00640 1 -- 172.21.15.102:0/2149008429 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aec77478b0 con 0x55aec77504f0 2024-04-25T02:07:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.472+0000 7f0115400640 1 --2- 172.21.15.102:0/2149008429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7764260 0x55aec777cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:07:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.474+0000 7f010fe00640 1 --2- 172.21.15.102:0/2149008429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec777c830 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0096c0 tx=0x7f00fc004aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba594cc2101a426a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.474+0000 7f0101200640 1 -- 172.21.15.102:0/2149008429 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc053070 con 0x55aec77504f0 2024-04-25T02:07:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.475+0000 7f0101200640 1 -- 172.21.15.102:0/2149008429 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f010800f9a0 con 0x55aec77504f0 2024-04-25T02:07:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.475+0000 7f0101200640 1 -- 172.21.15.102:0/2149008429 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc005420 con 0x55aec77504f0 2024-04-25T02:07:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.476+0000 7f011a3b8740 1 -- 172.21.15.102:0/2149008429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 msgr2=0x55aec777c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.476+0000 7f011a3b8740 1 --2- 172.21.15.102:0/2149008429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec777c830 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f00fc0096c0 tx=0x7f00fc004aa0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.476+0000 7f011a3b8740 1 -- 172.21.15.102:0/2149008429 shutdown_connections 2024-04-25T02:07:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.476+0000 7f011a3b8740 1 --2- 172.21.15.102:0/2149008429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7764260 0x55aec777cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.476+0000 7f011a3b8740 1 --2- 172.21.15.102:0/2149008429 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec77504f0 0x55aec777c830 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.476+0000 7f011a3b8740 1 --2- 172.21.15.102:0/2149008429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec7749de0 0x55aec7743480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.476+0000 7f011a3b8740 1 -- 172.21.15.102:0/2149008429 >> 172.21.15.102:0/2149008429 conn(0x55aec774cb20 msgr2=0x55aec7751fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:37.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.476+0000 7f011a3b8740 1 -- 172.21.15.102:0/2149008429 shutdown_connections 2024-04-25T02:07:37.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.476+0000 7f011a3b8740 1 -- 172.21.15.102:0/2149008429 wait complete. 2024-04-25T02:07:37.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.477+0000 7f011a3b8740 1 -- start start 2024-04-25T02:07:37.486 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.477+0000 7f011a3b8740 1 Processor -- start 2024-04-25T02:07:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.478+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 0x55aec776c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.478+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec77504f0 0x55aec776c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.478+0000 7f011a3b8740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec7764260 0x55aec7767750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.478+0000 7f011a3b8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f01101a9a60 con 0x55aec7749de0 2024-04-25T02:07:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.478+0000 7f011a3b8740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f01101aaed0 con 0x55aec7764260 2024-04-25T02:07:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.478+0000 7f011a3b8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f01101aa130 con 0x55aec77504f0 2024-04-25T02:07:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.478+0000 7f011a3b8740 10 client.0 _ll_register_callbacks cb 0x7f01080093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.478+0000 7f0115400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec7764260 0x55aec7767750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f0115400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec7764260 0x55aec7767750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36420/0 (socket says 172.21.15.102:36420) 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f0115400640 1 -- 172.21.15.102:0/3995941494 learned_addr learned my addr 172.21.15.102:0/3995941494 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f010fe00640 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec77504f0 0x55aec776c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f0115400640 1 -- 172.21.15.102:0/3995941494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec77504f0 msgr2=0x55aec776c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f0115400640 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec77504f0 0x55aec776c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f0115400640 1 -- 172.21.15.102:0/3995941494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 msgr2=0x55aec776c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f0115400640 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 0x55aec776c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f0115400640 1 -- 172.21.15.102:0/3995941494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aec7767f50 con 0x55aec7764260 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f0115400640 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec7764260 0x55aec7767750 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f01080040c0 tx=0x7f0108004050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c61a7178cbdaed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.479+0000 7f00f6000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.480+0000 7f00f6000640 1 -- 172.21.15.102:0/3995941494 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010801e040 con 0x55aec7764260 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.480+0000 7f00f6000640 1 -- 172.21.15.102:0/3995941494 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7f0108003a30 con 0x55aec7764260 2024-04-25T02:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.480+0000 7f011a3b8740 1 -- 172.21.15.102:0/3995941494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f010800f9a0 con 0x55aec7764260 2024-04-25T02:07:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.481+0000 7f00f6000640 1 -- 172.21.15.102:0/3995941494 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0108003c30 con 0x55aec7764260 2024-04-25T02:07:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.481+0000 7f011a3b8740 10 client.27524 fetch_fsmap learned FSMap version 44 2024-04-25T02:07:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.482+0000 7f011a3b8740 1 -- 172.21.15.102:0/3995941494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55aec77dad50 con 0x55aec7764260 2024-04-25T02:07:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.482+0000 7f00f6000640 1 -- 172.21.15.102:0/3995941494 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f010800f9a0 con 0x55aec7764260 2024-04-25T02:07:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.483+0000 7f00f6000640 1 -- 172.21.15.102:0/3995941494 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 44) ==== 8354+0+0 (secure 0 0 0) 0x7f0108024040 con 0x55aec7764260 2024-04-25T02:07:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.483+0000 7f011a3b8740 10 client.27524 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T02:07:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.483+0000 7f011a3b8740 10 client.27524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.483+0000 7f011a3b8740 20 client.27524 populate_metadata read hostname 'smithi102' 2024-04-25T02:07:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.483+0000 7f011a3b8740 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aec77db3c0 0x55aec77dd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.483+0000 7f011a3b8740 4 client.27524 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:07:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.483+0000 7f011a3b8740 1 -- 172.21.15.102:0/3995941494 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aec76b3340 con 0x55aec77db3c0 2024-04-25T02:07:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.484+0000 7f010fe00640 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aec77db3c0 0x55aec77dd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.486+0000 7f010fe00640 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aec77db3c0 0x55aec77dd7e0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be961a28e6c95b89 server_cookie=8161c8b5effb3538 in_seq=0 out_seq=0 2024-04-25T02:07:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.488+0000 7f00f6000640 10 client.27524 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.513+0000 7f00f6000640 1 -- 172.21.15.102:0/3995941494 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aec76b3340 con 0x55aec77db3c0 2024-04-25T02:07:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.513+0000 7f00f6000640 10 client.27524 handle_command_reply: tid=0 2024-04-25T02:07:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.513+0000 7f011a3b8740 1 client.27524 shutdown 2024-04-25T02:07:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.514+0000 7f011a3b8740 1 -- 172.21.15.102:0/3995941494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec7764260 msgr2=0x55aec7767750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:37.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.514+0000 7f011a3b8740 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec7764260 0x55aec7767750 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f01080040c0 tx=0x7f0108004050 comp rx=0 tx=0).stop 2024-04-25T02:07:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.514+0000 7f011a3b8740 1 -- 172.21.15.102:0/3995941494 shutdown_connections 2024-04-25T02:07:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.514+0000 7f011a3b8740 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55aec77db3c0 0x55aec77dd7e0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.515+0000 7f011a3b8740 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55aec7764260 0x55aec7767750 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.515+0000 7f011a3b8740 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aec77504f0 0x55aec776c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.515+0000 7f011a3b8740 1 --2- 172.21.15.102:0/3995941494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aec7749de0 0x55aec776c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.515+0000 7f011a3b8740 1 -- 172.21.15.102:0/3995941494 >> 172.21.15.102:0/3995941494 conn(0x55aec774cb20 msgr2=0x55aec7751fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:37.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.515+0000 7f00f6000640 0 client.27524 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.515+0000 7f011a3b8740 1 -- 172.21.15.102:0/3995941494 shutdown_connections 2024-04-25T02:07:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.515+0000 7f011a3b8740 1 -- 172.21.15.102:0/3995941494 wait complete. 2024-04-25T02:07:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.517+0000 7f011a3b8740 20 client.27524 trim_cache size 0 max 16384 2024-04-25T02:07:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:07:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:07:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:07:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:07:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.517+0000 7f0117400640 1 -- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00f80a2f80 msgr2=0x7f00f80a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.517+0000 7f0117400640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00f80a2f80 0x7f00f80a5440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55aec774de10 tx=0x7f0104009700 comp rx=0 tx=0).stop 2024-04-25T02:07:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 -- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0110100550 msgr2=0x7f011019a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0110100550 0x7f011019a1d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f010400e9f0 tx=0x7f010400eeb0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 -- 172.21.15.102:0/2958313684 shutdown_connections 2024-04-25T02:07:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f00f80a2f80 0x7f00f80a5440 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f011010e100 0x7f011019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0110100e70 0x7f011019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 --2- 172.21.15.102:0/2958313684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0110100550 0x7f011019a1d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 -- 172.21.15.102:0/2958313684 >> 172.21.15.102:0/2958313684 conn(0x7f01100777c0 msgr2=0x7f011010f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 -- 172.21.15.102:0/2958313684 shutdown_connections 2024-04-25T02:07:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:37.518+0000 7f0117400640 1 -- 172.21.15.102:0/2958313684 wait complete. 2024-04-25T02:07:37.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:37 smithi110 ceph-mon[32848]: pgmap v383: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 5.3 MiB/s wr, 543 op/s 2024-04-25T02:07:37.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:37 smithi193 ceph-mon[32981]: pgmap v383: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 5.3 MiB/s wr, 543 op/s 2024-04-25T02:07:37.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:37 smithi102 ceph-mon[27796]: pgmap v383: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 5.3 MiB/s wr, 543 op/s 2024-04-25T02:07:38.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 705.0 s, 1869.2 tps, lat 53.145 ms stddev 63.461 2024-04-25T02:07:39.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:39 smithi110 ceph-mon[32848]: pgmap v384: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-04-25T02:07:39.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:39 smithi193 ceph-mon[32981]: pgmap v384: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-04-25T02:07:39.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:39 smithi102 ceph-mon[27796]: pgmap v384: 97 pgs: 97 active+clean; 5.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 7.2 MiB/s wr, 737 op/s 2024-04-25T02:07:41.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:41 smithi193 ceph-mon[32981]: pgmap v385: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.9 MiB/s wr, 688 op/s 2024-04-25T02:07:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:41 smithi102 ceph-mon[27796]: pgmap v385: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.9 MiB/s wr, 688 op/s 2024-04-25T02:07:41.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:41 smithi110 ceph-mon[32848]: pgmap v385: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.9 MiB/s wr, 688 op/s 2024-04-25T02:07:43.079 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 710.0 s, 1496.9 tps, lat 64.848 ms stddev 71.468 2024-04-25T02:07:43.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:43 smithi193 ceph-mon[32981]: pgmap v386: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.7 MiB/s wr, 571 op/s 2024-04-25T02:07:43.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:43 smithi102 ceph-mon[27796]: pgmap v386: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.7 MiB/s wr, 571 op/s 2024-04-25T02:07:43.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:43 smithi110 ceph-mon[32848]: pgmap v386: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.7 MiB/s wr, 571 op/s 2024-04-25T02:07:45.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:45 smithi193 ceph-mon[32981]: pgmap v387: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-25T02:07:45.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:45 smithi102 ceph-mon[27796]: pgmap v387: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-25T02:07:45.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:45 smithi110 ceph-mon[32848]: pgmap v387: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-25T02:07:47.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:47 smithi102 ceph-mon[27796]: pgmap v388: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 9.0 MiB/s wr, 811 op/s 2024-04-25T02:07:47.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:47 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:07:47.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:47 smithi110 ceph-mon[32848]: pgmap v388: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 9.0 MiB/s wr, 811 op/s 2024-04-25T02:07:47.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:47 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:07:48.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:47 smithi193 ceph-mon[32981]: pgmap v388: 97 pgs: 97 active+clean; 5.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 9.0 MiB/s wr, 811 op/s 2024-04-25T02:07:48.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:47 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:07:48.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 715.0 s, 1291.1 tps, lat 79.773 ms stddev 165.983 2024-04-25T02:07:49.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:49 smithi102 ceph-mon[27796]: pgmap v389: 97 pgs: 97 active+clean; 5.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 22 MiB/s wr, 1.69k op/s 2024-04-25T02:07:49.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:49 smithi110 ceph-mon[32848]: pgmap v389: 97 pgs: 97 active+clean; 5.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 22 MiB/s wr, 1.69k op/s 2024-04-25T02:07:50.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:49 smithi193 ceph-mon[32981]: pgmap v389: 97 pgs: 97 active+clean; 5.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 22 MiB/s wr, 1.69k op/s 2024-04-25T02:07:50.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.691+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2395549252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e4101120 msgr2=0x7ff5e410dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:50.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.691+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2395549252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e4101120 0x7ff5e410dc10 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009900 tx=0x7ff5d002a420 comp rx=0 tx=0).stop 2024-04-25T02:07:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.694+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2395549252 shutdown_connections 2024-04-25T02:07:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.694+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2395549252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5e410e150 0x7ff5e4110540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.694+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2395549252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e4101120 0x7ff5e410dc10 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.694+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2395549252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5e4100800 0x7ff5e4100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.694+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2395549252 >> 172.21.15.102:0/2395549252 conn(0x7ff5e40777c0 msgr2=0x7ff5e4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.695+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2395549252 shutdown_connections 2024-04-25T02:07:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.695+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2395549252 wait complete. 2024-04-25T02:07:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.696+0000 7ff5e9e00640 1 Processor -- start 2024-04-25T02:07:50.706 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.700+0000 7ff5e9e00640 1 -- start start 2024-04-25T02:07:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.700+0000 7ff5e9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5e4100800 0x7ff5e41a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.700+0000 7ff5e9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5e4101120 0x7ff5e41a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.700+0000 7ff5e9e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e410e150 0x7ff5e419e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.700+0000 7ff5e9e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e410aa10 con 0x7ff5e4101120 2024-04-25T02:07:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.700+0000 7ff5e9e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e410a890 con 0x7ff5e410e150 2024-04-25T02:07:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.700+0000 7ff5e9e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e410ab90 con 0x7ff5e4100800 2024-04-25T02:07:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.701+0000 7ff5e2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5e4100800 0x7ff5e41a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.701+0000 7ff5e2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5e4100800 0x7ff5e41a4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56916/0 (socket says 172.21.15.102:56916) 2024-04-25T02:07:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.701+0000 7ff5e2a00640 1 -- 172.21.15.102:0/2603585083 learned_addr learned my addr 172.21.15.102:0/2603585083 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.701+0000 7ff5e3400640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e410e150 0x7ff5e419e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.701+0000 7ff5e2a00640 1 -- 172.21.15.102:0/2603585083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e410e150 msgr2=0x7ff5e419e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.701+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e410e150 0x7ff5e419e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.701+0000 7ff5e2a00640 1 -- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5e4101120 msgr2=0x7ff5e41a4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.701+0000 7ff5e2000640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5e4101120 0x7ff5e41a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.702+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5e4101120 0x7ff5e41a4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.702+0000 7ff5e2a00640 1 -- 172.21.15.102:0/2603585083 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e419ebc0 con 0x7ff5e4100800 2024-04-25T02:07:50.709 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.702+0000 7ff5e2000640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5e4101120 0x7ff5e41a4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:07:50.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.704+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5e4100800 0x7ff5e41a4230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400d950 tx=0x7ff5d400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.707+0000 7ff5e8a00640 1 -- 172.21.15.102:0/2603585083 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4017070 con 0x7ff5e4100800 2024-04-25T02:07:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.707+0000 7ff5e8a00640 1 -- 172.21.15.102:0/2603585083 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d4009c30 con 0x7ff5e4100800 2024-04-25T02:07:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.707+0000 7ff5e8a00640 1 -- 172.21.15.102:0/2603585083 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d400f6b0 con 0x7ff5e4100800 2024-04-25T02:07:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.707+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2603585083 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e419eeb0 con 0x7ff5e4100800 2024-04-25T02:07:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.707+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2603585083 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e41ab080 con 0x7ff5e4100800 2024-04-25T02:07:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.713+0000 7ff5ecbd2740 1 Processor -- start 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.714+0000 7ff5ecbd2740 1 -- start start 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.714+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 0x55d96c03b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.714+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03b720 0x55d96c045aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.714+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 0x55d96c0483d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.714+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d96bf9eab0 con 0x55d96c03b720 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.714+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d96bf47640 con 0x55d96c03ae00 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.714+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d96bf9aa70 con 0x55d96c045fe0 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e8a00640 1 -- 172.21.15.102:0/2603585083 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff5d40109d0 con 0x7ff5e4100800 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 0x55d96c0483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 0x55d96c0483d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56928/0 (socket says 172.21.15.102:56928) 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e2000640 1 -- 172.21.15.102:0/2130114538 learned_addr learned my addr 172.21.15.102:0/2130114538 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/2130114538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03b720 0x55d96c045aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e2000640 1 -- 172.21.15.102:0/2130114538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 msgr2=0x55d96c03b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e2000640 1 --2- 172.21.15.102:0/2130114538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 0x55d96c03b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e2000640 1 -- 172.21.15.102:0/2130114538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03b720 msgr2=0x55d96c045aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e2000640 1 --2- 172.21.15.102:0/2130114538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03b720 0x55d96c045aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.715+0000 7ff5e2000640 1 -- 172.21.15.102:0/2130114538 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96bcb3210 con 0x55d96c045fe0 2024-04-25T02:07:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.716+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/2130114538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03b720 0x55d96c045aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:07:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.716+0000 7ff5e2000640 1 --2- 172.21.15.102:0/2130114538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 0x55d96c0483d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000c00 tx=0x7ff5d00037c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d62294ed9838014a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.716+0000 7ff5cfe00640 1 -- 172.21.15.102:0/2130114538 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0040070 con 0x55d96c045fe0 2024-04-25T02:07:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.716+0000 7ff5cfe00640 1 -- 172.21.15.102:0/2130114538 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d0005000 con 0x55d96c045fe0 2024-04-25T02:07:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.716+0000 7ff5cfe00640 1 -- 172.21.15.102:0/2130114538 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0045920 con 0x55d96c045fe0 2024-04-25T02:07:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.717+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2130114538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 msgr2=0x55d96c0483d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.717+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2130114538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 0x55d96c0483d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000c00 tx=0x7ff5d00037c0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.718+0000 7ff5e8a00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff5c80a2f50 0x7ff5c80a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.719+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2130114538 shutdown_connections 2024-04-25T02:07:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.719+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2130114538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 0x55d96c0483d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.719+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2130114538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03b720 0x55d96c045aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.719+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2130114538 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 0x55d96c03b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.719+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2130114538 >> 172.21.15.102:0/2130114538 conn(0x55d96c037310 msgr2=0x55d96c0376f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.719+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2130114538 shutdown_connections 2024-04-25T02:07:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.719+0000 7ff5e8a00640 1 -- 172.21.15.102:0/2603585083 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7ff5d40cef60 con 0x7ff5e4100800 2024-04-25T02:07:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.721+0000 7ff5e3400640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff5c80a2f50 0x7ff5c80a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.722+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2130114538 wait complete. 2024-04-25T02:07:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.722+0000 7ff5ecbd2740 1 -- start start 2024-04-25T02:07:50.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.722+0000 7ff5ecbd2740 1 Processor -- start 2024-04-25T02:07:50.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5e3400640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff5c80a2f50 0x7ff5c80a5410 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff5e406af20 tx=0x7ff5dc004a40 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:50.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 0x55d96c042f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03b720 0x55d96c0434a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 0x55d96c0439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55d96c059bc0 con 0x55d96c03ae00 2024-04-25T02:07:50.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55d96bf5a990 con 0x55d96c03b720 2024-04-25T02:07:50.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d96c059d40 con 0x55d96c045fe0 2024-04-25T02:07:50.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5ecbd2740 10 client.0 _ll_register_callbacks cb 0x55d96bf5c950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:50.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5e2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03b720 0x55d96c0434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5e2000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03b720 0x55d96c0434a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56110/0 (socket says 172.21.15.102:56110) 2024-04-25T02:07:50.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.723+0000 7ff5e2000640 1 -- 172.21.15.102:0/2762960419 learned_addr learned my addr 172.21.15.102:0/2762960419 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:50.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.724+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 0x55d96c042f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.724+0000 7ff5e2000640 1 -- 172.21.15.102:0/2762960419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 msgr2=0x55d96c0439e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.724+0000 7ff5e2000640 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 0x55d96c0439e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.724+0000 7ff5e2000640 1 -- 172.21.15.102:0/2762960419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 msgr2=0x55d96c042f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.724+0000 7ff5e2000640 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 0x55d96c042f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.724+0000 7ff5e2000640 1 -- 172.21.15.102:0/2762960419 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d96bd7f620 con 0x55d96c03b720 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.724+0000 7ff5e2000640 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03b720 0x55d96c0434a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00092a0 tx=0x7ff5d0004b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=706ce6a6b9839b31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.724+0000 7ff5ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.725+0000 7ff5ccc00640 1 -- 172.21.15.102:0/2762960419 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0040070 con 0x55d96c03b720 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.725+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2762960419 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d0005000 con 0x55d96c03b720 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.725+0000 7ff5ccc00640 1 -- 172.21.15.102:0/2762960419 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7ff5d00053c0 con 0x55d96c03b720 2024-04-25T02:07:50.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.725+0000 7ff5ccc00640 1 -- 172.21.15.102:0/2762960419 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff5d00055c0 con 0x55d96c03b720 2024-04-25T02:07:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.725+0000 7ff5ecbd2740 10 client.27533 fetch_fsmap learned FSMap version 45 2024-04-25T02:07:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.725+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2762960419 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55d96c0659e0 con 0x55d96c03b720 2024-04-25T02:07:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.726+0000 7ff5ccc00640 1 -- 172.21.15.102:0/2762960419 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d0005000 con 0x55d96c03b720 2024-04-25T02:07:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.726+0000 7ff5ccc00640 1 -- 172.21.15.102:0/2762960419 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7ff5d00317a0 con 0x55d96c03b720 2024-04-25T02:07:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.726+0000 7ff5ecbd2740 10 client.27533 fetch_fsmap finished waiting for FSMap version 45 2024-04-25T02:07:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.726+0000 7ff5ecbd2740 10 client.27533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.726+0000 7ff5ecbd2740 20 client.27533 populate_metadata read hostname 'smithi102' 2024-04-25T02:07:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.726+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d96c066d30 0x55d96c069120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.726+0000 7ff5ecbd2740 4 client.27533 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:07:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.726+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2762960419 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d96bfabff0 con 0x55d96c066d30 2024-04-25T02:07:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.726+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d96c066d30 0x55d96c069120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.727+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d96c066d30 0x55d96c069120 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7ee5cca2466427d server_cookie=9cc16fdb4c2a1449 in_seq=0 out_seq=0 2024-04-25T02:07:50.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.727+0000 7ff5ccc00640 10 client.27533 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.728+0000 7ff5ccc00640 1 -- 172.21.15.102:0/2762960419 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d96bfabff0 con 0x55d96c066d30 2024-04-25T02:07:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.728+0000 7ff5ccc00640 10 client.27533 handle_command_reply: tid=0 2024-04-25T02:07:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.728+0000 7ff5ecbd2740 1 client.27533 shutdown 2024-04-25T02:07:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.728+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2762960419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03b720 msgr2=0x55d96c0434a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:50.735 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.728+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03b720 0x55d96c0434a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00092a0 tx=0x7ff5d0004b90 comp rx=0 tx=0).stop 2024-04-25T02:07:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.729+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2762960419 shutdown_connections 2024-04-25T02:07:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.729+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d96c066d30 0x55d96c069120 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.729+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c045fe0 0x55d96c0439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.729+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03b720 0x55d96c0434a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.729+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/2762960419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 0x55d96c042f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.729+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2762960419 >> 172.21.15.102:0/2762960419 conn(0x55d96c037310 msgr2=0x55d96c03d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.729+0000 7ff5ccc00640 0 client.27533 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:50.736 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.729+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2762960419 shutdown_connections 2024-04-25T02:07:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.730+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/2762960419 wait complete. 2024-04-25T02:07:50.737 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.731+0000 7ff5ecbd2740 20 client.27533 trim_cache size 0 max 16384 2024-04-25T02:07:50.762 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.756+0000 7ff5ecbd2740 1 Processor -- start 2024-04-25T02:07:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.756+0000 7ff5ecbd2740 1 -- start start 2024-04-25T02:07:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.756+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 0x55d96c041850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.756+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c036c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.756+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c045fe0 0x55d96c03d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.756+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e41adb70 con 0x55d96c03ae00 2024-04-25T02:07:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.756+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e41ab3b0 con 0x55d96c045fe0 2024-04-25T02:07:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.757+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e404f500 con 0x55d96c03b720 2024-04-25T02:07:50.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.757+0000 7ff5e2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c036c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.758+0000 7ff5e3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c045fe0 0x55d96c03d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.758+0000 7ff5e2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c036c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56948/0 (socket says 172.21.15.102:56948) 2024-04-25T02:07:50.764 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.758+0000 7ff5e2000640 1 -- 172.21.15.102:0/3410628631 learned_addr learned my addr 172.21.15.102:0/3410628631 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:50.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.758+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/3410628631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 0x55d96c041850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.758+0000 7ff5e2000640 1 -- 172.21.15.102:0/3410628631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c045fe0 msgr2=0x55d96c03d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:50.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.758+0000 7ff5e2000640 1 --2- 172.21.15.102:0/3410628631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c045fe0 0x55d96c03d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.758+0000 7ff5e2000640 1 -- 172.21.15.102:0/3410628631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 msgr2=0x55d96c041850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:50.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.758+0000 7ff5e2000640 1 --2- 172.21.15.102:0/3410628631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 0x55d96c041850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.765 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.758+0000 7ff5e2000640 1 -- 172.21.15.102:0/3410628631 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96c041f30 con 0x55d96c03b720 2024-04-25T02:07:50.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.759+0000 7ff5e2000640 1 --2- 172.21.15.102:0/3410628631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c036c00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009900 tx=0x7ff5d0003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a8234340a5c3df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:50.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.759+0000 7ff5cf400640 1 -- 172.21.15.102:0/3410628631 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0040040 con 0x55d96c03b720 2024-04-25T02:07:50.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.759+0000 7ff5cf400640 1 -- 172.21.15.102:0/3410628631 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d00053c0 con 0x55d96c03b720 2024-04-25T02:07:50.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.759+0000 7ff5cf400640 1 -- 172.21.15.102:0/3410628631 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d004fb90 con 0x55d96c03b720 2024-04-25T02:07:50.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.760+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/3410628631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 msgr2=0x55d96c036c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:50.766 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.760+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/3410628631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c036c00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009900 tx=0x7ff5d0003680 comp rx=0 tx=0).stop 2024-04-25T02:07:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.760+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/3410628631 shutdown_connections 2024-04-25T02:07:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.760+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/3410628631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c045fe0 0x55d96c03d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.760+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/3410628631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c036c00 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.760+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/3410628631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c03ae00 0x55d96c041850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.760+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/3410628631 >> 172.21.15.102:0/3410628631 conn(0x55d96c037310 msgr2=0x55d96c0437b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:50.767 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.761+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/3410628631 shutdown_connections 2024-04-25T02:07:50.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.761+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/3410628631 wait complete. 2024-04-25T02:07:50.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.761+0000 7ff5ecbd2740 1 -- start start 2024-04-25T02:07:50.768 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.762+0000 7ff5ecbd2740 1 Processor -- start 2024-04-25T02:07:50.769 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 0x55d96c05f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.769 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c05a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5ecbd2740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c045fe0 0x55d96c05aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.770 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5e2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 0x55d96c05f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5e2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 0x55d96c05f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56124/0 (socket says 172.21.15.102:56124) 2024-04-25T02:07:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4071f10 con 0x55d96c045fe0 2024-04-25T02:07:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4073360 con 0x55d96c03ae00 2024-04-25T02:07:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5ecbd2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e40725c0 con 0x55d96c03b720 2024-04-25T02:07:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c045fe0 0x55d96c05aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c045fe0 0x55d96c05aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46030/0 (socket says 172.21.15.102:46030) 2024-04-25T02:07:50.771 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.763+0000 7ff5e3400640 1 -- 172.21.15.102:0/698195982 learned_addr learned my addr 172.21.15.102:0/698195982 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.764+0000 7ff5e2000640 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c05a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.764+0000 7ff5ecbd2740 10 client.0 _ll_register_callbacks cb 0x7ff5d00036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.764+0000 7ff5e2a00640 1 -- 172.21.15.102:0/698195982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 msgr2=0x55d96c05a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.764+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c05a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.764+0000 7ff5e2a00640 1 -- 172.21.15.102:0/698195982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c045fe0 msgr2=0x55d96c05aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.764+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c045fe0 0x55d96c05aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.764+0000 7ff5e2a00640 1 -- 172.21.15.102:0/698195982 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d96c05afd0 con 0x55d96c03ae00 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.764+0000 7ff5e3400640 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c045fe0 0x55d96c05aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.765+0000 7ff5e2a00640 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 0x55d96c05f060 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff5d40cc6b0 tx=0x7ff5d400bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=506b0dbd06fb08b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:07:50.772 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.765+0000 7ff5ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:07:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.765+0000 7ff5ccc00640 1 -- 172.21.15.102:0/698195982 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4020040 con 0x55d96c03ae00 2024-04-25T02:07:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.765+0000 7ff5ccc00640 1 -- 172.21.15.102:0/698195982 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7ff5d40cc930 con 0x55d96c03ae00 2024-04-25T02:07:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.765+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/698195982 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5d00053c0 con 0x55d96c03ae00 2024-04-25T02:07:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.765+0000 7ff5ccc00640 1 -- 172.21.15.102:0/698195982 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7ff5d409f830 con 0x55d96c03ae00 2024-04-25T02:07:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.765+0000 7ff5ecbd2740 10 client.27539 fetch_fsmap learned FSMap version 45 2024-04-25T02:07:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.765+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/698195982 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55d96c0ccdc0 con 0x55d96c03ae00 2024-04-25T02:07:50.773 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.766+0000 7ff5ccc00640 1 -- 172.21.15.102:0/698195982 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d00053c0 con 0x55d96c03ae00 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.766+0000 7ff5ccc00640 1 -- 172.21.15.102:0/698195982 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7ff5d40b4260 con 0x55d96c03ae00 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.766+0000 7ff5ecbd2740 10 client.27539 fetch_fsmap finished waiting for FSMap version 45 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.766+0000 7ff5ecbd2740 10 client.27539 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.766+0000 7ff5ecbd2740 20 client.27539 populate_metadata read hostname 'smithi102' 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.766+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d96c0cd4d0 0x55d96c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.766+0000 7ff5ecbd2740 4 client.27539 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.766+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/698195982 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d96bfa6340 con 0x55d96c0cd4d0 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.766+0000 7ff5e2000640 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d96c0cd4d0 0x55d96c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.767+0000 7ff5e2000640 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d96c0cd4d0 0x55d96c0cf8f0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40d5633fef9987 server_cookie=e5fcac23c723aace in_seq=0 out_seq=0 2024-04-25T02:07:50.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.767+0000 7ff5ccc00640 10 client.27539 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.779+0000 7ff5ccc00640 1 -- 172.21.15.102:0/698195982 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d96bfa6340 con 0x55d96c0cd4d0 2024-04-25T02:07:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.779+0000 7ff5ccc00640 10 client.27539 handle_command_reply: tid=0 2024-04-25T02:07:50.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.779+0000 7ff5ecbd2740 1 client.27539 shutdown 2024-04-25T02:07:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.780+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/698195982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 msgr2=0x55d96c05f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.780+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 0x55d96c05f060 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff5d40cc6b0 tx=0x7ff5d400bf60 comp rx=0 tx=0).stop 2024-04-25T02:07:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.780+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/698195982 shutdown_connections 2024-04-25T02:07:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.780+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55d96c0cd4d0 0x55d96c0cf8f0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.780+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d96c045fe0 0x55d96c05aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.780+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d96c03b720 0x55d96c05a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.780+0000 7ff5ecbd2740 1 --2- 172.21.15.102:0/698195982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55d96c03ae00 0x55d96c05f060 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.780+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/698195982 >> 172.21.15.102:0/698195982 conn(0x55d96c037310 msgr2=0x55d96c0437b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:50.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.780+0000 7ff5ccc00640 0 client.27539 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:07:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.781+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/698195982 shutdown_connections 2024-04-25T02:07:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.781+0000 7ff5ecbd2740 1 -- 172.21.15.102:0/698195982 wait complete. 2024-04-25T02:07:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5ecbd2740 20 client.27539 trim_cache size 0 max 16384 2024-04-25T02:07:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:07:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:07:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:07:50.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:07:50.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff5c80a2f50 msgr2=0x7ff5c80a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:50.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff5c80a2f50 0x7ff5c80a5410 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff5e406af20 tx=0x7ff5dc004a40 comp rx=0 tx=0).stop 2024-04-25T02:07:50.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2603585083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5e4100800 msgr2=0x7ff5e41a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:07:50.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5e4100800 0x7ff5e41a4230 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400d950 tx=0x7ff5d400de10 comp rx=0 tx=0).stop 2024-04-25T02:07:50.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2603585083 shutdown_connections 2024-04-25T02:07:50.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7ff5c80a2f50 0x7ff5c80a5410 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e410e150 0x7ff5e419e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5e4101120 0x7ff5e41a4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 --2- 172.21.15.102:0/2603585083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5e4100800 0x7ff5e41a4230 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:07:50.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.782+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2603585083 >> 172.21.15.102:0/2603585083 conn(0x7ff5e40777c0 msgr2=0x7ff5e410b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:07:50.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.783+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2603585083 shutdown_connections 2024-04-25T02:07:50.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:07:50.783+0000 7ff5e9e00640 1 -- 172.21.15.102:0/2603585083 wait complete. 2024-04-25T02:07:51.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:51 smithi102 ceph-mon[27796]: pgmap v390: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-25T02:07:51.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:51 smithi110 ceph-mon[32848]: pgmap v390: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-25T02:07:52.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:51 smithi193 ceph-mon[32981]: pgmap v390: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 23 MiB/s wr, 1.72k op/s 2024-04-25T02:07:52.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:52 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:07:52.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:52 smithi102 ceph-mon[27796]: pgmap v391: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 22 MiB/s wr, 1.62k op/s 2024-04-25T02:07:52.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:52 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:07:52.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:52 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:07:52.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:52 smithi110 ceph-mon[32848]: pgmap v391: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 22 MiB/s wr, 1.62k op/s 2024-04-25T02:07:52.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:52 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:07:53.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:52 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:07:53.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:52 smithi193 ceph-mon[32981]: pgmap v391: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 22 MiB/s wr, 1.62k op/s 2024-04-25T02:07:53.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:52 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:07:53.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 720.0 s, 1263.4 tps, lat 78.475 ms stddev 89.588 2024-04-25T02:07:53.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:53 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:07:53.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:53 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:07:53.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:53 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:07:53.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:53 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:07:54.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:53 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:07:54.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:53 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:07:54.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:54 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:07:54.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:54 smithi102 ceph-mon[27796]: pgmap v392: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 24 MiB/s wr, 1.84k op/s 2024-04-25T02:07:54.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:54 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:07:54.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:54 smithi110 ceph-mon[32848]: pgmap v392: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 24 MiB/s wr, 1.84k op/s 2024-04-25T02:07:55.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:54 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:07:55.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:54 smithi193 ceph-mon[32981]: pgmap v392: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 24 MiB/s wr, 1.84k op/s 2024-04-25T02:07:57.243 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:56 smithi110 ceph-mon[32848]: pgmap v393: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-25T02:07:57.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:56 smithi193 ceph-mon[32981]: pgmap v393: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-25T02:07:57.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:56 smithi102 ceph-mon[27796]: pgmap v393: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-25T02:07:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 725.0 s, 1376.4 tps, lat 73.018 ms stddev 94.944 2024-04-25T02:07:59.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:07:58 smithi193 ceph-mon[32981]: pgmap v394: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 20 MiB/s wr, 1.50k op/s 2024-04-25T02:07:59.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:07:58 smithi102 ceph-mon[27796]: pgmap v394: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 20 MiB/s wr, 1.50k op/s 2024-04-25T02:07:59.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:07:58 smithi110 ceph-mon[32848]: pgmap v394: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 20 MiB/s wr, 1.50k op/s 2024-04-25T02:08:01.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:00 smithi193 ceph-mon[32981]: pgmap v395: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.6 MiB/s wr, 734 op/s 2024-04-25T02:08:01.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:00 smithi102 ceph-mon[27796]: pgmap v395: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.6 MiB/s wr, 734 op/s 2024-04-25T02:08:01.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:00 smithi110 ceph-mon[32848]: pgmap v395: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.6 MiB/s wr, 734 op/s 2024-04-25T02:08:03.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 730.0 s, 915.4 tps, lat 108.170 ms stddev 164.209 2024-04-25T02:08:03.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:03 smithi102 ceph-mon[27796]: pgmap v396: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 5.3 MiB/s wr, 584 op/s 2024-04-25T02:08:03.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:03 smithi110 ceph-mon[32848]: pgmap v396: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 5.3 MiB/s wr, 584 op/s 2024-04-25T02:08:03.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:03 smithi193 ceph-mon[32981]: pgmap v396: 97 pgs: 97 active+clean; 5.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 5.3 MiB/s wr, 584 op/s 2024-04-25T02:08:03.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.967+0000 7f0b36200640 1 -- 172.21.15.102:0/1547044697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b30108740 msgr2=0x7f0b30108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:03.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.967+0000 7f0b36200640 1 --2- 172.21.15.102:0/1547044697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b30108740 0x7f0b30108b20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009930 tx=0x7f0b2002a4d0 comp rx=0 tx=0).stop 2024-04-25T02:08:03.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.968+0000 7f0b36200640 1 -- 172.21.15.102:0/1547044697 shutdown_connections 2024-04-25T02:08:03.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.968+0000 7f0b36200640 1 --2- 172.21.15.102:0/1547044697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b30114890 0x7f0b30116c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:03.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.968+0000 7f0b36200640 1 --2- 172.21.15.102:0/1547044697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b30109060 0x7f0b30114350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:03.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.968+0000 7f0b36200640 1 --2- 172.21.15.102:0/1547044697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b30108740 0x7f0b30108b20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:03.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.968+0000 7f0b36200640 1 -- 172.21.15.102:0/1547044697 >> 172.21.15.102:0/1547044697 conn(0x7f0b3006cca0 msgr2=0x7f0b3006d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:03.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.969+0000 7f0b36200640 1 -- 172.21.15.102:0/1547044697 shutdown_connections 2024-04-25T02:08:03.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.969+0000 7f0b36200640 1 -- 172.21.15.102:0/1547044697 wait complete. 2024-04-25T02:08:03.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.970+0000 7f0b36200640 1 Processor -- start 2024-04-25T02:08:03.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.972+0000 7f0b36200640 1 -- start start 2024-04-25T02:08:03.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.972+0000 7f0b36200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b30108740 0x7f0b301a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:03.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.972+0000 7f0b36200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b30109060 0x7f0b301a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:03.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.972+0000 7f0b36200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b30114890 0x7f0b301a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:03.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b36200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3011a350 con 0x7f0b30114890 2024-04-25T02:08:03.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b36200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3011a1d0 con 0x7f0b30108740 2024-04-25T02:08:03.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b36200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3011a4d0 con 0x7f0b30109060 2024-04-25T02:08:03.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b30108740 0x7f0b301a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:03.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2f400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b30108740 0x7f0b301a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36614/0 (socket says 172.21.15.102:36614) 2024-04-25T02:08:03.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2f400640 1 -- 172.21.15.102:0/2809556997 learned_addr learned my addr 172.21.15.102:0/2809556997 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:03.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b30114890 0x7f0b301a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:03.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b30109060 0x7f0b301a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:03.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2f400640 1 -- 172.21.15.102:0/2809556997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b30109060 msgr2=0x7f0b301a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:03.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2f400640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b30109060 0x7f0b301a69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:03.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2f400640 1 -- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b30114890 msgr2=0x7f0b301a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:03.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2f400640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b30114890 0x7f0b301a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:03.980 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2f400640 1 -- 172.21.15.102:0/2809556997 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b301a0d90 con 0x7f0b30108740 2024-04-25T02:08:03.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.973+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b30109060 0x7f0b301a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:08:03.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.974+0000 7f0b2f400640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b30108740 0x7f0b301a6460 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0b2002a9f0 tx=0x7f0b20003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:03.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.975+0000 7f0b27e00640 1 -- 172.21.15.102:0/2809556997 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2003f070 con 0x7f0b30108740 2024-04-25T02:08:03.981 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.975+0000 7f0b27e00640 1 -- 172.21.15.102:0/2809556997 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b20031c90 con 0x7f0b30108740 2024-04-25T02:08:03.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.975+0000 7f0b27e00640 1 -- 172.21.15.102:0/2809556997 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20035030 con 0x7f0b30108740 2024-04-25T02:08:03.982 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.976+0000 7f0b36200640 1 -- 172.21.15.102:0/2809556997 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b301a1020 con 0x7f0b30108740 2024-04-25T02:08:03.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.976+0000 7f0b36200640 1 -- 172.21.15.102:0/2809556997 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b3010a0e0 con 0x7f0b30108740 2024-04-25T02:08:03.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.984+0000 7f0b27e00640 1 -- 172.21.15.102:0/2809556997 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0b20042070 con 0x7f0b30108740 2024-04-25T02:08:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.987+0000 7f0b27e00640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0b140a2e20 0x7f0b140a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.987+0000 7f0b27e00640 1 -- 172.21.15.102:0/2809556997 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7f0b200ef8b0 con 0x7f0b30108740 2024-04-25T02:08:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.988+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0b140a2e20 0x7f0b140a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.988+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0b140a2e20 0x7f0b140a52e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0b30071270 tx=0x7f0b1c005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.990+0000 7f0b38ffe740 1 Processor -- start 2024-04-25T02:08:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.990+0000 7f0b38ffe740 1 -- start start 2024-04-25T02:08:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.991+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 0x5621b56168b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.991+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b5611d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.991+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b562a260 0x5621b562c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.991+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5621b5571ab0 con 0x5621b562a260 2024-04-25T02:08:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.991+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5621b551a640 con 0x5621b5616df0 2024-04-25T02:08:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.991+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5621b556da70 con 0x5621b56164d0 2024-04-25T02:08:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.992+0000 7f0b2ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b5611d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.992+0000 7f0b2ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b5611d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36628/0 (socket says 172.21.15.102:36628) 2024-04-25T02:08:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.992+0000 7f0b2ea00640 1 -- 172.21.15.102:0/2568371134 learned_addr learned my addr 172.21.15.102:0/2568371134 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.992+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/2568371134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b562a260 0x5621b562c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.992+0000 7f0b2ea00640 1 -- 172.21.15.102:0/2568371134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 msgr2=0x5621b56168b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.992+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/2568371134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 0x5621b56168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.992+0000 7f0b2ea00640 1 -- 172.21.15.102:0/2568371134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b562a260 msgr2=0x5621b562c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.992+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/2568371134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b562a260 0x5621b562c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.992+0000 7f0b2ea00640 1 -- 172.21.15.102:0/2568371134 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621b5286210 con 0x5621b5616df0 2024-04-25T02:08:04.002 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.996+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/2568371134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b5611d20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c010690 tx=0x7f0b1c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63b2e3ad5c390584 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:04.003 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.996+0000 7f0b1be00640 1 -- 172.21.15.102:0/2568371134 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c00f440 con 0x5621b5616df0 2024-04-25T02:08:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.997+0000 7f0b1be00640 1 -- 172.21.15.102:0/2568371134 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b1c00fac0 con 0x5621b5616df0 2024-04-25T02:08:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.997+0000 7f0b1be00640 1 -- 172.21.15.102:0/2568371134 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c01ec30 con 0x5621b5616df0 2024-04-25T02:08:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.998+0000 7f0b38ffe740 1 -- 172.21.15.102:0/2568371134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 msgr2=0x5621b5611d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.998+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/2568371134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b5611d20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c010690 tx=0x7f0b1c018000 comp rx=0 tx=0).stop 2024-04-25T02:08:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.998+0000 7f0b38ffe740 1 -- 172.21.15.102:0/2568371134 shutdown_connections 2024-04-25T02:08:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.998+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/2568371134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b562a260 0x5621b562c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.998+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/2568371134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b5611d20 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.998+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/2568371134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 0x5621b56168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.998+0000 7f0b38ffe740 1 -- 172.21.15.102:0/2568371134 >> 172.21.15.102:0/2568371134 conn(0x5621b5612b20 msgr2=0x5621b5612f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.998+0000 7f0b38ffe740 1 -- 172.21.15.102:0/2568371134 shutdown_connections 2024-04-25T02:08:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.998+0000 7f0b38ffe740 1 -- 172.21.15.102:0/2568371134 wait complete. 2024-04-25T02:08:04.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.999+0000 7f0b38ffe740 1 -- start start 2024-04-25T02:08:04.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:03.999+0000 7f0b38ffe740 1 Processor -- start 2024-04-25T02:08:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.001+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b560f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.002+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b560f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.002+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b560fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.002+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5621b562cbf0 con 0x5621b56164d0 2024-04-25T02:08:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.002+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5621b552d990 con 0x5621b5616df0 2024-04-25T02:08:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.002+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5621b562cd70 con 0x5621b562a260 2024-04-25T02:08:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.002+0000 7f0b38ffe740 10 client.0 _ll_register_callbacks cb 0x5621b552f950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.002+0000 7f0b2f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b560f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.003+0000 7f0b2ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b560f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.003+0000 7f0b2f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b560f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56034/0 (socket says 172.21.15.102:56034) 2024-04-25T02:08:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.003+0000 7f0b2f400640 1 -- 172.21.15.102:0/4103485458 learned_addr learned my addr 172.21.15.102:0/4103485458 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.003+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b560fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.004+0000 7f0b2fe00640 1 -- 172.21.15.102:0/4103485458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 msgr2=0x5621b560f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.008+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b560f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.008+0000 7f0b2fe00640 1 -- 172.21.15.102:0/4103485458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 msgr2=0x5621b560f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.008+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b560f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.008+0000 7f0b2fe00640 1 -- 172.21.15.102:0/4103485458 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621b5352620 con 0x5621b562a260 2024-04-25T02:08:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.008+0000 7f0b2f400640 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b560f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.008+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b560fd40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0b28005fd0 tx=0x7f0b28009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea437f7ec1c4587f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.008+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b560f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.008+0000 7f0b18c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:08:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.012+0000 7f0b18c00640 1 -- 172.21.15.102:0/4103485458 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2800ca40 con 0x5621b562a260 2024-04-25T02:08:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.012+0000 7f0b18c00640 1 -- 172.21.15.102:0/4103485458 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7f0b2800cbe0 con 0x5621b562a260 2024-04-25T02:08:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.012+0000 7f0b38ffe740 1 -- 172.21.15.102:0/4103485458 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b1c00fac0 con 0x5621b562a260 2024-04-25T02:08:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.012+0000 7f0b18c00640 1 -- 172.21.15.102:0/4103485458 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0b2800cde0 con 0x5621b562a260 2024-04-25T02:08:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.013+0000 7f0b38ffe740 10 client.27505 fetch_fsmap learned FSMap version 46 2024-04-25T02:08:04.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.013+0000 7f0b38ffe740 1 -- 172.21.15.102:0/4103485458 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5621b5638cd0 con 0x5621b562a260 2024-04-25T02:08:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.013+0000 7f0b18c00640 1 -- 172.21.15.102:0/4103485458 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b1c00fac0 con 0x5621b562a260 2024-04-25T02:08:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.015+0000 7f0b18c00640 1 -- 172.21.15.102:0/4103485458 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7f0b2800f040 con 0x5621b562a260 2024-04-25T02:08:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.016+0000 7f0b38ffe740 10 client.27505 fetch_fsmap finished waiting for FSMap version 46 2024-04-25T02:08:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.016+0000 7f0b38ffe740 10 client.27505 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.016+0000 7f0b38ffe740 20 client.27505 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.016+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621b563a190 0x5621b563c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.016+0000 7f0b38ffe740 4 client.27505 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:08:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.016+0000 7f0b38ffe740 1 -- 172.21.15.102:0/4103485458 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621b557eff0 con 0x5621b563a190 2024-04-25T02:08:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.016+0000 7f0b2f400640 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621b563a190 0x5621b563c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.017+0000 7f0b2f400640 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621b563a190 0x5621b563c580 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=833e392cc7409e87 server_cookie=95f001ac283de50f in_seq=0 out_seq=0 2024-04-25T02:08:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.017+0000 7f0b18c00640 10 client.27505 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.018+0000 7f0b18c00640 1 -- 172.21.15.102:0/4103485458 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621b557eff0 con 0x5621b563a190 2024-04-25T02:08:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.018+0000 7f0b18c00640 10 client.27505 handle_command_reply: tid=0 2024-04-25T02:08:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.018+0000 7f0b38ffe740 1 client.27505 shutdown 2024-04-25T02:08:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.019+0000 7f0b38ffe740 1 -- 172.21.15.102:0/4103485458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 msgr2=0x5621b560fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.019+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b560fd40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0b28005fd0 tx=0x7f0b28009f90 comp rx=0 tx=0).stop 2024-04-25T02:08:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.020+0000 7f0b38ffe740 1 -- 172.21.15.102:0/4103485458 shutdown_connections 2024-04-25T02:08:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.020+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621b563a190 0x5621b563c580 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.020+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b560fd40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.020+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b560f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.020+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/4103485458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b560f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.020+0000 7f0b38ffe740 1 -- 172.21.15.102:0/4103485458 >> 172.21.15.102:0/4103485458 conn(0x5621b5612b20 msgr2=0x5621b56188c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.020+0000 7f0b18c00640 0 client.27505 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.020+0000 7f0b38ffe740 1 -- 172.21.15.102:0/4103485458 shutdown_connections 2024-04-25T02:08:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.020+0000 7f0b38ffe740 1 -- 172.21.15.102:0/4103485458 wait complete. 2024-04-25T02:08:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.021+0000 7f0b38ffe740 20 client.27505 trim_cache size 0 max 16384 2024-04-25T02:08:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.044+0000 7f0b38ffe740 1 Processor -- start 2024-04-25T02:08:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.044+0000 7f0b38ffe740 1 -- start start 2024-04-25T02:08:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b5635af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b56099e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b5641e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0b301b6440 con 0x5621b56164d0 2024-04-25T02:08:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0b301b3c50 con 0x5621b5616df0 2024-04-25T02:08:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0b301b1ac0 con 0x5621b562a260 2024-04-25T02:08:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b2ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b56099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b2f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b5635af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b2ea00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b56099e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:36644/0 (socket says 172.21.15.102:36644) 2024-04-25T02:08:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.045+0000 7f0b2ea00640 1 -- 172.21.15.102:0/489972859 learned_addr learned my addr 172.21.15.102:0/489972859 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/489972859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b5641e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b2f400640 1 -- 172.21.15.102:0/489972859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 msgr2=0x5621b5641e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b2f400640 1 --2- 172.21.15.102:0/489972859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b5641e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b2f400640 1 -- 172.21.15.102:0/489972859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 msgr2=0x5621b56099e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:04.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b2f400640 1 --2- 172.21.15.102:0/489972859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b56099e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b2f400640 1 -- 172.21.15.102:0/489972859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621b563c8f0 con 0x5621b56164d0 2024-04-25T02:08:04.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/489972859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b5641e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:08:04.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b2f400640 1 --2- 172.21.15.102:0/489972859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b5635af0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b200b87a0 tx=0x7f0b2003b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792c313ca3fa372c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:04.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b1b400640 1 -- 172.21.15.102:0/489972859 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b20049040 con 0x5621b56164d0 2024-04-25T02:08:04.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b1b400640 1 -- 172.21.15.102:0/489972859 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b200c4a40 con 0x5621b56164d0 2024-04-25T02:08:04.054 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.046+0000 7f0b1b400640 1 -- 172.21.15.102:0/489972859 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b200c45b0 con 0x5621b56164d0 2024-04-25T02:08:04.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.047+0000 7f0b38ffe740 1 -- 172.21.15.102:0/489972859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 msgr2=0x5621b5635af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:04.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.047+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/489972859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b5635af0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b200b87a0 tx=0x7f0b2003b790 comp rx=0 tx=0).stop 2024-04-25T02:08:04.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.048+0000 7f0b38ffe740 1 -- 172.21.15.102:0/489972859 shutdown_connections 2024-04-25T02:08:04.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.048+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/489972859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b562a260 0x5621b5641e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.048+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/489972859 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b5616df0 0x5621b56099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.048+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/489972859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b56164d0 0x5621b5635af0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.055 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.048+0000 7f0b38ffe740 1 -- 172.21.15.102:0/489972859 >> 172.21.15.102:0/489972859 conn(0x5621b5612b20 msgr2=0x5621b5604fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:04.056 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.048+0000 7f0b38ffe740 1 -- 172.21.15.102:0/489972859 shutdown_connections 2024-04-25T02:08:04.056 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.049+0000 7f0b38ffe740 1 -- 172.21.15.102:0/489972859 wait complete. 2024-04-25T02:08:04.056 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.049+0000 7f0b38ffe740 1 -- start start 2024-04-25T02:08:04.056 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.049+0000 7f0b38ffe740 1 Processor -- start 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.050+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 0x5621b5632020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.050+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b5616df0 0x5621b5632560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.050+0000 7f0b38ffe740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b562a260 0x5621b562d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.050+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0b301b2110 con 0x5621b5616df0 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.050+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0b301b3580 con 0x5621b562a260 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.050+0000 7f0b38ffe740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0b301b27e0 con 0x5621b56164d0 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.050+0000 7f0b38ffe740 10 client.0 _ll_register_callbacks cb 0x7f0b28009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 0x5621b5632020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b562a260 0x5621b562d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 0x5621b5632020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51004/0 (socket says 172.21.15.102:51004) 2024-04-25T02:08:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2f400640 1 -- 172.21.15.102:0/362280376 learned_addr learned my addr 172.21.15.102:0/362280376 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b5616df0 0x5621b5632560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2fe00640 1 -- 172.21.15.102:0/362280376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 msgr2=0x5621b5632020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 0x5621b5632020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2fe00640 1 -- 172.21.15.102:0/362280376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b5616df0 msgr2=0x5621b5632560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b5616df0 0x5621b5632560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2fe00640 1 -- 172.21.15.102:0/362280376 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621b562df90 con 0x5621b562a260 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2f400640 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 0x5621b5632020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2ea00640 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b5616df0 0x5621b5632560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b2fe00640 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b562a260 0x5621b562d7c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0b28026820 tx=0x7f0b28005c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfb5569175baf084 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.051+0000 7f0b18c00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:08:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.052+0000 7f0b18c00640 1 -- 172.21.15.102:0/362280376 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2802b040 con 0x5621b562a260 2024-04-25T02:08:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.052+0000 7f0b18c00640 1 -- 172.21.15.102:0/362280376 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8513+0+0 (secure 0 0 0) 0x7f0b28026a10 con 0x5621b562a260 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.052+0000 7f0b38ffe740 1 -- 172.21.15.102:0/362280376 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b200c4a40 con 0x5621b562a260 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.052+0000 7f0b18c00640 1 -- 172.21.15.102:0/362280376 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0b28026c10 con 0x5621b562a260 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.052+0000 7f0b38ffe740 10 client.27557 fetch_fsmap learned FSMap version 46 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.052+0000 7f0b38ffe740 1 -- 172.21.15.102:0/362280376 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5621b56a0370 con 0x5621b562a260 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.052+0000 7f0b18c00640 1 -- 172.21.15.102:0/362280376 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b200c4a40 con 0x5621b562a260 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.053+0000 7f0b18c00640 1 -- 172.21.15.102:0/362280376 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 46) ==== 8362+0+0 (secure 0 0 0) 0x7f0b280051c0 con 0x5621b562a260 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.053+0000 7f0b38ffe740 10 client.27557 fetch_fsmap finished waiting for FSMap version 46 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.053+0000 7f0b38ffe740 10 client.27557 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.053+0000 7f0b38ffe740 20 client.27557 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.053+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621b56a0b00 0x5621b56a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.053+0000 7f0b38ffe740 4 client.27557 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:08:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.053+0000 7f0b38ffe740 1 -- 172.21.15.102:0/362280376 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5621b5579340 con 0x5621b56a0b00 2024-04-25T02:08:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.053+0000 7f0b2f400640 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621b56a0b00 0x5621b56a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.054+0000 7f0b2f400640 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621b56a0b00 0x5621b56a2f20 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c07bb48af1286576 server_cookie=758e8d576d8a629f in_seq=0 out_seq=0 2024-04-25T02:08:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.054+0000 7f0b18c00640 10 client.27557 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b18c00640 1 -- 172.21.15.102:0/362280376 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5621b5579340 con 0x5621b56a0b00 2024-04-25T02:08:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b18c00640 10 client.27557 handle_command_reply: tid=0 2024-04-25T02:08:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 client.27557 shutdown 2024-04-25T02:08:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 -- 172.21.15.102:0/362280376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b562a260 msgr2=0x5621b562d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b562a260 0x5621b562d7c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0b28026820 tx=0x7f0b28005c60 comp rx=0 tx=0).stop 2024-04-25T02:08:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 -- 172.21.15.102:0/362280376 shutdown_connections 2024-04-25T02:08:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5621b56a0b00 0x5621b56a2f20 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5621b562a260 0x5621b562d7c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5621b5616df0 0x5621b5632560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 --2- 172.21.15.102:0/362280376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621b56164d0 0x5621b5632020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 -- 172.21.15.102:0/362280376 >> 172.21.15.102:0/362280376 conn(0x5621b5612b20 msgr2=0x5621b560f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b18c00640 0 client.27557 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 -- 172.21.15.102:0/362280376 shutdown_connections 2024-04-25T02:08:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.086+0000 7f0b38ffe740 1 -- 172.21.15.102:0/362280376 wait complete. 2024-04-25T02:08:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b38ffe740 20 client.27557 trim_cache size 0 max 16384 2024-04-25T02:08:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:08:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:08:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:08:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 -- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0b140a2e20 msgr2=0x7f0b140a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0b140a2e20 0x7f0b140a52e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0b30071270 tx=0x7f0b1c005f90 comp rx=0 tx=0).stop 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 -- 172.21.15.102:0/2809556997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b30108740 msgr2=0x7f0b301a6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b30108740 0x7f0b301a6460 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0b2002a9f0 tx=0x7f0b20003b90 comp rx=0 tx=0).stop 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 -- 172.21.15.102:0/2809556997 shutdown_connections 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f0b140a2e20 0x7f0b140a52e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b30114890 0x7f0b301a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b30109060 0x7f0b301a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 --2- 172.21.15.102:0/2809556997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b30108740 0x7f0b301a6460 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.087+0000 7f0b36200640 1 -- 172.21.15.102:0/2809556997 >> 172.21.15.102:0/2809556997 conn(0x7f0b3006cca0 msgr2=0x7f0b30070380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.088+0000 7f0b36200640 1 -- 172.21.15.102:0/2809556997 shutdown_connections 2024-04-25T02:08:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:04.088+0000 7f0b36200640 1 -- 172.21.15.102:0/2809556997 wait complete. 2024-04-25T02:08:05.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:05 smithi102 ceph-mon[27796]: pgmap v397: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 14 MiB/s wr, 1.38k op/s 2024-04-25T02:08:05.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:05 smithi110 ceph-mon[32848]: pgmap v397: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 14 MiB/s wr, 1.38k op/s 2024-04-25T02:08:05.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:05 smithi193 ceph-mon[32981]: pgmap v397: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 14 MiB/s wr, 1.38k op/s 2024-04-25T02:08:07.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:07 smithi102 ceph-mon[27796]: pgmap v398: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 14 MiB/s wr, 1.33k op/s 2024-04-25T02:08:07.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:07 smithi110 ceph-mon[32848]: pgmap v398: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 14 MiB/s wr, 1.33k op/s 2024-04-25T02:08:07.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:07 smithi193 ceph-mon[32981]: pgmap v398: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 14 MiB/s wr, 1.33k op/s 2024-04-25T02:08:08.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 735.0 s, 1096.0 tps, lat 91.286 ms stddev 115.624 2024-04-25T02:08:08.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:08 smithi102 ceph-mon[27796]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T02:08:08.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:08 smithi110 ceph-mon[32848]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T02:08:08.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:08 smithi193 ceph-mon[32981]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T02:08:09.395 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:09 smithi110 ceph-mon[32848]: pgmap v400: 97 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 74 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 20 MiB/s wr, 1.62k op/s 2024-04-25T02:08:09.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:09 smithi102 ceph-mon[27796]: pgmap v400: 97 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 74 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 20 MiB/s wr, 1.62k op/s 2024-04-25T02:08:09.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:09 smithi193 ceph-mon[32981]: pgmap v400: 97 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 74 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 20 MiB/s wr, 1.62k op/s 2024-04-25T02:08:11.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:11 smithi102 ceph-mon[27796]: pgmap v401: 97 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 74 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-04-25T02:08:11.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:11 smithi110 ceph-mon[32848]: pgmap v401: 97 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 74 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-04-25T02:08:11.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:11 smithi193 ceph-mon[32981]: pgmap v401: 97 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 74 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 21 MiB/s wr, 1.60k op/s 2024-04-25T02:08:13.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 740.0 s, 1419.8 tps, lat 71.251 ms stddev 98.099 2024-04-25T02:08:13.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:13 smithi102 ceph-mon[27796]: pgmap v402: 97 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 74 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 20 MiB/s wr, 1.51k op/s 2024-04-25T02:08:13.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:13 smithi110 ceph-mon[32848]: pgmap v402: 97 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 74 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 20 MiB/s wr, 1.51k op/s 2024-04-25T02:08:13.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:13 smithi193 ceph-mon[32981]: pgmap v402: 97 pgs: 13 active+clean+snaptrim_wait, 10 active+clean+snaptrim, 74 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 20 MiB/s wr, 1.51k op/s 2024-04-25T02:08:14.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:14 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:08:14.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:14 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:08:14.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:14 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:08:15.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:15 smithi102 ceph-mon[27796]: pgmap v403: 97 pgs: 97 active+clean; 5.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 15 MiB/s wr, 808 op/s 2024-04-25T02:08:15.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:15 smithi110 ceph-mon[32848]: pgmap v403: 97 pgs: 97 active+clean; 5.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 15 MiB/s wr, 808 op/s 2024-04-25T02:08:15.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:15 smithi193 ceph-mon[32981]: pgmap v403: 97 pgs: 97 active+clean; 5.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 15 MiB/s wr, 808 op/s 2024-04-25T02:08:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.257+0000 7f471b600640 1 -- 172.21.15.102:0/1076976050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4714107fa0 msgr2=0x7f4714108380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.257+0000 7f471b600640 1 --2- 172.21.15.102:0/1076976050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4714107fa0 0x7f4714108380 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f4708009900 tx=0x7f470802a410 comp rx=0 tx=0).stop 2024-04-25T02:08:17.265 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.258+0000 7f471b600640 1 -- 172.21.15.102:0/1076976050 shutdown_connections 2024-04-25T02:08:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.258+0000 7f471b600640 1 --2- 172.21.15.102:0/1076976050 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f471410d070 0x7f4714114c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.258+0000 7f471b600640 1 --2- 172.21.15.102:0/1076976050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4714108950 0x7f471410c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.258+0000 7f471b600640 1 --2- 172.21.15.102:0/1076976050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4714107fa0 0x7f4714108380 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.258+0000 7f471b600640 1 -- 172.21.15.102:0/1076976050 >> 172.21.15.102:0/1076976050 conn(0x7f4714075cd0 msgr2=0x7f47140780f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.258+0000 7f471b600640 1 -- 172.21.15.102:0/1076976050 shutdown_connections 2024-04-25T02:08:17.266 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.258+0000 7f471b600640 1 -- 172.21.15.102:0/1076976050 wait complete. 2024-04-25T02:08:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.260+0000 7f471b600640 1 Processor -- start 2024-04-25T02:08:17.267 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.260+0000 7f471b600640 1 -- start start 2024-04-25T02:08:17.269 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.260+0000 7f471b600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4714107fa0 0x7f471419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.260+0000 7f471b600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d070 0x7f471419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.260+0000 7f471b600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47141a2e60 0x7f47141a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.260+0000 7f471b600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4714118150 con 0x7f471410d070 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.260+0000 7f471b600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f4714117fd0 con 0x7f4714107fa0 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.260+0000 7f471b600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47141182d0 con 0x7f47141a2e60 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4719600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47141a2e60 0x7f47141a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4719600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47141a2e60 0x7f47141a3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56208/0 (socket says 172.21.15.102:56208) 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4719600640 1 -- 172.21.15.102:0/4272994789 learned_addr learned my addr 172.21.15.102:0/4272994789 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4718c00640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4714107fa0 0x7f471419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4713e00640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d070 0x7f471419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.270 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4719600640 1 -- 172.21.15.102:0/4272994789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4714107fa0 msgr2=0x7f471419e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4719600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4714107fa0 0x7f471419e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4719600640 1 -- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d070 msgr2=0x7f471419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4719600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d070 0x7f471419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4719600640 1 -- 172.21.15.102:0/4272994789 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47141a3950 con 0x7f47141a2e60 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.261+0000 7f4713e00640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d070 0x7f471419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.262+0000 7f4719600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47141a2e60 0x7f47141a3310 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f470c00eb10 tx=0x7f470c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.262+0000 7f4711600640 1 -- 172.21.15.102:0/4272994789 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c004b20 con 0x7f47141a2e60 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.262+0000 7f4711600640 1 -- 172.21.15.102:0/4272994789 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f470c004cc0 con 0x7f47141a2e60 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.262+0000 7f4711600640 1 -- 172.21.15.102:0/4272994789 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c010760 con 0x7f47141a2e60 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.262+0000 7f471b600640 1 -- 172.21.15.102:0/4272994789 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47141a3c40 con 0x7f47141a2e60 2024-04-25T02:08:17.271 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.262+0000 7f471b600640 1 -- 172.21.15.102:0/4272994789 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47141ab3e0 con 0x7f47141a2e60 2024-04-25T02:08:17.273 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.265+0000 7f471e365740 1 Processor -- start 2024-04-25T02:08:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.267+0000 7f4711600640 1 -- 172.21.15.102:0/4272994789 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f470c005020 con 0x7f47141a2e60 2024-04-25T02:08:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.267+0000 7f471e365740 1 -- start start 2024-04-25T02:08:17.275 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.268+0000 7f471e365740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469a71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.268+0000 7f471e365740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 0x5581469b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.268+0000 7f471e365740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 0x5581469b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.268+0000 7f471e365740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55814690aab0 con 0x5581469a7720 2024-04-25T02:08:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.268+0000 7f471e365740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5581468b3640 con 0x5581469a6e00 2024-04-25T02:08:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.268+0000 7f471e365740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558146906a70 con 0x5581469b1fe0 2024-04-25T02:08:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.268+0000 7f4713e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469a71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.276 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.269+0000 7f4718c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 0x5581469b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.269+0000 7f4713e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469a71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34276/0 (socket says 172.21.15.102:34276) 2024-04-25T02:08:17.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.269+0000 7f4713e00640 1 -- 172.21.15.102:0/1968819610 learned_addr learned my addr 172.21.15.102:0/1968819610 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:17.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.269+0000 7f4713e00640 1 --2- 172.21.15.102:0/1968819610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 0x5581469b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4713e00640 1 -- 172.21.15.102:0/1968819610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 msgr2=0x5581469b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:17.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4713e00640 1 --2- 172.21.15.102:0/1968819610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 0x5581469b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4713e00640 1 -- 172.21.15.102:0/1968819610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 msgr2=0x5581469b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:17.277 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4713e00640 1 --2- 172.21.15.102:0/1968819610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 0x5581469b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4713e00640 1 -- 172.21.15.102:0/1968819610 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55814661f210 con 0x5581469a6e00 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4718c00640 1 --2- 172.21.15.102:0/1968819610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 0x5581469b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4713e00640 1 --2- 172.21.15.102:0/1968819610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469a71e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4700005d00 tx=0x7f4700009410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10ea3c5ec09a6377 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4705c00640 1 -- 172.21.15.102:0/1968819610 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700004220 con 0x5581469a6e00 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4711600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e20 0x7f46fc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4705c00640 1 -- 172.21.15.102:0/1968819610 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47000043c0 con 0x5581469a6e00 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.270+0000 7f4705c00640 1 -- 172.21.15.102:0/1968819610 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700004dd0 con 0x5581469a6e00 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f4711600640 1 -- 172.21.15.102:0/4272994789 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8557+0+0 (secure 0 0 0) 0x7f470c0cfac0 con 0x7f47141a2e60 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f4718c00640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e20 0x7f46fc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f471e365740 1 -- 172.21.15.102:0/1968819610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 msgr2=0x5581469a71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f471e365740 1 --2- 172.21.15.102:0/1968819610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469a71e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4700005d00 tx=0x7f4700009410 comp rx=0 tx=0).stop 2024-04-25T02:08:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f471e365740 1 -- 172.21.15.102:0/1968819610 shutdown_connections 2024-04-25T02:08:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f471e365740 1 --2- 172.21.15.102:0/1968819610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 0x5581469b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f471e365740 1 --2- 172.21.15.102:0/1968819610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 0x5581469b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f471e365740 1 --2- 172.21.15.102:0/1968819610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469a71e0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f471e365740 1 -- 172.21.15.102:0/1968819610 >> 172.21.15.102:0/1968819610 conn(0x5581469a3310 msgr2=0x5581469a36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.271+0000 7f471e365740 1 -- 172.21.15.102:0/1968819610 shutdown_connections 2024-04-25T02:08:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.272+0000 7f4718c00640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e20 0x7f46fc0a52e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4708000c00 tx=0x7f4708005b50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.272+0000 7f471e365740 1 -- 172.21.15.102:0/1968819610 wait complete. 2024-04-25T02:08:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.272+0000 7f471e365740 1 -- start start 2024-04-25T02:08:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.273+0000 7f471e365740 1 Processor -- start 2024-04-25T02:08:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.273+0000 7f471e365740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.273+0000 7f471e365740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a7720 0x5581469af420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.273+0000 7f471e365740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469b1fe0 0x5581469af960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.273+0000 7f471e365740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5581469b4bb0 con 0x5581469b1fe0 2024-04-25T02:08:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.273+0000 7f471e365740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5581468c6990 con 0x5581469a6e00 2024-04-25T02:08:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.273+0000 7f471e365740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5581469b4d30 con 0x5581469a7720 2024-04-25T02:08:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.273+0000 7f471e365740 10 client.0 _ll_register_callbacks cb 0x5581468c8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.274+0000 7f4719600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469b1fe0 0x5581469af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.274+0000 7f4719600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469b1fe0 0x5581469af960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44234/0 (socket says 172.21.15.102:44234) 2024-04-25T02:08:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.274+0000 7f4719600640 1 -- 172.21.15.102:0/2677351664 learned_addr learned my addr 172.21.15.102:0/2677351664 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.274+0000 7f4713e00640 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a7720 0x5581469af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.274+0000 7f4718c00640 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.274+0000 7f4719600640 1 -- 172.21.15.102:0/2677351664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a7720 msgr2=0x5581469af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.275+0000 7f4719600640 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a7720 0x5581469af420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.275+0000 7f4719600640 1 -- 172.21.15.102:0/2677351664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 msgr2=0x5581469aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.275+0000 7f4719600640 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469aeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.275+0000 7f4719600640 1 -- 172.21.15.102:0/2677351664 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581466eb620 con 0x5581469b1fe0 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.275+0000 7f4719600640 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469b1fe0 0x5581469af960 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f470c098990 tx=0x7f470c09bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457e124b99d0828 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.275+0000 7f46fa000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.275+0000 7f46fa000640 1 -- 172.21.15.102:0/2677351664 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c09bfc0 con 0x5581469b1fe0 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.276+0000 7f46fa000640 1 -- 172.21.15.102:0/2677351664 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8557+0+0 (secure 0 0 0) 0x7f470c0a47a0 con 0x5581469b1fe0 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.276+0000 7f471e365740 1 -- 172.21.15.102:0/2677351664 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47000043c0 con 0x5581469b1fe0 2024-04-25T02:08:17.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.276+0000 7f46fa000640 1 -- 172.21.15.102:0/2677351664 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f470c0a0370 con 0x5581469b1fe0 2024-04-25T02:08:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.276+0000 7f471e365740 10 client.16851 fetch_fsmap learned FSMap version 47 2024-04-25T02:08:17.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.276+0000 7f471e365740 1 -- 172.21.15.102:0/2677351664 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5581469c0a40 con 0x5581469b1fe0 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.277+0000 7f46fa000640 1 -- 172.21.15.102:0/2677351664 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47000043c0 con 0x5581469b1fe0 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.277+0000 7f46fa000640 1 -- 172.21.15.102:0/2677351664 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f470c09e070 con 0x5581469b1fe0 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.277+0000 7f471e365740 10 client.16851 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.277+0000 7f471e365740 10 client.16851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.277+0000 7f471e365740 20 client.16851 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.277+0000 7f471e365740 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5581469c1fc0 0x5581469c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.277+0000 7f471e365740 4 client.16851 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.277+0000 7f471e365740 1 -- 172.21.15.102:0/2677351664 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558146917ff0 con 0x5581469c1fc0 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.278+0000 7f4718c00640 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5581469c1fc0 0x5581469c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.278+0000 7f4718c00640 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5581469c1fc0 0x5581469c43f0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72bc942aaa5651b8 server_cookie=50f7e3d84140ffbc in_seq=0 out_seq=0 2024-04-25T02:08:17.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.278+0000 7f46fa000640 10 client.16851 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f46fa000640 1 -- 172.21.15.102:0/2677351664 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558146917ff0 con 0x5581469c1fc0 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f46fa000640 10 client.16851 handle_command_reply: tid=0 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f471e365740 1 client.16851 shutdown 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f471e365740 1 -- 172.21.15.102:0/2677351664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469b1fe0 msgr2=0x5581469af960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f471e365740 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469b1fe0 0x5581469af960 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f470c098990 tx=0x7f470c09bbc0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f471e365740 1 -- 172.21.15.102:0/2677351664 shutdown_connections 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f471e365740 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5581469c1fc0 0x5581469c43f0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f471e365740 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469b1fe0 0x5581469af960 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f46fa000640 0 client.16851 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f471e365740 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a7720 0x5581469af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f471e365740 1 --2- 172.21.15.102:0/2677351664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a6e00 0x5581469aeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.279+0000 7f471e365740 1 -- 172.21.15.102:0/2677351664 >> 172.21.15.102:0/2677351664 conn(0x5581469a3310 msgr2=0x5581469a9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.280+0000 7f471e365740 1 -- 172.21.15.102:0/2677351664 shutdown_connections 2024-04-25T02:08:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.280+0000 7f471e365740 1 -- 172.21.15.102:0/2677351664 wait complete. 2024-04-25T02:08:17.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.280+0000 7f471e365740 20 client.16851 trim_cache size 0 max 16384 2024-04-25T02:08:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.302+0000 7f471e365740 1 Processor -- start 2024-04-25T02:08:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.302+0000 7f471e365740 1 -- start start 2024-04-25T02:08:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.302+0000 7f471e365740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a6e00 0x5581469bda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.302+0000 7f471e365740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a7720 0x5581469a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.302+0000 7f471e365740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 0x5581469af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.302+0000 7f471e365740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f47141b63f0 con 0x5581469a6e00 2024-04-25T02:08:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.302+0000 7f471e365740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f47141b3c30 con 0x5581469a7720 2024-04-25T02:08:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.302+0000 7f471e365740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47141b1aa0 con 0x5581469b1fe0 2024-04-25T02:08:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.303+0000 7f4718c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a6e00 0x5581469bda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.303+0000 7f4718c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a6e00 0x5581469bda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44244/0 (socket says 172.21.15.102:44244) 2024-04-25T02:08:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.303+0000 7f4718c00640 1 -- 172.21.15.102:0/2994601096 learned_addr learned my addr 172.21.15.102:0/2994601096 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.303+0000 7f4719600640 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 0x5581469af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.303+0000 7f4713e00640 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a7720 0x5581469a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.304+0000 7f4718c00640 1 -- 172.21.15.102:0/2994601096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 msgr2=0x5581469af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.304+0000 7f4718c00640 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 0x5581469af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.304+0000 7f4718c00640 1 -- 172.21.15.102:0/2994601096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a7720 msgr2=0x5581469a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.304+0000 7f4718c00640 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a7720 0x5581469a2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.304+0000 7f4718c00640 1 -- 172.21.15.102:0/2994601096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581469a3140 con 0x5581469a6e00 2024-04-25T02:08:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.304+0000 7f4713e00640 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a7720 0x5581469a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:08:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.305+0000 7f4718c00640 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a6e00 0x5581469bda70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4708009230 tx=0x7f470803e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0759b85b0970748 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.305+0000 7f4705200640 1 -- 172.21.15.102:0/2994601096 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708035040 con 0x5581469a6e00 2024-04-25T02:08:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.305+0000 7f4705200640 1 -- 172.21.15.102:0/2994601096 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4708059070 con 0x5581469a6e00 2024-04-25T02:08:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.305+0000 7f4705200640 1 -- 172.21.15.102:0/2994601096 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47080325c0 con 0x5581469a6e00 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 -- 172.21.15.102:0/2994601096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a6e00 msgr2=0x5581469bda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a6e00 0x5581469bda70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4708009230 tx=0x7f470803e000 comp rx=0 tx=0).stop 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 -- 172.21.15.102:0/2994601096 shutdown_connections 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469b1fe0 0x5581469af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469a7720 0x5581469a2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 --2- 172.21.15.102:0/2994601096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a6e00 0x5581469bda70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 -- 172.21.15.102:0/2994601096 >> 172.21.15.102:0/2994601096 conn(0x5581469a3310 msgr2=0x5581469aeea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 -- 172.21.15.102:0/2994601096 shutdown_connections 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 -- 172.21.15.102:0/2994601096 wait complete. 2024-04-25T02:08:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.306+0000 7f471e365740 1 -- start start 2024-04-25T02:08:17.316 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f471e365740 1 Processor -- start 2024-04-25T02:08:17.316 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f471e365740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a6e00 0x5581469b6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.316 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f471e365740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 0x5581469b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.316 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f471e365740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469b1fe0 0x558146a27570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.316 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f471e365740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f47141b20f0 con 0x5581469a7720 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f471e365740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f47141b3560 con 0x5581469b1fe0 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f471e365740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47141b27c0 con 0x5581469a6e00 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f471e365740 10 client.0 _ll_register_callbacks cb 0x7f470c09bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f4719600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469b1fe0 0x558146a27570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f4718c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a6e00 0x5581469b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f4718c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a6e00 0x5581469b6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:56244/0 (socket says 172.21.15.102:56244) 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f4718c00640 1 -- 172.21.15.102:0/3124691145 learned_addr learned my addr 172.21.15.102:0/3124691145 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f4719600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469b1fe0 0x558146a27570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:34298/0 (socket says 172.21.15.102:34298) 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.307+0000 7f4713e00640 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 0x5581469b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f4718c00640 1 -- 172.21.15.102:0/3124691145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469b1fe0 msgr2=0x558146a27570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f4718c00640 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469b1fe0 0x558146a27570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f4718c00640 1 -- 172.21.15.102:0/3124691145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 msgr2=0x5581469b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f4718c00640 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 0x5581469b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f4718c00640 1 -- 172.21.15.102:0/3124691145 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558146a27b60 con 0x5581469a6e00 2024-04-25T02:08:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f4719600640 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469b1fe0 0x558146a27570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f4718c00640 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a6e00 0x5581469b6170 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4708056040 tx=0x7f4708004820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1f26c969da7be6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f46fa000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:08:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f46fa000640 1 -- 172.21.15.102:0/3124691145 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4708035040 con 0x5581469a6e00 2024-04-25T02:08:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f46fa000640 1 -- 172.21.15.102:0/3124691145 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8557+0+0 (secure 0 0 0) 0x7f47080533a0 con 0x5581469a6e00 2024-04-25T02:08:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.308+0000 7f471e365740 1 -- 172.21.15.102:0/3124691145 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4708059070 con 0x5581469a6e00 2024-04-25T02:08:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.309+0000 7f46fa000640 1 -- 172.21.15.102:0/3124691145 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f47080535a0 con 0x5581469a6e00 2024-04-25T02:08:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.309+0000 7f471e365740 10 client.27526 fetch_fsmap learned FSMap version 47 2024-04-25T02:08:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.309+0000 7f471e365740 1 -- 172.21.15.102:0/3124691145 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558146a28e50 con 0x5581469a6e00 2024-04-25T02:08:17.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.309+0000 7f46fa000640 1 -- 172.21.15.102:0/3124691145 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4708059070 con 0x5581469a6e00 2024-04-25T02:08:17.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.310+0000 7f46fa000640 1 -- 172.21.15.102:0/3124691145 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f47080035f0 con 0x5581469a6e00 2024-04-25T02:08:17.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.310+0000 7f471e365740 10 client.27526 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T02:08:17.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.310+0000 7f471e365740 10 client.27526 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:17.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.310+0000 7f471e365740 20 client.27526 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:17.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.311+0000 7f471e365740 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5581469c1fc0 0x558146a2b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:17.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.311+0000 7f471e365740 4 client.27526 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:08:17.319 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.311+0000 7f471e365740 1 -- 172.21.15.102:0/3124691145 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558146912340 con 0x5581469c1fc0 2024-04-25T02:08:17.320 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.312+0000 7f4713e00640 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5581469c1fc0 0x558146a2b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:17.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.314+0000 7f4713e00640 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5581469c1fc0 0x558146a2b500 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a670672e214d1536 server_cookie=7d968cd55797bf48 in_seq=0 out_seq=0 2024-04-25T02:08:17.321 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.314+0000 7f46fa000640 10 client.27526 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.327+0000 7f46fa000640 1 -- 172.21.15.102:0/3124691145 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558146912340 con 0x5581469c1fc0 2024-04-25T02:08:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.327+0000 7f46fa000640 10 client.27526 handle_command_reply: tid=0 2024-04-25T02:08:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.327+0000 7f471e365740 1 client.27526 shutdown 2024-04-25T02:08:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 -- 172.21.15.102:0/3124691145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a6e00 msgr2=0x5581469b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a6e00 0x5581469b6170 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4708056040 tx=0x7f4708004820 comp rx=0 tx=0).stop 2024-04-25T02:08:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 -- 172.21.15.102:0/3124691145 shutdown_connections 2024-04-25T02:08:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5581469c1fc0 0x558146a2b500 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5581469b1fe0 0x558146a27570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5581469a7720 0x5581469b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f46fa000640 0 client.27526 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:17.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 --2- 172.21.15.102:0/3124691145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581469a6e00 0x5581469b6170 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 -- 172.21.15.102:0/3124691145 >> 172.21.15.102:0/3124691145 conn(0x5581469a3310 msgr2=0x5581469af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 -- 172.21.15.102:0/3124691145 shutdown_connections 2024-04-25T02:08:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 1 -- 172.21.15.102:0/3124691145 wait complete. 2024-04-25T02:08:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.328+0000 7f471e365740 20 client.27526 trim_cache size 0 max 16384 2024-04-25T02:08:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:08:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:08:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:08:17.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:08:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 -- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e20 msgr2=0x7f46fc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e20 0x7f46fc0a52e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4708000c00 tx=0x7f4708005b50 comp rx=0 tx=0).stop 2024-04-25T02:08:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 -- 172.21.15.102:0/4272994789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47141a2e60 msgr2=0x7f47141a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47141a2e60 0x7f47141a3310 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f470c00eb10 tx=0x7f470c00c2d0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 -- 172.21.15.102:0/4272994789 shutdown_connections 2024-04-25T02:08:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f46fc0a2e20 0x7f46fc0a52e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47141a2e60 0x7f47141a3310 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f471410d070 0x7f471419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 --2- 172.21.15.102:0/4272994789 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4714107fa0 0x7f471419e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 -- 172.21.15.102:0/4272994789 >> 172.21.15.102:0/4272994789 conn(0x7f4714075cd0 msgr2=0x7f4714077fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.329+0000 7f471b600640 1 -- 172.21.15.102:0/4272994789 shutdown_connections 2024-04-25T02:08:17.339 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:17.330+0000 7f471b600640 1 -- 172.21.15.102:0/4272994789 wait complete. 2024-04-25T02:08:17.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:17 smithi102 ceph-mon[27796]: pgmap v404: 97 pgs: 97 active+clean; 5.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 15 MiB/s wr, 700 op/s 2024-04-25T02:08:17.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:17 smithi110 ceph-mon[32848]: pgmap v404: 97 pgs: 97 active+clean; 5.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 15 MiB/s wr, 700 op/s 2024-04-25T02:08:17.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:17 smithi193 ceph-mon[32981]: pgmap v404: 97 pgs: 97 active+clean; 5.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 15 MiB/s wr, 700 op/s 2024-04-25T02:08:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 745.0 s, 1618.0 tps, lat 61.295 ms stddev 72.371 2024-04-25T02:08:18.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:18 smithi110 ceph-mon[32848]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T02:08:18.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:18 smithi193 ceph-mon[32981]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T02:08:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:18 smithi102 ceph-mon[27796]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T02:08:19.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:19 smithi110 ceph-mon[32848]: pgmap v406: 97 pgs: 97 active+clean; 5.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 13 MiB/s wr, 656 op/s 2024-04-25T02:08:19.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:19 smithi193 ceph-mon[32981]: pgmap v406: 97 pgs: 97 active+clean; 5.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 13 MiB/s wr, 656 op/s 2024-04-25T02:08:19.830 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:19 smithi102 ceph-mon[27796]: pgmap v406: 97 pgs: 97 active+clean; 5.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 13 MiB/s wr, 656 op/s 2024-04-25T02:08:21.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:21 smithi110 ceph-mon[32848]: pgmap v407: 97 pgs: 97 active+clean; 5.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 15 MiB/s wr, 652 op/s 2024-04-25T02:08:21.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:21 smithi193 ceph-mon[32981]: pgmap v407: 97 pgs: 97 active+clean; 5.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 15 MiB/s wr, 652 op/s 2024-04-25T02:08:21.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:21 smithi102 ceph-mon[27796]: pgmap v407: 97 pgs: 97 active+clean; 5.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 15 MiB/s wr, 652 op/s 2024-04-25T02:08:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 750.0 s, 1405.6 tps, lat 72.234 ms stddev 81.201 2024-04-25T02:08:23.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:23 smithi110 ceph-mon[32848]: pgmap v408: 97 pgs: 97 active+clean; 5.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 15 MiB/s wr, 652 op/s 2024-04-25T02:08:23.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:23 smithi193 ceph-mon[32981]: pgmap v408: 97 pgs: 97 active+clean; 5.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 15 MiB/s wr, 652 op/s 2024-04-25T02:08:23.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:23 smithi102 ceph-mon[27796]: pgmap v408: 97 pgs: 97 active+clean; 5.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 15 MiB/s wr, 652 op/s 2024-04-25T02:08:25.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:25 smithi110 ceph-mon[32848]: pgmap v409: 97 pgs: 97 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 13 MiB/s wr, 640 op/s 2024-04-25T02:08:25.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:25 smithi193 ceph-mon[32981]: pgmap v409: 97 pgs: 97 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 13 MiB/s wr, 640 op/s 2024-04-25T02:08:25.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:25 smithi102 ceph-mon[27796]: pgmap v409: 97 pgs: 97 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 13 MiB/s wr, 640 op/s 2024-04-25T02:08:27.743 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:27 smithi110 ceph-mon[32848]: pgmap v410: 97 pgs: 97 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 13 MiB/s wr, 648 op/s 2024-04-25T02:08:27.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:27 smithi193 ceph-mon[32981]: pgmap v410: 97 pgs: 97 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 13 MiB/s wr, 648 op/s 2024-04-25T02:08:27.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:27 smithi102 ceph-mon[27796]: pgmap v410: 97 pgs: 97 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 13 MiB/s wr, 648 op/s 2024-04-25T02:08:28.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 755.0 s, 1439.4 tps, lat 69.012 ms stddev 80.705 2024-04-25T02:08:29.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:29 smithi110 ceph-mon[32848]: pgmap v411: 97 pgs: 97 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 14 MiB/s wr, 651 op/s 2024-04-25T02:08:29.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:29 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:08:29.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:29 smithi193 ceph-mon[32981]: pgmap v411: 97 pgs: 97 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 14 MiB/s wr, 651 op/s 2024-04-25T02:08:29.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:29 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:08:29.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:29 smithi102 ceph-mon[27796]: pgmap v411: 97 pgs: 97 active+clean; 5.2 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 14 MiB/s wr, 651 op/s 2024-04-25T02:08:29.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:29 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:08:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.513+0000 7fc8e7000640 1 -- 172.21.15.102:0/3841360212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e011af20 msgr2=0x7fc8e011d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.513+0000 7fc8e7000640 1 --2- 172.21.15.102:0/3841360212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e011af20 0x7fc8e011d310 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8009860 tx=0x7fc8c802a3d0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.513+0000 7fc8e7000640 1 -- 172.21.15.102:0/3841360212 shutdown_connections 2024-04-25T02:08:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.513+0000 7fc8e7000640 1 --2- 172.21.15.102:0/3841360212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e011af20 0x7fc8e011d310 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.513+0000 7fc8e7000640 1 --2- 172.21.15.102:0/3841360212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e0110310 0x7fc8e0108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.513+0000 7fc8e7000640 1 --2- 172.21.15.102:0/3841360212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e010ff30 0x7fc8e0108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.513+0000 7fc8e7000640 1 -- 172.21.15.102:0/3841360212 >> 172.21.15.102:0/3841360212 conn(0x7fc8e006c810 msgr2=0x7fc8e006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.513+0000 7fc8e7000640 1 -- 172.21.15.102:0/3841360212 shutdown_connections 2024-04-25T02:08:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.513+0000 7fc8e7000640 1 -- 172.21.15.102:0/3841360212 wait complete. 2024-04-25T02:08:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.515+0000 7fc8e7000640 1 Processor -- start 2024-04-25T02:08:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.515+0000 7fc8e7000640 1 -- start start 2024-04-25T02:08:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.515+0000 7fc8e7000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e010ff30 0x7fc8e01a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.524 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.516+0000 7fc8e7000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e011af20 0x7fc8e01a7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.516+0000 7fc8e7000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e01ab710 0x7fc8e01abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.516+0000 7fc8e7000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01209f0 con 0x7fc8e01ab710 2024-04-25T02:08:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.516+0000 7fc8e7000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e0120870 con 0x7fc8e011af20 2024-04-25T02:08:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.516+0000 7fc8e7000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e0120b70 con 0x7fc8e010ff30 2024-04-25T02:08:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.517+0000 7fc8e5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e01ab710 0x7fc8e01abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.517+0000 7fc8e5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e01ab710 0x7fc8e01abbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47774/0 (socket says 172.21.15.102:47774) 2024-04-25T02:08:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.517+0000 7fc8e5000640 1 -- 172.21.15.102:0/786416866 learned_addr learned my addr 172.21.15.102:0/786416866 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.517+0000 7fc8df400640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e011af20 0x7fc8e01a7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.518+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e010ff30 0x7fc8e01a6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.518+0000 7fc8df400640 1 -- 172.21.15.102:0/786416866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e010ff30 msgr2=0x7fc8e01a6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.518+0000 7fc8df400640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e010ff30 0x7fc8e01a6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.518+0000 7fc8df400640 1 -- 172.21.15.102:0/786416866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e01ab710 msgr2=0x7fc8e01abbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.518+0000 7fc8df400640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e01ab710 0x7fc8e01abbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.518+0000 7fc8df400640 1 -- 172.21.15.102:0/786416866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e01ac200 con 0x7fc8e011af20 2024-04-25T02:08:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.518+0000 7fc8df400640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e011af20 0x7fc8e01a7390 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000b3c0 tx=0x7fc8d000b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.519+0000 7fc8dcc00640 1 -- 172.21.15.102:0/786416866 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0011830 con 0x7fc8e011af20 2024-04-25T02:08:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.519+0000 7fc8dcc00640 1 -- 172.21.15.102:0/786416866 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8d0016070 con 0x7fc8e011af20 2024-04-25T02:08:30.527 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.519+0000 7fc8dcc00640 1 -- 172.21.15.102:0/786416866 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000f2e0 con 0x7fc8e011af20 2024-04-25T02:08:30.528 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.519+0000 7fc8e7000640 1 -- 172.21.15.102:0/786416866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e01ac4f0 con 0x7fc8e011af20 2024-04-25T02:08:30.529 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.520+0000 7fc8e7000640 1 -- 172.21.15.102:0/786416866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e01b3c90 con 0x7fc8e011af20 2024-04-25T02:08:30.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.525+0000 7fc8e9f9a740 1 Processor -- start 2024-04-25T02:08:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.525+0000 7fc8e9f9a740 1 -- start start 2024-04-25T02:08:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.526+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5e91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.526+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 0x56325e5f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.527+0000 7fc8e5000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5e91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.527+0000 7fc8e5000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5e91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:50828/0 (socket says 172.21.15.102:50828) 2024-04-25T02:08:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.527+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5f3fe0 0x56325e5f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.527+0000 7fc8e5000640 1 -- 172.21.15.102:0/360320289 learned_addr learned my addr 172.21.15.102:0/360320289 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.527+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/360320289 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56325e54cab0 con 0x56325e5e9720 2024-04-25T02:08:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.527+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/360320289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56325e4f5640 con 0x56325e5e8e00 2024-04-25T02:08:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.527+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/360320289 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56325e548a70 con 0x56325e5f3fe0 2024-04-25T02:08:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.528+0000 7fc8dcc00640 1 -- 172.21.15.102:0/786416866 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc8d000f480 con 0x7fc8e011af20 2024-04-25T02:08:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.528+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/360320289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 0x56325e5f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.528+0000 7fc8e5000640 1 -- 172.21.15.102:0/360320289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5f3fe0 msgr2=0x56325e5f63d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:08:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.528+0000 7fc8e5000640 1 --2- 172.21.15.102:0/360320289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5f3fe0 0x56325e5f63d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.528+0000 7fc8e5000640 1 -- 172.21.15.102:0/360320289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 msgr2=0x56325e5f3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.528+0000 7fc8e5000640 1 --2- 172.21.15.102:0/360320289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 0x56325e5f3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.528+0000 7fc8e5000640 1 -- 172.21.15.102:0/360320289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56325e261210 con 0x56325e5e8e00 2024-04-25T02:08:30.536 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.529+0000 7fc8e5000640 1 --2- 172.21.15.102:0/360320289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5e91e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8030db0 tx=0x7fc8c8030de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba59abfcc0f0fa79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.530+0000 7fc8dcc00640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc8c40a2ea0 0x7fc8c40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.530+0000 7fc8dcc00640 1 -- 172.21.15.102:0/786416866 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8513+0+0 (secure 0 0 0) 0x7fc8d00ce8d0 con 0x7fc8e011af20 2024-04-25T02:08:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.531+0000 7fc8d5600640 1 -- 172.21.15.102:0/360320289 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c80305a0 con 0x56325e5e8e00 2024-04-25T02:08:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.531+0000 7fc8d5600640 1 -- 172.21.15.102:0/360320289 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8c8040070 con 0x56325e5e8e00 2024-04-25T02:08:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.531+0000 7fc8d5600640 1 -- 172.21.15.102:0/360320289 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8004c60 con 0x56325e5e8e00 2024-04-25T02:08:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.531+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc8c40a2ea0 0x7fc8c40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.532+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/360320289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 msgr2=0x56325e5e91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.532+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/360320289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5e91e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8030db0 tx=0x7fc8c8030de0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.534+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/360320289 shutdown_connections 2024-04-25T02:08:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.534+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/360320289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5f3fe0 0x56325e5f63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.534+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/360320289 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 0x56325e5f3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.534+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/360320289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5e91e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.534+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/360320289 >> 172.21.15.102:0/360320289 conn(0x56325e5e5310 msgr2=0x56325e5e56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.534+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc8c40a2ea0 0x7fc8c40a5360 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x56325e5ea3a0 tx=0x7fc8d8009680 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.535+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/360320289 shutdown_connections 2024-04-25T02:08:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.535+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/360320289 wait complete. 2024-04-25T02:08:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.537+0000 7fc8e9f9a740 1 -- start start 2024-04-25T02:08:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.537+0000 7fc8e9f9a740 1 Processor -- start 2024-04-25T02:08:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.539+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5f1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.539+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.539+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5f3fe0 0x56325e5f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.540+0000 7fc8df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.540+0000 7fc8df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51202/0 (socket says 172.21.15.102:51202) 2024-04-25T02:08:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.540+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.540+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5f1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:50838/0 (socket says 172.21.15.102:50838) 2024-04-25T02:08:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.541+0000 7fc8df400640 1 -- 172.21.15.102:0/2763588514 learned_addr learned my addr 172.21.15.102:0/2763588514 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:30.549 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.542+0000 7fc8e5000640 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5f3fe0 0x56325e5f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.543+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56325e5f6bb0 con 0x56325e5f3fe0 2024-04-25T02:08:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.543+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56325e508990 con 0x56325e5e8e00 2024-04-25T02:08:30.552 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.543+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56325e5f6d30 con 0x56325e5e9720 2024-04-25T02:08:30.553 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.545+0000 7fc8e9f9a740 10 client.0 _ll_register_callbacks cb 0x56325e50a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.545+0000 7fc8dfe00640 1 -- 172.21.15.102:0/2763588514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 msgr2=0x56325e5f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.545+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.545+0000 7fc8dfe00640 1 -- 172.21.15.102:0/2763588514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5f3fe0 msgr2=0x56325e5f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.545+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5f3fe0 0x56325e5f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.545+0000 7fc8dfe00640 1 -- 172.21.15.102:0/2763588514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56325e32d620 con 0x56325e5e8e00 2024-04-25T02:08:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.545+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8c8040070 con 0x56325e5e8e00 2024-04-25T02:08:30.554 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.546+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5f1010 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8005fd0 tx=0x7fc8d8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8124ebec3968463b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:30.555 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.546+0000 7fc8ce000640 10 client.27587 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:08:30.555 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.547+0000 7fc8ce000640 1 -- 172.21.15.102:0/2763588514 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d802c060 con 0x56325e5e8e00 2024-04-25T02:08:30.556 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.547+0000 7fc8ce000640 1 -- 172.21.15.102:0/2763588514 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8513+0+0 (secure 0 0 0) 0x7fc8d8016030 con 0x56325e5e8e00 2024-04-25T02:08:30.556 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.548+0000 7fc8ce000640 1 -- 172.21.15.102:0/2763588514 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc8d801cdc0 con 0x56325e5e8e00 2024-04-25T02:08:30.556 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.548+0000 7fc8e9f9a740 10 client.27587 fetch_fsmap learned FSMap version 48 2024-04-25T02:08:30.556 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.548+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({fsmap=48}) -- 0x56325e602af0 con 0x56325e5e8e00 2024-04-25T02:08:30.556 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.548+0000 7fc8ce000640 1 -- 172.21.15.102:0/2763588514 <== mon.2 v2:172.21.15.110:3300/0 4 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fc8d800b040 con 0x56325e5e8e00 2024-04-25T02:08:30.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.549+0000 7fc8e9f9a740 10 client.27587 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T02:08:30.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.549+0000 7fc8e9f9a740 10 client.27587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:30.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.549+0000 7fc8e9f9a740 20 client.27587 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:30.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.549+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56325e603ea0 0x56325e606290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.549+0000 7fc8e9f9a740 4 client.27587 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:08:30.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.549+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56325e559ff0 con 0x56325e603ea0 2024-04-25T02:08:30.557 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.550+0000 7fc8df400640 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56325e603ea0 0x56325e606290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.550+0000 7fc8df400640 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56325e603ea0 0x56325e606290 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb2fcc1c314631c server_cookie=a88ba38a2d0f9df7 in_seq=0 out_seq=0 2024-04-25T02:08:30.558 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.551+0000 7fc8ce000640 10 client.27587 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.551+0000 7fc8ce000640 1 -- 172.21.15.102:0/2763588514 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56325e559ff0 con 0x56325e603ea0 2024-04-25T02:08:30.559 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.551+0000 7fc8ce000640 10 client.27587 handle_command_reply: tid=0 2024-04-25T02:08:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.552+0000 7fc8e9f9a740 1 client.27587 shutdown 2024-04-25T02:08:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.553+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 msgr2=0x56325e5f1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.553+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5f1010 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8005fd0 tx=0x7fc8d8004a40 comp rx=0 tx=0).stop 2024-04-25T02:08:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.555+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 shutdown_connections 2024-04-25T02:08:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.555+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56325e603ea0 0x56325e606290 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.555+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5f3fe0 0x56325e5f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.555+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.555+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2763588514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5e8e00 0x56325e5f1010 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.555+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 >> 172.21.15.102:0/2763588514 conn(0x56325e5e5310 msgr2=0x56325e5eb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.555+0000 7fc8ce000640 0 client.27587 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.556+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 shutdown_connections 2024-04-25T02:08:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.556+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2763588514 wait complete. 2024-04-25T02:08:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.556+0000 7fc8e9f9a740 20 client.27587 trim_cache size 0 max 16384 2024-04-25T02:08:30.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.587+0000 7fc8e9f9a740 1 Processor -- start 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.587+0000 7fc8e9f9a740 1 -- start start 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.587+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e8e00 0x56325e5eb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.587+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.587+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 0x56325e60bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.587+0000 7fc8e9f9a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01bec90 con 0x56325e5e8e00 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.587+0000 7fc8e9f9a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01bc500 con 0x56325e5f3fe0 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.587+0000 7fc8e9f9a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01ba370 con 0x56325e5e9720 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51212/0 (socket says 172.21.15.102:51212) 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8df400640 1 -- 172.21.15.102:0/2103266695 learned_addr learned my addr 172.21.15.102:0/2103266695 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:30.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e8e00 0x56325e5eb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8e5000640 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 0x56325e60bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8df400640 1 -- 172.21.15.102:0/2103266695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 msgr2=0x56325e60bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8df400640 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 0x56325e60bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8df400640 1 -- 172.21.15.102:0/2103266695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e8e00 msgr2=0x56325e5eb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8df400640 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e8e00 0x56325e5eb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8df400640 1 -- 172.21.15.102:0/2103266695 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56325e5e3890 con 0x56325e5e9720 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.588+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e8e00 0x56325e5eb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.589+0000 7fc8df400640 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f18d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0097870 tx=0x7fc8d0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e6680ef7d9d9da1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.589+0000 7fc8d4c00640 1 -- 172.21.15.102:0/2103266695 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d009f250 con 0x56325e5e9720 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.589+0000 7fc8d4c00640 1 -- 172.21.15.102:0/2103266695 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56325e559ff0 con 0x56325e5e9720 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.589+0000 7fc8d4c00640 1 -- 172.21.15.102:0/2103266695 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0097510 con 0x56325e5e9720 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.590+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2103266695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 msgr2=0x56325e5f18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:30.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.590+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f18d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0097870 tx=0x7fc8d0000bc0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.590+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2103266695 shutdown_connections 2024-04-25T02:08:30.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.590+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 0x56325e60bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.590+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e9720 0x56325e5f18d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.590+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2103266695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e8e00 0x56325e5eb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.590+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2103266695 >> 172.21.15.102:0/2103266695 conn(0x56325e5e5310 msgr2=0x56325e5e4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:30.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.591+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2103266695 shutdown_connections 2024-04-25T02:08:30.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.591+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2103266695 wait complete. 2024-04-25T02:08:30.600 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.591+0000 7fc8e9f9a740 1 -- start start 2024-04-25T02:08:30.600 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.591+0000 7fc8e9f9a740 1 Processor -- start 2024-04-25T02:08:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e8e00 0x56325e5ee410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 0x56325e5ee950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8e9f9a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 0x56325e669410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8e9f9a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01ba9c0 con 0x56325e5e9720 2024-04-25T02:08:30.602 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8e9f9a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01bbe30 con 0x56325e5f3fe0 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8e9f9a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e01bb090 con 0x56325e5e8e00 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8e9f9a740 10 client.0 _ll_register_callbacks cb 0x7fc8d8009280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e8e00 0x56325e5ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8dfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e8e00 0x56325e5ee410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51226/0 (socket says 172.21.15.102:51226) 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8dfe00640 1 -- 172.21.15.102:0/2537261520 learned_addr learned my addr 172.21.15.102:0/2537261520 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8df400640 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 0x56325e5ee950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.592+0000 7fc8e5000640 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 0x56325e669410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8dfe00640 1 -- 172.21.15.102:0/2537261520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 msgr2=0x56325e669410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 0x56325e669410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8dfe00640 1 -- 172.21.15.102:0/2537261520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 msgr2=0x56325e5ee950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 0x56325e5ee950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8dfe00640 1 -- 172.21.15.102:0/2537261520 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56325e669950 con 0x56325e5e8e00 2024-04-25T02:08:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8df400640 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 0x56325e5ee950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:08:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8dfe00640 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e8e00 0x56325e5ee410 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8010ce0 tx=0x7fc8d801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4519d8e4b376ed07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8ce000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:08:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8ce000640 1 -- 172.21.15.102:0/2537261520 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d802c040 con 0x56325e5e8e00 2024-04-25T02:08:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2537261520 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56325e559ff0 con 0x56325e5e8e00 2024-04-25T02:08:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8ce000640 1 -- 172.21.15.102:0/2537261520 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8513+0+0 (secure 0 0 0) 0x7fc8d8031770 con 0x56325e5e8e00 2024-04-25T02:08:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.593+0000 7fc8ce000640 1 -- 172.21.15.102:0/2537261520 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fc8d8031970 con 0x56325e5e8e00 2024-04-25T02:08:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.594+0000 7fc8e9f9a740 10 client.27541 fetch_fsmap learned FSMap version 48 2024-04-25T02:08:30.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.594+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2537261520 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56325e66ac00 con 0x56325e5e8e00 2024-04-25T02:08:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.595+0000 7fc8ce000640 1 -- 172.21.15.102:0/2537261520 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56325e559ff0 con 0x56325e5e8e00 2024-04-25T02:08:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.595+0000 7fc8ce000640 1 -- 172.21.15.102:0/2537261520 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fc8d800b040 con 0x56325e5e8e00 2024-04-25T02:08:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.595+0000 7fc8e9f9a740 10 client.27541 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T02:08:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.595+0000 7fc8e9f9a740 10 client.27541 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.595+0000 7fc8e9f9a740 20 client.27541 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.596+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56325e66b3d0 0x56325e66d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.596+0000 7fc8e9f9a740 4 client.27541 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:08:30.606 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.596+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2537261520 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56325e554340 con 0x56325e66b3d0 2024-04-25T02:08:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.596+0000 7fc8df400640 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56325e66b3d0 0x56325e66d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.596+0000 7fc8ce000640 10 client.27541 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:30.607 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.596+0000 7fc8df400640 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56325e66b3d0 0x56325e66d7f0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d006270dc7c6d9eb server_cookie=48cd36150dffb258 in_seq=0 out_seq=0 2024-04-25T02:08:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8ce000640 1 -- 172.21.15.102:0/2537261520 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56325e554340 con 0x56325e66b3d0 2024-04-25T02:08:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8ce000640 10 client.27541 handle_command_reply: tid=0 2024-04-25T02:08:30.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 client.27541 shutdown 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2537261520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e8e00 msgr2=0x56325e5ee410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e8e00 0x56325e5ee410 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8010ce0 tx=0x7fc8d801a000 comp rx=0 tx=0).stop 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2537261520 shutdown_connections 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56325e66b3d0 0x56325e66d7f0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56325e5f3fe0 0x56325e669410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56325e5e9720 0x56325e5ee950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8ce000640 0 client.27541 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 --2- 172.21.15.102:0/2537261520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56325e5e8e00 0x56325e5ee410 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2537261520 >> 172.21.15.102:0/2537261520 conn(0x56325e5e5310 msgr2=0x56325e5f2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2537261520 shutdown_connections 2024-04-25T02:08:30.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 1 -- 172.21.15.102:0/2537261520 wait complete. 2024-04-25T02:08:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.614+0000 7fc8e9f9a740 20 client.27541 trim_cache size 0 max 16384 2024-04-25T02:08:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:08:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:08:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:08:30.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:08:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 -- 172.21.15.102:0/786416866 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc8c40a2ea0 msgr2=0x7fc8c40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc8c40a2ea0 0x7fc8c40a5360 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x56325e5ea3a0 tx=0x7fc8d8009680 comp rx=0 tx=0).stop 2024-04-25T02:08:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 -- 172.21.15.102:0/786416866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e011af20 msgr2=0x7fc8e01a7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e011af20 0x7fc8e01a7390 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000b3c0 tx=0x7fc8d000b880 comp rx=0 tx=0).stop 2024-04-25T02:08:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 -- 172.21.15.102:0/786416866 shutdown_connections 2024-04-25T02:08:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc8c40a2ea0 0x7fc8c40a5360 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e01ab710 0x7fc8e01abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e011af20 0x7fc8e01a7390 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 --2- 172.21.15.102:0/786416866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e010ff30 0x7fc8e01a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.615+0000 7fc8e7000640 1 -- 172.21.15.102:0/786416866 >> 172.21.15.102:0/786416866 conn(0x7fc8e006c810 msgr2=0x7fc8e010dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.616+0000 7fc8e7000640 1 -- 172.21.15.102:0/786416866 shutdown_connections 2024-04-25T02:08:30.625 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:30.616+0000 7fc8e7000640 1 -- 172.21.15.102:0/786416866 wait complete. 2024-04-25T02:08:31.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:31 smithi193 ceph-mon[32981]: pgmap v412: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 13 MiB/s wr, 638 op/s 2024-04-25T02:08:31.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:31 smithi102 ceph-mon[27796]: pgmap v412: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 13 MiB/s wr, 638 op/s 2024-04-25T02:08:31.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:31 smithi110 ceph-mon[32848]: pgmap v412: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 13 MiB/s wr, 638 op/s 2024-04-25T02:08:33.075 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 760.0 s, 1350.4 tps, lat 73.284 ms stddev 89.910 2024-04-25T02:08:33.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:33 smithi193 ceph-mon[32981]: pgmap v413: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 10 MiB/s wr, 542 op/s 2024-04-25T02:08:33.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:33 smithi102 ceph-mon[27796]: pgmap v413: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 10 MiB/s wr, 542 op/s 2024-04-25T02:08:33.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:33 smithi110 ceph-mon[32848]: pgmap v413: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 10 MiB/s wr, 542 op/s 2024-04-25T02:08:35.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:35 smithi102 ceph-mon[27796]: pgmap v414: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 13 MiB/s wr, 760 op/s 2024-04-25T02:08:35.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:35 smithi110 ceph-mon[32848]: pgmap v414: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 13 MiB/s wr, 760 op/s 2024-04-25T02:08:36.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:35 smithi193 ceph-mon[32981]: pgmap v414: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 13 MiB/s wr, 760 op/s 2024-04-25T02:08:37.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:37 smithi102 ceph-mon[27796]: pgmap v415: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 11 MiB/s wr, 652 op/s 2024-04-25T02:08:37.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:37 smithi110 ceph-mon[32848]: pgmap v415: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 11 MiB/s wr, 652 op/s 2024-04-25T02:08:38.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:37 smithi193 ceph-mon[32981]: pgmap v415: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 11 MiB/s wr, 652 op/s 2024-04-25T02:08:38.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 765.0 s, 1420.8 tps, lat 70.084 ms stddev 87.537 2024-04-25T02:08:39.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:39 smithi110 ceph-mon[32848]: pgmap v416: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 741 op/s 2024-04-25T02:08:39.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:39 smithi102 ceph-mon[27796]: pgmap v416: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 741 op/s 2024-04-25T02:08:40.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:39 smithi193 ceph-mon[32981]: pgmap v416: 97 pgs: 97 active+clean; 5.3 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 741 op/s 2024-04-25T02:08:41.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:41 smithi102 ceph-mon[27796]: pgmap v417: 97 pgs: 97 active+clean; 5.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 11 MiB/s wr, 662 op/s 2024-04-25T02:08:41.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:41 smithi110 ceph-mon[32848]: pgmap v417: 97 pgs: 97 active+clean; 5.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 11 MiB/s wr, 662 op/s 2024-04-25T02:08:42.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:41 smithi193 ceph-mon[32981]: pgmap v417: 97 pgs: 97 active+clean; 5.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 11 MiB/s wr, 662 op/s 2024-04-25T02:08:43.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 770.0 s, 1506.8 tps, lat 65.361 ms stddev 81.538 2024-04-25T02:08:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.833+0000 7fa1dd400640 1 -- 172.21.15.102:0/916193027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d009cbc0 msgr2=0x7fa1d009cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:43.840 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.833+0000 7fa1dd400640 1 --2- 172.21.15.102:0/916193027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d009cbc0 0x7fa1d009cfa0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009900 tx=0x7fa1cc02a400 comp rx=0 tx=0).stop 2024-04-25T02:08:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.836+0000 7fa1dd400640 1 -- 172.21.15.102:0/916193027 shutdown_connections 2024-04-25T02:08:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.836+0000 7fa1dd400640 1 --2- 172.21.15.102:0/916193027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d00a16d0 0x7fa1d00a9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.836+0000 7fa1dd400640 1 --2- 172.21.15.102:0/916193027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d009d570 0x7fa1d00a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.836+0000 7fa1dd400640 1 --2- 172.21.15.102:0/916193027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d009cbc0 0x7fa1d009cfa0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.836+0000 7fa1dd400640 1 -- 172.21.15.102:0/916193027 >> 172.21.15.102:0/916193027 conn(0x7fa1d0009290 msgr2=0x7fa1d00096a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.836+0000 7fa1dd400640 1 -- 172.21.15.102:0/916193027 shutdown_connections 2024-04-25T02:08:43.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.836+0000 7fa1dd400640 1 -- 172.21.15.102:0/916193027 wait complete. 2024-04-25T02:08:43.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.838+0000 7fa1dd400640 1 Processor -- start 2024-04-25T02:08:43.845 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.838+0000 7fa1dd400640 1 -- start start 2024-04-25T02:08:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.839+0000 7fa1dd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d009cbc0 0x7fa1d00150f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.840+0000 7fa1dd400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d00a16d0 0x7fa1d0015630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.840+0000 7fa1dd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d000e280 0x7fa1d000e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.840+0000 7fa1dd400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d00ac710 con 0x7fa1d009cbc0 2024-04-25T02:08:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.840+0000 7fa1dd400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d00ac590 con 0x7fa1d00a16d0 2024-04-25T02:08:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.840+0000 7fa1dd400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d00ac890 con 0x7fa1d000e280 2024-04-25T02:08:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.841+0000 7fa1d7400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d00a16d0 0x7fa1d0015630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.841+0000 7fa1dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d000e280 0x7fa1d000e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.841+0000 7fa1dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d000e280 0x7fa1d000e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:54080/0 (socket says 172.21.15.102:54080) 2024-04-25T02:08:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.841+0000 7fa1dca00640 1 -- 172.21.15.102:0/3347346931 learned_addr learned my addr 172.21.15.102:0/3347346931 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:43.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.841+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d009cbc0 0x7fa1d00150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.843+0000 7fa1dca00640 1 -- 172.21.15.102:0/3347346931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d00a16d0 msgr2=0x7fa1d0015630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.843+0000 7fa1dca00640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d00a16d0 0x7fa1d0015630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.843+0000 7fa1dca00640 1 -- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d009cbc0 msgr2=0x7fa1d00150f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:43.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.843+0000 7fa1dca00640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d009cbc0 0x7fa1d00150f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.843+0000 7fa1dca00640 1 -- 172.21.15.102:0/3347346931 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d0012c10 con 0x7fa1d000e280 2024-04-25T02:08:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.845+0000 7fa1dca00640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d000e280 0x7fa1d000e730 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8066f20 tx=0x7fa1d807bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.845+0000 7fa1d4c00640 1 -- 172.21.15.102:0/3347346931 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8082070 con 0x7fa1d000e280 2024-04-25T02:08:43.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.845+0000 7fa1d4c00640 1 -- 172.21.15.102:0/3347346931 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1d804fdc0 con 0x7fa1d000e280 2024-04-25T02:08:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.845+0000 7fa1d4c00640 1 -- 172.21.15.102:0/3347346931 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d807d3c0 con 0x7fa1d000e280 2024-04-25T02:08:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.845+0000 7fa1dd400640 1 -- 172.21.15.102:0/3347346931 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d0012f00 con 0x7fa1d000e280 2024-04-25T02:08:43.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.846+0000 7fa1dd400640 1 -- 172.21.15.102:0/3347346931 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d00133d0 con 0x7fa1d000e280 2024-04-25T02:08:43.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.858+0000 7fa1d4c00640 1 -- 172.21.15.102:0/3347346931 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa1d8085070 con 0x7fa1d000e280 2024-04-25T02:08:43.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.862+0000 7fa1e1854740 1 Processor -- start 2024-04-25T02:08:43.869 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.863+0000 7fa1d4c00640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa1c00a2e80 0x7fa1c00a5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.863+0000 7fa1e1854740 1 -- start start 2024-04-25T02:08:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.863+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e3fb0 0x56313b2e4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.863+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e48d0 0x56313b2eec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.863+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2f1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.863+0000 7fa1e1854740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56313b247ab0 con 0x56313b2e3fb0 2024-04-25T02:08:43.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.863+0000 7fa1e1854740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56313b1f0640 con 0x56313b2ef190 2024-04-25T02:08:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.865+0000 7fa1dca00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2f1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.865+0000 7fa1dca00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2f1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53990/0 (socket says 172.21.15.102:53990) 2024-04-25T02:08:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.865+0000 7fa1dca00640 1 -- 172.21.15.102:0/3109895112 learned_addr learned my addr 172.21.15.102:0/3109895112 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.865+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/3109895112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e3fb0 0x56313b2e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.863+0000 7fa1d4c00640 1 -- 172.21.15.102:0/3347346931 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8513+0+0 (secure 0 0 0) 0x7fa1d813ae00 con 0x7fa1d000e280 2024-04-25T02:08:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.865+0000 7fa1e1854740 1 -- 172.21.15.102:0/3109895112 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56313b243a70 con 0x56313b2e48d0 2024-04-25T02:08:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.866+0000 7fa1d7400640 1 --2- 172.21.15.102:0/3109895112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e48d0 0x56313b2eec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.866+0000 7fa1d7400640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa1c00a2e80 0x7fa1c00a5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.867+0000 7fa1dca00640 1 -- 172.21.15.102:0/3109895112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e48d0 msgr2=0x56313b2eec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:43.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.867+0000 7fa1dca00640 1 --2- 172.21.15.102:0/3109895112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e48d0 0x56313b2eec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.867+0000 7fa1dca00640 1 -- 172.21.15.102:0/3109895112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e3fb0 msgr2=0x56313b2e4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:43.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.867+0000 7fa1dca00640 1 --2- 172.21.15.102:0/3109895112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e3fb0 0x56313b2e4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.875 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.867+0000 7fa1dca00640 1 -- 172.21.15.102:0/3109895112 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56313af5c210 con 0x56313b2ef190 2024-04-25T02:08:43.875 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.868+0000 7fa1dca00640 1 --2- 172.21.15.102:0/3109895112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2f1580 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8000b60 tx=0x7fa1d807e8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14d041011576a44a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:43.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.869+0000 7fa1c9200640 1 -- 172.21.15.102:0/3109895112 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d808c050 con 0x56313b2ef190 2024-04-25T02:08:43.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.869+0000 7fa1c9200640 1 -- 172.21.15.102:0/3109895112 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1d8111020 con 0x56313b2ef190 2024-04-25T02:08:43.876 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.869+0000 7fa1c9200640 1 -- 172.21.15.102:0/3109895112 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d81078b0 con 0x56313b2ef190 2024-04-25T02:08:43.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1d7400640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa1c00a2e80 0x7fa1c00a5340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x56313b2e5550 tx=0x7fa1c4005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:43.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1e1854740 1 -- 172.21.15.102:0/3109895112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 msgr2=0x56313b2f1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:43.878 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3109895112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2f1580 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8000b60 tx=0x7fa1d807e8a0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1e1854740 1 -- 172.21.15.102:0/3109895112 shutdown_connections 2024-04-25T02:08:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3109895112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2f1580 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3109895112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e48d0 0x56313b2eec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3109895112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e3fb0 0x56313b2e4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1e1854740 1 -- 172.21.15.102:0/3109895112 >> 172.21.15.102:0/3109895112 conn(0x56313b2e04c0 msgr2=0x56313b2e08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1e1854740 1 -- 172.21.15.102:0/3109895112 shutdown_connections 2024-04-25T02:08:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.872+0000 7fa1e1854740 1 -- 172.21.15.102:0/3109895112 wait complete. 2024-04-25T02:08:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.873+0000 7fa1e1854740 1 -- start start 2024-04-25T02:08:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.873+0000 7fa1e1854740 1 Processor -- start 2024-04-25T02:08:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.876+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2ec0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.877+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.877+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b2ecb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.878+0000 7fa1e1854740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56313b302d10 con 0x56313b2e48d0 2024-04-25T02:08:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.878+0000 7fa1e1854740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x56313b203990 con 0x56313b2e3fb0 2024-04-25T02:08:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.878+0000 7fa1e1854740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56313b302e90 con 0x56313b2ef190 2024-04-25T02:08:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.878+0000 7fa1e1854740 10 client.0 _ll_register_callbacks cb 0x56313b205950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.878+0000 7fa1d7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2ec0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.878+0000 7fa1dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b2ecb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.878+0000 7fa1dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b2ecb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:54102/0 (socket says 172.21.15.102:54102) 2024-04-25T02:08:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.878+0000 7fa1dca00640 1 -- 172.21.15.102:0/3782661689 learned_addr learned my addr 172.21.15.102:0/3782661689 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.880+0000 7fa1dca00640 1 -- 172.21.15.102:0/3782661689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 msgr2=0x56313b2ec0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.880+0000 7fa1d7400640 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.880+0000 7fa1dca00640 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2ec0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.880+0000 7fa1dca00640 1 -- 172.21.15.102:0/3782661689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 msgr2=0x56313b2ec600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.880+0000 7fa1dca00640 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.880+0000 7fa1dca00640 1 -- 172.21.15.102:0/3782661689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56313b028620 con 0x56313b2ef190 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.880+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2ec0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.881+0000 7fa1dca00640 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b2ecb40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8110af0 tx=0x7fa1d8110f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae6236d92a77a5ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.881+0000 7fa1d7400640 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:08:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.881+0000 7fa1bd600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:08:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.882+0000 7fa1bd600640 1 -- 172.21.15.102:0/3782661689 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8104ad0 con 0x56313b2ef190 2024-04-25T02:08:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.882+0000 7fa1bd600640 1 -- 172.21.15.102:0/3782661689 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8513+0+0 (secure 0 0 0) 0x7fa1d81041b0 con 0x56313b2ef190 2024-04-25T02:08:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.883+0000 7fa1e1854740 1 -- 172.21.15.102:0/3782661689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1d8111020 con 0x56313b2ef190 2024-04-25T02:08:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.884+0000 7fa1bd600640 1 -- 172.21.15.102:0/3782661689 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fa1d808c030 con 0x56313b2ef190 2024-04-25T02:08:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.884+0000 7fa1e1854740 10 client.27550 fetch_fsmap learned FSMap version 48 2024-04-25T02:08:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.884+0000 7fa1e1854740 1 -- 172.21.15.102:0/3782661689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56313b302d10 con 0x56313b2ef190 2024-04-25T02:08:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.885+0000 7fa1bd600640 1 -- 172.21.15.102:0/3782661689 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1d8111020 con 0x56313b2ef190 2024-04-25T02:08:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.885+0000 7fa1bd600640 1 -- 172.21.15.102:0/3782661689 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fa1d810d8f0 con 0x56313b2ef190 2024-04-25T02:08:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.886+0000 7fa1e1854740 10 client.27550 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T02:08:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.886+0000 7fa1e1854740 10 client.27550 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.886+0000 7fa1e1854740 20 client.27550 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.886+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56313b30fd40 0x56313b312130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.886+0000 7fa1e1854740 4 client.27550 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:08:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.886+0000 7fa1e1854740 1 -- 172.21.15.102:0/3782661689 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56313b254ff0 con 0x56313b30fd40 2024-04-25T02:08:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.887+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56313b30fd40 0x56313b312130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.888+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56313b30fd40 0x56313b312130 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d592bbd9d293eae server_cookie=871034545578725d in_seq=0 out_seq=0 2024-04-25T02:08:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.888+0000 7fa1bd600640 10 client.27550 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:43.896 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.889+0000 7fa1bd600640 1 -- 172.21.15.102:0/3782661689 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56313b254ff0 con 0x56313b30fd40 2024-04-25T02:08:43.896 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.889+0000 7fa1bd600640 10 client.27550 handle_command_reply: tid=0 2024-04-25T02:08:43.896 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.889+0000 7fa1e1854740 1 client.27550 shutdown 2024-04-25T02:08:43.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.890+0000 7fa1e1854740 1 -- 172.21.15.102:0/3782661689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 msgr2=0x56313b2ecb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:43.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.890+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b2ecb40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8110af0 tx=0x7fa1d8110f30 comp rx=0 tx=0).stop 2024-04-25T02:08:43.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.890+0000 7fa1e1854740 1 -- 172.21.15.102:0/3782661689 shutdown_connections 2024-04-25T02:08:43.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.890+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56313b30fd40 0x56313b312130 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.891+0000 7fa1bd600640 0 client.27550 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:43.897 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.891+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b2ecb40 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.891+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.891+0000 7fa1e1854740 1 --2- 172.21.15.102:0/3782661689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2ec0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.891+0000 7fa1e1854740 1 -- 172.21.15.102:0/3782661689 >> 172.21.15.102:0/3782661689 conn(0x56313b2e04c0 msgr2=0x56313b2e6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.891+0000 7fa1e1854740 1 -- 172.21.15.102:0/3782661689 shutdown_connections 2024-04-25T02:08:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.891+0000 7fa1e1854740 1 -- 172.21.15.102:0/3782661689 wait complete. 2024-04-25T02:08:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.892+0000 7fa1e1854740 20 client.27550 trim_cache size 0 max 16384 2024-04-25T02:08:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.922+0000 7fa1e1854740 1 Processor -- start 2024-04-25T02:08:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.922+0000 7fa1e1854740 1 -- start start 2024-04-25T02:08:43.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:43 smithi102 ceph-mon[27796]: pgmap v418: 97 pgs: 97 active+clean; 5.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.4 MiB/s wr, 576 op/s 2024-04-25T02:08:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.923+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2dfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.923+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.923+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b317990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.924+0000 7fa1e1854740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b80110e0 con 0x56313b2e48d0 2024-04-25T02:08:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.924+0000 7fa1e1854740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b800e950 con 0x56313b2e3fb0 2024-04-25T02:08:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.924+0000 7fa1e1854740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b800c7c0 con 0x56313b2ef190 2024-04-25T02:08:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.924+0000 7fa1d7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2dfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.924+0000 7fa1dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b317990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.924+0000 7fa1d7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2dfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:54004/0 (socket says 172.21.15.102:54004) 2024-04-25T02:08:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.924+0000 7fa1d7e00640 1 -- 172.21.15.102:0/2157872334 learned_addr learned my addr 172.21.15.102:0/2157872334 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.925+0000 7fa1d7400640 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.925+0000 7fa1d7e00640 1 -- 172.21.15.102:0/2157872334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 msgr2=0x56313b317990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.925+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b317990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.925+0000 7fa1d7e00640 1 -- 172.21.15.102:0/2157872334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 msgr2=0x56313b2ec960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.925+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.926+0000 7fa1d7e00640 1 -- 172.21.15.102:0/2157872334 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56313b312760 con 0x56313b2e3fb0 2024-04-25T02:08:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.926+0000 7fa1d7400640 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:08:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.926+0000 7fa1dca00640 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b317990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:08:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.926+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2dfdd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc005a90 tx=0x7fa1cc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffce79b599fb2924 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.927+0000 7fa1bfe00640 1 -- 172.21.15.102:0/2157872334 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc04c170 con 0x56313b2e3fb0 2024-04-25T02:08:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.927+0000 7fa1bfe00640 1 -- 172.21.15.102:0/2157872334 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1cc04c7f0 con 0x56313b2e3fb0 2024-04-25T02:08:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.927+0000 7fa1bfe00640 1 -- 172.21.15.102:0/2157872334 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc04c170 con 0x56313b2e3fb0 2024-04-25T02:08:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.929+0000 7fa1e1854740 1 -- 172.21.15.102:0/2157872334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 msgr2=0x56313b2dfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.929+0000 7fa1e1854740 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2dfdd0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc005a90 tx=0x7fa1cc000bc0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.930+0000 7fa1e1854740 1 -- 172.21.15.102:0/2157872334 shutdown_connections 2024-04-25T02:08:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.930+0000 7fa1e1854740 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2ef190 0x56313b317990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.930+0000 7fa1e1854740 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2ec960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.930+0000 7fa1e1854740 1 --2- 172.21.15.102:0/2157872334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2e3fb0 0x56313b2dfdd0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.930+0000 7fa1e1854740 1 -- 172.21.15.102:0/2157872334 >> 172.21.15.102:0/2157872334 conn(0x56313b2e04c0 msgr2=0x56313b2e61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.930+0000 7fa1e1854740 1 -- 172.21.15.102:0/2157872334 shutdown_connections 2024-04-25T02:08:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.931+0000 7fa1e1854740 1 -- 172.21.15.102:0/2157872334 wait complete. 2024-04-25T02:08:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.931+0000 7fa1e1854740 1 -- start start 2024-04-25T02:08:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.932+0000 7fa1e1854740 1 Processor -- start 2024-04-25T02:08:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.933+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e3fb0 0x56313b3083b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.933+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.933+0000 7fa1e1854740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2e8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.933+0000 7fa1e1854740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b800ce10 con 0x56313b2e48d0 2024-04-25T02:08:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.933+0000 7fa1e1854740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b800e280 con 0x56313b2ef190 2024-04-25T02:08:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.933+0000 7fa1e1854740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b800d4e0 con 0x56313b2e3fb0 2024-04-25T02:08:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.933+0000 7fa1e1854740 10 client.0 _ll_register_callbacks cb 0x7fa1cc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:43.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.934+0000 7fa1d7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e3fb0 0x56313b3083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.934+0000 7fa1d7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e3fb0 0x56313b3083b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:54108/0 (socket says 172.21.15.102:54108) 2024-04-25T02:08:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.934+0000 7fa1d7e00640 1 -- 172.21.15.102:0/4125321215 learned_addr learned my addr 172.21.15.102:0/4125321215 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:43.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.934+0000 7fa1dca00640 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2e8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.942 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.935+0000 7fa1d7e00640 1 -- 172.21.15.102:0/4125321215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 msgr2=0x56313b2e8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:43.942 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.935+0000 7fa1d7400640 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.942 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.935+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2e8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.942 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.935+0000 7fa1d7e00640 1 -- 172.21.15.102:0/4125321215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 msgr2=0x56313b2e89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:43.942 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.935+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2e89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:43.943 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.937+0000 7fa1d7e00640 1 -- 172.21.15.102:0/4125321215 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56313b2e9670 con 0x56313b2e3fb0 2024-04-25T02:08:43.944 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.937+0000 7fa1dca00640 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:08:43.944 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.937+0000 7fa1d7400640 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:08:43.944 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.937+0000 7fa1d7e00640 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e3fb0 0x56313b3083b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc004890 tx=0x7fa1cc004ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e08654ab242bb2df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:43.944 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.937+0000 7fa1bd600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:08:43.945 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.938+0000 7fa1bd600640 1 -- 172.21.15.102:0/4125321215 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc003b50 con 0x56313b2e3fb0 2024-04-25T02:08:43.945 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.938+0000 7fa1bd600640 1 -- 172.21.15.102:0/4125321215 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8513+0+0 (secure 0 0 0) 0x7fa1cc003cf0 con 0x56313b2e3fb0 2024-04-25T02:08:43.945 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.938+0000 7fa1e1854740 1 -- 172.21.15.102:0/4125321215 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1cc04c7f0 con 0x56313b2e3fb0 2024-04-25T02:08:43.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.939+0000 7fa1bd600640 1 -- 172.21.15.102:0/4125321215 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fa1cc003ef0 con 0x56313b2e3fb0 2024-04-25T02:08:43.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.940+0000 7fa1e1854740 10 client.27556 fetch_fsmap learned FSMap version 48 2024-04-25T02:08:43.946 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.940+0000 7fa1e1854740 1 -- 172.21.15.102:0/4125321215 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56313b376140 con 0x56313b2e3fb0 2024-04-25T02:08:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.940+0000 7fa1bd600640 1 -- 172.21.15.102:0/4125321215 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1cc04c7f0 con 0x56313b2e3fb0 2024-04-25T02:08:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.940+0000 7fa1bd600640 1 -- 172.21.15.102:0/4125321215 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 48) ==== 8354+0+0 (secure 0 0 0) 0x7fa1cc05fc00 con 0x56313b2e3fb0 2024-04-25T02:08:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.942+0000 7fa1e1854740 10 client.27556 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T02:08:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.942+0000 7fa1e1854740 10 client.27556 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.942+0000 7fa1e1854740 20 client.27556 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.942+0000 7fa1e1854740 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56313b3768f0 0x56313b378d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.942+0000 7fa1e1854740 4 client.27556 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:08:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.942+0000 7fa1e1854740 1 -- 172.21.15.102:0/4125321215 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56313b24f340 con 0x56313b3768f0 2024-04-25T02:08:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.943+0000 7fa1d7400640 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56313b3768f0 0x56313b378d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:43.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.946+0000 7fa1d7400640 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56313b3768f0 0x56313b378d10 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d59b17d61c0e689 server_cookie=80819ef779d2a7c1 in_seq=0 out_seq=0 2024-04-25T02:08:43.954 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:43.947+0000 7fa1bd600640 10 client.27556 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:43.993 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:43 smithi110 ceph-mon[32848]: pgmap v418: 97 pgs: 97 active+clean; 5.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.4 MiB/s wr, 576 op/s 2024-04-25T02:08:44.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:43 smithi193 ceph-mon[32981]: pgmap v418: 97 pgs: 97 active+clean; 5.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.4 MiB/s wr, 576 op/s 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1bd600640 1 -- 172.21.15.102:0/4125321215 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56313b24f340 con 0x56313b3768f0 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1bd600640 10 client.27556 handle_command_reply: tid=0 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1e1854740 1 client.27556 shutdown 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1e1854740 1 -- 172.21.15.102:0/4125321215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e3fb0 msgr2=0x56313b3083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1e1854740 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e3fb0 0x56313b3083b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc004890 tx=0x7fa1cc004ac0 comp rx=0 tx=0).stop 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1e1854740 1 -- 172.21.15.102:0/4125321215 shutdown_connections 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1e1854740 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56313b3768f0 0x56313b378d10 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1e1854740 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56313b2ef190 0x56313b2e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1bd600640 0 client.27556 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.056+0000 7fa1e1854740 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56313b2e48d0 0x56313b2e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.057+0000 7fa1e1854740 1 --2- 172.21.15.102:0/4125321215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56313b2e3fb0 0x56313b3083b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.057+0000 7fa1e1854740 1 -- 172.21.15.102:0/4125321215 >> 172.21.15.102:0/4125321215 conn(0x56313b2e04c0 msgr2=0x56313b2ed8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.057+0000 7fa1e1854740 1 -- 172.21.15.102:0/4125321215 shutdown_connections 2024-04-25T02:08:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.057+0000 7fa1e1854740 1 -- 172.21.15.102:0/4125321215 wait complete. 2024-04-25T02:08:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.057+0000 7fa1e1854740 20 client.27556 trim_cache size 0 max 16384 2024-04-25T02:08:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:08:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:08:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:08:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:08:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 -- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa1c00a2e80 msgr2=0x7fa1c00a5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa1c00a2e80 0x7fa1c00a5340 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x56313b2e5550 tx=0x7fa1c4005f90 comp rx=0 tx=0).stop 2024-04-25T02:08:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 -- 172.21.15.102:0/3347346931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d000e280 msgr2=0x7fa1d000e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d000e280 0x7fa1d000e730 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8066f20 tx=0x7fa1d807bd70 comp rx=0 tx=0).stop 2024-04-25T02:08:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 -- 172.21.15.102:0/3347346931 shutdown_connections 2024-04-25T02:08:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fa1c00a2e80 0x7fa1c00a5340 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d000e280 0x7fa1d000e730 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1d00a16d0 0x7fa1d0015630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 --2- 172.21.15.102:0/3347346931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d009cbc0 0x7fa1d00150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 -- 172.21.15.102:0/3347346931 >> 172.21.15.102:0/3347346931 conn(0x7fa1d0009290 msgr2=0x7fa1d009bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 -- 172.21.15.102:0/3347346931 shutdown_connections 2024-04-25T02:08:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:44.058+0000 7fa1dd400640 1 -- 172.21.15.102:0/3347346931 wait complete. 2024-04-25T02:08:45.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:45 smithi102 ceph-mon[27796]: pgmap v419: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 843 op/s 2024-04-25T02:08:45.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:45 smithi110 ceph-mon[32848]: pgmap v419: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 843 op/s 2024-04-25T02:08:46.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:45 smithi193 ceph-mon[32981]: pgmap v419: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 13 MiB/s wr, 843 op/s 2024-04-25T02:08:47.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:47 smithi102 ceph-mon[27796]: pgmap v420: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 16 MiB/s wr, 991 op/s 2024-04-25T02:08:47.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:47 smithi110 ceph-mon[32848]: pgmap v420: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 16 MiB/s wr, 991 op/s 2024-04-25T02:08:48.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:47 smithi193 ceph-mon[32981]: pgmap v420: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 16 MiB/s wr, 991 op/s 2024-04-25T02:08:48.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 775.0 s, 1229.8 tps, lat 82.966 ms stddev 159.164 2024-04-25T02:08:48.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:48 smithi102 ceph-mon[27796]: pgmap v421: 97 pgs: 97 active+clean; 5.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 27 MiB/s wr, 1.75k op/s 2024-04-25T02:08:48.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:48 smithi110 ceph-mon[32848]: pgmap v421: 97 pgs: 97 active+clean; 5.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 27 MiB/s wr, 1.75k op/s 2024-04-25T02:08:49.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:48 smithi193 ceph-mon[32981]: pgmap v421: 97 pgs: 97 active+clean; 5.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 27 MiB/s wr, 1.75k op/s 2024-04-25T02:08:51.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:50 smithi193 ceph-mon[32981]: pgmap v422: 97 pgs: 97 active+clean; 5.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 27 MiB/s wr, 1.69k op/s 2024-04-25T02:08:51.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:50 smithi102 ceph-mon[27796]: pgmap v422: 97 pgs: 97 active+clean; 5.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 27 MiB/s wr, 1.69k op/s 2024-04-25T02:08:51.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:50 smithi110 ceph-mon[32848]: pgmap v422: 97 pgs: 97 active+clean; 5.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 27 MiB/s wr, 1.69k op/s 2024-04-25T02:08:53.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 780.0 s, 1036.6 tps, lat 97.295 ms stddev 136.816 2024-04-25T02:08:53.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:53 smithi102 ceph-mon[27796]: pgmap v423: 97 pgs: 97 active+clean; 5.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 26 MiB/s wr, 1.73k op/s 2024-04-25T02:08:53.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:53 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:08:53.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:53 smithi110 ceph-mon[32848]: pgmap v423: 97 pgs: 97 active+clean; 5.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 26 MiB/s wr, 1.73k op/s 2024-04-25T02:08:53.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:53 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:08:53.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:53 smithi193 ceph-mon[32981]: pgmap v423: 97 pgs: 97 active+clean; 5.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 26 MiB/s wr, 1.73k op/s 2024-04-25T02:08:53.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:53 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:08:54.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:54 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:08:54.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:54 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:08:54.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:54 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:08:55.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:55 smithi102 ceph-mon[27796]: pgmap v424: 97 pgs: 97 active+clean; 5.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 36 MiB/s wr, 2.40k op/s 2024-04-25T02:08:55.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:55 smithi110 ceph-mon[32848]: pgmap v424: 97 pgs: 97 active+clean; 5.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 36 MiB/s wr, 2.40k op/s 2024-04-25T02:08:55.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:55 smithi193 ceph-mon[32981]: pgmap v424: 97 pgs: 97 active+clean; 5.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 36 MiB/s wr, 2.40k op/s 2024-04-25T02:08:56.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:56 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:08:56.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:56 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:08:56.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:56 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:08:56.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:56 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:08:56.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:56 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:08:56.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:56 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:08:56.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:56 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:08:56.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:56 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:08:56.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:56 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:08:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.223+0000 7fb238200640 1 -- 172.21.15.102:0/953478660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2301028f0 msgr2=0x7fb230102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.223+0000 7fb238200640 1 --2- 172.21.15.102:0/953478660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2301028f0 0x7fb230102cd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb224009900 tx=0x7fb22402a480 comp rx=0 tx=0).stop 2024-04-25T02:08:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.224+0000 7fb238200640 1 -- 172.21.15.102:0/953478660 shutdown_connections 2024-04-25T02:08:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.224+0000 7fb238200640 1 --2- 172.21.15.102:0/953478660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb23010bd10 0x7fb23010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.224+0000 7fb238200640 1 --2- 172.21.15.102:0/953478660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb230103210 0x7fb23010b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.224+0000 7fb238200640 1 --2- 172.21.15.102:0/953478660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2301028f0 0x7fb230102cd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.224+0000 7fb238200640 1 -- 172.21.15.102:0/953478660 >> 172.21.15.102:0/953478660 conn(0x7fb2300fc110 msgr2=0x7fb2300fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.224+0000 7fb238200640 1 -- 172.21.15.102:0/953478660 shutdown_connections 2024-04-25T02:08:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.224+0000 7fb238200640 1 -- 172.21.15.102:0/953478660 wait complete. 2024-04-25T02:08:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.225+0000 7fb238200640 1 Processor -- start 2024-04-25T02:08:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.225+0000 7fb238200640 1 -- start start 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb238200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2301028f0 0x7fb23010a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb238200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb230103210 0x7fb23010aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb238200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb23010bd10 0x7fb2301a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb238200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb230111ba0 con 0x7fb2301028f0 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb238200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb230111a20 con 0x7fb230103210 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb238200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb230111d20 con 0x7fb23010bd10 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb234e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb230103210 0x7fb23010aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb235800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2301028f0 0x7fb23010a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb234e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb230103210 0x7fb23010aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:33390/0 (socket says 172.21.15.102:33390) 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb234e00640 1 -- 172.21.15.102:0/1206363134 learned_addr learned my addr 172.21.15.102:0/1206363134 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.226+0000 7fb236200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb23010bd10 0x7fb2301a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb234e00640 1 -- 172.21.15.102:0/1206363134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb23010bd10 msgr2=0x7fb2301a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb234e00640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb23010bd10 0x7fb2301a2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb234e00640 1 -- 172.21.15.102:0/1206363134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2301028f0 msgr2=0x7fb23010a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb234e00640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2301028f0 0x7fb23010a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb234e00640 1 -- 172.21.15.102:0/1206363134 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2301a35e0 con 0x7fb230103210 2024-04-25T02:08:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb236200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb23010bd10 0x7fb2301a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:08:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb234e00640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb230103210 0x7fb23010aee0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00b3f0 tx=0x7fb21c00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb22e000640 1 -- 172.21.15.102:0/1206363134 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c015020 con 0x7fb230103210 2024-04-25T02:08:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb22e000640 1 -- 172.21.15.102:0/1206363134 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb21c011d10 con 0x7fb230103210 2024-04-25T02:08:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb238200640 1 -- 172.21.15.102:0/1206363134 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2301a38d0 con 0x7fb230103210 2024-04-25T02:08:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb22e000640 1 -- 172.21.15.102:0/1206363134 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c00f2e0 con 0x7fb230103210 2024-04-25T02:08:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.227+0000 7fb238200640 1 -- 172.21.15.102:0/1206363134 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2301ab160 con 0x7fb230103210 2024-04-25T02:08:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.230+0000 7fb23b1c0740 1 Processor -- start 2024-04-25T02:08:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.231+0000 7fb23b1c0740 1 -- start start 2024-04-25T02:08:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.232+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f7fe00 0x560189f801e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.232+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f80720 0x560189f8aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.232+0000 7fb22e000640 1 -- 172.21.15.102:0/1206363134 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb21c018020 con 0x7fb230103210 2024-04-25T02:08:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.233+0000 7fb236200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f7fe00 0x560189f801e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.233+0000 7fb235800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f80720 0x560189f8aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.233+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f8afe0 0x560189f8d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.233+0000 7fb235800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f80720 0x560189f8aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40354/0 (socket says 172.21.15.102:40354) 2024-04-25T02:08:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.233+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560189ee3ab0 con 0x560189f80720 2024-04-25T02:08:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.233+0000 7fb235800640 1 -- 172.21.15.102:0/2745640666 learned_addr learned my addr 172.21.15.102:0/2745640666 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.233+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2745640666 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560189e8c640 con 0x560189f8afe0 2024-04-25T02:08:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.233+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2745640666 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560189edfa70 con 0x560189f7fe00 2024-04-25T02:08:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.234+0000 7fb234e00640 1 --2- 172.21.15.102:0/2745640666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f8afe0 0x560189f8d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.234+0000 7fb235800640 1 -- 172.21.15.102:0/2745640666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f7fe00 msgr2=0x560189f801e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.234+0000 7fb235800640 1 --2- 172.21.15.102:0/2745640666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f7fe00 0x560189f801e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.234+0000 7fb235800640 1 -- 172.21.15.102:0/2745640666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f8afe0 msgr2=0x560189f8d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.234+0000 7fb235800640 1 --2- 172.21.15.102:0/2745640666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f8afe0 0x560189f8d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.234+0000 7fb235800640 1 -- 172.21.15.102:0/2745640666 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560189bf8210 con 0x560189f80720 2024-04-25T02:08:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.235+0000 7fb236200640 1 --2- 172.21.15.102:0/2745640666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f7fe00 0x560189f801e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:08:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.235+0000 7fb235800640 1 --2- 172.21.15.102:0/2745640666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f80720 0x560189f8aaa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb224009230 tx=0x7fb2240040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23c934b84e600ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.236+0000 7fb22e000640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2180a2e20 0x7fb2180a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.236+0000 7fb222a00640 1 -- 172.21.15.102:0/2745640666 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224035040 con 0x560189f80720 2024-04-25T02:08:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.236+0000 7fb22e000640 1 -- 172.21.15.102:0/1206363134 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8513+0+0 (secure 0 0 0) 0x7fb21c0cebd0 con 0x7fb230103210 2024-04-25T02:08:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.236+0000 7fb222a00640 1 -- 172.21.15.102:0/2745640666 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb224005520 con 0x560189f80720 2024-04-25T02:08:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.236+0000 7fb222a00640 1 -- 172.21.15.102:0/2745640666 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224003850 con 0x560189f80720 2024-04-25T02:08:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.237+0000 7fb236200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2180a2e20 0x7fb2180a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.237+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2745640666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f80720 msgr2=0x560189f8aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.237+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/2745640666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f80720 0x560189f8aaa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb224009230 tx=0x7fb2240040f0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.237+0000 7fb236200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2180a2e20 0x7fb2180a52e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb228005fd0 tx=0x7fb228009500 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.238+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2745640666 shutdown_connections 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.238+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/2745640666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f8afe0 0x560189f8d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.238+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/2745640666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f80720 0x560189f8aaa0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.238+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/2745640666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f7fe00 0x560189f801e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.238+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2745640666 >> 172.21.15.102:0/2745640666 conn(0x560189f7c310 msgr2=0x560189f7c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.238+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2745640666 shutdown_connections 2024-04-25T02:08:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.238+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2745640666 wait complete. 2024-04-25T02:08:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.238+0000 7fb23b1c0740 1 -- start start 2024-04-25T02:08:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.238+0000 7fb23b1c0740 1 Processor -- start 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.239+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 0x560189f88000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.239+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f88540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.239+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f8afe0 0x560189f88a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.239+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560189f8dbb0 con 0x560189f8afe0 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.239+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560189e9f990 con 0x560189f7fe00 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.239+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560189f8dd30 con 0x560189f80720 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.239+0000 7fb23b1c0740 10 client.0 _ll_register_callbacks cb 0x560189ea1950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb236200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f8afe0 0x560189f88a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb234e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f88540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb234e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f88540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35404/0 (socket says 172.21.15.102:35404) 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb234e00640 1 -- 172.21.15.102:0/3336126528 learned_addr learned my addr 172.21.15.102:0/3336126528 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb235800640 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 0x560189f88000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb234e00640 1 -- 172.21.15.102:0/3336126528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 msgr2=0x560189f88000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb234e00640 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 0x560189f88000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb234e00640 1 -- 172.21.15.102:0/3336126528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f8afe0 msgr2=0x560189f88a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb234e00640 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f8afe0 0x560189f88a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb234e00640 1 -- 172.21.15.102:0/3336126528 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560189cc4620 con 0x560189f80720 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb234e00640 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f88540 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00bdb0 tx=0x7fb21c09b030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9365d921b9c7325 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.240+0000 7fb217400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb217400640 1 -- 172.21.15.102:0/3336126528 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c010eb0 con 0x560189f80720 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3336126528 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb224005520 con 0x560189f80720 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb217400640 1 -- 172.21.15.102:0/3336126528 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8513+0+0 (secure 0 0 0) 0x7fb21c097810 con 0x560189f80720 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb217400640 1 -- 172.21.15.102:0/3336126528 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fb21c09b8a0 con 0x560189f80720 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb23b1c0740 10 client.27568 fetch_fsmap learned FSMap version 49 2024-04-25T02:08:57.249 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3336126528 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x560189f99b60 con 0x560189f80720 2024-04-25T02:08:57.251 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb217400640 1 -- 172.21.15.102:0/3336126528 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb224005520 con 0x560189f80720 2024-04-25T02:08:57.251 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb217400640 1 -- 172.21.15.102:0/3336126528 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 49) ==== 8346+0+0 (secure 0 0 0) 0x7fb21c0bb0b0 con 0x560189f80720 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb23b1c0740 10 client.27568 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb23b1c0740 10 client.27568 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.241+0000 7fb23b1c0740 20 client.27568 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.242+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560189f9afd0 0x560189f9d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.242+0000 7fb23b1c0740 4 client.27568 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.242+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3336126528 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560189ef0ff0 con 0x560189f9afd0 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.242+0000 7fb235800640 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560189f9afd0 0x560189f9d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.242+0000 7fb235800640 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560189f9afd0 0x560189f9d3c0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19b64a61b6c2025b server_cookie=26e11303e9cb3656 in_seq=0 out_seq=0 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.242+0000 7fb217400640 10 client.27568 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb217400640 1 -- 172.21.15.102:0/3336126528 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560189ef0ff0 con 0x560189f9afd0 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb217400640 10 client.27568 handle_command_reply: tid=0 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb23b1c0740 1 client.27568 shutdown 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3336126528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 msgr2=0x560189f88540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:57.252 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f88540 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00bdb0 tx=0x7fb21c09b030 comp rx=0 tx=0).stop 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3336126528 shutdown_connections 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560189f9afd0 0x560189f9d3c0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f8afe0 0x560189f88a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f88540 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3336126528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 0x560189f88000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3336126528 >> 172.21.15.102:0/3336126528 conn(0x560189f7c310 msgr2=0x560189f82090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.243+0000 7fb217400640 0 client.27568 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.244+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3336126528 shutdown_connections 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.244+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3336126528 wait complete. 2024-04-25T02:08:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.244+0000 7fb23b1c0740 20 client.27568 trim_cache size 0 max 16384 2024-04-25T02:08:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.272+0000 7fb23b1c0740 1 Processor -- start 2024-04-25T02:08:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.272+0000 7fb23b1c0740 1 -- start start 2024-04-25T02:08:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.273+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f7fe00 0x560189f96a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.273+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f888a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.273+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f8afe0 0x560189f7b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.273+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301b63e0 con 0x560189f7fe00 2024-04-25T02:08:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.273+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301b3c50 con 0x560189f8afe0 2024-04-25T02:08:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.273+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301b1ac0 con 0x560189f80720 2024-04-25T02:08:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.273+0000 7fb234e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f888a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.274+0000 7fb234e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f888a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35406/0 (socket says 172.21.15.102:35406) 2024-04-25T02:08:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.274+0000 7fb234e00640 1 -- 172.21.15.102:0/2122794565 learned_addr learned my addr 172.21.15.102:0/2122794565 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.274+0000 7fb234e00640 1 -- 172.21.15.102:0/2122794565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f8afe0 msgr2=0x560189f7b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.274+0000 7fb234e00640 1 --2- 172.21.15.102:0/2122794565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f8afe0 0x560189f7b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.274+0000 7fb234e00640 1 -- 172.21.15.102:0/2122794565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f7fe00 msgr2=0x560189f96a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.274+0000 7fb234e00640 1 --2- 172.21.15.102:0/2122794565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f7fe00 0x560189f96a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.274+0000 7fb234e00640 1 -- 172.21.15.102:0/2122794565 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560189f8d600 con 0x560189f80720 2024-04-25T02:08:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.278+0000 7fb234e00640 1 --2- 172.21.15.102:0/2122794565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f888a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb21c09aea0 tx=0x7fb21c010720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3148a7a46cca58e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.278+0000 7fb222000640 1 -- 172.21.15.102:0/2122794565 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c0b3070 con 0x560189f80720 2024-04-25T02:08:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.278+0000 7fb222000640 1 -- 172.21.15.102:0/2122794565 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb21c09b1a0 con 0x560189f80720 2024-04-25T02:08:57.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.278+0000 7fb222000640 1 -- 172.21.15.102:0/2122794565 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c09f550 con 0x560189f80720 2024-04-25T02:08:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.279+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2122794565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 msgr2=0x560189f888a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:57.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.279+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/2122794565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f888a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb21c09aea0 tx=0x7fb21c010720 comp rx=0 tx=0).stop 2024-04-25T02:08:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.279+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2122794565 shutdown_connections 2024-04-25T02:08:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.279+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/2122794565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f8afe0 0x560189f7b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.279+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/2122794565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f80720 0x560189f888a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.279+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/2122794565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f7fe00 0x560189f96a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.279+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2122794565 >> 172.21.15.102:0/2122794565 conn(0x560189f7c310 msgr2=0x560189f7abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.280+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2122794565 shutdown_connections 2024-04-25T02:08:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.280+0000 7fb23b1c0740 1 -- 172.21.15.102:0/2122794565 wait complete. 2024-04-25T02:08:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.280+0000 7fb23b1c0740 1 -- start start 2024-04-25T02:08:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.281+0000 7fb23b1c0740 1 Processor -- start 2024-04-25T02:08:57.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.281+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 0x560189f84eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.288 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.281+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f8afe0 0x560189f853f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.282+0000 7fb235800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 0x560189f84eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.282+0000 7fb23b1c0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f85930 0x56018a011510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.282+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301b2110 con 0x560189f8afe0 2024-04-25T02:08:57.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.282+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301b3580 con 0x560189f7fe00 2024-04-25T02:08:57.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.282+0000 7fb23b1c0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb2301b27e0 con 0x560189f85930 2024-04-25T02:08:57.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.282+0000 7fb23b1c0740 10 client.0 _ll_register_callbacks cb 0x7fb21c097440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:08:57.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.282+0000 7fb236200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f85930 0x56018a011510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.289 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.282+0000 7fb236200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f85930 0x56018a011510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:35412/0 (socket says 172.21.15.102:35412) 2024-04-25T02:08:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.282+0000 7fb236200640 1 -- 172.21.15.102:0/3617730817 learned_addr learned my addr 172.21.15.102:0/3617730817 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:08:57.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.283+0000 7fb235800640 1 -- 172.21.15.102:0/3617730817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f85930 msgr2=0x56018a011510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:57.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.283+0000 7fb235800640 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f85930 0x56018a011510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.283+0000 7fb235800640 1 -- 172.21.15.102:0/3617730817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f8afe0 msgr2=0x560189f853f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:08:57.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.283+0000 7fb235800640 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f8afe0 0x560189f853f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.283+0000 7fb235800640 1 -- 172.21.15.102:0/3617730817 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56018a011b50 con 0x560189f7fe00 2024-04-25T02:08:57.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.283+0000 7fb236200640 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f85930 0x56018a011510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:08:57.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.285+0000 7fb235800640 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 0x560189f84eb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb224009900 tx=0x7fb224005a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bb23eb4b79e23fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:08:57.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.285+0000 7fb217400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:08:57.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.285+0000 7fb217400640 1 -- 172.21.15.102:0/3617730817 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224035040 con 0x560189f7fe00 2024-04-25T02:08:57.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.285+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3617730817 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb21c09b1a0 con 0x560189f7fe00 2024-04-25T02:08:57.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.285+0000 7fb217400640 1 -- 172.21.15.102:0/3617730817 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8513+0+0 (secure 0 0 0) 0x7fb224051120 con 0x560189f7fe00 2024-04-25T02:08:57.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.286+0000 7fb217400640 1 -- 172.21.15.102:0/3617730817 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fb224051340 con 0x560189f7fe00 2024-04-25T02:08:57.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.286+0000 7fb23b1c0740 10 client.27626 fetch_fsmap learned FSMap version 49 2024-04-25T02:08:57.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.286+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3617730817 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x56018a012e40 con 0x560189f7fe00 2024-04-25T02:08:57.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.286+0000 7fb217400640 1 -- 172.21.15.102:0/3617730817 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb21c09b1a0 con 0x560189f7fe00 2024-04-25T02:08:57.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.287+0000 7fb217400640 1 -- 172.21.15.102:0/3617730817 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 49) ==== 8346+0+0 (secure 0 0 0) 0x7fb22404d070 con 0x560189f7fe00 2024-04-25T02:08:57.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.287+0000 7fb23b1c0740 10 client.27626 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T02:08:57.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.287+0000 7fb23b1c0740 10 client.27626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:08:57.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.287+0000 7fb23b1c0740 20 client.27626 populate_metadata read hostname 'smithi102' 2024-04-25T02:08:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.287+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56018a013120 0x56018a015510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:08:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.287+0000 7fb23b1c0740 4 client.27626 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:08:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.287+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3617730817 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560189eeb340 con 0x56018a013120 2024-04-25T02:08:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.288+0000 7fb234e00640 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56018a013120 0x56018a015510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:08:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.288+0000 7fb234e00640 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56018a013120 0x56018a015510 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3c5036da7b6bb54 server_cookie=438413c81a4e5baa in_seq=0 out_seq=0 2024-04-25T02:08:57.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.288+0000 7fb217400640 10 client.27626 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb217400640 1 -- 172.21.15.102:0/3617730817 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560189eeb340 con 0x56018a013120 2024-04-25T02:08:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb217400640 10 client.27626 handle_command_reply: tid=0 2024-04-25T02:08:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb23b1c0740 1 client.27626 shutdown 2024-04-25T02:08:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3617730817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 msgr2=0x560189f84eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 0x560189f84eb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb224009900 tx=0x7fb224005a60 comp rx=0 tx=0).stop 2024-04-25T02:08:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3617730817 shutdown_connections 2024-04-25T02:08:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x56018a013120 0x56018a015510 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560189f85930 0x56018a011510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560189f8afe0 0x560189f853f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb23b1c0740 1 --2- 172.21.15.102:0/3617730817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560189f7fe00 0x560189f84eb0 secure :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb224009900 tx=0x7fb224005a60 comp rx=0 tx=0).stop 2024-04-25T02:08:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3617730817 >> 172.21.15.102:0/3617730817 conn(0x560189f7c310 msgr2=0x560189f7b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.373+0000 7fb217400640 0 client.27626 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:08:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.374+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3617730817 shutdown_connections 2024-04-25T02:08:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.375+0000 7fb23b1c0740 1 -- 172.21.15.102:0/3617730817 wait complete. 2024-04-25T02:08:57.381 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.375+0000 7fb23b1c0740 20 client.27626 trim_cache size 0 max 16384 2024-04-25T02:08:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:08:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:08:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:08:57.382 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:08:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.376+0000 7fb238200640 1 -- 172.21.15.102:0/1206363134 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2180a2e20 msgr2=0x7fb2180a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.376+0000 7fb238200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2180a2e20 0x7fb2180a52e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb228005fd0 tx=0x7fb228009500 comp rx=0 tx=0).stop 2024-04-25T02:08:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.376+0000 7fb238200640 1 -- 172.21.15.102:0/1206363134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb230103210 msgr2=0x7fb23010aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:08:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.376+0000 7fb238200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb230103210 0x7fb23010aee0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00b3f0 tx=0x7fb21c00b8b0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.376+0000 7fb238200640 1 -- 172.21.15.102:0/1206363134 shutdown_connections 2024-04-25T02:08:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.376+0000 7fb238200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb2180a2e20 0x7fb2180a52e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.376+0000 7fb238200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb23010bd10 0x7fb2301a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.376+0000 7fb238200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb230103210 0x7fb23010aee0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.383 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.376+0000 7fb238200640 1 --2- 172.21.15.102:0/1206363134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2301028f0 0x7fb23010a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:08:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.377+0000 7fb238200640 1 -- 172.21.15.102:0/1206363134 >> 172.21.15.102:0/1206363134 conn(0x7fb2300fc110 msgr2=0x7fb230100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:08:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.377+0000 7fb238200640 1 -- 172.21.15.102:0/1206363134 shutdown_connections 2024-04-25T02:08:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:08:57.377+0000 7fb238200640 1 -- 172.21.15.102:0/1206363134 wait complete. 2024-04-25T02:08:57.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:57 smithi102 ceph-mon[27796]: pgmap v425: 97 pgs: 97 active+clean; 5.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 36 MiB/s wr, 2.42k op/s 2024-04-25T02:08:57.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:57 smithi110 ceph-mon[32848]: pgmap v425: 97 pgs: 97 active+clean; 5.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 36 MiB/s wr, 2.42k op/s 2024-04-25T02:08:57.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:57 smithi193 ceph-mon[32981]: pgmap v425: 97 pgs: 97 active+clean; 5.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 36 MiB/s wr, 2.42k op/s 2024-04-25T02:08:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 785.0 s, 1581.5 tps, lat 63.581 ms stddev 74.112 2024-04-25T02:08:59.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:08:59 smithi102 ceph-mon[27796]: pgmap v426: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 33 MiB/s wr, 2.24k op/s 2024-04-25T02:08:59.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:08:59 smithi110 ceph-mon[32848]: pgmap v426: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 33 MiB/s wr, 2.24k op/s 2024-04-25T02:08:59.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:08:59 smithi193 ceph-mon[32981]: pgmap v426: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 33 MiB/s wr, 2.24k op/s 2024-04-25T02:09:01.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:01 smithi102 ceph-mon[27796]: pgmap v427: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 21 MiB/s wr, 1.48k op/s 2024-04-25T02:09:01.493 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:01 smithi110 ceph-mon[32848]: pgmap v427: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 21 MiB/s wr, 1.48k op/s 2024-04-25T02:09:01.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:01 smithi193 ceph-mon[32981]: pgmap v427: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 21 MiB/s wr, 1.48k op/s 2024-04-25T02:09:03.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 790.0 s, 1453.5 tps, lat 68.090 ms stddev 95.005 2024-04-25T02:09:03.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:03 smithi102 ceph-mon[27796]: pgmap v428: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 21 MiB/s wr, 1.47k op/s 2024-04-25T02:09:03.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:03 smithi110 ceph-mon[32848]: pgmap v428: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 21 MiB/s wr, 1.47k op/s 2024-04-25T02:09:03.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:03 smithi193 ceph-mon[32981]: pgmap v428: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 21 MiB/s wr, 1.47k op/s 2024-04-25T02:09:04.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:04 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:04.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:04 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:04.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:04 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:05.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:05 smithi102 ceph-mon[27796]: pgmap v429: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 25 MiB/s wr, 1.86k op/s 2024-04-25T02:09:05.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:05 smithi110 ceph-mon[32848]: pgmap v429: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 25 MiB/s wr, 1.86k op/s 2024-04-25T02:09:05.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:05 smithi193 ceph-mon[32981]: pgmap v429: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 25 MiB/s wr, 1.86k op/s 2024-04-25T02:09:07.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:07 smithi102 ceph-mon[27796]: pgmap v430: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 17 MiB/s wr, 1.42k op/s 2024-04-25T02:09:07.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:07 smithi110 ceph-mon[32848]: pgmap v430: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 17 MiB/s wr, 1.42k op/s 2024-04-25T02:09:07.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:07 smithi193 ceph-mon[32981]: pgmap v430: 97 pgs: 97 active+clean; 5.7 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 17 MiB/s wr, 1.42k op/s 2024-04-25T02:09:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 795.0 s, 1288.2 tps, lat 76.439 ms stddev 102.516 2024-04-25T02:09:08.431 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:08 smithi102 ceph-mon[27796]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T02:09:08.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:08 smithi110 ceph-mon[32848]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T02:09:08.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:08 smithi193 ceph-mon[32981]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T02:09:09.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:09 smithi102 ceph-mon[27796]: pgmap v432: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 5.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 16 MiB/s wr, 1.33k op/s 2024-04-25T02:09:09.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:09 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:09.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:09 smithi110 ceph-mon[32848]: pgmap v432: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 5.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 16 MiB/s wr, 1.33k op/s 2024-04-25T02:09:09.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:09 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:09.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:09 smithi193 ceph-mon[32981]: pgmap v432: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 5.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 16 MiB/s wr, 1.33k op/s 2024-04-25T02:09:09.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:09 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.565+0000 7f5880000640 1 -- 172.21.15.102:0/226486065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5878107770 msgr2=0x7f5878107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.565+0000 7f5880000640 1 --2- 172.21.15.102:0/226486065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5878107770 0x7f5878107b50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f586c009900 tx=0x7f586c02a3f0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.565+0000 7f5880000640 1 -- 172.21.15.102:0/226486065 shutdown_connections 2024-04-25T02:09:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.565+0000 7f5880000640 1 --2- 172.21.15.102:0/226486065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5878102120 0x7f587810e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.565+0000 7f5880000640 1 --2- 172.21.15.102:0/226486065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5878101760 0x7f5878101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.565+0000 7f5880000640 1 --2- 172.21.15.102:0/226486065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5878107770 0x7f5878107b50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.565+0000 7f5880000640 1 -- 172.21.15.102:0/226486065 >> 172.21.15.102:0/226486065 conn(0x7f58780fd1f0 msgr2=0x7f58780ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.565+0000 7f5880000640 1 -- 172.21.15.102:0/226486065 shutdown_connections 2024-04-25T02:09:10.574 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.565+0000 7f5880000640 1 -- 172.21.15.102:0/226486065 wait complete. 2024-04-25T02:09:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.567+0000 7f5880000640 1 Processor -- start 2024-04-25T02:09:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.567+0000 7f5880000640 1 -- start start 2024-04-25T02:09:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f5880000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5878101760 0x7f5878073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.575 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f5880000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5878102120 0x7f5878073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f5880000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5878107770 0x7f58780744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f5880000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5878111b90 con 0x7f5878101760 2024-04-25T02:09:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f5880000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f5878111a10 con 0x7f5878102120 2024-04-25T02:09:10.576 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f5880000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5878111d10 con 0x7f5878107770 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5878101760 0x7f5878073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5878101760 0x7f5878073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53706/0 (socket says 172.21.15.102:53706) 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587d600640 1 -- 172.21.15.102:0/631663099 learned_addr learned my addr 172.21.15.102:0/631663099 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587cc00640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5878102120 0x7f5878073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587e000640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5878107770 0x7f58780744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587cc00640 1 -- 172.21.15.102:0/631663099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5878107770 msgr2=0x7f58780744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587cc00640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5878107770 0x7f58780744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587cc00640 1 -- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5878101760 msgr2=0x7f5878073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587cc00640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5878101760 0x7f5878073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.577 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587cc00640 1 -- 172.21.15.102:0/631663099 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58781a6470 con 0x7f5878102120 2024-04-25T02:09:10.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.568+0000 7f587d600640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5878101760 0x7f5878073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:10.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.569+0000 7f587cc00640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5878102120 0x7f5878073f80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5864005fd0 tx=0x7f586400dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:10.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.569+0000 7f5876000640 1 -- 172.21.15.102:0/631663099 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864009c30 con 0x7f5878102120 2024-04-25T02:09:10.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.569+0000 7f5876000640 1 -- 172.21.15.102:0/631663099 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5864009dd0 con 0x7f5878102120 2024-04-25T02:09:10.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.569+0000 7f5876000640 1 -- 172.21.15.102:0/631663099 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864010460 con 0x7f5878102120 2024-04-25T02:09:10.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.570+0000 7f5880000640 1 -- 172.21.15.102:0/631663099 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58781a6760 con 0x7f5878102120 2024-04-25T02:09:10.578 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.570+0000 7f5880000640 1 -- 172.21.15.102:0/631663099 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58781a6c60 con 0x7f5878102120 2024-04-25T02:09:10.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.574+0000 7f5876000640 1 -- 172.21.15.102:0/631663099 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5864003ef0 con 0x7f5878102120 2024-04-25T02:09:10.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.575+0000 7f5882df0740 1 Processor -- start 2024-04-25T02:09:10.585 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.577+0000 7f5882df0740 1 -- start start 2024-04-25T02:09:10.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.577+0000 7f5882df0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a016009f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.577+0000 7f5882df0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a0160b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.578+0000 7f5882df0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 0x559a0160dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.578+0000 7f5882df0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559a0155bab0 con 0x559a0160b7f0 2024-04-25T02:09:10.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.578+0000 7f5882df0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559a01504640 con 0x559a01600f30 2024-04-25T02:09:10.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.578+0000 7f5882df0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559a01557a70 con 0x559a01600610 2024-04-25T02:09:10.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.578+0000 7f587e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a016009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.578+0000 7f587e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a016009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50710/0 (socket says 172.21.15.102:50710) 2024-04-25T02:09:10.586 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.578+0000 7f587e000640 1 -- 172.21.15.102:0/755775956 learned_addr learned my addr 172.21.15.102:0/755775956 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:10.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.578+0000 7f587d600640 1 --2- 172.21.15.102:0/755775956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a0160b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.579+0000 7f587e000640 1 -- 172.21.15.102:0/755775956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 msgr2=0x559a0160b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:10.587 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.579+0000 7f587e000640 1 --2- 172.21.15.102:0/755775956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a0160b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.579+0000 7f587e000640 1 -- 172.21.15.102:0/755775956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 msgr2=0x559a0160dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.579+0000 7f587e000640 1 --2- 172.21.15.102:0/755775956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 0x559a0160dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.579+0000 7f587e000640 1 -- 172.21.15.102:0/755775956 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a01270210 con 0x559a01600610 2024-04-25T02:09:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.579+0000 7f587d600640 1 --2- 172.21.15.102:0/755775956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a0160b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.579+0000 7f587e000640 1 --2- 172.21.15.102:0/755775956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a016009f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5870009ea0 tx=0x7f5870009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abbef599dce389e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.579+0000 7f586a600640 1 -- 172.21.15.102:0/755775956 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870004d20 con 0x559a01600610 2024-04-25T02:09:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.580+0000 7f586a600640 1 -- 172.21.15.102:0/755775956 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5870004050 con 0x559a01600610 2024-04-25T02:09:10.588 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.580+0000 7f586a600640 1 -- 172.21.15.102:0/755775956 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870005280 con 0x559a01600610 2024-04-25T02:09:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.580+0000 7f5876000640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f58600a2ea0 0x7f58600a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.580+0000 7f5882df0740 1 -- 172.21.15.102:0/755775956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 msgr2=0x559a016009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.580+0000 7f5882df0740 1 --2- 172.21.15.102:0/755775956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a016009f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5870009ea0 tx=0x7f5870009f90 comp rx=0 tx=0).stop 2024-04-25T02:09:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f5876000640 1 -- 172.21.15.102:0/631663099 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8557+0+0 (secure 0 0 0) 0x7f58640cef60 con 0x7f5878102120 2024-04-25T02:09:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f5882df0740 1 -- 172.21.15.102:0/755775956 shutdown_connections 2024-04-25T02:09:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f5882df0740 1 --2- 172.21.15.102:0/755775956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 0x559a0160dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f5882df0740 1 --2- 172.21.15.102:0/755775956 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a0160b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.590 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f5882df0740 1 --2- 172.21.15.102:0/755775956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a016009f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f5882df0740 1 -- 172.21.15.102:0/755775956 >> 172.21.15.102:0/755775956 conn(0x559a015fcb20 msgr2=0x559a015fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f5882df0740 1 -- 172.21.15.102:0/755775956 shutdown_connections 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f5882df0740 1 -- 172.21.15.102:0/755775956 wait complete. 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f587d600640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f58600a2ea0 0x7f58600a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.581+0000 7f5882df0740 1 -- start start 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f5882df0740 1 Processor -- start 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f5882df0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600610 0x559a01608880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f5882df0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600f30 0x559a01608dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f5882df0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a0160b7f0 0x559a01609300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f5882df0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559a0160e420 con 0x559a01600610 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f5882df0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559a01517990 con 0x559a0160b7f0 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f5882df0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559a0160e5a0 con 0x559a01600f30 2024-04-25T02:09:10.591 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f5882df0740 10 client.0 _ll_register_callbacks cb 0x559a01519950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f587cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600f30 0x559a01608dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f587e000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a0160b7f0 0x559a01609300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f587cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600f30 0x559a01608dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50716/0 (socket says 172.21.15.102:50716) 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f587e000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a0160b7f0 0x559a01609300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56042/0 (socket says 172.21.15.102:56042) 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.582+0000 7f587cc00640 1 -- 172.21.15.102:0/2495729791 learned_addr learned my addr 172.21.15.102:0/2495729791 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f587d600640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f58600a2ea0 0x7f58600a5360 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x559a01601bb0 tx=0x7f586c0046c0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f587d600640 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600610 0x559a01608880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f587e000640 1 -- 172.21.15.102:0/2495729791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600f30 msgr2=0x559a01608dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f587e000640 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600f30 0x559a01608dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f587e000640 1 -- 172.21.15.102:0/2495729791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600610 msgr2=0x559a01608880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:10.593 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f587e000640 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600610 0x559a01608880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f587e000640 1 -- 172.21.15.102:0/2495729791 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a0133c620 con 0x559a0160b7f0 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f587cc00640 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600f30 0x559a01608dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f587e000640 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a0160b7f0 0x559a01609300 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f587001b040 tx=0x7f5870004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d07ded1eb4e4b52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f585ea00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f585ea00640 1 -- 172.21.15.102:0/2495729791 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870029070 con 0x559a0160b7f0 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f585ea00640 1 -- 172.21.15.102:0/2495729791 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8557+0+0 (secure 0 0 0) 0x7f5870004fa0 con 0x559a0160b7f0 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.583+0000 7f5882df0740 1 -- 172.21.15.102:0/2495729791 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5870004050 con 0x559a0160b7f0 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.584+0000 7f585ea00640 1 -- 172.21.15.102:0/2495729791 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f5870035050 con 0x559a0160b7f0 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.584+0000 7f5882df0740 10 client.27638 fetch_fsmap learned FSMap version 51 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.584+0000 7f5882df0740 1 -- 172.21.15.102:0/2495729791 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x559a0161a3c0 con 0x559a0160b7f0 2024-04-25T02:09:10.594 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.584+0000 7f585ea00640 1 -- 172.21.15.102:0/2495729791 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5870004050 con 0x559a0160b7f0 2024-04-25T02:09:10.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.586+0000 7f585ea00640 1 -- 172.21.15.102:0/2495729791 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f58700051a0 con 0x559a0160b7f0 2024-04-25T02:09:10.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.586+0000 7f5882df0740 10 client.27638 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:09:10.595 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.586+0000 7f5882df0740 10 client.27638 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.586+0000 7f5882df0740 20 client.27638 populate_metadata read hostname 'smithi102' 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.586+0000 7f5882df0740 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a0161b770 0x559a0161db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.586+0000 7f5882df0740 4 client.27638 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.586+0000 7f5882df0740 1 -- 172.21.15.102:0/2495729791 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a01568ff0 con 0x559a0161b770 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.586+0000 7f587d600640 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a0161b770 0x559a0161db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.587+0000 7f587d600640 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a0161b770 0x559a0161db60 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1e4edc151da2639 server_cookie=d0aa6f0b881a625a in_seq=0 out_seq=0 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.587+0000 7f585ea00640 10 client.27638 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f585ea00640 1 -- 172.21.15.102:0/2495729791 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a01568ff0 con 0x559a0161b770 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f585ea00640 10 client.27638 handle_command_reply: tid=0 2024-04-25T02:09:10.596 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f5882df0740 1 client.27638 shutdown 2024-04-25T02:09:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f5882df0740 1 -- 172.21.15.102:0/2495729791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a0160b7f0 msgr2=0x559a01609300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f5882df0740 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a0160b7f0 0x559a01609300 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f587001b040 tx=0x7f5870004430 comp rx=0 tx=0).stop 2024-04-25T02:09:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f5882df0740 1 -- 172.21.15.102:0/2495729791 shutdown_connections 2024-04-25T02:09:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f5882df0740 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a0161b770 0x559a0161db60 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f5882df0740 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a0160b7f0 0x559a01609300 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f5882df0740 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600f30 0x559a01608dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f5882df0740 1 --2- 172.21.15.102:0/2495729791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600610 0x559a01608880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.588+0000 7f5882df0740 1 -- 172.21.15.102:0/2495729791 >> 172.21.15.102:0/2495729791 conn(0x559a015fcb20 msgr2=0x559a015f3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.589+0000 7f585ea00640 0 client.27638 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.589+0000 7f5882df0740 1 -- 172.21.15.102:0/2495729791 shutdown_connections 2024-04-25T02:09:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.589+0000 7f5882df0740 1 -- 172.21.15.102:0/2495729791 wait complete. 2024-04-25T02:09:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.589+0000 7f5882df0740 20 client.27638 trim_cache size 0 max 16384 2024-04-25T02:09:10.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.614+0000 7f5882df0740 1 Processor -- start 2024-04-25T02:09:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.614+0000 7f5882df0740 1 -- start start 2024-04-25T02:09:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.614+0000 7f5882df0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a01602fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.614+0000 7f5882df0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a015f3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.614+0000 7f5882df0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 0x559a01623270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.614+0000 7f5882df0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f58781adc20 con 0x559a0160b7f0 2024-04-25T02:09:10.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.614+0000 7f5882df0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f58781ab430 con 0x559a01600f30 2024-04-25T02:09:10.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.614+0000 7f5882df0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f58781a9300 con 0x559a01600610 2024-04-25T02:09:10.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587cc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a015f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587d600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a01602fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587cc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a015f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:56054/0 (socket says 172.21.15.102:56054) 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587cc00640 1 -- 172.21.15.102:0/2806043883 learned_addr learned my addr 172.21.15.102:0/2806043883 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587e000640 1 --2- 172.21.15.102:0/2806043883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 0x559a01623270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587cc00640 1 -- 172.21.15.102:0/2806043883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 msgr2=0x559a01602fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587cc00640 1 --2- 172.21.15.102:0/2806043883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a01602fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587cc00640 1 -- 172.21.15.102:0/2806043883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 msgr2=0x559a01623270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587cc00640 1 --2- 172.21.15.102:0/2806043883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 0x559a01623270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587cc00640 1 -- 172.21.15.102:0/2806043883 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a016044a0 con 0x559a01600f30 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587e000640 1 --2- 172.21.15.102:0/2806043883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 0x559a01623270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f587cc00640 1 --2- 172.21.15.102:0/2806043883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a015f3300 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5864000c00 tx=0x7f586400df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eddf309df86afcb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:10.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.615+0000 7f5869c00640 1 -- 172.21.15.102:0/2806043883 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58640b4070 con 0x559a01600f30 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.616+0000 7f5869c00640 1 -- 172.21.15.102:0/2806043883 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f586409b4b0 con 0x559a01600f30 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.616+0000 7f5869c00640 1 -- 172.21.15.102:0/2806043883 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864020040 con 0x559a01600f30 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.617+0000 7f5882df0740 1 -- 172.21.15.102:0/2806043883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 msgr2=0x559a015f3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.617+0000 7f5882df0740 1 --2- 172.21.15.102:0/2806043883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a015f3300 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5864000c00 tx=0x7f586400df90 comp rx=0 tx=0).stop 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.617+0000 7f5882df0740 1 -- 172.21.15.102:0/2806043883 shutdown_connections 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.617+0000 7f5882df0740 1 --2- 172.21.15.102:0/2806043883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a0160b7f0 0x559a01623270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.617+0000 7f5882df0740 1 --2- 172.21.15.102:0/2806043883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600f30 0x559a015f3300 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.617+0000 7f5882df0740 1 --2- 172.21.15.102:0/2806043883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a01600610 0x559a01602fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.617+0000 7f5882df0740 1 -- 172.21.15.102:0/2806043883 >> 172.21.15.102:0/2806043883 conn(0x559a015fcb20 msgr2=0x559a015f2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.618+0000 7f5882df0740 1 -- 172.21.15.102:0/2806043883 shutdown_connections 2024-04-25T02:09:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.618+0000 7f5882df0740 1 -- 172.21.15.102:0/2806043883 wait complete. 2024-04-25T02:09:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.619+0000 7f5882df0740 1 -- start start 2024-04-25T02:09:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.619+0000 7f5882df0740 1 Processor -- start 2024-04-25T02:09:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f5882df0740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600610 0x559a01613800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f5882df0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600f30 0x559a01613d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f5882df0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a0160b7f0 0x559a0160e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f5882df0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f58781a9950 con 0x559a01600f30 2024-04-25T02:09:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f5882df0740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f58781aad60 con 0x559a01600610 2024-04-25T02:09:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f5882df0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f58781a9fc0 con 0x559a0160b7f0 2024-04-25T02:09:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f5882df0740 10 client.0 _ll_register_callbacks cb 0x7f586c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600f30 0x559a01613d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a0160b7f0 0x559a0160e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600f30 0x559a01613d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53744/0 (socket says 172.21.15.102:53744) 2024-04-25T02:09:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587cc00640 1 -- 172.21.15.102:0/4024074108 learned_addr learned my addr 172.21.15.102:0/4024074108 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a0160b7f0 0x559a0160e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50746/0 (socket says 172.21.15.102:50746) 2024-04-25T02:09:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587d600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600610 0x559a01613800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587cc00640 1 -- 172.21.15.102:0/4024074108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a0160b7f0 msgr2=0x559a0160e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:10.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587cc00640 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a0160b7f0 0x559a0160e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587cc00640 1 -- 172.21.15.102:0/4024074108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600610 msgr2=0x559a01613800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587cc00640 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600610 0x559a01613800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587cc00640 1 -- 172.21.15.102:0/4024074108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a0160edf0 con 0x559a01600f30 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587d600640 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600610 0x559a01613800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.620+0000 7f587e000640 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a0160b7f0 0x559a0160e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.621+0000 7f587cc00640 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600f30 0x559a01613d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f586409e000 tx=0x7f5864097110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39dd2a2d6684384e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.621+0000 7f585ea00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.621+0000 7f585ea00640 1 -- 172.21.15.102:0/4024074108 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864020050 con 0x559a01600f30 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.621+0000 7f5882df0740 1 -- 172.21.15.102:0/4024074108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f586409b4b0 con 0x559a01600f30 2024-04-25T02:09:10.630 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.621+0000 7f585ea00640 1 -- 172.21.15.102:0/4024074108 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8557+0+0 (secure 0 0 0) 0x7f58640b41e0 con 0x559a01600f30 2024-04-25T02:09:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.622+0000 7f585ea00640 1 -- 172.21.15.102:0/4024074108 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f58640b4040 con 0x559a01600f30 2024-04-25T02:09:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.623+0000 7f5882df0740 10 client.16902 fetch_fsmap learned FSMap version 51 2024-04-25T02:09:10.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.623+0000 7f5882df0740 1 -- 172.21.15.102:0/4024074108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x559a015f9490 con 0x559a01600f30 2024-04-25T02:09:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.624+0000 7f585ea00640 1 -- 172.21.15.102:0/4024074108 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f586409b4b0 con 0x559a01600f30 2024-04-25T02:09:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.624+0000 7f585ea00640 1 -- 172.21.15.102:0/4024074108 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f586409aa90 con 0x559a01600f30 2024-04-25T02:09:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.624+0000 7f5882df0740 10 client.16902 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:09:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.624+0000 7f5882df0740 10 client.16902 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.624+0000 7f5882df0740 20 client.16902 populate_metadata read hostname 'smithi102' 2024-04-25T02:09:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.624+0000 7f5882df0740 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a015f9b90 0x559a0168cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.624+0000 7f5882df0740 4 client.16902 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:09:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.624+0000 7f5882df0740 1 -- 172.21.15.102:0/4024074108 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559a01563340 con 0x559a015f9b90 2024-04-25T02:09:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.625+0000 7f587d600640 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a015f9b90 0x559a0168cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.625+0000 7f587d600640 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a015f9b90 0x559a0168cac0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd8dc921d8ec600c server_cookie=158dfbbae2dbccd1 in_seq=0 out_seq=0 2024-04-25T02:09:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.625+0000 7f585ea00640 10 client.16902 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.649+0000 7f585ea00640 1 -- 172.21.15.102:0/4024074108 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559a01563340 con 0x559a015f9b90 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.649+0000 7f585ea00640 10 client.16902 handle_command_reply: tid=0 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.649+0000 7f5882df0740 1 client.16902 shutdown 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 -- 172.21.15.102:0/4024074108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600f30 msgr2=0x559a01613d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600f30 0x559a01613d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f586409e000 tx=0x7f5864097110 comp rx=0 tx=0).stop 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 -- 172.21.15.102:0/4024074108 shutdown_connections 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559a015f9b90 0x559a0168cac0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559a0160b7f0 0x559a0160e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559a01600f30 0x559a01613d40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 --2- 172.21.15.102:0/4024074108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559a01600610 0x559a01613800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f585ea00640 0 client.16902 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 -- 172.21.15.102:0/4024074108 >> 172.21.15.102:0/4024074108 conn(0x559a015fcb20 msgr2=0x559a0160a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 -- 172.21.15.102:0/4024074108 shutdown_connections 2024-04-25T02:09:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 1 -- 172.21.15.102:0/4024074108 wait complete. 2024-04-25T02:09:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.650+0000 7f5882df0740 20 client.16902 trim_cache size 0 max 16384 2024-04-25T02:09:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:09:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:09:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:09:10.659 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:09:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 -- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f58600a2ea0 msgr2=0x7f58600a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f58600a2ea0 0x7f58600a5360 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x559a01601bb0 tx=0x7f586c0046c0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 -- 172.21.15.102:0/631663099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5878102120 msgr2=0x7f5878073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5878102120 0x7f5878073f80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5864005fd0 tx=0x7f586400dde0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 -- 172.21.15.102:0/631663099 shutdown_connections 2024-04-25T02:09:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f58600a2ea0 0x7f58600a5360 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5878107770 0x7f58780744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.660 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5878102120 0x7f5878073f80 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 --2- 172.21.15.102:0/631663099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5878101760 0x7f5878073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 -- 172.21.15.102:0/631663099 >> 172.21.15.102:0/631663099 conn(0x7f58780fd1f0 msgr2=0x7f58780fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 -- 172.21.15.102:0/631663099 shutdown_connections 2024-04-25T02:09:10.661 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:10.651+0000 7f5880000640 1 -- 172.21.15.102:0/631663099 wait complete. 2024-04-25T02:09:11.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:11 smithi102 ceph-mon[27796]: pgmap v433: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 17 MiB/s wr, 1.33k op/s 2024-04-25T02:09:11.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:11 smithi110 ceph-mon[32848]: pgmap v433: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 17 MiB/s wr, 1.33k op/s 2024-04-25T02:09:11.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:11 smithi193 ceph-mon[32981]: pgmap v433: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 17 MiB/s wr, 1.33k op/s 2024-04-25T02:09:13.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 800.0 s, 1360.0 tps, lat 74.996 ms stddev 99.792 2024-04-25T02:09:13.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:13 smithi102 ceph-mon[27796]: pgmap v434: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 15 MiB/s wr, 1.21k op/s 2024-04-25T02:09:13.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:13 smithi110 ceph-mon[32848]: pgmap v434: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 15 MiB/s wr, 1.21k op/s 2024-04-25T02:09:13.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:13 smithi193 ceph-mon[32981]: pgmap v434: 97 pgs: 11 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 78 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 15 MiB/s wr, 1.21k op/s 2024-04-25T02:09:15.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:15 smithi102 ceph-mon[27796]: pgmap v435: 97 pgs: 97 active+clean; 5.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 11 MiB/s wr, 814 op/s 2024-04-25T02:09:15.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:15 smithi110 ceph-mon[32848]: pgmap v435: 97 pgs: 97 active+clean; 5.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 11 MiB/s wr, 814 op/s 2024-04-25T02:09:15.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:15 smithi193 ceph-mon[32981]: pgmap v435: 97 pgs: 97 active+clean; 5.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 11 MiB/s wr, 814 op/s 2024-04-25T02:09:17.431 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:17 smithi102 ceph-mon[27796]: pgmap v436: 97 pgs: 97 active+clean; 5.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.0 MiB/s wr, 634 op/s 2024-04-25T02:09:17.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:17 smithi110 ceph-mon[32848]: pgmap v436: 97 pgs: 97 active+clean; 5.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.0 MiB/s wr, 634 op/s 2024-04-25T02:09:17.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:17 smithi193 ceph-mon[32981]: pgmap v436: 97 pgs: 97 active+clean; 5.3 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 9.0 MiB/s wr, 634 op/s 2024-04-25T02:09:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 805.0 s, 1318.2 tps, lat 75.035 ms stddev 89.956 2024-04-25T02:09:18.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:18 smithi110 ceph-mon[32848]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T02:09:18.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:18 smithi193 ceph-mon[32981]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T02:09:18.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:18 smithi102 ceph-mon[27796]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T02:09:19.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:19 smithi110 ceph-mon[32848]: pgmap v438: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 8.6 MiB/s wr, 654 op/s 2024-04-25T02:09:19.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:19 smithi193 ceph-mon[32981]: pgmap v438: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 8.6 MiB/s wr, 654 op/s 2024-04-25T02:09:19.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:19 smithi102 ceph-mon[27796]: pgmap v438: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 8.6 MiB/s wr, 654 op/s 2024-04-25T02:09:21.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:21 smithi102 ceph-mon[27796]: pgmap v439: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 7.0 MiB/s wr, 631 op/s 2024-04-25T02:09:21.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:21 smithi110 ceph-mon[32848]: pgmap v439: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 7.0 MiB/s wr, 631 op/s 2024-04-25T02:09:21.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:21 smithi193 ceph-mon[32981]: pgmap v439: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 7.0 MiB/s wr, 631 op/s 2024-04-25T02:09:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 810.0 s, 1314.0 tps, lat 76.529 ms stddev 92.767 2024-04-25T02:09:23.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:23 smithi110 ceph-mon[32848]: pgmap v440: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 7.0 MiB/s wr, 631 op/s 2024-04-25T02:09:23.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:23 smithi193 ceph-mon[32981]: pgmap v440: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 7.0 MiB/s wr, 631 op/s 2024-04-25T02:09:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.822+0000 7f99c5400640 1 -- 172.21.15.102:0/205494596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c010bd10 msgr2=0x7f99c010e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.822+0000 7f99c5400640 1 --2- 172.21.15.102:0/205494596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c010bd10 0x7f99c010e1b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f99b800f0a0 tx=0x7f99b802e7a0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.822+0000 7f99c5400640 1 -- 172.21.15.102:0/205494596 shutdown_connections 2024-04-25T02:09:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.822+0000 7f99c5400640 1 --2- 172.21.15.102:0/205494596 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c010bd10 0x7f99c010e1b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.822+0000 7f99c5400640 1 --2- 172.21.15.102:0/205494596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c0103210 0x7f99c010b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.822+0000 7f99c5400640 1 --2- 172.21.15.102:0/205494596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c01028f0 0x7f99c0102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.822+0000 7f99c5400640 1 -- 172.21.15.102:0/205494596 >> 172.21.15.102:0/205494596 conn(0x7f99c00fc110 msgr2=0x7f99c00fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.823+0000 7f99c5400640 1 -- 172.21.15.102:0/205494596 shutdown_connections 2024-04-25T02:09:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.823+0000 7f99c5400640 1 -- 172.21.15.102:0/205494596 wait complete. 2024-04-25T02:09:23.831 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.824+0000 7f99c5400640 1 Processor -- start 2024-04-25T02:09:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.828+0000 7f99c5400640 1 -- start start 2024-04-25T02:09:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.828+0000 7f99c5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c01028f0 0x7f99c0195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.828+0000 7f99c5400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c0103210 0x7f99c0196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.828+0000 7f99c5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c010bd10 0x7f99c019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.828+0000 7f99c5400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99c010f9e0 con 0x7f99c010bd10 2024-04-25T02:09:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.828+0000 7f99c5400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99c010f860 con 0x7f99c0103210 2024-04-25T02:09:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.828+0000 7f99c5400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99c010fb60 con 0x7f99c01028f0 2024-04-25T02:09:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.829+0000 7f99be800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c01028f0 0x7f99c0195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.829+0000 7f99be800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c01028f0 0x7f99c0195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:36814/0 (socket says 172.21.15.102:36814) 2024-04-25T02:09:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.829+0000 7f99be800640 1 -- 172.21.15.102:0/1855135891 learned_addr learned my addr 172.21.15.102:0/1855135891 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.829+0000 7f99bde00640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c0103210 0x7f99c0196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.830+0000 7f99be800640 1 -- 172.21.15.102:0/1855135891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c0103210 msgr2=0x7f99c0196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.830+0000 7f99be800640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c0103210 0x7f99c0196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.830+0000 7f99be800640 1 -- 172.21.15.102:0/1855135891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c010bd10 msgr2=0x7f99c019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.830+0000 7f99be800640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c010bd10 0x7f99c019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.830+0000 7f99be800640 1 -- 172.21.15.102:0/1855135891 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c019ae60 con 0x7f99c01028f0 2024-04-25T02:09:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.831+0000 7f99be800640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c01028f0 0x7f99c0195ec0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00b450 tx=0x7f99ac00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.832+0000 7f99b7400640 1 -- 172.21.15.102:0/1855135891 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac015020 con 0x7f99c01028f0 2024-04-25T02:09:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.832+0000 7f99c5400640 1 -- 172.21.15.102:0/1855135891 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c019b150 con 0x7f99c01028f0 2024-04-25T02:09:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.832+0000 7f99b7400640 1 -- 172.21.15.102:0/1855135891 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99ac011d70 con 0x7f99c01028f0 2024-04-25T02:09:23.839 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.832+0000 7f99b7400640 1 -- 172.21.15.102:0/1855135891 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac00f340 con 0x7f99c01028f0 2024-04-25T02:09:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.832+0000 7f99c5400640 1 -- 172.21.15.102:0/1855135891 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c01a29e0 con 0x7f99c01028f0 2024-04-25T02:09:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.836+0000 7f99c81f4740 1 Processor -- start 2024-04-25T02:09:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.837+0000 7f99c81f4740 1 -- start start 2024-04-25T02:09:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.839+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.839+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 0x55b1adb3f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.839+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb41be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.839+0000 7f99c81f4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b1ada8fab0 con 0x55b1adb3f7f0 2024-04-25T02:09:23.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.839+0000 7f99c81f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b1ada38640 con 0x55b1adb34610 2024-04-25T02:09:23.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.839+0000 7f99c81f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b1ada8ba70 con 0x55b1adb34f30 2024-04-25T02:09:23.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.839+0000 7f99bf200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.839+0000 7f99bf200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59258/0 (socket says 172.21.15.102:59258) 2024-04-25T02:09:23.847 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.839+0000 7f99bf200640 1 -- 172.21.15.102:0/3847788742 learned_addr learned my addr 172.21.15.102:0/3847788742 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.840+0000 7f99bf200640 1 -- 172.21.15.102:0/3847788742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 msgr2=0x55b1adb3f2b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:09:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.840+0000 7f99bf200640 1 --2- 172.21.15.102:0/3847788742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 0x55b1adb3f2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.840+0000 7f99bf200640 1 -- 172.21.15.102:0/3847788742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 msgr2=0x55b1adb41be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:09:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.843+0000 7f99bde00640 1 --2- 172.21.15.102:0/3847788742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb41be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.843+0000 7f99bf200640 1 --2- 172.21.15.102:0/3847788742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb41be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.843+0000 7f99bf200640 1 -- 172.21.15.102:0/3847788742 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1ad7a4210 con 0x55b1adb34610 2024-04-25T02:09:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.844+0000 7f99bde00640 1 --2- 172.21.15.102:0/3847788742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb41be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:09:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.844+0000 7f99bf200640 1 --2- 172.21.15.102:0/3847788742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb349f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f99b8010090 tx=0x7f99b800d500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de7c5128ed7ae58e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.844+0000 7f99b7400640 1 -- 172.21.15.102:0/1855135891 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f99ac018020 con 0x7f99c01028f0 2024-04-25T02:09:23.851 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.844+0000 7f99ab400640 1 -- 172.21.15.102:0/3847788742 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8043070 con 0x55b1adb34610 2024-04-25T02:09:23.852 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.845+0000 7f99ab400640 1 -- 172.21.15.102:0/3847788742 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b8003e50 con 0x55b1adb34610 2024-04-25T02:09:23.853 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.846+0000 7f99c81f4740 1 -- 172.21.15.102:0/3847788742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 msgr2=0x55b1adb349f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.847+0000 7f99b7400640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a40a2e20 0x7f99a40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.847+0000 7f99b7400640 1 -- 172.21.15.102:0/1855135891 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7f99ac0ced60 con 0x7f99c01028f0 2024-04-25T02:09:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.847+0000 7f99ab400640 1 -- 172.21.15.102:0/3847788742 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8043070 con 0x55b1adb34610 2024-04-25T02:09:23.854 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.848+0000 7f99c81f4740 1 --2- 172.21.15.102:0/3847788742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb349f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f99b8010090 tx=0x7f99b800d500 comp rx=0 tx=0).stop 2024-04-25T02:09:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.848+0000 7f99c81f4740 1 -- 172.21.15.102:0/3847788742 shutdown_connections 2024-04-25T02:09:23.855 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.848+0000 7f99c81f4740 1 --2- 172.21.15.102:0/3847788742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb41be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.848+0000 7f99c81f4740 1 --2- 172.21.15.102:0/3847788742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 0x55b1adb3f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.848+0000 7f99c81f4740 1 --2- 172.21.15.102:0/3847788742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb349f0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.848+0000 7f99c81f4740 1 -- 172.21.15.102:0/3847788742 >> 172.21.15.102:0/3847788742 conn(0x55b1adb30b20 msgr2=0x55b1adb30f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.848+0000 7f99be800640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a40a2e20 0x7f99a40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.849+0000 7f99c81f4740 1 -- 172.21.15.102:0/3847788742 shutdown_connections 2024-04-25T02:09:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.849+0000 7f99be800640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a40a2e20 0x7f99a40a52e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55b1adb35bb0 tx=0x7f99ac010fd0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:23.857 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.849+0000 7f99c81f4740 1 -- 172.21.15.102:0/3847788742 wait complete. 2024-04-25T02:09:23.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.850+0000 7f99c81f4740 1 -- start start 2024-04-25T02:09:23.858 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.850+0000 7f99c81f4740 1 Processor -- start 2024-04-25T02:09:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.853+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb3c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.853+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 0x55b1adb3ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.853+0000 7f99bf200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb3c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.853+0000 7f99bf200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb3c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59266/0 (socket says 172.21.15.102:59266) 2024-04-25T02:09:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.854+0000 7f99bf200640 1 -- 172.21.15.102:0/1773119348 learned_addr learned my addr 172.21.15.102:0/1773119348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.854+0000 7f99c81f4740 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb3d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.854+0000 7f99bde00640 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 0x55b1adb3ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.861 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.854+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55b1adb423c0 con 0x55b1adb3f7f0 2024-04-25T02:09:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.855+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55b1ada4b990 con 0x55b1adb34610 2024-04-25T02:09:23.862 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.855+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b1adb42540 con 0x55b1adb34f30 2024-04-25T02:09:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.855+0000 7f99c81f4740 10 client.0 _ll_register_callbacks cb 0x55b1ada4d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.856+0000 7f99bf200640 1 -- 172.21.15.102:0/1773119348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 msgr2=0x55b1adb3ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.856+0000 7f99bf200640 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 0x55b1adb3ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.857+0000 7f99bf200640 1 -- 172.21.15.102:0/1773119348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 msgr2=0x55b1adb3d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.857+0000 7f99bf200640 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb3d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.857+0000 7f99bf200640 1 -- 172.21.15.102:0/1773119348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1ad870620 con 0x55b1adb34610 2024-04-25T02:09:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.857+0000 7f99bde00640 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 0x55b1adb3ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.858+0000 7f99bf200640 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb3c780 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f99b802ecc0 tx=0x7f99b800d2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8762cbf330eea3a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.858+0000 7f999fe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:09:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.858+0000 7f999fe00640 1 -- 172.21.15.102:0/1773119348 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b804f060 con 0x55b1adb34610 2024-04-25T02:09:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.858+0000 7f999fe00640 1 -- 172.21.15.102:0/1773119348 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7f99b8004000 con 0x55b1adb34610 2024-04-25T02:09:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.858+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b8003e50 con 0x55b1adb34610 2024-04-25T02:09:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.859+0000 7f999fe00640 1 -- 172.21.15.102:0/1773119348 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f99b8043040 con 0x55b1adb34610 2024-04-25T02:09:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.859+0000 7f99c81f4740 10 client.27656 fetch_fsmap learned FSMap version 51 2024-04-25T02:09:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.859+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b1adb4e080 con 0x55b1adb34610 2024-04-25T02:09:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.859+0000 7f999fe00640 1 -- 172.21.15.102:0/1773119348 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b8003e50 con 0x55b1adb34610 2024-04-25T02:09:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.859+0000 7f999fe00640 1 -- 172.21.15.102:0/1773119348 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f99b803e7e0 con 0x55b1adb34610 2024-04-25T02:09:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.860+0000 7f99c81f4740 10 client.27656 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:09:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.860+0000 7f99c81f4740 10 client.27656 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.861+0000 7f99c81f4740 20 client.27656 populate_metadata read hostname 'smithi102' 2024-04-25T02:09:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.861+0000 7f99c81f4740 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b1adb4f3d0 0x55b1adb517c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.861+0000 7f99c81f4740 4 client.27656 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:09:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.861+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1ada9cff0 con 0x55b1adb4f3d0 2024-04-25T02:09:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.861+0000 7f99bde00640 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b1adb4f3d0 0x55b1adb517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.869 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.862+0000 7f99bde00640 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b1adb4f3d0 0x55b1adb517c0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24a71c2b1bb8ce53 server_cookie=bdce46cfebdfb802 in_seq=0 out_seq=0 2024-04-25T02:09:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.864+0000 7f999fe00640 10 client.27656 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.864+0000 7f999fe00640 1 -- 172.21.15.102:0/1773119348 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b1ada9cff0 con 0x55b1adb4f3d0 2024-04-25T02:09:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.864+0000 7f999fe00640 10 client.27656 handle_command_reply: tid=0 2024-04-25T02:09:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.864+0000 7f99c81f4740 1 client.27656 shutdown 2024-04-25T02:09:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.866+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 msgr2=0x55b1adb3c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.866+0000 7f99c81f4740 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb3c780 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f99b802ecc0 tx=0x7f99b800d2e0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.866+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 shutdown_connections 2024-04-25T02:09:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.866+0000 7f99c81f4740 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b1adb4f3d0 0x55b1adb517c0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.866+0000 7f99c81f4740 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb3d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.866+0000 7f99c81f4740 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34f30 0x55b1adb3ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.866+0000 7f99c81f4740 1 --2- 172.21.15.102:0/1773119348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34610 0x55b1adb3c780 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.866+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 >> 172.21.15.102:0/1773119348 conn(0x55b1adb30b20 msgr2=0x55b1adb27480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.866+0000 7f999fe00640 0 client.27656 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.867+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 shutdown_connections 2024-04-25T02:09:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.867+0000 7f99c81f4740 1 -- 172.21.15.102:0/1773119348 wait complete. 2024-04-25T02:09:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.867+0000 7f99c81f4740 20 client.27656 trim_cache size 0 max 16384 2024-04-25T02:09:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.892+0000 7f99c81f4740 1 Processor -- start 2024-04-25T02:09:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.892+0000 7f99c81f4740 1 -- start start 2024-04-25T02:09:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb4b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb34f30 0x55b1adb56e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb3f7f0 0x55b1adb57360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99c81f4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99c01adc20 con 0x55b1adb34f30 2024-04-25T02:09:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99c81f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99c01ab430 con 0x55b1adb3f7f0 2024-04-25T02:09:23.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99c81f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99c01a92a0 con 0x55b1adb34610 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99bf200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb4b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99bf200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb4b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:36834/0 (socket says 172.21.15.102:36834) 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99bf200640 1 -- 172.21.15.102:0/3655350707 learned_addr learned my addr 172.21.15.102:0/3655350707 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.893+0000 7f99bf200640 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb3f7f0 0x55b1adb57360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99bde00640 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb34f30 0x55b1adb56e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99bf200640 1 -- 172.21.15.102:0/3655350707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb3f7f0 msgr2=0x55b1adb57360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99bf200640 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb3f7f0 0x55b1adb57360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99bf200640 1 -- 172.21.15.102:0/3655350707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb34f30 msgr2=0x55b1adb56e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99bf200640 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb34f30 0x55b1adb56e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99bf200640 1 -- 172.21.15.102:0/3655350707 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1adb42540 con 0x55b1adb34610 2024-04-25T02:09:23.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99bde00640 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb34f30 0x55b1adb56e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:09:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99bf200640 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb4b2d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x55b1adb578a0 tx=0x7f99b8040040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d840bb9322b62173 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99aaa00640 1 -- 172.21.15.102:0/3655350707 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b803a030 con 0x55b1adb34610 2024-04-25T02:09:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99aaa00640 1 -- 172.21.15.102:0/3655350707 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b80493e0 con 0x55b1adb34610 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.894+0000 7f99aaa00640 1 -- 172.21.15.102:0/3655350707 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8049a80 con 0x55b1adb34610 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.895+0000 7f99c81f4740 1 -- 172.21.15.102:0/3655350707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 msgr2=0x55b1adb4b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.895+0000 7f99c81f4740 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb4b2d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x55b1adb578a0 tx=0x7f99b8040040 comp rx=0 tx=0).stop 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.895+0000 7f99c81f4740 1 -- 172.21.15.102:0/3655350707 shutdown_connections 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.895+0000 7f99c81f4740 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb3f7f0 0x55b1adb57360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.895+0000 7f99c81f4740 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb34f30 0x55b1adb56e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.895+0000 7f99c81f4740 1 --2- 172.21.15.102:0/3655350707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb4b2d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.895+0000 7f99c81f4740 1 -- 172.21.15.102:0/3655350707 >> 172.21.15.102:0/3655350707 conn(0x55b1adb30b20 msgr2=0x55b1adb3b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.895+0000 7f99c81f4740 1 -- 172.21.15.102:0/3655350707 shutdown_connections 2024-04-25T02:09:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.895+0000 7f99c81f4740 1 -- 172.21.15.102:0/3655350707 wait complete. 2024-04-25T02:09:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.896+0000 7f99c81f4740 1 -- start start 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.896+0000 7f99c81f4740 1 Processor -- start 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb3a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34f30 0x55b1adb477e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99c81f4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb38a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99c81f4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99c01a98f0 con 0x55b1adb3f7f0 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99c81f4740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f99c01aad60 con 0x55b1adb34f30 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99c81f4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99c01a9fc0 con 0x55b1adb34610 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99c81f4740 10 client.0 _ll_register_callbacks cb 0x7f99b8041070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99bde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34f30 0x55b1adb477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99bde00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34f30 0x55b1adb477e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59286/0 (socket says 172.21.15.102:59286) 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99bde00640 1 -- 172.21.15.102:0/830902138 learned_addr learned my addr 172.21.15.102:0/830902138 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99bf200640 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb3a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99bf200640 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb38a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99bde00640 1 -- 172.21.15.102:0/830902138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 msgr2=0x55b1adb3a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99bde00640 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb3a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.897+0000 7f99bde00640 1 -- 172.21.15.102:0/830902138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 msgr2=0x55b1adb38a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.898+0000 7f99bde00640 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb38a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.898+0000 7f99bde00640 1 -- 172.21.15.102:0/830902138 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1adb38f50 con 0x55b1adb34f30 2024-04-25T02:09:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.898+0000 7f99bf200640 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb38a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:09:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.898+0000 7f99bde00640 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34f30 0x55b1adb477e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x55b1adb31f40 tx=0x7f99b001df20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85267cc02ef8a39f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.898+0000 7f999fe00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.898+0000 7f999fe00640 1 -- 172.21.15.102:0/830902138 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b000f040 con 0x55b1adb34f30 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.898+0000 7f999fe00640 1 -- 172.21.15.102:0/830902138 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7f99b0018b60 con 0x55b1adb34f30 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.898+0000 7f99c81f4740 1 -- 172.21.15.102:0/830902138 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b80493e0 con 0x55b1adb34f30 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f999fe00640 1 -- 172.21.15.102:0/830902138 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f99b002d050 con 0x55b1adb34f30 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f99c81f4740 10 client.27662 fetch_fsmap learned FSMap version 51 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f99c81f4740 1 -- 172.21.15.102:0/830902138 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x55b1adbb4de0 con 0x55b1adb34f30 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f999fe00640 1 -- 172.21.15.102:0/830902138 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b80493e0 con 0x55b1adb34f30 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f999fe00640 1 -- 172.21.15.102:0/830902138 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f99b0016640 con 0x55b1adb34f30 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f99c81f4740 10 client.27662 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f99c81f4740 10 client.27662 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f99c81f4740 20 client.27662 populate_metadata read hostname 'smithi102' 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f99c81f4740 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b1adbb5590 0x55b1adbb79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:23.908 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f99c81f4740 4 client.27662 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:09:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f99c81f4740 1 -- 172.21.15.102:0/830902138 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b1ada97340 con 0x55b1adbb5590 2024-04-25T02:09:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.899+0000 7f99bf200640 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b1adbb5590 0x55b1adbb79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.900+0000 7f99bf200640 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b1adbb5590 0x55b1adbb79b0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7460296758fdae09 server_cookie=1b14d25d7bab04f5 in_seq=0 out_seq=0 2024-04-25T02:09:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.900+0000 7f999fe00640 10 client.27662 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f999fe00640 1 -- 172.21.15.102:0/830902138 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b1ada97340 con 0x55b1adbb5590 2024-04-25T02:09:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f999fe00640 10 client.27662 handle_command_reply: tid=0 2024-04-25T02:09:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 client.27662 shutdown 2024-04-25T02:09:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 -- 172.21.15.102:0/830902138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34f30 msgr2=0x55b1adb477e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34f30 0x55b1adb477e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x55b1adb31f40 tx=0x7f99b001df20 comp rx=0 tx=0).stop 2024-04-25T02:09:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 -- 172.21.15.102:0/830902138 shutdown_connections 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55b1adbb5590 0x55b1adbb79b0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b1adb3f7f0 0x55b1adb38a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55b1adb34f30 0x55b1adb477e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 --2- 172.21.15.102:0/830902138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b1adb34610 0x55b1adb3a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 -- 172.21.15.102:0/830902138 >> 172.21.15.102:0/830902138 conn(0x55b1adb30b20 msgr2=0x55b1adb3c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f999fe00640 0 client.27662 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 -- 172.21.15.102:0/830902138 shutdown_connections 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 1 -- 172.21.15.102:0/830902138 wait complete. 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.918+0000 7f99c81f4740 20 client.27662 trim_cache size 0 max 16384 2024-04-25T02:09:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:09:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:09:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:09:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:09:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 -- 172.21.15.102:0/1855135891 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a40a2e20 msgr2=0x7f99a40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a40a2e20 0x7f99a40a52e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x55b1adb35bb0 tx=0x7f99ac010fd0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 -- 172.21.15.102:0/1855135891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c01028f0 msgr2=0x7f99c0195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c01028f0 0x7f99c0195ec0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00b450 tx=0x7f99ac00b910 comp rx=0 tx=0).stop 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 -- 172.21.15.102:0/1855135891 shutdown_connections 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f99a40a2e20 0x7f99a40a52e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99c010bd10 0x7f99c019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99c0103210 0x7f99c0196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 --2- 172.21.15.102:0/1855135891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99c01028f0 0x7f99c0195ec0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.919+0000 7f99c5400640 1 -- 172.21.15.102:0/1855135891 >> 172.21.15.102:0/1855135891 conn(0x7f99c00fc110 msgr2=0x7f99c00fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.920+0000 7f99c5400640 1 -- 172.21.15.102:0/1855135891 shutdown_connections 2024-04-25T02:09:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:23.920+0000 7f99c5400640 1 -- 172.21.15.102:0/1855135891 wait complete. 2024-04-25T02:09:23.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:23 smithi102 ceph-mon[27796]: pgmap v440: 97 pgs: 97 active+clean; 5.4 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 7.0 MiB/s wr, 631 op/s 2024-04-25T02:09:25.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:25 smithi110 ceph-mon[32848]: pgmap v441: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.7 MiB/s wr, 604 op/s 2024-04-25T02:09:25.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:25 smithi193 ceph-mon[32981]: pgmap v441: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.7 MiB/s wr, 604 op/s 2024-04-25T02:09:25.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:25 smithi102 ceph-mon[27796]: pgmap v441: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.7 MiB/s wr, 604 op/s 2024-04-25T02:09:27.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:27 smithi110 ceph-mon[32848]: pgmap v442: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.9 MiB/s wr, 600 op/s 2024-04-25T02:09:27.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:27 smithi193 ceph-mon[32981]: pgmap v442: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.9 MiB/s wr, 600 op/s 2024-04-25T02:09:27.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:27 smithi102 ceph-mon[27796]: pgmap v442: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.9 MiB/s wr, 600 op/s 2024-04-25T02:09:28.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 815.0 s, 1734.0 tps, lat 58.211 ms stddev 73.815 2024-04-25T02:09:29.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:29 smithi110 ceph-mon[32848]: pgmap v443: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 8.1 MiB/s wr, 701 op/s 2024-04-25T02:09:29.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:29 smithi193 ceph-mon[32981]: pgmap v443: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 8.1 MiB/s wr, 701 op/s 2024-04-25T02:09:29.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:29 smithi102 ceph-mon[27796]: pgmap v443: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 8.1 MiB/s wr, 701 op/s 2024-04-25T02:09:31.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:31 smithi110 ceph-mon[32848]: pgmap v444: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.9 MiB/s wr, 630 op/s 2024-04-25T02:09:31.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:31 smithi193 ceph-mon[32981]: pgmap v444: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.9 MiB/s wr, 630 op/s 2024-04-25T02:09:31.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:31 smithi102 ceph-mon[27796]: pgmap v444: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.9 MiB/s wr, 630 op/s 2024-04-25T02:09:33.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 820.0 s, 1772.4 tps, lat 56.368 ms stddev 64.661 2024-04-25T02:09:33.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:33 smithi110 ceph-mon[32848]: pgmap v445: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.4 MiB/s wr, 562 op/s 2024-04-25T02:09:33.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:33 smithi193 ceph-mon[32981]: pgmap v445: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.4 MiB/s wr, 562 op/s 2024-04-25T02:09:33.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:33 smithi102 ceph-mon[27796]: pgmap v445: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.4 MiB/s wr, 562 op/s 2024-04-25T02:09:35.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:35 smithi110 ceph-mon[32848]: pgmap v446: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 8.7 MiB/s wr, 775 op/s 2024-04-25T02:09:35.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:35 smithi193 ceph-mon[32981]: pgmap v446: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 8.7 MiB/s wr, 775 op/s 2024-04-25T02:09:35.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:35 smithi102 ceph-mon[27796]: pgmap v446: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 8.7 MiB/s wr, 775 op/s 2024-04-25T02:09:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.098+0000 7f785e600640 1 -- 172.21.15.102:0/1811902724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 msgr2=0x7f78580ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.098+0000 7f785e600640 1 --2- 172.21.15.102:0/1811902724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 0x7f78580ffe60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7840009930 tx=0x7f784002a420 comp rx=0 tx=0).stop 2024-04-25T02:09:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.098+0000 7f785e600640 1 -- 172.21.15.102:0/1811902724 shutdown_connections 2024-04-25T02:09:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.098+0000 7f785e600640 1 --2- 172.21.15.102:0/1811902724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7858109f90 0x7f7858110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.105 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.098+0000 7f785e600640 1 --2- 172.21.15.102:0/1811902724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7858100430 0x7f7858109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.098+0000 7f785e600640 1 --2- 172.21.15.102:0/1811902724 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 0x7f78580ffe60 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.106 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.098+0000 7f785e600640 1 -- 172.21.15.102:0/1811902724 >> 172.21.15.102:0/1811902724 conn(0x7f7858076740 msgr2=0x7f78580fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.098+0000 7f785e600640 1 -- 172.21.15.102:0/1811902724 shutdown_connections 2024-04-25T02:09:37.107 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.100+0000 7f785e600640 1 -- 172.21.15.102:0/1811902724 wait complete. 2024-04-25T02:09:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.101+0000 7f785e600640 1 Processor -- start 2024-04-25T02:09:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.101+0000 7f785e600640 1 -- start start 2024-04-25T02:09:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.102+0000 7f785e600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 0x7f7858072f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.102+0000 7f785e600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7858109f90 0x7f785806d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.102+0000 7f785e600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785806d540 0x7f785806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.109 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.102+0000 7f785e600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f785806bb70 con 0x7f7858109f90 2024-04-25T02:09:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.102+0000 7f785e600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f785806b9f0 con 0x7f78580ffa80 2024-04-25T02:09:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.102+0000 7f785e600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f785806bcf0 con 0x7f785806d540 2024-04-25T02:09:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.102+0000 7f7857400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 0x7f7858072f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.102+0000 7f7857400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 0x7f7858072f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53444/0 (socket says 172.21.15.102:53444) 2024-04-25T02:09:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.102+0000 7f7857400640 1 -- 172.21.15.102:0/3563812786 learned_addr learned my addr 172.21.15.102:0/3563812786 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.103+0000 7f7857e00640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785806d540 0x7f785806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.103+0000 7f7856a00640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7858109f90 0x7f785806d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.103+0000 7f7857400640 1 -- 172.21.15.102:0/3563812786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785806d540 msgr2=0x7f785806d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.103+0000 7f7857400640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785806d540 0x7f785806d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.103+0000 7f7857400640 1 -- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7858109f90 msgr2=0x7f785806d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.103+0000 7f7857400640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7858109f90 0x7f785806d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.103+0000 7f7857400640 1 -- 172.21.15.102:0/3563812786 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78581af100 con 0x7f78580ffa80 2024-04-25T02:09:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.104+0000 7f7857e00640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785806d540 0x7f785806d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:09:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.104+0000 7f7857400640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 0x7f7858072f90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f784002a940 tx=0x7f7840003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.105+0000 7f784fe00640 1 -- 172.21.15.102:0/3563812786 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784003f070 con 0x7f78580ffa80 2024-04-25T02:09:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.105+0000 7f784fe00640 1 -- 172.21.15.102:0/3563812786 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7840031c90 con 0x7f78580ffa80 2024-04-25T02:09:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.105+0000 7f784fe00640 1 -- 172.21.15.102:0/3563812786 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7840035030 con 0x7f78580ffa80 2024-04-25T02:09:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.105+0000 7f785e600640 1 -- 172.21.15.102:0/3563812786 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78581af330 con 0x7f78580ffa80 2024-04-25T02:09:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.105+0000 7f785e600640 1 -- 172.21.15.102:0/3563812786 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78581af7f0 con 0x7f78580ffa80 2024-04-25T02:09:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.110+0000 7f786139b740 1 Processor -- start 2024-04-25T02:09:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.111+0000 7f786139b740 1 -- start start 2024-04-25T02:09:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.111+0000 7f786139b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdc8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.111+0000 7f786139b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdc94a0 0x559aefdc9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.111+0000 7f786139b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 0x559aefdd6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.111+0000 7f786139b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559aefd2cab0 con 0x559aefdc94a0 2024-04-25T02:09:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.111+0000 7f786139b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559aefcd5640 con 0x559aefdd3da0 2024-04-25T02:09:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.111+0000 7f786139b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559aefd28a70 con 0x559aefdca690 2024-04-25T02:09:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.112+0000 7f7856a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 0x559aefdd6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.112+0000 7f7856a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 0x559aefdd6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53454/0 (socket says 172.21.15.102:53454) 2024-04-25T02:09:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.112+0000 7f7856a00640 1 -- 172.21.15.102:0/2472092214 learned_addr learned my addr 172.21.15.102:0/2472092214 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.115+0000 7f7856a00640 1 -- 172.21.15.102:0/2472092214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 msgr2=0x559aefdc8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.114+0000 7f7857e00640 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdc8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.117+0000 7f7856a00640 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdc8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.117+0000 7f784fe00640 1 -- 172.21.15.102:0/3563812786 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7840042070 con 0x7f78580ffa80 2024-04-25T02:09:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.117+0000 7f7856a00640 1 -- 172.21.15.102:0/2472092214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdc94a0 msgr2=0x559aefdc9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.117+0000 7f7857400640 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdc94a0 0x559aefdc9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.118+0000 7f7856a00640 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdc94a0 0x559aefdc9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.118+0000 7f7856a00640 1 -- 172.21.15.102:0/2472092214 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aefa41210 con 0x559aefdd3da0 2024-04-25T02:09:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.118+0000 7f7857e00640 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdc8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:09:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.119+0000 7f7856a00640 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 0x559aefdd6190 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f784800cb10 tx=0x7f784800e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1753b596141e98f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.119+0000 7f7857400640 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdc94a0 0x559aefdc9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:09:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.119+0000 7f7847e00640 1 -- 172.21.15.102:0/2472092214 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848004020 con 0x559aefdd3da0 2024-04-25T02:09:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.119+0000 7f7847e00640 1 -- 172.21.15.102:0/2472092214 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78480041c0 con 0x559aefdd3da0 2024-04-25T02:09:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.119+0000 7f7847e00640 1 -- 172.21.15.102:0/2472092214 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848004020 con 0x559aefdd3da0 2024-04-25T02:09:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.120+0000 7f784fe00640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f783c0a2ea0 0x7f783c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.120+0000 7f786139b740 1 -- 172.21.15.102:0/2472092214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 msgr2=0x559aefdd6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.120+0000 7f786139b740 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 0x559aefdd6190 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f784800cb10 tx=0x7f784800e350 comp rx=0 tx=0).stop 2024-04-25T02:09:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.123+0000 7f786139b740 1 -- 172.21.15.102:0/2472092214 shutdown_connections 2024-04-25T02:09:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.123+0000 7f786139b740 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 0x559aefdd6190 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.123+0000 7f786139b740 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdc94a0 0x559aefdc9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.123+0000 7f786139b740 1 --2- 172.21.15.102:0/2472092214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdc8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.123+0000 7f786139b740 1 -- 172.21.15.102:0/2472092214 >> 172.21.15.102:0/2472092214 conn(0x559aefdc5610 msgr2=0x559aefdc59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.123+0000 7f786139b740 1 -- 172.21.15.102:0/2472092214 shutdown_connections 2024-04-25T02:09:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.123+0000 7f786139b740 1 -- 172.21.15.102:0/2472092214 wait complete. 2024-04-25T02:09:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.123+0000 7f784fe00640 1 -- 172.21.15.102:0/3563812786 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7f78400efbc0 con 0x7f78580ffa80 2024-04-25T02:09:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.124+0000 7f786139b740 1 -- start start 2024-04-25T02:09:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.124+0000 7f786139b740 1 Processor -- start 2024-04-25T02:09:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.124+0000 7f7857e00640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f783c0a2ea0 0x7f783c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.125+0000 7f786139b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdc94a0 0x559aefdd0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.125+0000 7f786139b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdca690 0x559aefdd1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.125+0000 7f786139b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 0x559aefdd16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.125+0000 7f786139b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559aefdd6be0 con 0x559aefdca690 2024-04-25T02:09:37.133 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.125+0000 7f786139b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559aefce8990 con 0x559aefdd3da0 2024-04-25T02:09:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.125+0000 7f786139b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559aefdd6d60 con 0x559aefdc94a0 2024-04-25T02:09:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.125+0000 7f786139b740 10 client.0 _ll_register_callbacks cb 0x559aefcea950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.126+0000 7f7857400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdc94a0 0x559aefdd0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.126+0000 7f7857400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdc94a0 0x559aefdd0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41872/0 (socket says 172.21.15.102:41872) 2024-04-25T02:09:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.126+0000 7f7857400640 1 -- 172.21.15.102:0/2790117728 learned_addr learned my addr 172.21.15.102:0/2790117728 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.127+0000 7f7857400640 1 -- 172.21.15.102:0/2790117728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 msgr2=0x559aefdd16a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:09:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.127+0000 7f7857400640 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 0x559aefdd16a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.127+0000 7f7857400640 1 -- 172.21.15.102:0/2790117728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdca690 msgr2=0x559aefdd1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.127+0000 7f7856a00640 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdca690 0x559aefdd1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.128+0000 7f7857e00640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f783c0a2ea0 0x7f783c0a5360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7850005fd0 tx=0x7f7850009600 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.128+0000 7f7857400640 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdca690 0x559aefdd1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.128+0000 7f7857400640 1 -- 172.21.15.102:0/2790117728 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aefb0d620 con 0x559aefdc94a0 2024-04-25T02:09:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.128+0000 7f7856a00640 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdca690 0x559aefdd1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:09:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.129+0000 7f7857400640 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdc94a0 0x559aefdd0c20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f78400b84e0 tx=0x7f78400b83e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=800dc91991e67cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.129+0000 7f7845600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:09:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.130+0000 7f7845600640 1 -- 172.21.15.102:0/2790117728 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7840049050 con 0x559aefdc94a0 2024-04-25T02:09:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.130+0000 7f7845600640 1 -- 172.21.15.102:0/2790117728 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7f78400ed530 con 0x559aefdc94a0 2024-04-25T02:09:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.132+0000 7f786139b740 1 -- 172.21.15.102:0/2790117728 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78480041c0 con 0x559aefdc94a0 2024-04-25T02:09:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.133+0000 7f7845600640 1 -- 172.21.15.102:0/2790117728 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f78400bc780 con 0x559aefdc94a0 2024-04-25T02:09:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.133+0000 7f786139b740 10 client.27613 fetch_fsmap learned FSMap version 51 2024-04-25T02:09:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.133+0000 7f786139b740 1 -- 172.21.15.102:0/2790117728 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x559aefdd6be0 con 0x559aefdc94a0 2024-04-25T02:09:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.133+0000 7f7845600640 1 -- 172.21.15.102:0/2790117728 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78480041c0 con 0x559aefdc94a0 2024-04-25T02:09:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.135+0000 7f7845600640 1 -- 172.21.15.102:0/2790117728 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f784003b5f0 con 0x559aefdc94a0 2024-04-25T02:09:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.135+0000 7f786139b740 10 client.27613 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:09:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.135+0000 7f786139b740 10 client.27613 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.135+0000 7f786139b740 20 client.27613 populate_metadata read hostname 'smithi102' 2024-04-25T02:09:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.135+0000 7f786139b740 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559aefddab90 0x559aefddd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.135+0000 7f786139b740 4 client.27613 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:09:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.135+0000 7f786139b740 1 -- 172.21.15.102:0/2790117728 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559aefd39ff0 con 0x559aefddab90 2024-04-25T02:09:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.135+0000 7f7856a00640 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559aefddab90 0x559aefddd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.136+0000 7f7845600640 10 client.27613 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.136+0000 7f7856a00640 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559aefddab90 0x559aefddd030 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=942bcbec3e055b65 server_cookie=2d5f9a488996644c in_seq=0 out_seq=0 2024-04-25T02:09:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.137+0000 7f7845600640 1 -- 172.21.15.102:0/2790117728 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559aefd39ff0 con 0x559aefddab90 2024-04-25T02:09:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.137+0000 7f7845600640 10 client.27613 handle_command_reply: tid=0 2024-04-25T02:09:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.137+0000 7f786139b740 1 client.27613 shutdown 2024-04-25T02:09:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.138+0000 7f786139b740 1 -- 172.21.15.102:0/2790117728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdc94a0 msgr2=0x559aefdd0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.138+0000 7f786139b740 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdc94a0 0x559aefdd0c20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f78400b84e0 tx=0x7f78400b83e0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f786139b740 1 -- 172.21.15.102:0/2790117728 shutdown_connections 2024-04-25T02:09:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f786139b740 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559aefddab90 0x559aefddd030 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f7845600640 0 client.27613 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f786139b740 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdd3da0 0x559aefdd16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f786139b740 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdca690 0x559aefdd1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f786139b740 1 --2- 172.21.15.102:0/2790117728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdc94a0 0x559aefdd0c20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f786139b740 1 -- 172.21.15.102:0/2790117728 >> 172.21.15.102:0/2790117728 conn(0x559aefdc5610 msgr2=0x559aefdc48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f786139b740 1 -- 172.21.15.102:0/2790117728 shutdown_connections 2024-04-25T02:09:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f786139b740 1 -- 172.21.15.102:0/2790117728 wait complete. 2024-04-25T02:09:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.139+0000 7f786139b740 20 client.27613 trim_cache size 0 max 16384 2024-04-25T02:09:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.166+0000 7f786139b740 1 Processor -- start 2024-04-25T02:09:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.166+0000 7f786139b740 1 -- start start 2024-04-25T02:09:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f786139b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdfe900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f786139b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdfee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f786139b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdd3da0 0x559aefdd2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f786139b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f78581b6130 con 0x559aefdd3da0 2024-04-25T02:09:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f786139b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f78581b3940 con 0x559aefdc94a0 2024-04-25T02:09:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f786139b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f78581b1860 con 0x559aefdca690 2024-04-25T02:09:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f7856a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdfee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f7857400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdfe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f7856a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdfee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:41884/0 (socket says 172.21.15.102:41884) 2024-04-25T02:09:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.167+0000 7f7856a00640 1 -- 172.21.15.102:0/302448535 learned_addr learned my addr 172.21.15.102:0/302448535 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.168+0000 7f7856a00640 1 -- 172.21.15.102:0/302448535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 msgr2=0x559aefdfe900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.168+0000 7f7856a00640 1 --2- 172.21.15.102:0/302448535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdfe900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.168+0000 7f7857e00640 1 --2- 172.21.15.102:0/302448535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdd3da0 0x559aefdd2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.168+0000 7f7856a00640 1 -- 172.21.15.102:0/302448535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdd3da0 msgr2=0x559aefdd2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:37.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.168+0000 7f7856a00640 1 --2- 172.21.15.102:0/302448535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdd3da0 0x559aefdd2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.168+0000 7f7856a00640 1 -- 172.21.15.102:0/302448535 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aefdc5440 con 0x559aefdca690 2024-04-25T02:09:37.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.168+0000 7f7856a00640 1 --2- 172.21.15.102:0/302448535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdfee40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f784800c8d0 tx=0x7f7848003e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffe6570a57236129 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:37.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.169+0000 7f785e600640 1 -- 172.21.15.102:0/302448535 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78480040d0 con 0x559aefdca690 2024-04-25T02:09:37.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.169+0000 7f785e600640 1 -- 172.21.15.102:0/302448535 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848004270 con 0x559aefdca690 2024-04-25T02:09:37.176 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.169+0000 7f785e600640 1 -- 172.21.15.102:0/302448535 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848003bf0 con 0x559aefdca690 2024-04-25T02:09:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.170+0000 7f786139b740 1 -- 172.21.15.102:0/302448535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 msgr2=0x559aefdfee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.170+0000 7f786139b740 1 --2- 172.21.15.102:0/302448535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdfee40 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f784800c8d0 tx=0x7f7848003e60 comp rx=0 tx=0).stop 2024-04-25T02:09:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.170+0000 7f786139b740 1 -- 172.21.15.102:0/302448535 shutdown_connections 2024-04-25T02:09:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.170+0000 7f786139b740 1 --2- 172.21.15.102:0/302448535 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdd3da0 0x559aefdd2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.177 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.170+0000 7f786139b740 1 --2- 172.21.15.102:0/302448535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdfee40 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.170+0000 7f786139b740 1 --2- 172.21.15.102:0/302448535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdfe900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.170+0000 7f786139b740 1 -- 172.21.15.102:0/302448535 >> 172.21.15.102:0/302448535 conn(0x559aefdc5610 msgr2=0x559aefdc59f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.170+0000 7f786139b740 1 -- 172.21.15.102:0/302448535 shutdown_connections 2024-04-25T02:09:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.170+0000 7f786139b740 1 -- 172.21.15.102:0/302448535 wait complete. 2024-04-25T02:09:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.171+0000 7f786139b740 1 -- start start 2024-04-25T02:09:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.171+0000 7f786139b740 1 Processor -- start 2024-04-25T02:09:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.171+0000 7f786139b740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdcd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.178 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.171+0000 7f786139b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdcddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.171+0000 7f786139b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdd3da0 0x559aefdce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.171+0000 7f786139b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f78581b1e20 con 0x559aefdd3da0 2024-04-25T02:09:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.171+0000 7f786139b740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f78581b3270 con 0x559aefdc94a0 2024-04-25T02:09:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.171+0000 7f786139b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f78581b24d0 con 0x559aefdca690 2024-04-25T02:09:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f786139b740 10 client.0 _ll_register_callbacks cb 0x7f7840009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7857400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdcd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.180 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7857400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdcd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:53476/0 (socket says 172.21.15.102:53476) 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7857400640 1 -- 172.21.15.102:0/2105576509 learned_addr learned my addr 172.21.15.102:0/2105576509 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7856a00640 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdcddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7857400640 1 -- 172.21.15.102:0/2105576509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 msgr2=0x559aefdcddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7857400640 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdcddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7857400640 1 -- 172.21.15.102:0/2105576509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdd3da0 msgr2=0x559aefdce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7857400640 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdd3da0 0x559aefdce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7857400640 1 -- 172.21.15.102:0/2105576509 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aefe5a160 con 0x559aefdc94a0 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.172+0000 7f7857400640 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdcd870 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f78400bbd80 tx=0x7f784003b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70b822a6cefa8aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.173+0000 7f7845600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.173+0000 7f7845600640 1 -- 172.21.15.102:0/2105576509 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7840049040 con 0x559aefdc94a0 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.173+0000 7f7845600640 1 -- 172.21.15.102:0/2105576509 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7f78400bb6d0 con 0x559aefdc94a0 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.173+0000 7f786139b740 1 -- 172.21.15.102:0/2105576509 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7848004270 con 0x559aefdc94a0 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.173+0000 7f7845600640 1 -- 172.21.15.102:0/2105576509 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f78400bbbc0 con 0x559aefdc94a0 2024-04-25T02:09:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.173+0000 7f786139b740 10 client.27677 fetch_fsmap learned FSMap version 51 2024-04-25T02:09:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.173+0000 7f786139b740 1 -- 172.21.15.102:0/2105576509 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) -- 0x559aefe5b330 con 0x559aefdc94a0 2024-04-25T02:09:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.173+0000 7f7845600640 1 -- 172.21.15.102:0/2105576509 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848004270 con 0x559aefdc94a0 2024-04-25T02:09:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.174+0000 7f7845600640 1 -- 172.21.15.102:0/2105576509 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 51) ==== 8354+0+0 (secure 0 0 0) 0x7f78400b7dd0 con 0x559aefdc94a0 2024-04-25T02:09:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.174+0000 7f786139b740 10 client.27677 fetch_fsmap finished waiting for FSMap version 51 2024-04-25T02:09:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.174+0000 7f786139b740 10 client.27677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.174+0000 7f786139b740 20 client.27677 populate_metadata read hostname 'smithi102' 2024-04-25T02:09:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.174+0000 7f786139b740 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559aefddab90 0x559aefe5d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.174+0000 7f786139b740 4 client.27677 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:09:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.174+0000 7f786139b740 1 -- 172.21.15.102:0/2105576509 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559aefd34340 con 0x559aefddab90 2024-04-25T02:09:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.175+0000 7f7856a00640 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559aefddab90 0x559aefe5d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.175+0000 7f7856a00640 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559aefddab90 0x559aefe5d890 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a8a82bc2cf0ad47 server_cookie=4ed0e69bae6592e4 in_seq=0 out_seq=0 2024-04-25T02:09:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.175+0000 7f7845600640 10 client.27677 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.196+0000 7f7845600640 1 -- 172.21.15.102:0/2105576509 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559aefd34340 con 0x559aefddab90 2024-04-25T02:09:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.196+0000 7f7845600640 10 client.27677 handle_command_reply: tid=0 2024-04-25T02:09:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.196+0000 7f786139b740 1 client.27677 shutdown 2024-04-25T02:09:37.203 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.196+0000 7f786139b740 1 -- 172.21.15.102:0/2105576509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 msgr2=0x559aefdcd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.196+0000 7f786139b740 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdcd870 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f78400bbd80 tx=0x7f784003b990 comp rx=0 tx=0).stop 2024-04-25T02:09:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f786139b740 1 -- 172.21.15.102:0/2105576509 shutdown_connections 2024-04-25T02:09:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f786139b740 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559aefddab90 0x559aefe5d890 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f786139b740 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559aefdd3da0 0x559aefdce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f786139b740 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aefdca690 0x559aefdcddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f786139b740 1 --2- 172.21.15.102:0/2105576509 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559aefdc94a0 0x559aefdcd870 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f786139b740 1 -- 172.21.15.102:0/2105576509 >> 172.21.15.102:0/2105576509 conn(0x559aefdc5610 msgr2=0x559aefdd2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:37.204 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f7845600640 0 client.27677 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f786139b740 1 -- 172.21.15.102:0/2105576509 shutdown_connections 2024-04-25T02:09:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f786139b740 1 -- 172.21.15.102:0/2105576509 wait complete. 2024-04-25T02:09:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.197+0000 7f786139b740 20 client.27677 trim_cache size 0 max 16384 2024-04-25T02:09:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:09:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:09:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:09:37.205 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:09:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.198+0000 7f784d600640 1 -- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f783c0a2ea0 msgr2=0x7f783c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.198+0000 7f784d600640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f783c0a2ea0 0x7f783c0a5360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7850005fd0 tx=0x7f7850009600 comp rx=0 tx=0).stop 2024-04-25T02:09:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.198+0000 7f784d600640 1 -- 172.21.15.102:0/3563812786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 msgr2=0x7f7858072f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.198+0000 7f784d600640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 0x7f7858072f90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f784002a940 tx=0x7f7840003b90 comp rx=0 tx=0).stop 2024-04-25T02:09:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.199+0000 7f784d600640 1 -- 172.21.15.102:0/3563812786 shutdown_connections 2024-04-25T02:09:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.199+0000 7f784d600640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f783c0a2ea0 0x7f783c0a5360 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.199+0000 7f784d600640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f785806d540 0x7f785806d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.199+0000 7f784d600640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7858109f90 0x7f785806d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.199+0000 7f784d600640 1 --2- 172.21.15.102:0/3563812786 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78580ffa80 0x7f7858072f90 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:37.207 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.199+0000 7f784d600640 1 -- 172.21.15.102:0/3563812786 >> 172.21.15.102:0/3563812786 conn(0x7f7858076740 msgr2=0x7f78581086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.201+0000 7f784d600640 1 -- 172.21.15.102:0/3563812786 shutdown_connections 2024-04-25T02:09:37.208 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:37.201+0000 7f784d600640 1 -- 172.21.15.102:0/3563812786 wait complete. 2024-04-25T02:09:37.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:37 smithi193 ceph-mon[32981]: pgmap v447: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.8 MiB/s wr, 677 op/s 2024-04-25T02:09:37.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:37 smithi102 ceph-mon[27796]: pgmap v447: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.8 MiB/s wr, 677 op/s 2024-04-25T02:09:37.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:37 smithi110 ceph-mon[32848]: pgmap v447: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.8 MiB/s wr, 677 op/s 2024-04-25T02:09:38.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 825.0 s, 1656.2 tps, lat 59.938 ms stddev 67.135 2024-04-25T02:09:38.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:38 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:38.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:38 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:38.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:38 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:39.895 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:39 smithi110 ceph-mon[32848]: pgmap v448: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 8.6 MiB/s wr, 780 op/s 2024-04-25T02:09:39.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:39 smithi102 ceph-mon[27796]: pgmap v448: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 8.6 MiB/s wr, 780 op/s 2024-04-25T02:09:40.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:39 smithi193 ceph-mon[32981]: pgmap v448: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 8.6 MiB/s wr, 780 op/s 2024-04-25T02:09:41.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:41 smithi102 ceph-mon[27796]: pgmap v449: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.2 MiB/s wr, 650 op/s 2024-04-25T02:09:41.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:41 smithi110 ceph-mon[32848]: pgmap v449: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.2 MiB/s wr, 650 op/s 2024-04-25T02:09:42.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:41 smithi193 ceph-mon[32981]: pgmap v449: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.2 MiB/s wr, 650 op/s 2024-04-25T02:09:42.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:42 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:42.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:42 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:43.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:42 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:09:43.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 830.0 s, 1342.2 tps, lat 75.019 ms stddev 86.690 2024-04-25T02:09:43.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:43 smithi102 ceph-mon[27796]: pgmap v450: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 6.5 MiB/s wr, 571 op/s 2024-04-25T02:09:43.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:43 smithi110 ceph-mon[32848]: pgmap v450: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 6.5 MiB/s wr, 571 op/s 2024-04-25T02:09:44.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:43 smithi193 ceph-mon[32981]: pgmap v450: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 6.5 MiB/s wr, 571 op/s 2024-04-25T02:09:45.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:45 smithi102 ceph-mon[27796]: pgmap v451: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 8.6 MiB/s wr, 782 op/s 2024-04-25T02:09:45.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:45 smithi110 ceph-mon[32848]: pgmap v451: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 8.6 MiB/s wr, 782 op/s 2024-04-25T02:09:46.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:45 smithi193 ceph-mon[32981]: pgmap v451: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 8.6 MiB/s wr, 782 op/s 2024-04-25T02:09:46.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:46 smithi102 ceph-mon[27796]: pgmap v452: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.2 MiB/s wr, 655 op/s 2024-04-25T02:09:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:46 smithi110 ceph-mon[32848]: pgmap v452: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.2 MiB/s wr, 655 op/s 2024-04-25T02:09:47.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:46 smithi193 ceph-mon[32981]: pgmap v452: 97 pgs: 97 active+clean; 5.5 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.2 MiB/s wr, 655 op/s 2024-04-25T02:09:48.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 835.0 s, 1468.4 tps, lat 67.250 ms stddev 80.930 2024-04-25T02:09:49.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:49 smithi102 ceph-mon[27796]: pgmap v453: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 7.8 MiB/s wr, 739 op/s 2024-04-25T02:09:49.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:49 smithi110 ceph-mon[32848]: pgmap v453: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 7.8 MiB/s wr, 739 op/s 2024-04-25T02:09:49.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:49 smithi193 ceph-mon[32981]: pgmap v453: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 7.8 MiB/s wr, 739 op/s 2024-04-25T02:09:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.377+0000 7fb5d9a00640 1 -- 172.21.15.102:0/2561087443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc0969c0 msgr2=0x7fb5cc0a2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.377+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/2561087443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc0969c0 0x7fb5cc0a2bb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4067720 tx=0x7fb5d4096140 comp rx=0 tx=0).stop 2024-04-25T02:09:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.377+0000 7fb5d9a00640 1 -- 172.21.15.102:0/2561087443 shutdown_connections 2024-04-25T02:09:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.377+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/2561087443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc0969c0 0x7fb5cc0a2bb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.377+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/2561087443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5cc096000 0x7fb5cc096480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.377+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/2561087443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5cc09ad20 0x7fb5cc09b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.377+0000 7fb5d9a00640 1 -- 172.21.15.102:0/2561087443 >> 172.21.15.102:0/2561087443 conn(0x7fb5cc00b830 msgr2=0x7fb5cc00bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.377+0000 7fb5d9a00640 1 -- 172.21.15.102:0/2561087443 shutdown_connections 2024-04-25T02:09:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.377+0000 7fb5d9a00640 1 -- 172.21.15.102:0/2561087443 wait complete. 2024-04-25T02:09:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.379+0000 7fb5d9a00640 1 Processor -- start 2024-04-25T02:09:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.379+0000 7fb5d9a00640 1 -- start start 2024-04-25T02:09:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.379+0000 7fb5d9a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5cc096000 0x7fb5cc138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.379+0000 7fb5d9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5cc0969c0 0x7fb5cc138c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.379+0000 7fb5d9a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc09ad20 0x7fb5cc0a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.379+0000 7fb5d9a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc0a6060 con 0x7fb5cc0969c0 2024-04-25T02:09:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.379+0000 7fb5d9a00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc0a5ee0 con 0x7fb5cc09ad20 2024-04-25T02:09:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.379+0000 7fb5d9a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc0a61e0 con 0x7fb5cc096000 2024-04-25T02:09:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5cc096000 0x7fb5cc138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5cc096000 0x7fb5cc138710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:43602/0 (socket says 172.21.15.102:43602) 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d3e00640 1 -- 172.21.15.102:0/56818452 learned_addr learned my addr 172.21.15.102:0/56818452 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d9000640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc09ad20 0x7fb5cc0a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d3400640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5cc0969c0 0x7fb5cc138c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d9000640 1 -- 172.21.15.102:0/56818452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5cc096000 msgr2=0x7fb5cc138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d9000640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5cc096000 0x7fb5cc138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d9000640 1 -- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5cc0969c0 msgr2=0x7fb5cc138c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d9000640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5cc0969c0 0x7fb5cc138c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d9000640 1 -- 172.21.15.102:0/56818452 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc0a2450 con 0x7fb5cc09ad20 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d3400640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5cc0969c0 0x7fb5cc138c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.380+0000 7fb5d9000640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc09ad20 0x7fb5cc0a1cd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4050ce0 tx=0x7fb5d409deb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.381+0000 7fb5d0c00640 1 -- 172.21.15.102:0/56818452 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40ab020 con 0x7fb5cc09ad20 2024-04-25T02:09:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.381+0000 7fb5d0c00640 1 -- 172.21.15.102:0/56818452 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5d40a7d30 con 0x7fb5cc09ad20 2024-04-25T02:09:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.381+0000 7fb5d9a00640 1 -- 172.21.15.102:0/56818452 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5cc0a26e0 con 0x7fb5cc09ad20 2024-04-25T02:09:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.381+0000 7fb5d0c00640 1 -- 172.21.15.102:0/56818452 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d406a070 con 0x7fb5cc09ad20 2024-04-25T02:09:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.381+0000 7fb5d9a00640 1 -- 172.21.15.102:0/56818452 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5cc13f5d0 con 0x7fb5cc09ad20 2024-04-25T02:09:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.384+0000 7fb5dde05740 1 Processor -- start 2024-04-25T02:09:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.384+0000 7fb5d0c00640 1 -- 172.21.15.102:0/56818452 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb5d40ae020 con 0x7fb5cc09ad20 2024-04-25T02:09:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.384+0000 7fb5dde05740 1 -- start start 2024-04-25T02:09:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.385+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84abffb0 0x558a84ac0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.385+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84ac08d0 0x558a84acac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.385+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84acb190 0x558a84acd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.385+0000 7fb5dde05740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x558a84a23ab0 con 0x558a84acb190 2024-04-25T02:09:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.385+0000 7fb5dde05740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558a849cc640 con 0x558a84abffb0 2024-04-25T02:09:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.385+0000 7fb5dde05740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558a84a1fa70 con 0x558a84ac08d0 2024-04-25T02:09:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.385+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84ac08d0 0x558a84acac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.385+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84ac08d0 0x558a84acac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:43614/0 (socket says 172.21.15.102:43614) 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.385+0000 7fb5d3e00640 1 -- 172.21.15.102:0/2207958373 learned_addr learned my addr 172.21.15.102:0/2207958373 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.386+0000 7fb5d3400640 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84abffb0 0x558a84ac0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.386+0000 7fb5d3400640 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84acb190 0x558a84acd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.386+0000 7fb5d0c00640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5bc0a2d80 0x7fb5bc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.387+0000 7fb5d0c00640 1 -- 172.21.15.102:0/56818452 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7fb5d415b3f0 con 0x7fb5cc09ad20 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.387+0000 7fb5d3e00640 1 -- 172.21.15.102:0/2207958373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84abffb0 msgr2=0x558a84ac0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.387+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84abffb0 0x558a84ac0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.387+0000 7fb5d3e00640 1 -- 172.21.15.102:0/2207958373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84acb190 msgr2=0x558a84acd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.387+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84acb190 0x558a84acd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.387+0000 7fb5d9000640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5bc0a2d80 0x7fb5bc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.387+0000 7fb5d3e00640 1 -- 172.21.15.102:0/2207958373 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a84738210 con 0x558a84ac08d0 2024-04-25T02:09:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.387+0000 7fb5d3400640 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84acb190 0x558a84acd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:09:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.388+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84ac08d0 0x558a84acac50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009d50 tx=0x7fb5c8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aab5e80e6e6b7e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.389+0000 7fb5d9000640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5bc0a2d80 0x7fb5bc0a5240 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4127950 tx=0x7fb5d404fa70 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.389+0000 7fb5c5600640 1 -- 172.21.15.102:0/2207958373 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c801e070 con 0x558a84ac08d0 2024-04-25T02:09:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.389+0000 7fb5c5600640 1 -- 172.21.15.102:0/2207958373 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5c8019b10 con 0x558a84ac08d0 2024-04-25T02:09:50.398 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.389+0000 7fb5c5600640 1 -- 172.21.15.102:0/2207958373 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c800eb70 con 0x558a84ac08d0 2024-04-25T02:09:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.390+0000 7fb5dde05740 1 -- 172.21.15.102:0/2207958373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84ac08d0 msgr2=0x558a84acac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.390+0000 7fb5dde05740 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84ac08d0 0x558a84acac50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009d50 tx=0x7fb5c8011810 comp rx=0 tx=0).stop 2024-04-25T02:09:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.391+0000 7fb5dde05740 1 -- 172.21.15.102:0/2207958373 shutdown_connections 2024-04-25T02:09:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.391+0000 7fb5dde05740 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84acb190 0x558a84acd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.391+0000 7fb5dde05740 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84ac08d0 0x558a84acac50 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.391+0000 7fb5dde05740 1 --2- 172.21.15.102:0/2207958373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84abffb0 0x558a84ac0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.391+0000 7fb5dde05740 1 -- 172.21.15.102:0/2207958373 >> 172.21.15.102:0/2207958373 conn(0x558a84abc4c0 msgr2=0x558a84abc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.391+0000 7fb5dde05740 1 -- 172.21.15.102:0/2207958373 shutdown_connections 2024-04-25T02:09:50.399 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.391+0000 7fb5dde05740 1 -- 172.21.15.102:0/2207958373 wait complete. 2024-04-25T02:09:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.391+0000 7fb5dde05740 1 -- start start 2024-04-25T02:09:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.391+0000 7fb5dde05740 1 Processor -- start 2024-04-25T02:09:50.401 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ac8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ac8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5dde05740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x558a84acddc0 con 0x558a84abffb0 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5dde05740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x558a849df990 con 0x558a84ac08d0 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5dde05740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558a84acdf40 con 0x558a84acb190 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5dde05740 10 client.0 _ll_register_callbacks cb 0x558a849e1950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:50764/0 (socket says 172.21.15.102:50764) 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3e00640 1 -- 172.21.15.102:0/3247484493 learned_addr learned my addr 172.21.15.102:0/3247484493 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3400640 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ac8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3400640 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ac8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3e00640 1 -- 172.21.15.102:0/3247484493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 msgr2=0x558a84ac8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:50.402 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ac8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3e00640 1 -- 172.21.15.102:0/3247484493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 msgr2=0x558a84ac8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ac8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.392+0000 7fb5d3e00640 1 -- 172.21.15.102:0/3247484493 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a84804620 con 0x558a84ac08d0 2024-04-25T02:09:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.393+0000 7fb5d3400640 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ac8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:09:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.393+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8680 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8000c00 tx=0x7fb5c8009a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a349bc01ee87d19d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.393+0000 7fb5ba000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:09:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.393+0000 7fb5ba000640 1 -- 172.21.15.102:0/3247484493 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8029020 con 0x558a84ac08d0 2024-04-25T02:09:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.393+0000 7fb5dde05740 1 -- 172.21.15.102:0/3247484493 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5c8019b10 con 0x558a84ac08d0 2024-04-25T02:09:50.403 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.393+0000 7fb5ba000640 1 -- 172.21.15.102:0/3247484493 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7fb5c8019770 con 0x558a84ac08d0 2024-04-25T02:09:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.394+0000 7fb5ba000640 1 -- 172.21.15.102:0/3247484493 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fb5c8034050 con 0x558a84ac08d0 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.394+0000 7fb5dde05740 10 client.27689 fetch_fsmap learned FSMap version 53 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.394+0000 7fb5dde05740 1 -- 172.21.15.102:0/3247484493 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x558a84ad9d50 con 0x558a84ac08d0 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.394+0000 7fb5ba000640 1 -- 172.21.15.102:0/3247484493 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5c8019b10 con 0x558a84ac08d0 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.395+0000 7fb5ba000640 1 -- 172.21.15.102:0/3247484493 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 53) ==== 8358+0+0 (secure 0 0 0) 0x7fb5c80046c0 con 0x558a84ac08d0 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.395+0000 7fb5dde05740 10 client.27689 fetch_fsmap finished waiting for FSMap version 53 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.395+0000 7fb5dde05740 10 client.27689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.395+0000 7fb5dde05740 20 client.27689 populate_metadata read hostname 'smithi102' 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.395+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558a84adb150 0x558a84add540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.395+0000 7fb5dde05740 4 client.27689 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:09:50.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.395+0000 7fb5dde05740 1 -- 172.21.15.102:0/3247484493 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a84a30ff0 con 0x558a84adb150 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.395+0000 7fb5d3400640 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558a84adb150 0x558a84add540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.396+0000 7fb5d3400640 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558a84adb150 0x558a84add540 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b2af849d77c0e19 server_cookie=d029938972876842 in_seq=0 out_seq=0 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.396+0000 7fb5ba000640 10 client.27689 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5ba000640 1 -- 172.21.15.102:0/3247484493 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558a84a30ff0 con 0x558a84adb150 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5ba000640 10 client.27689 handle_command_reply: tid=0 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 client.27689 shutdown 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 -- 172.21.15.102:0/3247484493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 msgr2=0x558a84ac8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8680 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8000c00 tx=0x7fb5c8009a90 comp rx=0 tx=0).stop 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 -- 172.21.15.102:0/3247484493 shutdown_connections 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558a84adb150 0x558a84add540 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ac8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8680 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5ba000640 0 client.27689 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3247484493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ac8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 -- 172.21.15.102:0/3247484493 >> 172.21.15.102:0/3247484493 conn(0x558a84abc4c0 msgr2=0x558a84ac21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 -- 172.21.15.102:0/3247484493 shutdown_connections 2024-04-25T02:09:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 1 -- 172.21.15.102:0/3247484493 wait complete. 2024-04-25T02:09:50.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.397+0000 7fb5dde05740 20 client.27689 trim_cache size 0 max 16384 2024-04-25T02:09:50.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5dde05740 1 Processor -- start 2024-04-25T02:09:50.430 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5dde05740 1 -- start start 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ad6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ae2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5dde05740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc14a7f0 con 0x558a84abffb0 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5dde05740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc148030 con 0x558a84ac08d0 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5dde05740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc145ea0 con 0x558a84acb190 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5d3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ad6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:50776/0 (socket says 172.21.15.102:50776) 2024-04-25T02:09:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.421+0000 7fb5d3e00640 1 -- 172.21.15.102:0/2984622516 learned_addr learned my addr 172.21.15.102:0/2984622516 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5d3400640 1 --2- 172.21.15.102:0/2984622516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ae2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5d3e00640 1 -- 172.21.15.102:0/2984622516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 msgr2=0x558a84ae2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/2984622516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ae2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5d3e00640 1 -- 172.21.15.102:0/2984622516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 msgr2=0x558a84ad6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/2984622516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ad6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5d3e00640 1 -- 172.21.15.102:0/2984622516 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a84ac70f0 con 0x558a84ac08d0 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5d3400640 1 --2- 172.21.15.102:0/2984622516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ad6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/2984622516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8004c00 tx=0x7fb5c8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c19972f47c31af40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5c4c00640 1 -- 172.21.15.102:0/2984622516 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8029070 con 0x558a84ac08d0 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5c4c00640 1 -- 172.21.15.102:0/2984622516 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5c8004030 con 0x558a84ac08d0 2024-04-25T02:09:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.422+0000 7fb5c4c00640 1 -- 172.21.15.102:0/2984622516 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c801e040 con 0x558a84ac08d0 2024-04-25T02:09:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 -- 172.21.15.102:0/2984622516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 msgr2=0x558a84ac8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 --2- 172.21.15.102:0/2984622516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8004c00 tx=0x7fb5c8009260 comp rx=0 tx=0).stop 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 -- 172.21.15.102:0/2984622516 shutdown_connections 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 --2- 172.21.15.102:0/2984622516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ae2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 --2- 172.21.15.102:0/2984622516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac8990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 --2- 172.21.15.102:0/2984622516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ad6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 -- 172.21.15.102:0/2984622516 >> 172.21.15.102:0/2984622516 conn(0x558a84abc4c0 msgr2=0x558a84abc8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 -- 172.21.15.102:0/2984622516 shutdown_connections 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 -- 172.21.15.102:0/2984622516 wait complete. 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.423+0000 7fb5dde05740 1 -- start start 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5dde05740 1 Processor -- start 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ad3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.436 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5dde05740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ac5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5dde05740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc1464f0 con 0x558a84abffb0 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5dde05740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc147960 con 0x558a84ac08d0 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5dde05740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc146bc0 con 0x558a84acb190 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5dde05740 10 client.0 _ll_register_callbacks cb 0x7fb5c801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5d3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ad3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:50788/0 (socket says 172.21.15.102:50788) 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5d3e00640 1 -- 172.21.15.102:0/3947305097 learned_addr learned my addr 172.21.15.102:0/3947305097 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.424+0000 7fb5d3400640 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ac5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5d3e00640 1 -- 172.21.15.102:0/3947305097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 msgr2=0x558a84ac5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:50.437 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ac5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5d3e00640 1 -- 172.21.15.102:0/3947305097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 msgr2=0x558a84ad3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ad3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5d3e00640 1 -- 172.21.15.102:0/3947305097 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a84ac5a60 con 0x558a84ac08d0 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5d3e00640 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac4c20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8004b80 tx=0x7fb5c8004eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d207392568581d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5ba000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5ba000640 1 -- 172.21.15.102:0/3947305097 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c801e070 con 0x558a84ac08d0 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5ba000640 1 -- 172.21.15.102:0/3947305097 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7fb5c8004fa0 con 0x558a84ac08d0 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.425+0000 7fb5dde05740 1 -- 172.21.15.102:0/3947305097 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5c8004030 con 0x558a84ac08d0 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.426+0000 7fb5ba000640 1 -- 172.21.15.102:0/3947305097 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fb5c8029040 con 0x558a84ac08d0 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.426+0000 7fb5dde05740 10 client.27695 fetch_fsmap learned FSMap version 53 2024-04-25T02:09:50.438 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.426+0000 7fb5dde05740 1 -- 172.21.15.102:0/3947305097 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x558a84b41720 con 0x558a84ac08d0 2024-04-25T02:09:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.427+0000 7fb5ba000640 1 -- 172.21.15.102:0/3947305097 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5c8004030 con 0x558a84ac08d0 2024-04-25T02:09:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.427+0000 7fb5ba000640 1 -- 172.21.15.102:0/3947305097 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 53) ==== 8358+0+0 (secure 0 0 0) 0x7fb5c8003c90 con 0x558a84ac08d0 2024-04-25T02:09:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.427+0000 7fb5dde05740 10 client.27695 fetch_fsmap finished waiting for FSMap version 53 2024-04-25T02:09:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.427+0000 7fb5dde05740 10 client.27695 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:09:50.439 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.427+0000 7fb5dde05740 20 client.27695 populate_metadata read hostname 'smithi102' 2024-04-25T02:09:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.427+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558a84b41e10 0x558a84b44230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:09:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.427+0000 7fb5dde05740 4 client.27695 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:09:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.427+0000 7fb5dde05740 1 -- 172.21.15.102:0/3947305097 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558a84a2b340 con 0x558a84b41e10 2024-04-25T02:09:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.428+0000 7fb5d3400640 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558a84b41e10 0x558a84b44230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:09:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.429+0000 7fb5d3400640 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558a84b41e10 0x558a84b44230 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feceb110ec5eb96b server_cookie=74c53c972f06ca51 in_seq=0 out_seq=0 2024-04-25T02:09:50.440 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.429+0000 7fb5ba000640 10 client.27695 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:50.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.523+0000 7fb5ba000640 1 -- 172.21.15.102:0/3947305097 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558a84a2b340 con 0x558a84b41e10 2024-04-25T02:09:50.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.523+0000 7fb5ba000640 10 client.27695 handle_command_reply: tid=0 2024-04-25T02:09:50.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.523+0000 7fb5dde05740 1 client.27695 shutdown 2024-04-25T02:09:50.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.523+0000 7fb5dde05740 1 -- 172.21.15.102:0/3947305097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 msgr2=0x558a84ac4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:50.531 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.523+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac4c20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8004b80 tx=0x7fb5c8004eb0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.523+0000 7fb5dde05740 1 -- 172.21.15.102:0/3947305097 shutdown_connections 2024-04-25T02:09:50.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.524+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x558a84b41e10 0x558a84b44230 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.524+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558a84acb190 0x558a84ac5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.524+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x558a84ac08d0 0x558a84ac4c20 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.524+0000 7fb5dde05740 1 --2- 172.21.15.102:0/3947305097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558a84abffb0 0x558a84ad3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.524+0000 7fb5dde05740 1 -- 172.21.15.102:0/3947305097 >> 172.21.15.102:0/3947305097 conn(0x558a84abc4c0 msgr2=0x558a84ac21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:50.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.524+0000 7fb5ba000640 0 client.27695 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:09:50.532 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.524+0000 7fb5dde05740 1 -- 172.21.15.102:0/3947305097 shutdown_connections 2024-04-25T02:09:50.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.524+0000 7fb5dde05740 1 -- 172.21.15.102:0/3947305097 wait complete. 2024-04-25T02:09:50.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.524+0000 7fb5dde05740 20 client.27695 trim_cache size 0 max 16384 2024-04-25T02:09:50.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:09:50.533 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:09:50.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:09:50.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:09:50.534 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 -- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5bc0a2d80 msgr2=0x7fb5bc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5bc0a2d80 0x7fb5bc0a5240 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4127950 tx=0x7fb5d404fa70 comp rx=0 tx=0).stop 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 -- 172.21.15.102:0/56818452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc09ad20 msgr2=0x7fb5cc0a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc09ad20 0x7fb5cc0a1cd0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4050ce0 tx=0x7fb5d409deb0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 -- 172.21.15.102:0/56818452 shutdown_connections 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb5bc0a2d80 0x7fb5bc0a5240 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc09ad20 0x7fb5cc0a1cd0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5cc0969c0 0x7fb5cc138c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 --2- 172.21.15.102:0/56818452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5cc096000 0x7fb5cc138710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 -- 172.21.15.102:0/56818452 >> 172.21.15.102:0/56818452 conn(0x7fb5cc00b830 msgr2=0x7fb5cc094a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.526+0000 7fb5d9a00640 1 -- 172.21.15.102:0/56818452 shutdown_connections 2024-04-25T02:09:50.535 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:09:50.527+0000 7fb5d9a00640 1 -- 172.21.15.102:0/56818452 wait complete. 2024-04-25T02:09:51.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:51 smithi102 ceph-mon[27796]: pgmap v454: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 7.0 MiB/s wr, 659 op/s 2024-04-25T02:09:51.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:51 smithi110 ceph-mon[32848]: pgmap v454: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 7.0 MiB/s wr, 659 op/s 2024-04-25T02:09:51.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:51 smithi193 ceph-mon[32981]: pgmap v454: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 7.0 MiB/s wr, 659 op/s 2024-04-25T02:09:53.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 840.0 s, 1567.0 tps, lat 64.392 ms stddev 83.531 2024-04-25T02:09:53.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:53 smithi102 ceph-mon[27796]: pgmap v455: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 5.7 MiB/s wr, 559 op/s 2024-04-25T02:09:53.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:53 smithi110 ceph-mon[32848]: pgmap v455: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 5.7 MiB/s wr, 559 op/s 2024-04-25T02:09:53.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:53 smithi193 ceph-mon[32981]: pgmap v455: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 5.7 MiB/s wr, 559 op/s 2024-04-25T02:09:55.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:55 smithi102 ceph-mon[27796]: pgmap v456: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 7.3 MiB/s wr, 738 op/s 2024-04-25T02:09:55.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:55 smithi110 ceph-mon[32848]: pgmap v456: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 7.3 MiB/s wr, 738 op/s 2024-04-25T02:09:55.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:55 smithi193 ceph-mon[32981]: pgmap v456: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 7.3 MiB/s wr, 738 op/s 2024-04-25T02:09:56.431 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:56 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:09:56.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:56 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:09:56.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:56 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:09:57.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:57 smithi102 ceph-mon[27796]: pgmap v457: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-04-25T02:09:57.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:57 smithi110 ceph-mon[32848]: pgmap v457: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-04-25T02:09:57.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:57 smithi193 ceph-mon[32981]: pgmap v457: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-04-25T02:09:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 845.0 s, 1999.6 tps, lat 50.118 ms stddev 62.339 2024-04-25T02:09:58.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:58 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:09:58.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:58 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:09:58.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:58 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:09:58.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:58 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:09:58.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:58 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:09:58.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:58 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:09:58.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:58 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:09:58.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:58 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:09:58.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:58 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:09:59.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:09:59 smithi102 ceph-mon[27796]: pgmap v458: 97 pgs: 97 active+clean; 5.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.2 MiB/s wr, 695 op/s 2024-04-25T02:09:59.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:09:59 smithi110 ceph-mon[32848]: pgmap v458: 97 pgs: 97 active+clean; 5.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.2 MiB/s wr, 695 op/s 2024-04-25T02:09:59.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:09:59 smithi193 ceph-mon[32981]: pgmap v458: 97 pgs: 97 active+clean; 5.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.2 MiB/s wr, 695 op/s 2024-04-25T02:10:00.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a[27792]: 2024-04-25T02:09:59.999+0000 7f4d19000700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:10:00.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a[27792]: 2024-04-25T02:09:59.999+0000 7f4d19000700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-25T02:10:00.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a[27792]: 2024-04-25T02:09:59.999+0000 7f4d19000700 -1 log_channel(cluster) log [ERR] : mds.i(mds.1): Metadata damage detected 2024-04-25T02:10:00.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:00 smithi102 ceph-mon[27796]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:10:00.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:00 smithi102 ceph-mon[27796]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-25T02:10:00.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:00 smithi102 ceph-mon[27796]: mds.i(mds.1): Metadata damage detected 2024-04-25T02:10:00.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:00 smithi110 ceph-mon[32848]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:10:00.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:00 smithi110 ceph-mon[32848]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-25T02:10:00.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:00 smithi110 ceph-mon[32848]: mds.i(mds.1): Metadata damage detected 2024-04-25T02:10:00.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:00 smithi193 ceph-mon[32981]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-04-25T02:10:00.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:00 smithi193 ceph-mon[32981]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-04-25T02:10:00.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:00 smithi193 ceph-mon[32981]: mds.i(mds.1): Metadata damage detected 2024-04-25T02:10:01.413 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:01 smithi193 ceph-mon[32981]: pgmap v459: 97 pgs: 97 active+clean; 5.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.3 MiB/s wr, 614 op/s 2024-04-25T02:10:01.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:01 smithi102 ceph-mon[27796]: pgmap v459: 97 pgs: 97 active+clean; 5.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.3 MiB/s wr, 614 op/s 2024-04-25T02:10:01.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:01 smithi110 ceph-mon[32848]: pgmap v459: 97 pgs: 97 active+clean; 5.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.3 MiB/s wr, 614 op/s 2024-04-25T02:10:02.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:02 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:10:02.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:02 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:10:02.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:02 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:10:03.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 850.0 s, 913.8 tps, lat 108.726 ms stddev 295.017 2024-04-25T02:10:03.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:03 smithi102 ceph-mon[27796]: pgmap v460: 97 pgs: 97 active+clean; 5.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 5.2 MiB/s wr, 515 op/s 2024-04-25T02:10:03.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:03 smithi110 ceph-mon[32848]: pgmap v460: 97 pgs: 97 active+clean; 5.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 5.2 MiB/s wr, 515 op/s 2024-04-25T02:10:03.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:03 smithi193 ceph-mon[32981]: pgmap v460: 97 pgs: 97 active+clean; 5.6 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 5.2 MiB/s wr, 515 op/s 2024-04-25T02:10:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.702+0000 7fb4a7e00640 1 -- 172.21.15.102:0/872023444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4a01125d0 msgr2=0x7fb4a01149c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.702+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/872023444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4a01125d0 0x7fb4a01149c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb498009860 tx=0x7fb49802a340 comp rx=0 tx=0).stop 2024-04-25T02:10:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.702+0000 7fb4a7e00640 1 -- 172.21.15.102:0/872023444 shutdown_connections 2024-04-25T02:10:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.702+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/872023444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4a01125d0 0x7fb4a01149c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.702+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/872023444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4a0076590 0x7fb4a0112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.702+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/872023444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a0075c70 0x7fb4a0076050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.702+0000 7fb4a7e00640 1 -- 172.21.15.102:0/872023444 >> 172.21.15.102:0/872023444 conn(0x7fb4a00ff440 msgr2=0x7fb4a0101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.703+0000 7fb4a7e00640 1 -- 172.21.15.102:0/872023444 shutdown_connections 2024-04-25T02:10:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.703+0000 7fb4a7e00640 1 -- 172.21.15.102:0/872023444 wait complete. 2024-04-25T02:10:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.704+0000 7fb4a7e00640 1 Processor -- start 2024-04-25T02:10:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.704+0000 7fb4a7e00640 1 -- start start 2024-04-25T02:10:03.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4a0075c70 0x7fb4a019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4a0076590 0x7fb4a019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a01125d0 0x7fb4a01a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a7e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a01180f0 con 0x7fb4a0075c70 2024-04-25T02:10:03.713 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a7e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a0117f70 con 0x7fb4a01125d0 2024-04-25T02:10:03.714 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a7e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a0118270 con 0x7fb4a0076590 2024-04-25T02:10:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a5e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a01125d0 0x7fb4a01a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a5e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a01125d0 0x7fb4a01a2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59438/0 (socket says 172.21.15.102:59438) 2024-04-25T02:10:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a5e00640 1 -- 172.21.15.102:0/3360370853 learned_addr learned my addr 172.21.15.102:0/3360370853 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.705+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4a0076590 0x7fb4a019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb4a5e00640 1 -- 172.21.15.102:0/3360370853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4a0076590 msgr2=0x7fb4a019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb4a5e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4a0076590 0x7fb4a019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb4a5e00640 1 -- 172.21.15.102:0/3360370853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4a0075c70 msgr2=0x7fb4a019e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb4a5e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4a0075c70 0x7fb4a019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb4a5e00640 1 -- 172.21.15.102:0/3360370853 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a01a3540 con 0x7fb4a01125d0 2024-04-25T02:10:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb4a5e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a01125d0 0x7fb4a01a2e10 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb49802a930 tx=0x7fb498033ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb49e000640 1 -- 172.21.15.102:0/3360370853 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb498037040 con 0x7fb4a01125d0 2024-04-25T02:10:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb49e000640 1 -- 172.21.15.102:0/3360370853 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb498033cd0 con 0x7fb4a01125d0 2024-04-25T02:10:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb49e000640 1 -- 172.21.15.102:0/3360370853 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4980053a0 con 0x7fb4a01125d0 2024-04-25T02:10:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.706+0000 7fb4a7e00640 1 -- 172.21.15.102:0/3360370853 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a01a37d0 con 0x7fb4a01125d0 2024-04-25T02:10:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.707+0000 7fb4a7e00640 1 -- 172.21.15.102:0/3360370853 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a01ab0c0 con 0x7fb4a01125d0 2024-04-25T02:10:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.710+0000 7fb4aac84740 1 Processor -- start 2024-04-25T02:10:03.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.711+0000 7fb49e000640 1 -- 172.21.15.102:0/3360370853 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb498042020 con 0x7fb4a01125d0 2024-04-25T02:10:03.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.711+0000 7fb4aac84740 1 -- start start 2024-04-25T02:10:03.719 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.711+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313c8e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.711+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313c98aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.711+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c9b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.711+0000 7fb4aac84740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55e313bf1ab0 con 0x55e313c8de00 2024-04-25T02:10:03.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.711+0000 7fb4aac84740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e313b9a640 con 0x55e313c8e720 2024-04-25T02:10:03.720 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.711+0000 7fb4aac84740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e313beda70 con 0x55e313c98fe0 2024-04-25T02:10:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.713+0000 7fb4a4a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c9b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.713+0000 7fb4a4a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c9b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:47802/0 (socket says 172.21.15.102:47802) 2024-04-25T02:10:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.713+0000 7fb4a4a00640 1 -- 172.21.15.102:0/1926754076 learned_addr learned my addr 172.21.15.102:0/1926754076 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.713+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313c8e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.713+0000 7fb4a5400640 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313c98aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.722 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.714+0000 7fb49e000640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4880a2e20 0x7fb4880a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.715+0000 7fb4a5400640 1 -- 172.21.15.102:0/1926754076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 msgr2=0x55e313c9b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.715+0000 7fb4a5400640 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c9b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.715+0000 7fb4a5400640 1 -- 172.21.15.102:0/1926754076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 msgr2=0x55e313c8e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:03.723 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.715+0000 7fb49e000640 1 -- 172.21.15.102:0/3360370853 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7fb4980eea60 con 0x7fb4a01125d0 2024-04-25T02:10:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.715+0000 7fb4a5400640 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313c8e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.715+0000 7fb4a5400640 1 -- 172.21.15.102:0/1926754076 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e313906210 con 0x55e313c8e720 2024-04-25T02:10:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.715+0000 7fb4a5e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4880a2e20 0x7fb4880a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.724 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.715+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313c8e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.716+0000 7fb4a5e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4880a2e20 0x7fb4880a52e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb49802a860 tx=0x7fb498005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.716+0000 7fb4a5400640 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313c98aa0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb4940096c0 tx=0x7fb494009f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63ca317d4ff871f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.716+0000 7fb492600640 1 -- 172.21.15.102:0/1926754076 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49400ed10 con 0x55e313c8e720 2024-04-25T02:10:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.716+0000 7fb492600640 1 -- 172.21.15.102:0/1926754076 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb494010910 con 0x55e313c8e720 2024-04-25T02:10:03.725 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.716+0000 7fb492600640 1 -- 172.21.15.102:0/1926754076 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb494018920 con 0x55e313c8e720 2024-04-25T02:10:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.717+0000 7fb4aac84740 1 -- 172.21.15.102:0/1926754076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 msgr2=0x55e313c98aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.717+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313c98aa0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fb4940096c0 tx=0x7fb494009f40 comp rx=0 tx=0).stop 2024-04-25T02:10:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.717+0000 7fb4aac84740 1 -- 172.21.15.102:0/1926754076 shutdown_connections 2024-04-25T02:10:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.717+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c9b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.717+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313c98aa0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.717+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1926754076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313c8e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.717+0000 7fb4aac84740 1 -- 172.21.15.102:0/1926754076 >> 172.21.15.102:0/1926754076 conn(0x55e313c8a310 msgr2=0x55e313c8a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:03.726 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.718+0000 7fb4aac84740 1 -- 172.21.15.102:0/1926754076 shutdown_connections 2024-04-25T02:10:03.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.718+0000 7fb4aac84740 1 -- 172.21.15.102:0/1926754076 wait complete. 2024-04-25T02:10:03.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.718+0000 7fb4aac84740 1 -- start start 2024-04-25T02:10:03.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4aac84740 1 Processor -- start 2024-04-25T02:10:03.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8de00 0x55e313c95f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.729 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8e720 0x55e313c964a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4aac84740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55e313c9bc10 con 0x55e313c8e720 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4aac84740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55e313bad990 con 0x55e313c8de00 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4aac84740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e313c9bd90 con 0x55e313c98fe0 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4aac84740 10 client.0 _ll_register_callbacks cb 0x55e313baf950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4a4a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8de00 0x55e313c95f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4a4a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8de00 0x55e313c95f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59454/0 (socket says 172.21.15.102:59454) 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4a4a00640 1 -- 172.21.15.102:0/1879827508 learned_addr learned my addr 172.21.15.102:0/1879827508 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.719+0000 7fb4a5400640 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8e720 0x55e313c964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb4a4a00640 1 -- 172.21.15.102:0/1879827508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 msgr2=0x55e313c969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.730 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb4a4a00640 1 -- 172.21.15.102:0/1879827508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8e720 msgr2=0x55e313c964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8e720 0x55e313c964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb4a4a00640 1 -- 172.21.15.102:0/1879827508 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3139d2620 con 0x55e313c8de00 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8de00 0x55e313c95f60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00f700 tx=0x7fb48c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47a1ff269a65c4b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb486a00640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb486a00640 1 -- 172.21.15.102:0/1879827508 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c003970 con 0x55e313c8de00 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb486a00640 1 -- 172.21.15.102:0/1879827508 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7fb48c003b10 con 0x55e313c8de00 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb4aac84740 1 -- 172.21.15.102:0/1879827508 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb494010910 con 0x55e313c8de00 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb486a00640 1 -- 172.21.15.102:0/1879827508 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fb48c003d10 con 0x55e313c8de00 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb4aac84740 10 client.27707 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.720+0000 7fb4aac84740 1 -- 172.21.15.102:0/1879827508 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55e313ca7b60 con 0x55e313c8de00 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.721+0000 7fb486a00640 1 -- 172.21.15.102:0/1879827508 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb494010910 con 0x55e313c8de00 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.721+0000 7fb486a00640 1 -- 172.21.15.102:0/1879827508 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7fb48c004030 con 0x55e313c8de00 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.721+0000 7fb4aac84740 10 client.27707 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:03.731 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.721+0000 7fb4aac84740 10 client.27707 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.721+0000 7fb4aac84740 20 client.27707 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.721+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e313ca8fd0 0x55e313cab420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.721+0000 7fb4aac84740 4 client.27707 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:10:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.721+0000 7fb4aac84740 1 -- 172.21.15.102:0/1879827508 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e313bfeff0 con 0x55e313ca8fd0 2024-04-25T02:10:03.732 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.721+0000 7fb4a5400640 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e313ca8fd0 0x55e313cab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.722+0000 7fb4a5400640 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e313ca8fd0 0x55e313cab420 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6626d9de68b986d8 server_cookie=8a2aff4b7113c798 in_seq=0 out_seq=0 2024-04-25T02:10:03.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.722+0000 7fb486a00640 10 client.27707 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:03.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb486a00640 1 -- 172.21.15.102:0/1879827508 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e313bfeff0 con 0x55e313ca8fd0 2024-04-25T02:10:03.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb486a00640 10 client.27707 handle_command_reply: tid=0 2024-04-25T02:10:03.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb4aac84740 1 client.27707 shutdown 2024-04-25T02:10:03.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb4aac84740 1 -- 172.21.15.102:0/1879827508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8de00 msgr2=0x55e313c95f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:03.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8de00 0x55e313c95f60 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00f700 tx=0x7fb48c004ab0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.733 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb4aac84740 1 -- 172.21.15.102:0/1879827508 shutdown_connections 2024-04-25T02:10:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e313ca8fd0 0x55e313cab420 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8e720 0x55e313c964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb4aac84740 1 --2- 172.21.15.102:0/1879827508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8de00 0x55e313c95f60 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb4aac84740 1 -- 172.21.15.102:0/1879827508 >> 172.21.15.102:0/1879827508 conn(0x55e313c8a310 msgr2=0x55e313c90090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.723+0000 7fb486a00640 0 client.27707 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.724+0000 7fb4aac84740 1 -- 172.21.15.102:0/1879827508 shutdown_connections 2024-04-25T02:10:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.724+0000 7fb4aac84740 1 -- 172.21.15.102:0/1879827508 wait complete. 2024-04-25T02:10:03.734 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.724+0000 7fb4aac84740 20 client.27707 trim_cache size 0 max 16384 2024-04-25T02:10:03.774 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.766+0000 7fb4aac84740 1 Processor -- start 2024-04-25T02:10:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.767+0000 7fb4aac84740 1 -- start start 2024-04-25T02:10:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.767+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313c98730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.767+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313ca4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.776 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.767+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c8f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.767+0000 7fb4aac84740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb480011140 con 0x55e313c8de00 2024-04-25T02:10:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.767+0000 7fb4aac84740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb48000e9b0 con 0x55e313c8e720 2024-04-25T02:10:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.767+0000 7fb4aac84740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb48000c820 con 0x55e313c98fe0 2024-04-25T02:10:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.768+0000 7fb4a5400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313ca4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.768+0000 7fb4a5400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313ca4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59470/0 (socket says 172.21.15.102:59470) 2024-04-25T02:10:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.768+0000 7fb4a5400640 1 -- 172.21.15.102:0/638604047 learned_addr learned my addr 172.21.15.102:0/638604047 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:03.777 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.768+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c8f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.769+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313c98730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.769+0000 7fb4a4a00640 1 -- 172.21.15.102:0/638604047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 msgr2=0x55e313ca4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.769+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313ca4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.770+0000 7fb4a4a00640 1 -- 172.21.15.102:0/638604047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 msgr2=0x55e313c98730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.770+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313c98730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.770+0000 7fb4a4a00640 1 -- 172.21.15.102:0/638604047 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e313c8aee0 con 0x55e313c98fe0 2024-04-25T02:10:03.778 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.770+0000 7fb4a5400640 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313ca4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:03.779 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.771+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c8f800 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00f700 tx=0x7fb48c000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e27a1a75f2d980e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:03.779 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.771+0000 7fb491c00640 1 -- 172.21.15.102:0/638604047 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c02f040 con 0x55e313c98fe0 2024-04-25T02:10:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.772+0000 7fb491c00640 1 -- 172.21.15.102:0/638604047 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb48c030040 con 0x55e313c98fe0 2024-04-25T02:10:03.780 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.772+0000 7fb491c00640 1 -- 172.21.15.102:0/638604047 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c011290 con 0x55e313c98fe0 2024-04-25T02:10:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.774+0000 7fb4aac84740 1 -- 172.21.15.102:0/638604047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 msgr2=0x55e313c8f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:03.782 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.774+0000 7fb4aac84740 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c8f800 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00f700 tx=0x7fb48c000f40 comp rx=0 tx=0).stop 2024-04-25T02:10:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.774+0000 7fb4aac84740 1 -- 172.21.15.102:0/638604047 shutdown_connections 2024-04-25T02:10:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.775+0000 7fb4aac84740 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313c8f800 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.775+0000 7fb4aac84740 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313ca4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.775+0000 7fb4aac84740 1 --2- 172.21.15.102:0/638604047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313c98730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.783 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.775+0000 7fb4aac84740 1 -- 172.21.15.102:0/638604047 >> 172.21.15.102:0/638604047 conn(0x55e313c8a310 msgr2=0x55e313c98020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.775+0000 7fb4aac84740 1 -- 172.21.15.102:0/638604047 shutdown_connections 2024-04-25T02:10:03.784 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.775+0000 7fb4aac84740 1 -- 172.21.15.102:0/638604047 wait complete. 2024-04-25T02:10:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.777+0000 7fb4aac84740 1 -- start start 2024-04-25T02:10:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.777+0000 7fb4aac84740 1 Processor -- start 2024-04-25T02:10:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313ca12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313d0a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4aac84740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313d0ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4aac84740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb48000ce70 con 0x55e313c8de00 2024-04-25T02:10:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4aac84740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fb48000e2e0 con 0x55e313c8e720 2024-04-25T02:10:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4aac84740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb48000d540 con 0x55e313c98fe0 2024-04-25T02:10:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4aac84740 10 client.0 _ll_register_callbacks cb 0x7fb48c0114a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4a5400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313d0a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4a5400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313d0a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:59480/0 (socket says 172.21.15.102:59480) 2024-04-25T02:10:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4a5400640 1 -- 172.21.15.102:0/3004866257 learned_addr learned my addr 172.21.15.102:0/3004866257 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:03.787 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.778+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313ca12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.779+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313d0ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.779+0000 7fb4a4a00640 1 -- 172.21.15.102:0/3004866257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 msgr2=0x55e313d0ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.779+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313d0ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.779+0000 7fb4a4a00640 1 -- 172.21.15.102:0/3004866257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 msgr2=0x55e313d0a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.779+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313d0a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.788 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.779+0000 7fb4a4a00640 1 -- 172.21.15.102:0/3004866257 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e313d0b2f0 con 0x55e313c8de00 2024-04-25T02:10:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.781+0000 7fb4a4a00640 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313ca12d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0044f0 tx=0x7fb48c005ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6a44b4cbe12e3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.781+0000 7fb486a00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-04-25T02:10:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.781+0000 7fb486a00640 1 -- 172.21.15.102:0/3004866257 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c02f070 con 0x55e313c8de00 2024-04-25T02:10:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.781+0000 7fb486a00640 1 -- 172.21.15.102:0/3004866257 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8513+0+0 (secure 0 0 0) 0x7fb48c011220 con 0x55e313c8de00 2024-04-25T02:10:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.782+0000 7fb4aac84740 1 -- 172.21.15.102:0/3004866257 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb48c030040 con 0x55e313c8de00 2024-04-25T02:10:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.782+0000 7fb486a00640 1 -- 172.21.15.102:0/3004866257 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fb48c003eb0 con 0x55e313c8de00 2024-04-25T02:10:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.782+0000 7fb4aac84740 10 client.16935 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.782+0000 7fb4aac84740 1 -- 172.21.15.102:0/3004866257 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55e313ca00a0 con 0x55e313c8de00 2024-04-25T02:10:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.783+0000 7fb486a00640 1 -- 172.21.15.102:0/3004866257 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb48c030040 con 0x55e313c8de00 2024-04-25T02:10:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.783+0000 7fb486a00640 1 -- 172.21.15.102:0/3004866257 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7fb48c033070 con 0x55e313c8de00 2024-04-25T02:10:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.783+0000 7fb4aac84740 10 client.16935 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.784+0000 7fb4aac84740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.784+0000 7fb4aac84740 20 client.16935 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.784+0000 7fb4aac84740 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e313ca0840 0x55e313d245d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.784+0000 7fb4aac84740 4 client.16935 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:10:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.784+0000 7fb4aac84740 1 -- 172.21.15.102:0/3004866257 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e313bf9340 con 0x55e313ca0840 2024-04-25T02:10:03.793 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.785+0000 7fb4a5400640 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e313ca0840 0x55e313d245d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.786+0000 7fb4a5400640 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e313ca0840 0x55e313d245d0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9f1dd6fe43d0de3 server_cookie=4a12a14971ec8e53 in_seq=0 out_seq=0 2024-04-25T02:10:03.794 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.786+0000 7fb486a00640 10 client.16935 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.809+0000 7fb486a00640 1 -- 172.21.15.102:0/3004866257 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e313bf9340 con 0x55e313ca0840 2024-04-25T02:10:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb486a00640 10 client.16935 handle_command_reply: tid=0 2024-04-25T02:10:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb4aac84740 1 client.16935 shutdown 2024-04-25T02:10:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb4aac84740 1 -- 172.21.15.102:0/3004866257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 msgr2=0x55e313ca12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb4aac84740 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313ca12d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0044f0 tx=0x7fb48c005ac0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb4aac84740 1 -- 172.21.15.102:0/3004866257 shutdown_connections 2024-04-25T02:10:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb4aac84740 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55e313ca0840 0x55e313d245d0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb4aac84740 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e313c98fe0 0x55e313d0ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb4aac84740 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e313c8e720 0x55e313d0a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb4aac84740 1 --2- 172.21.15.102:0/3004866257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e313c8de00 0x55e313ca12d0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0044f0 tx=0x7fb48c005ac0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb4aac84740 1 -- 172.21.15.102:0/3004866257 >> 172.21.15.102:0/3004866257 conn(0x55e313c8a310 msgr2=0x55e313ca3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.810+0000 7fb486a00640 0 client.16935 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.811+0000 7fb4aac84740 1 -- 172.21.15.102:0/3004866257 shutdown_connections 2024-04-25T02:10:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.811+0000 7fb4aac84740 1 -- 172.21.15.102:0/3004866257 wait complete. 2024-04-25T02:10:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.811+0000 7fb4aac84740 20 client.16935 trim_cache size 0 max 16384 2024-04-25T02:10:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:10:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:10:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:10:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:10:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 -- 172.21.15.102:0/3360370853 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4880a2e20 msgr2=0x7fb4880a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4880a2e20 0x7fb4880a52e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb49802a860 tx=0x7fb498005f90 comp rx=0 tx=0).stop 2024-04-25T02:10:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 -- 172.21.15.102:0/3360370853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a01125d0 msgr2=0x7fb4a01a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a01125d0 0x7fb4a01a2e10 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fb49802a930 tx=0x7fb498033ae0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 -- 172.21.15.102:0/3360370853 shutdown_connections 2024-04-25T02:10:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fb4880a2e20 0x7fb4880a52e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a01125d0 0x7fb4a01a2e10 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4a0076590 0x7fb4a019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 --2- 172.21.15.102:0/3360370853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb4a0075c70 0x7fb4a019e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.812+0000 7fb4a7e00640 1 -- 172.21.15.102:0/3360370853 >> 172.21.15.102:0/3360370853 conn(0x7fb4a00ff440 msgr2=0x7fb4a0114020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.813+0000 7fb4a7e00640 1 -- 172.21.15.102:0/3360370853 shutdown_connections 2024-04-25T02:10:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:03.813+0000 7fb4a7e00640 1 -- 172.21.15.102:0/3360370853 wait complete. 2024-04-25T02:10:05.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:05 smithi102 ceph-mon[27796]: pgmap v461: 97 pgs: 97 active+clean; 5.7 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-25T02:10:05.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:05 smithi110 ceph-mon[32848]: pgmap v461: 97 pgs: 97 active+clean; 5.7 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-25T02:10:05.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:05 smithi193 ceph-mon[32981]: pgmap v461: 97 pgs: 97 active+clean; 5.7 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 18 MiB/s wr, 1.35k op/s 2024-04-25T02:10:07.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:07 smithi102 ceph-mon[27796]: pgmap v462: 97 pgs: 97 active+clean; 5.7 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 22 MiB/s wr, 1.53k op/s 2024-04-25T02:10:07.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:07 smithi110 ceph-mon[32848]: pgmap v462: 97 pgs: 97 active+clean; 5.7 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 22 MiB/s wr, 1.53k op/s 2024-04-25T02:10:07.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:07 smithi193 ceph-mon[32981]: pgmap v462: 97 pgs: 97 active+clean; 5.7 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 22 MiB/s wr, 1.53k op/s 2024-04-25T02:10:08.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 855.0 s, 1035.6 tps, lat 96.063 ms stddev 131.744 2024-04-25T02:10:08.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:08 smithi102 ceph-mon[27796]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T02:10:08.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:08 smithi110 ceph-mon[32848]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T02:10:08.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:08 smithi193 ceph-mon[32981]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T02:10:09.395 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:09 smithi110 ceph-mon[32848]: pgmap v464: 97 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 85 active+clean; 5.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 27 MiB/s wr, 1.85k op/s 2024-04-25T02:10:09.429 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:09 smithi102 ceph-mon[27796]: pgmap v464: 97 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 85 active+clean; 5.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 27 MiB/s wr, 1.85k op/s 2024-04-25T02:10:09.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:09 smithi193 ceph-mon[32981]: pgmap v464: 97 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 85 active+clean; 5.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 27 MiB/s wr, 1.85k op/s 2024-04-25T02:10:11.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:11 smithi102 ceph-mon[27796]: pgmap v465: 97 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 85 active+clean; 5.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 28 MiB/s wr, 1.91k op/s 2024-04-25T02:10:11.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:11 smithi110 ceph-mon[32848]: pgmap v465: 97 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 85 active+clean; 5.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 28 MiB/s wr, 1.91k op/s 2024-04-25T02:10:11.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:11 smithi193 ceph-mon[32981]: pgmap v465: 97 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 85 active+clean; 5.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 28 MiB/s wr, 1.91k op/s 2024-04-25T02:10:13.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 860.0 s, 1446.4 tps, lat 68.111 ms stddev 86.809 2024-04-25T02:10:13.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:13 smithi102 ceph-mon[27796]: pgmap v466: 97 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 85 active+clean; 5.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 28 MiB/s wr, 1.91k op/s 2024-04-25T02:10:13.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:13 smithi110 ceph-mon[32848]: pgmap v466: 97 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 85 active+clean; 5.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 28 MiB/s wr, 1.91k op/s 2024-04-25T02:10:13.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:13 smithi193 ceph-mon[32981]: pgmap v466: 97 pgs: 7 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 85 active+clean; 5.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 28 MiB/s wr, 1.91k op/s 2024-04-25T02:10:15.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:15 smithi102 ceph-mon[27796]: pgmap v467: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 15 MiB/s wr, 1.11k op/s 2024-04-25T02:10:15.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:15 smithi110 ceph-mon[32848]: pgmap v467: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 15 MiB/s wr, 1.11k op/s 2024-04-25T02:10:15.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:15 smithi193 ceph-mon[32981]: pgmap v467: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 15 MiB/s wr, 1.11k op/s 2024-04-25T02:10:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.996+0000 7fe437400640 1 -- 172.21.15.102:0/2744084425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430102120 msgr2=0x7fe43010e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.996+0000 7fe437400640 1 --2- 172.21.15.102:0/2744084425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430102120 0x7fe43010e630 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe428009ca0 tx=0x7fe42802a8c0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.996+0000 7fe437400640 1 -- 172.21.15.102:0/2744084425 shutdown_connections 2024-04-25T02:10:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.996+0000 7fe437400640 1 --2- 172.21.15.102:0/2744084425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430102120 0x7fe43010e630 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.996+0000 7fe437400640 1 --2- 172.21.15.102:0/2744084425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe430101760 0x7fe430101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.996+0000 7fe437400640 1 --2- 172.21.15.102:0/2744084425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe430107770 0x7fe430107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.996+0000 7fe437400640 1 -- 172.21.15.102:0/2744084425 >> 172.21.15.102:0/2744084425 conn(0x7fe4300fd1f0 msgr2=0x7fe4300ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.996+0000 7fe437400640 1 -- 172.21.15.102:0/2744084425 shutdown_connections 2024-04-25T02:10:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.996+0000 7fe437400640 1 -- 172.21.15.102:0/2744084425 wait complete. 2024-04-25T02:10:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.997+0000 7fe437400640 1 Processor -- start 2024-04-25T02:10:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.997+0000 7fe437400640 1 -- start start 2024-04-25T02:10:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.998+0000 7fe437400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe430101760 0x7fe43019e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.998+0000 7fe437400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe430102120 0x7fe43019ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.998+0000 7fe437400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430107770 0x7fe4301a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.005 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.998+0000 7fe437400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4301181e0 con 0x7fe430102120 2024-04-25T02:10:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.998+0000 7fe437400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe430118060 con 0x7fe430107770 2024-04-25T02:10:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.998+0000 7fe437400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe430118360 con 0x7fe430101760 2024-04-25T02:10:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.998+0000 7fe434a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe430101760 0x7fe43019e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.999+0000 7fe435400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430107770 0x7fe4301a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.999+0000 7fe435400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430107770 0x7fe4301a2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:47254/0 (socket says 172.21.15.102:47254) 2024-04-25T02:10:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.999+0000 7fe435400640 1 -- 172.21.15.102:0/3754698341 learned_addr learned my addr 172.21.15.102:0/3754698341 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:16.999+0000 7fe42fe00640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe430102120 0x7fe43019ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.000+0000 7fe435400640 1 -- 172.21.15.102:0/3754698341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe430101760 msgr2=0x7fe43019e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.000+0000 7fe435400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe430101760 0x7fe43019e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.000+0000 7fe435400640 1 -- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe430102120 msgr2=0x7fe43019ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.000+0000 7fe435400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe430102120 0x7fe43019ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.000+0000 7fe435400640 1 -- 172.21.15.102:0/3754698341 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4301a34d0 con 0x7fe430107770 2024-04-25T02:10:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.000+0000 7fe434a00640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe430101760 0x7fe43019e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:10:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.001+0000 7fe435400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430107770 0x7fe4301a2da0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe428003ce0 tx=0x7fe428003d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.001+0000 7fe42d600640 1 -- 172.21.15.102:0/3754698341 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe428031650 con 0x7fe430107770 2024-04-25T02:10:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.001+0000 7fe42d600640 1 -- 172.21.15.102:0/3754698341 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4280317f0 con 0x7fe430107770 2024-04-25T02:10:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.001+0000 7fe42d600640 1 -- 172.21.15.102:0/3754698341 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42802f9f0 con 0x7fe430107770 2024-04-25T02:10:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.001+0000 7fe437400640 1 -- 172.21.15.102:0/3754698341 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4301a3760 con 0x7fe430107770 2024-04-25T02:10:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.002+0000 7fe437400640 1 -- 172.21.15.102:0/3754698341 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4301ab050 con 0x7fe430107770 2024-04-25T02:10:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.007+0000 7fe42d600640 1 -- 172.21.15.102:0/3754698341 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe42802fc10 con 0x7fe430107770 2024-04-25T02:10:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.008+0000 7fe43a21a740 1 Processor -- start 2024-04-25T02:10:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.008+0000 7fe43a21a740 1 -- start start 2024-04-25T02:10:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.009+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad8e00 0x559bcfad91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.009+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.009+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfae3fe0 0x559bcfae63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.009+0000 7fe43a21a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559bcfa3cab0 con 0x559bcfad8e00 2024-04-25T02:10:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.009+0000 7fe43a21a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559bcf9e5640 con 0x559bcfad9720 2024-04-25T02:10:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.009+0000 7fe43a21a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559bcfa38a70 con 0x559bcfae3fe0 2024-04-25T02:10:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.010+0000 7fe42d600640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4140a2e20 0x7fe4140a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.010+0000 7fe42fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfae3fe0 0x559bcfae63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.010+0000 7fe42fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfae3fe0 0x559bcfae63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:43390/0 (socket says 172.21.15.102:43390) 2024-04-25T02:10:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.010+0000 7fe42fe00640 1 -- 172.21.15.102:0/3802872401 learned_addr learned my addr 172.21.15.102:0/3802872401 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.011+0000 7fe42fe00640 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad8e00 0x559bcfad91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.010+0000 7fe434a00640 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.011+0000 7fe42d600640 1 -- 172.21.15.102:0/3754698341 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8557+0+0 (secure 0 0 0) 0x7fe4280f07c0 con 0x7fe430107770 2024-04-25T02:10:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.011+0000 7fe435400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4140a2e20 0x7fe4140a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.011+0000 7fe42fe00640 1 -- 172.21.15.102:0/3802872401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 msgr2=0x559bcfae3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.011+0000 7fe42fe00640 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.011+0000 7fe42fe00640 1 -- 172.21.15.102:0/3802872401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad8e00 msgr2=0x559bcfad91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.011+0000 7fe42fe00640 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad8e00 0x559bcfad91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.011+0000 7fe42fe00640 1 -- 172.21.15.102:0/3802872401 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bcf751210 con 0x559bcfae3fe0 2024-04-25T02:10:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.011+0000 7fe434a00640 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:10:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.012+0000 7fe435400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4140a2e20 0x7fe4140a52e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe4280b9230 tx=0x7fe42808ee50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.012+0000 7fe42fe00640 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfae3fe0 0x559bcfae63d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe420011470 tx=0x7fe420009850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=330ac6b3c4cc43b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.012+0000 7fe425c00640 1 -- 172.21.15.102:0/3802872401 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420004050 con 0x559bcfae3fe0 2024-04-25T02:10:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.012+0000 7fe425c00640 1 -- 172.21.15.102:0/3802872401 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe420005500 con 0x559bcfae3fe0 2024-04-25T02:10:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.012+0000 7fe425c00640 1 -- 172.21.15.102:0/3802872401 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4200035f0 con 0x559bcfae3fe0 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.013+0000 7fe43a21a740 1 -- 172.21.15.102:0/3802872401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfae3fe0 msgr2=0x559bcfae63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.013+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfae3fe0 0x559bcfae63d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe420011470 tx=0x7fe420009850 comp rx=0 tx=0).stop 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.013+0000 7fe43a21a740 1 -- 172.21.15.102:0/3802872401 shutdown_connections 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.013+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfae3fe0 0x559bcfae63d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.013+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.013+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3802872401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad8e00 0x559bcfad91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.013+0000 7fe43a21a740 1 -- 172.21.15.102:0/3802872401 >> 172.21.15.102:0/3802872401 conn(0x559bcfad5310 msgr2=0x559bcfad56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.013+0000 7fe43a21a740 1 -- 172.21.15.102:0/3802872401 shutdown_connections 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.013+0000 7fe43a21a740 1 -- 172.21.15.102:0/3802872401 wait complete. 2024-04-25T02:10:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.014+0000 7fe43a21a740 1 -- start start 2024-04-25T02:10:17.022 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.014+0000 7fe43a21a740 1 Processor -- start 2024-04-25T02:10:17.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfae0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad9720 0x559bcfae1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.023 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfae3fe0 0x559bcfae1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe42fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfae0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe43a21a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x559bcfae6c10 con 0x559bcfad9720 2024-04-25T02:10:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe43a21a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x559bcf9f8990 con 0x559bcfae3fe0 2024-04-25T02:10:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe43a21a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559bcfae6d90 con 0x559bcfad8e00 2024-04-25T02:10:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe42fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfae0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:43398/0 (socket says 172.21.15.102:43398) 2024-04-25T02:10:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe42fe00640 1 -- 172.21.15.102:0/1196982986 learned_addr learned my addr 172.21.15.102:0/1196982986 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:17.024 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.016+0000 7fe43a21a740 10 client.0 _ll_register_callbacks cb 0x559bcf9fa950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.017+0000 7fe42fe00640 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfae3fe0 0x559bcfae1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.017+0000 7fe434a00640 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad9720 0x559bcfae1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.017+0000 7fe42fe00640 1 -- 172.21.15.102:0/1196982986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfae3fe0 msgr2=0x559bcfae1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.017+0000 7fe42fe00640 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfae3fe0 0x559bcfae1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.017+0000 7fe42fe00640 1 -- 172.21.15.102:0/1196982986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad9720 msgr2=0x559bcfae1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.017+0000 7fe42fe00640 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad9720 0x559bcfae1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.017+0000 7fe42fe00640 1 -- 172.21.15.102:0/1196982986 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bcf81d620 con 0x559bcfad8e00 2024-04-25T02:10:17.025 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.017+0000 7fe42fe00640 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfae0fd0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe420011440 tx=0x7fe420012fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d0c769ca64db45e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.018+0000 7fe41a000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:10:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.019+0000 7fe41a000640 1 -- 172.21.15.102:0/1196982986 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe420028070 con 0x559bcfad8e00 2024-04-25T02:10:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.019+0000 7fe41a000640 1 -- 172.21.15.102:0/1196982986 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8557+0+0 (secure 0 0 0) 0x7fe42001e660 con 0x559bcfad8e00 2024-04-25T02:10:17.026 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.019+0000 7fe43a21a740 1 -- 172.21.15.102:0/1196982986 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe420005500 con 0x559bcfad8e00 2024-04-25T02:10:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.020+0000 7fe41a000640 1 -- 172.21.15.102:0/1196982986 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fe420005330 con 0x559bcfad8e00 2024-04-25T02:10:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.020+0000 7fe43a21a740 10 client.27664 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.020+0000 7fe43a21a740 1 -- 172.21.15.102:0/1196982986 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x559bcfaf2b00 con 0x559bcfad8e00 2024-04-25T02:10:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.021+0000 7fe41a000640 1 -- 172.21.15.102:0/1196982986 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe420005500 con 0x559bcfad8e00 2024-04-25T02:10:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.021+0000 7fe41a000640 1 -- 172.21.15.102:0/1196982986 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7fe42000e8c0 con 0x559bcfad8e00 2024-04-25T02:10:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.021+0000 7fe43a21a740 10 client.27664 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.021+0000 7fe43a21a740 10 client.27664 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.021+0000 7fe43a21a740 20 client.27664 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.021+0000 7fe43a21a740 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559bcfaf4380 0x559bcfaf6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.021+0000 7fe43a21a740 4 client.27664 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:10:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.021+0000 7fe43a21a740 1 -- 172.21.15.102:0/1196982986 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bcfa49ff0 con 0x559bcfaf4380 2024-04-25T02:10:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.022+0000 7fe434a00640 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559bcfaf4380 0x559bcfaf6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.022+0000 7fe434a00640 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559bcfaf4380 0x559bcfaf6800 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f931801ddce2346b server_cookie=231a36a08f889333 in_seq=0 out_seq=0 2024-04-25T02:10:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.022+0000 7fe41a000640 10 client.27664 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.023+0000 7fe41a000640 1 -- 172.21.15.102:0/1196982986 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559bcfa49ff0 con 0x559bcfaf4380 2024-04-25T02:10:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.023+0000 7fe41a000640 10 client.27664 handle_command_reply: tid=0 2024-04-25T02:10:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.023+0000 7fe43a21a740 1 client.27664 shutdown 2024-04-25T02:10:17.031 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.024+0000 7fe43a21a740 1 -- 172.21.15.102:0/1196982986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 msgr2=0x559bcfae0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.024+0000 7fe43a21a740 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfae0fd0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe420011440 tx=0x7fe420012fc0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.024+0000 7fe43a21a740 1 -- 172.21.15.102:0/1196982986 shutdown_connections 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.024+0000 7fe43a21a740 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559bcfaf4380 0x559bcfaf6800 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.024+0000 7fe43a21a740 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfae3fe0 0x559bcfae1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.024+0000 7fe43a21a740 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfad9720 0x559bcfae1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.024+0000 7fe43a21a740 1 --2- 172.21.15.102:0/1196982986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfae0fd0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.024+0000 7fe41a000640 0 client.27664 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.024+0000 7fe43a21a740 1 -- 172.21.15.102:0/1196982986 >> 172.21.15.102:0/1196982986 conn(0x559bcfad5310 msgr2=0x559bcfadb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.025+0000 7fe43a21a740 1 -- 172.21.15.102:0/1196982986 shutdown_connections 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.025+0000 7fe43a21a740 1 -- 172.21.15.102:0/1196982986 wait complete. 2024-04-25T02:10:17.032 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.025+0000 7fe43a21a740 20 client.27664 trim_cache size 0 max 16384 2024-04-25T02:10:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.049+0000 7fe43a21a740 1 Processor -- start 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.049+0000 7fe43a21a740 1 -- start start 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.049+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfaefb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.049+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.049+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfafbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.049+0000 7fe43a21a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4301b6390 con 0x559bcfae3fe0 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.049+0000 7fe43a21a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4301b3c00 con 0x559bcfad9720 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.049+0000 7fe43a21a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe4301b1a70 con 0x559bcfad8e00 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe434a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe434a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae1890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:47272/0 (socket says 172.21.15.102:47272) 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe434a00640 1 -- 172.21.15.102:0/2095157547 learned_addr learned my addr 172.21.15.102:0/2095157547 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe42fe00640 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfaefb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe42fe00640 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfafbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.058 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe434a00640 1 -- 172.21.15.102:0/2095157547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 msgr2=0x559bcfaefb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe434a00640 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfaefb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe434a00640 1 -- 172.21.15.102:0/2095157547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 msgr2=0x559bcfafbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe434a00640 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfafbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe434a00640 1 -- 172.21.15.102:0/2095157547 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bcfada1c0 con 0x559bcfad9720 2024-04-25T02:10:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe42fe00640 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfafbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:10:17.059 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.050+0000 7fe434a00640 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae1890 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x559bcfae0850 tx=0x7fe41c00f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef2f82fc73c6d9f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.051+0000 7fe425200640 1 -- 172.21.15.102:0/2095157547 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c004050 con 0x559bcfad9720 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.051+0000 7fe425200640 1 -- 172.21.15.102:0/2095157547 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c0041f0 con 0x559bcfad9720 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.051+0000 7fe425200640 1 -- 172.21.15.102:0/2095157547 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c01c3f0 con 0x559bcfad9720 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.051+0000 7fe43a21a740 1 -- 172.21.15.102:0/2095157547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 msgr2=0x559bcfae1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.051+0000 7fe43a21a740 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae1890 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x559bcfae0850 tx=0x7fe41c00f7f0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.052+0000 7fe43a21a740 1 -- 172.21.15.102:0/2095157547 shutdown_connections 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.052+0000 7fe43a21a740 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfafbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.052+0000 7fe43a21a740 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad9720 0x559bcfae1890 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.052+0000 7fe43a21a740 1 --2- 172.21.15.102:0/2095157547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad8e00 0x559bcfaefb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.052+0000 7fe43a21a740 1 -- 172.21.15.102:0/2095157547 >> 172.21.15.102:0/2095157547 conn(0x559bcfad5310 msgr2=0x559bcfae2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.053+0000 7fe43a21a740 1 -- 172.21.15.102:0/2095157547 shutdown_connections 2024-04-25T02:10:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.053+0000 7fe43a21a740 1 -- 172.21.15.102:0/2095157547 wait complete. 2024-04-25T02:10:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.054+0000 7fe43a21a740 1 -- start start 2024-04-25T02:10:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.054+0000 7fe43a21a740 1 Processor -- start 2024-04-25T02:10:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.054+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad8e00 0x559bcfaec200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad9720 0x559bcfadd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe43a21a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfaddde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe43a21a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4301b20c0 con 0x559bcfae3fe0 2024-04-25T02:10:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe43a21a740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe4301b3530 con 0x559bcfad8e00 2024-04-25T02:10:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe43a21a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe4301b2790 con 0x559bcfad9720 2024-04-25T02:10:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe43a21a740 10 client.0 _ll_register_callbacks cb 0x7fe420009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe42fe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad8e00 0x559bcfaec200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe434a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad9720 0x559bcfadd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe434a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad9720 0x559bcfadd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:43418/0 (socket says 172.21.15.102:43418) 2024-04-25T02:10:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe434a00640 1 -- 172.21.15.102:0/3901244519 learned_addr learned my addr 172.21.15.102:0/3901244519 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe42fe00640 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfaddde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe434a00640 1 -- 172.21.15.102:0/3901244519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad8e00 msgr2=0x559bcfaec200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:17.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe434a00640 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad8e00 0x559bcfaec200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe434a00640 1 -- 172.21.15.102:0/3901244519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 msgr2=0x559bcfaddde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:17.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe434a00640 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfaddde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe434a00640 1 -- 172.21.15.102:0/3901244519 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bcfade350 con 0x559bcfad9720 2024-04-25T02:10:17.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.055+0000 7fe42fe00640 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfaddde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:10:17.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.056+0000 7fe434a00640 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad9720 0x559bcfadd8a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe41c006010 tx=0x7fe41c00fe50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11194144ddb38023 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:17.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.057+0000 7fe41a000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:10:17.064 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.057+0000 7fe41a000640 1 -- 172.21.15.102:0/3901244519 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c005450 con 0x559bcfad9720 2024-04-25T02:10:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.057+0000 7fe41a000640 1 -- 172.21.15.102:0/3901244519 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8557+0+0 (secure 0 0 0) 0x7fe41c004730 con 0x559bcfad9720 2024-04-25T02:10:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.057+0000 7fe43a21a740 1 -- 172.21.15.102:0/3901244519 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe41c0041f0 con 0x559bcfad9720 2024-04-25T02:10:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe41a000640 1 -- 172.21.15.102:0/3901244519 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fe41c0044a0 con 0x559bcfad9720 2024-04-25T02:10:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe43a21a740 10 client.27670 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe43a21a740 1 -- 172.21.15.102:0/3901244519 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x559bcfb5a4d0 con 0x559bcfad9720 2024-04-25T02:10:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe41a000640 1 -- 172.21.15.102:0/3901244519 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c0041f0 con 0x559bcfad9720 2024-04-25T02:10:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe41a000640 1 -- 172.21.15.102:0/3901244519 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7fe41c003b90 con 0x559bcfad9720 2024-04-25T02:10:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe43a21a740 10 client.27670 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe43a21a740 10 client.27670 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe43a21a740 20 client.27670 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559bcfb5ac80 0x559bcfb5d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe43a21a740 4 client.27670 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:10:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.058+0000 7fe43a21a740 1 -- 172.21.15.102:0/3901244519 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559bcfa44340 con 0x559bcfb5ac80 2024-04-25T02:10:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.059+0000 7fe42fe00640 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559bcfb5ac80 0x559bcfb5d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:17.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.059+0000 7fe42fe00640 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559bcfb5ac80 0x559bcfb5d0a0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6620270ff97e3a5 server_cookie=15e558beb0b5bc53 in_seq=0 out_seq=0 2024-04-25T02:10:17.067 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.059+0000 7fe41a000640 10 client.27670 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.078+0000 7fe41a000640 1 -- 172.21.15.102:0/3901244519 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559bcfa44340 con 0x559bcfb5ac80 2024-04-25T02:10:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.078+0000 7fe41a000640 10 client.27670 handle_command_reply: tid=0 2024-04-25T02:10:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe43a21a740 1 client.27670 shutdown 2024-04-25T02:10:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe43a21a740 1 -- 172.21.15.102:0/3901244519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad9720 msgr2=0x559bcfadd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad9720 0x559bcfadd8a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe41c006010 tx=0x7fe41c00fe50 comp rx=0 tx=0).stop 2024-04-25T02:10:17.086 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe43a21a740 1 -- 172.21.15.102:0/3901244519 shutdown_connections 2024-04-25T02:10:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x559bcfb5ac80 0x559bcfb5d0a0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559bcfae3fe0 0x559bcfaddde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559bcfad9720 0x559bcfadd8a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe43a21a740 1 --2- 172.21.15.102:0/3901244519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x559bcfad8e00 0x559bcfaec200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe43a21a740 1 -- 172.21.15.102:0/3901244519 >> 172.21.15.102:0/3901244519 conn(0x559bcfad5310 msgr2=0x559bcfae2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:17.087 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.079+0000 7fe41a000640 0 client.27670 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.080+0000 7fe43a21a740 1 -- 172.21.15.102:0/3901244519 shutdown_connections 2024-04-25T02:10:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.080+0000 7fe43a21a740 1 -- 172.21.15.102:0/3901244519 wait complete. 2024-04-25T02:10:17.088 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.081+0000 7fe43a21a740 20 client.27670 trim_cache size 0 max 16384 2024-04-25T02:10:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:10:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:10:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:10:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:10:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 -- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4140a2e20 msgr2=0x7fe4140a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:17.089 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4140a2e20 0x7fe4140a52e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe4280b9230 tx=0x7fe42808ee50 comp rx=0 tx=0).stop 2024-04-25T02:10:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 -- 172.21.15.102:0/3754698341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430107770 msgr2=0x7fe4301a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430107770 0x7fe4301a2da0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe428003ce0 tx=0x7fe428003d10 comp rx=0 tx=0).stop 2024-04-25T02:10:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 -- 172.21.15.102:0/3754698341 shutdown_connections 2024-04-25T02:10:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe4140a2e20 0x7fe4140a52e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe430107770 0x7fe4301a2da0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe430102120 0x7fe43019ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 --2- 172.21.15.102:0/3754698341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe430101760 0x7fe43019e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.082+0000 7fe437400640 1 -- 172.21.15.102:0/3754698341 >> 172.21.15.102:0/3754698341 conn(0x7fe4300fd1f0 msgr2=0x7fe4300fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:17.090 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.083+0000 7fe437400640 1 -- 172.21.15.102:0/3754698341 shutdown_connections 2024-04-25T02:10:17.091 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:17.083+0000 7fe437400640 1 -- 172.21.15.102:0/3754698341 wait complete. 2024-04-25T02:10:17.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:17 smithi102 ceph-mon[27796]: pgmap v468: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 8.8 MiB/s wr, 773 op/s 2024-04-25T02:10:17.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:17 smithi110 ceph-mon[32848]: pgmap v468: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 8.8 MiB/s wr, 773 op/s 2024-04-25T02:10:17.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:17 smithi193 ceph-mon[32981]: pgmap v468: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 8.8 MiB/s wr, 773 op/s 2024-04-25T02:10:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 865.0 s, 1868.4 tps, lat 55.087 ms stddev 70.434 2024-04-25T02:10:18.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:18 smithi110 ceph-mon[32848]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T02:10:18.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:18 smithi193 ceph-mon[32981]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T02:10:18.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:18 smithi102 ceph-mon[27796]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T02:10:19.745 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:19 smithi110 ceph-mon[32848]: pgmap v470: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.8 MiB/s wr, 673 op/s 2024-04-25T02:10:19.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:19 smithi193 ceph-mon[32981]: pgmap v470: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.8 MiB/s wr, 673 op/s 2024-04-25T02:10:19.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:19 smithi102 ceph-mon[27796]: pgmap v470: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.8 MiB/s wr, 673 op/s 2024-04-25T02:10:21.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:21 smithi102 ceph-mon[27796]: pgmap v471: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 622 op/s 2024-04-25T02:10:21.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:21 smithi110 ceph-mon[32848]: pgmap v471: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 622 op/s 2024-04-25T02:10:21.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:21 smithi193 ceph-mon[32981]: pgmap v471: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 622 op/s 2024-04-25T02:10:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 870.0 s, 2139.6 tps, lat 45.780 ms stddev 58.374 2024-04-25T02:10:23.648 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:23 smithi102 ceph-mon[27796]: pgmap v472: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 622 op/s 2024-04-25T02:10:23.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:23 smithi110 ceph-mon[32848]: pgmap v472: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 622 op/s 2024-04-25T02:10:23.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:23 smithi193 ceph-mon[32981]: pgmap v472: 97 pgs: 97 active+clean; 5.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 622 op/s 2024-04-25T02:10:25.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:25 smithi110 ceph-mon[32848]: pgmap v473: 97 pgs: 97 active+clean; 5.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.9 MiB/s wr, 683 op/s 2024-04-25T02:10:25.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:25 smithi193 ceph-mon[32981]: pgmap v473: 97 pgs: 97 active+clean; 5.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.9 MiB/s wr, 683 op/s 2024-04-25T02:10:25.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:25 smithi102 ceph-mon[27796]: pgmap v473: 97 pgs: 97 active+clean; 5.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.9 MiB/s wr, 683 op/s 2024-04-25T02:10:27.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:27 smithi193 ceph-mon[32981]: pgmap v474: 97 pgs: 97 active+clean; 5.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.1 MiB/s wr, 683 op/s 2024-04-25T02:10:27.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:27 smithi102 ceph-mon[27796]: pgmap v474: 97 pgs: 97 active+clean; 5.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.1 MiB/s wr, 683 op/s 2024-04-25T02:10:27.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:27 smithi110 ceph-mon[32848]: pgmap v474: 97 pgs: 97 active+clean; 5.3 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.1 MiB/s wr, 683 op/s 2024-04-25T02:10:28.080 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 875.0 s, 1202.1 tps, lat 84.352 ms stddev 99.588 2024-04-25T02:10:29.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:29 smithi193 ceph-mon[32981]: pgmap v475: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.9 MiB/s wr, 715 op/s 2024-04-25T02:10:29.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:29 smithi102 ceph-mon[27796]: pgmap v475: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.9 MiB/s wr, 715 op/s 2024-04-25T02:10:29.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:29 smithi110 ceph-mon[32848]: pgmap v475: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.9 MiB/s wr, 715 op/s 2024-04-25T02:10:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.274+0000 7f66fa800640 1 -- 172.21.15.102:0/507486545 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66f4107770 msgr2=0x7f66f4107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.274+0000 7f66fa800640 1 --2- 172.21.15.102:0/507486545 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66f4107770 0x7f66f4107b50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009930 tx=0x7f66e402a440 comp rx=0 tx=0).stop 2024-04-25T02:10:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.274+0000 7f66fa800640 1 -- 172.21.15.102:0/507486545 shutdown_connections 2024-04-25T02:10:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.274+0000 7f66fa800640 1 --2- 172.21.15.102:0/507486545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f4102120 0x7f66f410e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.274+0000 7f66fa800640 1 --2- 172.21.15.102:0/507486545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f4101760 0x7f66f4101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.274+0000 7f66fa800640 1 --2- 172.21.15.102:0/507486545 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66f4107770 0x7f66f4107b50 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.274+0000 7f66fa800640 1 -- 172.21.15.102:0/507486545 >> 172.21.15.102:0/507486545 conn(0x7f66f40fd1f0 msgr2=0x7f66f40ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:30.282 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.274+0000 7f66fa800640 1 -- 172.21.15.102:0/507486545 shutdown_connections 2024-04-25T02:10:30.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.275+0000 7f66fa800640 1 -- 172.21.15.102:0/507486545 wait complete. 2024-04-25T02:10:30.283 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66fa800640 1 Processor -- start 2024-04-25T02:10:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66fa800640 1 -- start start 2024-04-25T02:10:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66fa800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f4101760 0x7f66f419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66fa800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66f4102120 0x7f66f419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66fa800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f4107770 0x7f66f419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66fa800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f66f4075960 con 0x7f66f4107770 2024-04-25T02:10:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66fa800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f66f40757e0 con 0x7f66f4101760 2024-04-25T02:10:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66fa800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66f4113fb0 con 0x7f66f4102120 2024-04-25T02:10:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66f3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f4101760 0x7f66f419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f4107770 0x7f66f419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.276+0000 7f66f3400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f4101760 0x7f66f419a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:60532/0 (socket says 172.21.15.102:60532) 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.277+0000 7f66f3400640 1 -- 172.21.15.102:0/1264360871 learned_addr learned my addr 172.21.15.102:0/1264360871 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.277+0000 7f66f2a00640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66f4102120 0x7f66f419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.277+0000 7f66f3400640 1 -- 172.21.15.102:0/1264360871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66f4102120 msgr2=0x7f66f419a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.277+0000 7f66f3400640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66f4102120 0x7f66f419a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.277+0000 7f66f3400640 1 -- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f4107770 msgr2=0x7f66f419e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.277+0000 7f66f3400640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f4107770 0x7f66f419e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.277+0000 7f66f3400640 1 -- 172.21.15.102:0/1264360871 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f419f040 con 0x7f66f4101760 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.277+0000 7f66f3e00640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f4107770 0x7f66f419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.277+0000 7f66f3400640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f4101760 0x7f66f419a050 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f66e402a960 tx=0x7f66e4003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.278+0000 7f66ebe00640 1 -- 172.21.15.102:0/1264360871 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e403f070 con 0x7f66f4101760 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.278+0000 7f66ebe00640 1 -- 172.21.15.102:0/1264360871 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66e4031c90 con 0x7f66f4101760 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.278+0000 7f66fa800640 1 -- 172.21.15.102:0/1264360871 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f419f2d0 con 0x7f66f4101760 2024-04-25T02:10:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.278+0000 7f66ebe00640 1 -- 172.21.15.102:0/1264360871 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4035030 con 0x7f66f4101760 2024-04-25T02:10:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.279+0000 7f66fa800640 1 -- 172.21.15.102:0/1264360871 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f41a6bc0 con 0x7f66f4101760 2024-04-25T02:10:30.290 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.282+0000 7f66fd7c5740 1 Processor -- start 2024-04-25T02:10:30.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.284+0000 7f66fd7c5740 1 -- start start 2024-04-25T02:10:30.291 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.284+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342d71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.285+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d7720 0x5638342e1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.292 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.285+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342e1fe0 0x5638342e43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.286+0000 7f66f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342d71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.286+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x56383423aab0 con 0x5638342d6e00 2024-04-25T02:10:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.286+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5638341e3640 con 0x5638342e1fe0 2024-04-25T02:10:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.286+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563834236a70 con 0x5638342d7720 2024-04-25T02:10:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.286+0000 7f66f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342d71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41996/0 (socket says 172.21.15.102:41996) 2024-04-25T02:10:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.286+0000 7f66f3e00640 1 -- 172.21.15.102:0/815012388 learned_addr learned my addr 172.21.15.102:0/815012388 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.286+0000 7f66ebe00640 1 -- 172.21.15.102:0/1264360871 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f66e4042070 con 0x7f66f4101760 2024-04-25T02:10:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.286+0000 7f66f2a00640 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342e1fe0 0x5638342e43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.286+0000 7f66f3400640 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d7720 0x5638342e1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.287+0000 7f66f3e00640 1 -- 172.21.15.102:0/815012388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d7720 msgr2=0x5638342e1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.287+0000 7f66f3e00640 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d7720 0x5638342e1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.287+0000 7f66f3e00640 1 -- 172.21.15.102:0/815012388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342e1fe0 msgr2=0x5638342e43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.287+0000 7f66f3e00640 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342e1fe0 0x5638342e43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.287+0000 7f66f3e00640 1 -- 172.21.15.102:0/815012388 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563833f4f210 con 0x5638342d6e00 2024-04-25T02:10:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.287+0000 7f66f2a00640 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342e1fe0 0x5638342e43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:10:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.287+0000 7f66f3e00640 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342d71e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00f790 tx=0x7f66ec00fc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4957cbdf242808 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.287+0000 7f66e8c00640 1 -- 172.21.15.102:0/815012388 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec004d20 con 0x5638342d6e00 2024-04-25T02:10:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.288+0000 7f66e8c00640 1 -- 172.21.15.102:0/815012388 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66ec00de00 con 0x5638342d6e00 2024-04-25T02:10:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.288+0000 7f66e8c00640 1 -- 172.21.15.102:0/815012388 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec010430 con 0x5638342d6e00 2024-04-25T02:10:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.288+0000 7f66ebe00640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f66d80a2ea0 0x7f66d80a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.288+0000 7f66ebe00640 1 -- 172.21.15.102:0/1264360871 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8513+0+0 (secure 0 0 0) 0x7f66e40ef7c0 con 0x7f66f4101760 2024-04-25T02:10:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.288+0000 7f66f3400640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f66d80a2ea0 0x7f66d80a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.288+0000 7f66fd7c5740 1 -- 172.21.15.102:0/815012388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 msgr2=0x5638342d71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.288+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342d71e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00f790 tx=0x7f66ec00fc50 comp rx=0 tx=0).stop 2024-04-25T02:10:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.289+0000 7f66fd7c5740 1 -- 172.21.15.102:0/815012388 shutdown_connections 2024-04-25T02:10:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.289+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342e1fe0 0x5638342e43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.289+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d7720 0x5638342e1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.289+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/815012388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342d71e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.289+0000 7f66fd7c5740 1 -- 172.21.15.102:0/815012388 >> 172.21.15.102:0/815012388 conn(0x5638342d3310 msgr2=0x5638342d36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.289+0000 7f66fd7c5740 1 -- 172.21.15.102:0/815012388 shutdown_connections 2024-04-25T02:10:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.289+0000 7f66fd7c5740 1 -- 172.21.15.102:0/815012388 wait complete. 2024-04-25T02:10:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.289+0000 7f66f3400640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f66d80a2ea0 0x7f66d80a5360 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x5638342d83a0 tx=0x7f66e4000bc0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.290+0000 7f66fd7c5740 1 -- start start 2024-04-25T02:10:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.290+0000 7f66fd7c5740 1 Processor -- start 2024-04-25T02:10:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342deef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342df430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342e1fe0 0x5638342df970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5638342e4c10 con 0x5638342d6e00 2024-04-25T02:10:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x5638341f6990 con 0x5638342d7720 2024-04-25T02:10:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5638342e4d90 con 0x5638342e1fe0 2024-04-25T02:10:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342deef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342deef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42000/0 (socket says 172.21.15.102:42000) 2024-04-25T02:10:30.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66f3e00640 1 -- 172.21.15.102:0/2635468241 learned_addr learned my addr 172.21.15.102:0/2635468241 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:30.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66f3e00640 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342e1fe0 0x5638342df970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66fd7c5740 10 client.0 _ll_register_callbacks cb 0x5638341f8950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:30.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.291+0000 7f66f2a00640 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342df430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.292+0000 7f66f3e00640 1 -- 172.21.15.102:0/2635468241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 msgr2=0x5638342df430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:30.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.292+0000 7f66f3e00640 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342df430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.292+0000 7f66f3e00640 1 -- 172.21.15.102:0/2635468241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 msgr2=0x5638342deef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:30.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.292+0000 7f66f3e00640 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342deef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.300 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.292+0000 7f66f3e00640 1 -- 172.21.15.102:0/2635468241 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56383401b620 con 0x5638342e1fe0 2024-04-25T02:10:30.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.292+0000 7f66f3e00640 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342e1fe0 0x5638342df970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00f520 tx=0x7f66ec007520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b608d0a321636ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:30.301 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.292+0000 7f66dd600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.293+0000 7f66dd600640 1 -- 172.21.15.102:0/2635468241 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec01d040 con 0x5638342e1fe0 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.293+0000 7f66fd7c5740 1 -- 172.21.15.102:0/2635468241 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66ec00de00 con 0x5638342e1fe0 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.293+0000 7f66dd600640 1 -- 172.21.15.102:0/2635468241 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8513+0+0 (secure 0 0 0) 0x7f66ec004ea0 con 0x5638342e1fe0 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.293+0000 7f66dd600640 1 -- 172.21.15.102:0/2635468241 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f66ec0050a0 con 0x5638342e1fe0 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.293+0000 7f66fd7c5740 10 client.27682 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.293+0000 7f66fd7c5740 1 -- 172.21.15.102:0/2635468241 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5638342f92c0 con 0x5638342e1fe0 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.293+0000 7f66dd600640 1 -- 172.21.15.102:0/2635468241 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66ec00de00 con 0x5638342e1fe0 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.294+0000 7f66dd600640 1 -- 172.21.15.102:0/2635468241 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f66ec028410 con 0x5638342e1fe0 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.294+0000 7f66fd7c5740 10 client.27682 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:30.302 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.294+0000 7f66fd7c5740 10 client.27682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.294+0000 7f66fd7c5740 20 client.27682 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.294+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638342fa730 0x5638342fcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.294+0000 7f66fd7c5740 4 client.27682 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:10:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.294+0000 7f66fd7c5740 1 -- 172.21.15.102:0/2635468241 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563834247ff0 con 0x5638342fa730 2024-04-25T02:10:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.294+0000 7f66f2a00640 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638342fa730 0x5638342fcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.303 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.295+0000 7f66f2a00640 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638342fa730 0x5638342fcb20 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=843bfe52ec7e0e87 server_cookie=3739ce138b2a4815 in_seq=0 out_seq=0 2024-04-25T02:10:30.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.295+0000 7f66dd600640 10 client.27682 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:30.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.296+0000 7f66dd600640 1 -- 172.21.15.102:0/2635468241 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563834247ff0 con 0x5638342fa730 2024-04-25T02:10:30.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.296+0000 7f66dd600640 10 client.27682 handle_command_reply: tid=0 2024-04-25T02:10:30.304 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.297+0000 7f66fd7c5740 1 client.27682 shutdown 2024-04-25T02:10:30.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.298+0000 7f66fd7c5740 1 -- 172.21.15.102:0/2635468241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342e1fe0 msgr2=0x5638342df970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:30.305 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.298+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342e1fe0 0x5638342df970 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00f520 tx=0x7f66ec007520 comp rx=0 tx=0).stop 2024-04-25T02:10:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.298+0000 7f66fd7c5740 1 -- 172.21.15.102:0/2635468241 shutdown_connections 2024-04-25T02:10:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.298+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5638342fa730 0x5638342fcb20 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.299+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342e1fe0 0x5638342df970 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.299+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342df430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.299+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/2635468241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342d6e00 0x5638342deef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.299+0000 7f66fd7c5740 1 -- 172.21.15.102:0/2635468241 >> 172.21.15.102:0/2635468241 conn(0x5638342d3310 msgr2=0x5638342d9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.299+0000 7f66dd600640 0 client.27682 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.299+0000 7f66fd7c5740 1 -- 172.21.15.102:0/2635468241 shutdown_connections 2024-04-25T02:10:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.299+0000 7f66fd7c5740 1 -- 172.21.15.102:0/2635468241 wait complete. 2024-04-25T02:10:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.300+0000 7f66fd7c5740 20 client.27682 trim_cache size 0 max 16384 2024-04-25T02:10:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.326+0000 7f66fd7c5740 1 Processor -- start 2024-04-25T02:10:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.326+0000 7f66fd7c5740 1 -- start start 2024-04-25T02:10:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.326+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d6e00 0x5638342f6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.326+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d7720 0x5638343022e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.326+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 0x563834302820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.326+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f66f41ad9e0 con 0x5638342e1fe0 2024-04-25T02:10:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.326+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f66f41ab250 con 0x5638342d6e00 2024-04-25T02:10:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.326+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66f41a9100 con 0x5638342d7720 2024-04-25T02:10:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.327+0000 7f66f3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d6e00 0x5638342f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.327+0000 7f66f3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d6e00 0x5638342f6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:60558/0 (socket says 172.21.15.102:60558) 2024-04-25T02:10:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.327+0000 7f66f3e00640 1 -- 172.21.15.102:0/758407889 learned_addr learned my addr 172.21.15.102:0/758407889 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.327+0000 7f66f3e00640 1 --2- 172.21.15.102:0/758407889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 0x563834302820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.327+0000 7f66f3e00640 1 -- 172.21.15.102:0/758407889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d7720 msgr2=0x5638343022e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T02:10:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.327+0000 7f66f3e00640 1 --2- 172.21.15.102:0/758407889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d7720 0x5638343022e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.327+0000 7f66f3e00640 1 -- 172.21.15.102:0/758407889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 msgr2=0x563834302820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.327+0000 7f66f3e00640 1 --2- 172.21.15.102:0/758407889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 0x563834302820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.327+0000 7f66f3e00640 1 -- 172.21.15.102:0/758407889 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638342ce620 con 0x5638342d6e00 2024-04-25T02:10:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.328+0000 7f66f3e00640 1 --2- 172.21.15.102:0/758407889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d6e00 0x5638342f6390 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00db90 tx=0x7f66ec01a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51c2320681bd23d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.328+0000 7f66dfe00640 1 -- 172.21.15.102:0/758407889 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec00ddd0 con 0x5638342d6e00 2024-04-25T02:10:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.328+0000 7f66dfe00640 1 -- 172.21.15.102:0/758407889 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66ec003e50 con 0x5638342d6e00 2024-04-25T02:10:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.328+0000 7f66dfe00640 1 -- 172.21.15.102:0/758407889 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec028c10 con 0x5638342d6e00 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.329+0000 7f66fd7c5740 1 -- 172.21.15.102:0/758407889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d6e00 msgr2=0x5638342f6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.329+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/758407889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d6e00 0x5638342f6390 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f66ec00db90 tx=0x7f66ec01a040 comp rx=0 tx=0).stop 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.329+0000 7f66fd7c5740 1 -- 172.21.15.102:0/758407889 shutdown_connections 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.329+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/758407889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 0x563834302820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.329+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/758407889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d7720 0x5638343022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.329+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/758407889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d6e00 0x5638342f6390 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.329+0000 7f66fd7c5740 1 -- 172.21.15.102:0/758407889 >> 172.21.15.102:0/758407889 conn(0x5638342d3310 msgr2=0x5638342fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.329+0000 7f66fd7c5740 1 -- 172.21.15.102:0/758407889 shutdown_connections 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.329+0000 7f66fd7c5740 1 -- 172.21.15.102:0/758407889 wait complete. 2024-04-25T02:10:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.330+0000 7f66fd7c5740 1 -- start start 2024-04-25T02:10:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.330+0000 7f66fd7c5740 1 Processor -- start 2024-04-25T02:10:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.330+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d6e00 0x5638342f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.330+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342db760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.330+0000 7f66fd7c5740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 0x5638342dbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.330+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f66f41a9750 con 0x5638342e1fe0 2024-04-25T02:10:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.330+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f66f41aab80 con 0x5638342d7720 2024-04-25T02:10:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.330+0000 7f66fd7c5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66f41a9de0 con 0x5638342d6e00 2024-04-25T02:10:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.330+0000 7f66fd7c5740 10 client.0 _ll_register_callbacks cb 0x7f66ec00f5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d6e00 0x5638342f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.340 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342db760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f2a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342db760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:60562/0 (socket says 172.21.15.102:60562) 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f2a00640 1 -- 172.21.15.102:0/3794704063 learned_addr learned my addr 172.21.15.102:0/3794704063 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f3e00640 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 0x5638342dbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f2a00640 1 -- 172.21.15.102:0/3794704063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d6e00 msgr2=0x5638342f2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f2a00640 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d6e00 0x5638342f2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f2a00640 1 -- 172.21.15.102:0/3794704063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 msgr2=0x5638342dbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f2a00640 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 0x5638342dbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f2a00640 1 -- 172.21.15.102:0/3794704063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638342dc340 con 0x5638342d7720 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f3e00640 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 0x5638342dbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66f2a00640 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342db760 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f66e000a5c0 tx=0x7f66e0013830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=606c23a4b5812633 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66dd600640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66dd600640 1 -- 172.21.15.102:0/3794704063 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0023060 con 0x5638342d7720 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66dd600640 1 -- 172.21.15.102:0/3794704063 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8513+0+0 (secure 0 0 0) 0x7f66e001d230 con 0x5638342d7720 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.331+0000 7f66fd7c5740 1 -- 172.21.15.102:0/3794704063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66ec003e50 con 0x5638342d7720 2024-04-25T02:10:30.341 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66dd600640 1 -- 172.21.15.102:0/3794704063 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f66e001d430 con 0x5638342d7720 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66fd7c5740 10 client.27743 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66fd7c5740 1 -- 172.21.15.102:0/3794704063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x563834360a80 con 0x5638342d7720 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66dd600640 1 -- 172.21.15.102:0/3794704063 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66ec003e50 con 0x5638342d7720 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66dd600640 1 -- 172.21.15.102:0/3794704063 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f66e0011440 con 0x5638342d7720 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66fd7c5740 10 client.27743 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66fd7c5740 10 client.27743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66fd7c5740 20 client.27743 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563834361290 0x5638343636b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66fd7c5740 4 client.27743 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66fd7c5740 1 -- 172.21.15.102:0/3794704063 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563834242340 con 0x563834361290 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.332+0000 7f66f3e00640 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563834361290 0x5638343636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.333+0000 7f66f3e00640 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563834361290 0x5638343636b0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a6c2f6b9f8b6cfb server_cookie=d0d78f21ac90eae4 in_seq=0 out_seq=0 2024-04-25T02:10:30.342 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.333+0000 7f66dd600640 10 client.27743 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.399+0000 7f66dd600640 1 -- 172.21.15.102:0/3794704063 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563834242340 con 0x563834361290 2024-04-25T02:10:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.399+0000 7f66dd600640 10 client.27743 handle_command_reply: tid=0 2024-04-25T02:10:30.406 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.399+0000 7f66fd7c5740 1 client.27743 shutdown 2024-04-25T02:10:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.399+0000 7f66fd7c5740 1 -- 172.21.15.102:0/3794704063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 msgr2=0x5638342db760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.399+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342db760 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f66e000a5c0 tx=0x7f66e0013830 comp rx=0 tx=0).stop 2024-04-25T02:10:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66fd7c5740 1 -- 172.21.15.102:0/3794704063 shutdown_connections 2024-04-25T02:10:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x563834361290 0x5638343636b0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5638342e1fe0 0x5638342dbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.407 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5638342d7720 0x5638342db760 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66fd7c5740 1 --2- 172.21.15.102:0/3794704063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5638342d6e00 0x5638342f2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66fd7c5740 1 -- 172.21.15.102:0/3794704063 >> 172.21.15.102:0/3794704063 conn(0x5638342d3310 msgr2=0x5638342dee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66dd600640 0 client.27743 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66fd7c5740 1 -- 172.21.15.102:0/3794704063 shutdown_connections 2024-04-25T02:10:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66fd7c5740 1 -- 172.21.15.102:0/3794704063 wait complete. 2024-04-25T02:10:30.408 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.400+0000 7f66fd7c5740 20 client.27743 trim_cache size 0 max 16384 2024-04-25T02:10:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:10:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:10:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:10:30.410 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:10:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.403+0000 7f66fa800640 1 -- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f66d80a2ea0 msgr2=0x7f66d80a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.403+0000 7f66fa800640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f66d80a2ea0 0x7f66d80a5360 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x5638342d83a0 tx=0x7f66e4000bc0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.403+0000 7f66fa800640 1 -- 172.21.15.102:0/1264360871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f4101760 msgr2=0x7f66f419a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.403+0000 7f66fa800640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f4101760 0x7f66f419a050 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f66e402a960 tx=0x7f66e4003c00 comp rx=0 tx=0).stop 2024-04-25T02:10:30.411 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.404+0000 7f66fa800640 1 -- 172.21.15.102:0/1264360871 shutdown_connections 2024-04-25T02:10:30.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.404+0000 7f66fa800640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f66d80a2ea0 0x7f66d80a5360 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.404+0000 7f66fa800640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66f4107770 0x7f66f419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.404+0000 7f66fa800640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66f4102120 0x7f66f419a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:30.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.404+0000 7f66fa800640 1 --2- 172.21.15.102:0/1264360871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66f4101760 0x7f66f419a050 secure :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f66e402a960 tx=0x7f66e4003c00 comp rx=0 tx=0).stop 2024-04-25T02:10:30.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.404+0000 7f66fa800640 1 -- 172.21.15.102:0/1264360871 >> 172.21.15.102:0/1264360871 conn(0x7f66f40fd1f0 msgr2=0x7f66f40fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:30.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.404+0000 7f66fa800640 1 -- 172.21.15.102:0/1264360871 shutdown_connections 2024-04-25T02:10:30.412 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:30.404+0000 7f66fa800640 1 -- 172.21.15.102:0/1264360871 wait complete. 2024-04-25T02:10:31.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:31 smithi193 ceph-mon[32981]: pgmap v476: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.7 MiB/s wr, 674 op/s 2024-04-25T02:10:31.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:31 smithi102 ceph-mon[27796]: pgmap v476: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.7 MiB/s wr, 674 op/s 2024-04-25T02:10:31.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:31 smithi110 ceph-mon[32848]: pgmap v476: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.7 MiB/s wr, 674 op/s 2024-04-25T02:10:33.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 880.0 s, 1291.0 tps, lat 76.513 ms stddev 90.124 2024-04-25T02:10:33.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:33 smithi102 ceph-mon[27796]: pgmap v477: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.0 MiB/s wr, 588 op/s 2024-04-25T02:10:33.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:33 smithi110 ceph-mon[32848]: pgmap v477: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.0 MiB/s wr, 588 op/s 2024-04-25T02:10:34.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:33 smithi193 ceph-mon[32981]: pgmap v477: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 6.0 MiB/s wr, 588 op/s 2024-04-25T02:10:35.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:35 smithi102 ceph-mon[27796]: pgmap v478: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 7.2 MiB/s wr, 743 op/s 2024-04-25T02:10:35.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:35 smithi110 ceph-mon[32848]: pgmap v478: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 7.2 MiB/s wr, 743 op/s 2024-04-25T02:10:36.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:35 smithi193 ceph-mon[32981]: pgmap v478: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 7.2 MiB/s wr, 743 op/s 2024-04-25T02:10:37.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:37 smithi102 ceph-mon[27796]: pgmap v479: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.1 MiB/s wr, 631 op/s 2024-04-25T02:10:37.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:37 smithi110 ceph-mon[32848]: pgmap v479: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.1 MiB/s wr, 631 op/s 2024-04-25T02:10:38.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:37 smithi193 ceph-mon[32981]: pgmap v479: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 6.1 MiB/s wr, 631 op/s 2024-04-25T02:10:38.077 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 885.0 s, 1334.6 tps, lat 74.594 ms stddev 95.791 2024-04-25T02:10:39.796 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:39 smithi110 ceph-mon[32848]: pgmap v480: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 6.3 MiB/s wr, 699 op/s 2024-04-25T02:10:39.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:39 smithi102 ceph-mon[27796]: pgmap v480: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 6.3 MiB/s wr, 699 op/s 2024-04-25T02:10:40.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:39 smithi193 ceph-mon[32981]: pgmap v480: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 6.3 MiB/s wr, 699 op/s 2024-04-25T02:10:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:41 smithi102 ceph-mon[27796]: pgmap v481: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.4 MiB/s wr, 670 op/s 2024-04-25T02:10:41.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:41 smithi110 ceph-mon[32848]: pgmap v481: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.4 MiB/s wr, 670 op/s 2024-04-25T02:10:42.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:41 smithi193 ceph-mon[32981]: pgmap v481: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.4 MiB/s wr, 670 op/s 2024-04-25T02:10:43.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 890.0 s, 1331.2 tps, lat 75.876 ms stddev 92.443 2024-04-25T02:10:43.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.591+0000 7f096c400640 1 -- 172.21.15.102:0/1238052861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0964103920 msgr2=0x7f0964103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:43.599 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.591+0000 7f096c400640 1 --2- 172.21.15.102:0/1238052861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0964103920 0x7f0964103da0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0950009900 tx=0x7f095002a410 comp rx=0 tx=0).stop 2024-04-25T02:10:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.592+0000 7f096c400640 1 -- 172.21.15.102:0/1238052861 shutdown_connections 2024-04-25T02:10:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.592+0000 7f096c400640 1 --2- 172.21.15.102:0/1238052861 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09641042e0 0x7f09641107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.592+0000 7f096c400640 1 --2- 172.21.15.102:0/1238052861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0964103920 0x7f0964103da0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.592+0000 7f096c400640 1 --2- 172.21.15.102:0/1238052861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0964109930 0x7f0964109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.592+0000 7f096c400640 1 -- 172.21.15.102:0/1238052861 >> 172.21.15.102:0/1238052861 conn(0x7f09640ff430 msgr2=0x7f0964101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.592+0000 7f096c400640 1 -- 172.21.15.102:0/1238052861 shutdown_connections 2024-04-25T02:10:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.592+0000 7f096c400640 1 -- 172.21.15.102:0/1238052861 wait complete. 2024-04-25T02:10:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.593+0000 7f096c400640 1 Processor -- start 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.593+0000 7f096c400640 1 -- start start 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.594+0000 7f096c400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0964103920 0x7f0964075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.594+0000 7f096c400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09641042e0 0x7f0964076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.594+0000 7f096c400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0964109930 0x7f09640766d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.594+0000 7f096c400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0964113d90 con 0x7f09641042e0 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.594+0000 7f096c400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0964113c10 con 0x7f0964109930 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.594+0000 7f096c400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0964113f10 con 0x7f0964103920 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.594+0000 7f0969a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0964103920 0x7f0964075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.594+0000 7f0969a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0964103920 0x7f0964075c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38598/0 (socket says 172.21.15.102:38598) 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.595+0000 7f0969a00640 1 -- 172.21.15.102:0/3524343726 learned_addr learned my addr 172.21.15.102:0/3524343726 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.595+0000 7f0969000640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09641042e0 0x7f0964076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.595+0000 7f096a400640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0964109930 0x7f09640766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.595+0000 7f0969000640 1 -- 172.21.15.102:0/3524343726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0964103920 msgr2=0x7f0964075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.595+0000 7f0969000640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0964103920 0x7f0964075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.596+0000 7f0969000640 1 -- 172.21.15.102:0/3524343726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0964109930 msgr2=0x7f09640766d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.596+0000 7f0969000640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0964109930 0x7f09640766d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.596+0000 7f0969000640 1 -- 172.21.15.102:0/3524343726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0964199ff0 con 0x7f09641042e0 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.596+0000 7f0969000640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09641042e0 0x7f0964076190 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f095002aa00 tx=0x7f0950003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.598+0000 7f0962000640 1 -- 172.21.15.102:0/3524343726 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095003f070 con 0x7f09641042e0 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.598+0000 7f096c400640 1 -- 172.21.15.102:0/3524343726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096419a280 con 0x7f09641042e0 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.598+0000 7f0962000640 1 -- 172.21.15.102:0/3524343726 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0950031c20 con 0x7f09641042e0 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.598+0000 7f0962000640 1 -- 172.21.15.102:0/3524343726 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095003f070 con 0x7f09641042e0 2024-04-25T02:10:43.611 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.598+0000 7f096c400640 1 -- 172.21.15.102:0/3524343726 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096419a790 con 0x7f09641042e0 2024-04-25T02:10:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.605+0000 7f096f3e1740 1 Processor -- start 2024-04-25T02:10:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.605+0000 7f0962000640 1 -- 172.21.15.102:0/3524343726 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0950042070 con 0x7f09641042e0 2024-04-25T02:10:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.606+0000 7f096f3e1740 1 -- start start 2024-04-25T02:10:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.606+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611251e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.606+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 0x56036112faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.606+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611323d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.606+0000 7f096f3e1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560361088ab0 con 0x560361125720 2024-04-25T02:10:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.607+0000 7f096f3e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560361031640 con 0x56036112ffe0 2024-04-25T02:10:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.607+0000 7f096f3e1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560361084a70 con 0x560361124e00 2024-04-25T02:10:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.607+0000 7f096a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.607+0000 7f096a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611251e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38606/0 (socket says 172.21.15.102:38606) 2024-04-25T02:10:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.607+0000 7f096a400640 1 -- 172.21.15.102:0/4191890949 learned_addr learned my addr 172.21.15.102:0/4191890949 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.608+0000 7f096a400640 1 -- 172.21.15.102:0/4191890949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 msgr2=0x5603611323d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.609+0000 7f0969000640 1 --2- 172.21.15.102:0/4191890949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.609+0000 7f096a400640 1 --2- 172.21.15.102:0/4191890949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611323d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.609+0000 7f096a400640 1 -- 172.21.15.102:0/4191890949 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 msgr2=0x56036112faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.609+0000 7f096a400640 1 --2- 172.21.15.102:0/4191890949 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 0x56036112faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.609+0000 7f096a400640 1 -- 172.21.15.102:0/4191890949 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560360d9d210 con 0x560361124e00 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.609+0000 7f0969000640 1 --2- 172.21.15.102:0/4191890949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611323d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.609+0000 7f0962000640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09480a3070 0x7f09480a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.610+0000 7f096a400640 1 --2- 172.21.15.102:0/4191890949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611251e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f095c00b700 tx=0x7f095c00bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f3c9c5b4b88c91d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.610+0000 7f0962000640 1 -- 172.21.15.102:0/3524343726 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8513+0+0 (secure 0 0 0) 0x7f09500ef760 con 0x7f09641042e0 2024-04-25T02:10:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.610+0000 7f095aa00640 1 -- 172.21.15.102:0/4191890949 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c003dd0 con 0x560361124e00 2024-04-25T02:10:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.610+0000 7f095aa00640 1 -- 172.21.15.102:0/4191890949 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f095c015070 con 0x560361124e00 2024-04-25T02:10:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.610+0000 7f095aa00640 1 -- 172.21.15.102:0/4191890949 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c0106a0 con 0x560361124e00 2024-04-25T02:10:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.610+0000 7f0969a00640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09480a3070 0x7f09480a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.613+0000 7f0969a00640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09480a3070 0x7f09480a5530 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5603611263a0 tx=0x7f09540095d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:43.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.613+0000 7f096f3e1740 1 -- 172.21.15.102:0/4191890949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 msgr2=0x5603611251e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:43.621 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.613+0000 7f096f3e1740 1 --2- 172.21.15.102:0/4191890949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611251e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f095c00b700 tx=0x7f095c00bbc0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.622 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.614+0000 7f096f3e1740 1 -- 172.21.15.102:0/4191890949 shutdown_connections 2024-04-25T02:10:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.614+0000 7f096f3e1740 1 --2- 172.21.15.102:0/4191890949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611323d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.614+0000 7f096f3e1740 1 --2- 172.21.15.102:0/4191890949 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 0x56036112faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.614+0000 7f096f3e1740 1 --2- 172.21.15.102:0/4191890949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611251e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.614+0000 7f096f3e1740 1 -- 172.21.15.102:0/4191890949 >> 172.21.15.102:0/4191890949 conn(0x560361121310 msgr2=0x5603611216f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.615+0000 7f096f3e1740 1 -- 172.21.15.102:0/4191890949 shutdown_connections 2024-04-25T02:10:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.615+0000 7f096f3e1740 1 -- 172.21.15.102:0/4191890949 wait complete. 2024-04-25T02:10:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.616+0000 7f096f3e1740 1 -- start start 2024-04-25T02:10:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.616+0000 7f096f3e1740 1 Processor -- start 2024-04-25T02:10:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.617+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361124e00 0x56036112d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.617+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 0x56036112d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.617+0000 7f0969a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361124e00 0x56036112d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.617+0000 7f0969a00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361124e00 0x56036112d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:49600/0 (socket says 172.21.15.102:49600) 2024-04-25T02:10:43.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.617+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 0x56036112dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.618+0000 7f096f3e1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x560361132c10 con 0x560361125720 2024-04-25T02:10:43.626 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.618+0000 7f096f3e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x560361044990 con 0x560361124e00 2024-04-25T02:10:43.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.618+0000 7f096f3e1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560361132d90 con 0x56036112ffe0 2024-04-25T02:10:43.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.618+0000 7f0969a00640 1 -- 172.21.15.102:0/1796050845 learned_addr learned my addr 172.21.15.102:0/1796050845 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:43.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.618+0000 7f096f3e1740 10 client.0 _ll_register_callbacks cb 0x560361046950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:43.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.618+0000 7f096a400640 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 0x56036112dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.619+0000 7f0969a00640 1 -- 172.21.15.102:0/1796050845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 msgr2=0x56036112dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:43.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.619+0000 7f0969a00640 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 0x56036112dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.627 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.619+0000 7f0969a00640 1 -- 172.21.15.102:0/1796050845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 msgr2=0x56036112d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:43.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.619+0000 7f0969a00640 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 0x56036112d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.619+0000 7f0969a00640 1 -- 172.21.15.102:0/1796050845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560360e69620 con 0x560361124e00 2024-04-25T02:10:43.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.619+0000 7f0969a00640 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361124e00 0x56036112d040 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0954009d30 tx=0x7f0954004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1fd529d7483f253 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:43.628 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.619+0000 7f094f400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:10:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.620+0000 7f094f400640 1 -- 172.21.15.102:0/1796050845 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954027060 con 0x560361124e00 2024-04-25T02:10:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.620+0000 7f096f3e1740 1 -- 172.21.15.102:0/1796050845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095c015070 con 0x560361124e00 2024-04-25T02:10:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.620+0000 7f094f400640 1 -- 172.21.15.102:0/1796050845 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8513+0+0 (secure 0 0 0) 0x7f0954015cc0 con 0x560361124e00 2024-04-25T02:10:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.620+0000 7f094f400640 1 -- 172.21.15.102:0/1796050845 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f095402b460 con 0x560361124e00 2024-04-25T02:10:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.620+0000 7f096f3e1740 10 client.27752 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.620+0000 7f096f3e1740 1 -- 172.21.15.102:0/1796050845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x56036113ebc0 con 0x560361124e00 2024-04-25T02:10:43.629 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.621+0000 7f094f400640 1 -- 172.21.15.102:0/1796050845 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f095c015070 con 0x560361124e00 2024-04-25T02:10:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.622+0000 7f094f400640 1 -- 172.21.15.102:0/1796050845 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f0954022700 con 0x560361124e00 2024-04-25T02:10:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.622+0000 7f096f3e1740 10 client.27752 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.622+0000 7f096f3e1740 10 client.27752 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.622+0000 7f096f3e1740 20 client.27752 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.622+0000 7f096f3e1740 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560361140030 0x5603611424a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.623+0000 7f096f3e1740 4 client.27752 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:10:43.631 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.623+0000 7f096f3e1740 1 -- 172.21.15.102:0/1796050845 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560361095ff0 con 0x560361140030 2024-04-25T02:10:43.633 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.624+0000 7f0969000640 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560361140030 0x5603611424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.634 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.626+0000 7f0969000640 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560361140030 0x5603611424a0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bbe739ffa3ab78c server_cookie=bd390f2d65457af0 in_seq=0 out_seq=0 2024-04-25T02:10:43.635 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.626+0000 7f094f400640 10 client.27752 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:43.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.627+0000 7f094f400640 1 -- 172.21.15.102:0/1796050845 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560361095ff0 con 0x560361140030 2024-04-25T02:10:43.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.627+0000 7f094f400640 10 client.27752 handle_command_reply: tid=0 2024-04-25T02:10:43.636 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.628+0000 7f096f3e1740 1 client.27752 shutdown 2024-04-25T02:10:43.637 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.629+0000 7f096f3e1740 1 -- 172.21.15.102:0/1796050845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361124e00 msgr2=0x56036112d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:43.637 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.629+0000 7f096f3e1740 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361124e00 0x56036112d040 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0954009d30 tx=0x7f0954004a40 comp rx=0 tx=0).stop 2024-04-25T02:10:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.629+0000 7f096f3e1740 1 -- 172.21.15.102:0/1796050845 shutdown_connections 2024-04-25T02:10:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.629+0000 7f096f3e1740 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x560361140030 0x5603611424a0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.629+0000 7f096f3e1740 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 0x56036112dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.629+0000 7f096f3e1740 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 0x56036112d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.629+0000 7f096f3e1740 1 --2- 172.21.15.102:0/1796050845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361124e00 0x56036112d040 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.629+0000 7f096f3e1740 1 -- 172.21.15.102:0/1796050845 >> 172.21.15.102:0/1796050845 conn(0x560361121310 msgr2=0x560361127090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.629+0000 7f094f400640 0 client.27752 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:43.638 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.630+0000 7f096f3e1740 1 -- 172.21.15.102:0/1796050845 shutdown_connections 2024-04-25T02:10:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.630+0000 7f096f3e1740 1 -- 172.21.15.102:0/1796050845 wait complete. 2024-04-25T02:10:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.630+0000 7f096f3e1740 20 client.27752 trim_cache size 0 max 16384 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.663+0000 7f096f3e1740 1 Processor -- start 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.663+0000 7f096f3e1740 1 -- start start 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.663+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361124e00 0x56036113bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.663+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361125720 0x56036112d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.663+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 0x560361147db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.663+0000 7f096f3e1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f09641bec90 con 0x560361124e00 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.663+0000 7f096f3e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f09641bc500 con 0x560361125720 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.663+0000 7f096f3e1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0964077130 con 0x56036112ffe0 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361125720 0x56036112d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361125720 0x56036112d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:49616/0 (socket says 172.21.15.102:49616) 2024-04-25T02:10:43.672 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969000640 1 -- 172.21.15.102:0/3943665112 learned_addr learned my addr 172.21.15.102:0/3943665112 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:43.673 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969a00640 1 --2- 172.21.15.102:0/3943665112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361124e00 0x56036113bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969000640 1 -- 172.21.15.102:0/3943665112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 msgr2=0x560361147db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f096a400640 1 --2- 172.21.15.102:0/3943665112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 0x560361147db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969000640 1 --2- 172.21.15.102:0/3943665112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 0x560361147db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969000640 1 -- 172.21.15.102:0/3943665112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361124e00 msgr2=0x56036113bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969000640 1 --2- 172.21.15.102:0/3943665112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361124e00 0x56036113bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969000640 1 -- 172.21.15.102:0/3943665112 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560361120af0 con 0x560361125720 2024-04-25T02:10:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.664+0000 7f0969000640 1 --2- 172.21.15.102:0/3943665112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361125720 0x56036112d8e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0950000c00 tx=0x7f09500f1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c103a048a614c4ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.665+0000 7f095a000640 1 -- 172.21.15.102:0/3943665112 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950049040 con 0x560361125720 2024-04-25T02:10:43.674 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.665+0000 7f095a000640 1 -- 172.21.15.102:0/3943665112 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09500bbc80 con 0x560361125720 2024-04-25T02:10:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.666+0000 7f096f3e1740 1 -- 172.21.15.102:0/3943665112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361125720 msgr2=0x56036112d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.666+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3943665112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361125720 0x56036112d8e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0950000c00 tx=0x7f09500f1040 comp rx=0 tx=0).stop 2024-04-25T02:10:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.667+0000 7f096f3e1740 1 -- 172.21.15.102:0/3943665112 shutdown_connections 2024-04-25T02:10:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.667+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3943665112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56036112ffe0 0x560361147db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.667+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3943665112 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x560361125720 0x56036112d8e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.667+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3943665112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361124e00 0x56036113bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.675 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.667+0000 7f096f3e1740 1 -- 172.21.15.102:0/3943665112 >> 172.21.15.102:0/3943665112 conn(0x560361121310 msgr2=0x56036112e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.667+0000 7f096f3e1740 1 -- 172.21.15.102:0/3943665112 shutdown_connections 2024-04-25T02:10:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.667+0000 7f096f3e1740 1 -- 172.21.15.102:0/3943665112 wait complete. 2024-04-25T02:10:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.668+0000 7f096f3e1740 1 -- start start 2024-04-25T02:10:43.676 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.668+0000 7f096f3e1740 1 Processor -- start 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611381d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 0x560361134fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f096f3e1740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611354f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f096f3e1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f09641a19d0 con 0x560361125720 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f096f3e1740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f09641bbe30 con 0x56036112ffe0 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f096f3e1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f09641a20a0 con 0x560361124e00 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f096f3e1740 10 client.0 _ll_register_callbacks cb 0x7f0954009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f0969a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f0969a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611381d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:38628/0 (socket says 172.21.15.102:38628) 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f0969a00640 1 -- 172.21.15.102:0/3349323966 learned_addr learned my addr 172.21.15.102:0/3349323966 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:43.678 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.669+0000 7f096a400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.670+0000 7f096a400640 1 -- 172.21.15.102:0/3349323966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 msgr2=0x5603611381d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:43.679 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.670+0000 7f096a400640 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611381d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.670+0000 7f096a400640 1 -- 172.21.15.102:0/3349323966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 msgr2=0x560361134fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.670+0000 7f096a400640 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 0x560361134fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.670+0000 7f096a400640 1 -- 172.21.15.102:0/3349323966 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560361135a30 con 0x56036112ffe0 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.670+0000 7f096a400640 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611354f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x560361122170 tx=0x7f095c011f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=858720913ebbf16c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.670+0000 7f094f400640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.670+0000 7f094f400640 1 -- 172.21.15.102:0/3349323966 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c005030 con 0x56036112ffe0 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.671+0000 7f094f400640 1 -- 172.21.15.102:0/3349323966 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8513+0+0 (secure 0 0 0) 0x7f095c0051d0 con 0x56036112ffe0 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.671+0000 7f096f3e1740 1 -- 172.21.15.102:0/3349323966 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560361090340 con 0x56036112ffe0 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.671+0000 7f094f400640 1 -- 172.21.15.102:0/3349323966 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f095c005410 con 0x56036112ffe0 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.671+0000 7f096f3e1740 10 client.27758 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:43.680 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.671+0000 7f096f3e1740 1 -- 172.21.15.102:0/3349323966 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x5603611a73b0 con 0x56036112ffe0 2024-04-25T02:10:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.671+0000 7f094f400640 1 -- 172.21.15.102:0/3349323966 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560361090340 con 0x56036112ffe0 2024-04-25T02:10:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.672+0000 7f094f400640 1 -- 172.21.15.102:0/3349323966 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f095c011600 con 0x56036112ffe0 2024-04-25T02:10:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.672+0000 7f096f3e1740 10 client.27758 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.672+0000 7f096f3e1740 10 client.27758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:43.681 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.672+0000 7f096f3e1740 20 client.27758 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.672+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5603611a7a20 0x5603611a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.672+0000 7f096f3e1740 4 client.27758 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:10:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.672+0000 7f096f3e1740 1 -- 172.21.15.102:0/3349323966 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56036100e7a0 con 0x5603611a7a20 2024-04-25T02:10:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.672+0000 7f0969a00640 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5603611a7a20 0x5603611a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.673+0000 7f0969a00640 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5603611a7a20 0x5603611a9e40 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd06c8139c5be776 server_cookie=141f68d4e3b9a2dd in_seq=0 out_seq=0 2024-04-25T02:10:43.682 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.673+0000 7f094f400640 10 client.27758 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f094f400640 1 -- 172.21.15.102:0/3349323966 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56036100e7a0 con 0x5603611a7a20 2024-04-25T02:10:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f094f400640 10 client.27758 handle_command_reply: tid=0 2024-04-25T02:10:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f096f3e1740 1 client.27758 shutdown 2024-04-25T02:10:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f096f3e1740 1 -- 172.21.15.102:0/3349323966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 msgr2=0x5603611354f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611354f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x560361122170 tx=0x7f095c011f70 comp rx=0 tx=0).stop 2024-04-25T02:10:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f096f3e1740 1 -- 172.21.15.102:0/3349323966 shutdown_connections 2024-04-25T02:10:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x5603611a7a20 0x5603611a9e40 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56036112ffe0 0x5603611354f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560361125720 0x560361134fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f096f3e1740 1 --2- 172.21.15.102:0/3349323966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560361124e00 0x5603611381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f096f3e1740 1 -- 172.21.15.102:0/3349323966 >> 172.21.15.102:0/3349323966 conn(0x560361121310 msgr2=0x56036112f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.687+0000 7f094f400640 0 client.27758 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.688+0000 7f096f3e1740 1 -- 172.21.15.102:0/3349323966 shutdown_connections 2024-04-25T02:10:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.688+0000 7f096f3e1740 1 -- 172.21.15.102:0/3349323966 wait complete. 2024-04-25T02:10:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.688+0000 7f096f3e1740 20 client.27758 trim_cache size 0 max 16384 2024-04-25T02:10:43.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:10:43.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:10:43.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:10:43.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:10:43.698 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 -- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09480a3070 msgr2=0x7f09480a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09480a3070 0x7f09480a5530 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x5603611263a0 tx=0x7f09540095d0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 -- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09641042e0 msgr2=0x7f0964076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09641042e0 0x7f0964076190 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f095002aa00 tx=0x7f0950003b90 comp rx=0 tx=0).stop 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 -- 172.21.15.102:0/3524343726 shutdown_connections 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09480a3070 0x7f09480a5530 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0964109930 0x7f09640766d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09641042e0 0x7f0964076190 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 --2- 172.21.15.102:0/3524343726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0964103920 0x7f0964075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 -- 172.21.15.102:0/3524343726 >> 172.21.15.102:0/3524343726 conn(0x7f09640ff430 msgr2=0x7f09640ff950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 -- 172.21.15.102:0/3524343726 shutdown_connections 2024-04-25T02:10:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:43.689+0000 7f096c400640 1 -- 172.21.15.102:0/3524343726 wait complete. 2024-04-25T02:10:43.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:43 smithi102 ceph-mon[27796]: pgmap v482: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T02:10:43.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:43 smithi110 ceph-mon[32848]: pgmap v482: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T02:10:44.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:43 smithi193 ceph-mon[32981]: pgmap v482: 97 pgs: 97 active+clean; 5.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.0 MiB/s wr, 537 op/s 2024-04-25T02:10:45.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:45 smithi102 ceph-mon[27796]: pgmap v483: 97 pgs: 97 active+clean; 5.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 6.2 MiB/s wr, 689 op/s 2024-04-25T02:10:45.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:45 smithi110 ceph-mon[32848]: pgmap v483: 97 pgs: 97 active+clean; 5.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 6.2 MiB/s wr, 689 op/s 2024-04-25T02:10:46.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:45 smithi193 ceph-mon[32981]: pgmap v483: 97 pgs: 97 active+clean; 5.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 6.2 MiB/s wr, 689 op/s 2024-04-25T02:10:47.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:47 smithi102 ceph-mon[27796]: pgmap v484: 97 pgs: 97 active+clean; 5.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.3 MiB/s wr, 637 op/s 2024-04-25T02:10:47.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:47 smithi110 ceph-mon[32848]: pgmap v484: 97 pgs: 97 active+clean; 5.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.3 MiB/s wr, 637 op/s 2024-04-25T02:10:48.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:47 smithi193 ceph-mon[32981]: pgmap v484: 97 pgs: 97 active+clean; 5.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.3 MiB/s wr, 637 op/s 2024-04-25T02:10:48.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 895.0 s, 1289.0 tps, lat 78.047 ms stddev 93.142 2024-04-25T02:10:49.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:49 smithi102 ceph-mon[27796]: pgmap v485: 97 pgs: 97 active+clean; 5.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.2 MiB/s wr, 679 op/s 2024-04-25T02:10:49.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:49 smithi110 ceph-mon[32848]: pgmap v485: 97 pgs: 97 active+clean; 5.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.2 MiB/s wr, 679 op/s 2024-04-25T02:10:50.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:49 smithi193 ceph-mon[32981]: pgmap v485: 97 pgs: 97 active+clean; 5.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.2 MiB/s wr, 679 op/s 2024-04-25T02:10:51.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:51 smithi102 ceph-mon[27796]: pgmap v486: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.3 MiB/s wr, 664 op/s 2024-04-25T02:10:51.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:51 smithi110 ceph-mon[32848]: pgmap v486: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.3 MiB/s wr, 664 op/s 2024-04-25T02:10:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:51 smithi193 ceph-mon[32981]: pgmap v486: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.3 MiB/s wr, 664 op/s 2024-04-25T02:10:53.076 INFO:teuthology.orchestra.run.smithi102.stderr:progress: 900.0 s, 1360.6 tps, lat 72.897 ms stddev 86.350 2024-04-25T02:10:53.362 INFO:teuthology.orchestra.run.smithi102.stdout:transaction type: 2024-04-25T02:10:53.362 INFO:teuthology.orchestra.run.smithi102.stdout:scaling factor: 32 2024-04-25T02:10:53.362 INFO:teuthology.orchestra.run.smithi102.stdout:query mode: simple 2024-04-25T02:10:53.363 INFO:teuthology.orchestra.run.smithi102.stdout:number of clients: 100 2024-04-25T02:10:53.363 INFO:teuthology.orchestra.run.smithi102.stdout:number of threads: 4 2024-04-25T02:10:53.363 INFO:teuthology.orchestra.run.smithi102.stdout:duration: 900 s 2024-04-25T02:10:53.363 INFO:teuthology.orchestra.run.smithi102.stdout:number of transactions actually processed: 1816416 2024-04-25T02:10:53.363 INFO:teuthology.orchestra.run.smithi102.stdout:latency average = 49.529 ms 2024-04-25T02:10:53.363 INFO:teuthology.orchestra.run.smithi102.stdout:latency stddev = 92.570 ms 2024-04-25T02:10:53.363 INFO:teuthology.orchestra.run.smithi102.stdout:tps = 2017.599771 (including connections establishing) 2024-04-25T02:10:53.363 INFO:teuthology.orchestra.run.smithi102.stdout:tps = 2017.615228 (excluding connections establishing) 2024-04-25T02:10:53.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-04-25T02:10:53.681 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:53 smithi102 ceph-mon[27796]: pgmap v487: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 4.9 MiB/s wr, 535 op/s 2024-04-25T02:10:53.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:53 smithi110 ceph-mon[32848]: pgmap v487: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 4.9 MiB/s wr, 535 op/s 2024-04-25T02:10:54.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:53 smithi193 ceph-mon[32981]: pgmap v487: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 4.9 MiB/s wr, 535 op/s 2024-04-25T02:10:55.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:55 smithi102 ceph-mon[27796]: pgmap v488: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 6.3 MiB/s wr, 694 op/s 2024-04-25T02:10:55.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:55 smithi110 ceph-mon[32848]: pgmap v488: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 6.3 MiB/s wr, 694 op/s 2024-04-25T02:10:56.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:55 smithi193 ceph-mon[32981]: pgmap v488: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 6.3 MiB/s wr, 694 op/s 2024-04-25T02:10:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.891+0000 7fd15f000640 1 -- 172.21.15.102:0/340623239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd15810e030 msgr2=0x7fd158110420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:56.899 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.891+0000 7fd15f000640 1 --2- 172.21.15.102:0/340623239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd15810e030 0x7fd158110420 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd150009be0 tx=0x7fd15002a750 comp rx=0 tx=0).stop 2024-04-25T02:10:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.892+0000 7fd15f000640 1 -- 172.21.15.102:0/340623239 shutdown_connections 2024-04-25T02:10:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.892+0000 7fd15f000640 1 --2- 172.21.15.102:0/340623239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd15810e030 0x7fd158110420 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.900 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.892+0000 7fd15f000640 1 --2- 172.21.15.102:0/340623239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd158074360 0x7fd15810daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.892+0000 7fd15f000640 1 --2- 172.21.15.102:0/340623239 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd158073a40 0x7fd158073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.901 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.892+0000 7fd15f000640 1 -- 172.21.15.102:0/340623239 >> 172.21.15.102:0/340623239 conn(0x7fd158078bd0 msgr2=0x7fd158078fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.893+0000 7fd15f000640 1 -- 172.21.15.102:0/340623239 shutdown_connections 2024-04-25T02:10:56.902 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.893+0000 7fd15f000640 1 -- 172.21.15.102:0/340623239 wait complete. 2024-04-25T02:10:56.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.895+0000 7fd15f000640 1 Processor -- start 2024-04-25T02:10:56.903 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.895+0000 7fd15f000640 1 -- start start 2024-04-25T02:10:56.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.895+0000 7fd15f000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd158073a40 0x7fd15819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.895+0000 7fd15f000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd158074360 0x7fd15819a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.895+0000 7fd15f000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15810e030 0x7fd15819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.895+0000 7fd15f000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd158113d60 con 0x7fd158074360 2024-04-25T02:10:56.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.895+0000 7fd15f000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd158113be0 con 0x7fd158073a40 2024-04-25T02:10:56.904 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.895+0000 7fd15f000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd158113ee0 con 0x7fd15810e030 2024-04-25T02:10:56.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd15d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15810e030 0x7fd15819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd15d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15810e030 0x7fd15819e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51428/0 (socket says 172.21.15.102:51428) 2024-04-25T02:10:56.905 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd15d000640 1 -- 172.21.15.102:0/1493913303 learned_addr learned my addr 172.21.15.102:0/1493913303 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:56.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd157400640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd158074360 0x7fd15819a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd15d000640 1 -- 172.21.15.102:0/1493913303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd158073a40 msgr2=0x7fd15819a060 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T02:10:56.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd15d000640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd158073a40 0x7fd15819a060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd15d000640 1 -- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd158074360 msgr2=0x7fd15819a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:56.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd15d000640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd158074360 0x7fd15819a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd15d000640 1 -- 172.21.15.102:0/1493913303 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15819f0a0 con 0x7fd15810e030 2024-04-25T02:10:56.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.896+0000 7fd157400640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd158074360 0x7fd15819a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:10:56.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.897+0000 7fd15d000640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15810e030 0x7fd15819e920 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd150009b30 tx=0x7fd1500305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:56.906 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.897+0000 7fd154c00640 1 -- 172.21.15.102:0/1493913303 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150041020 con 0x7fd15810e030 2024-04-25T02:10:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.897+0000 7fd154c00640 1 -- 172.21.15.102:0/1493913303 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd150003e50 con 0x7fd15810e030 2024-04-25T02:10:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.897+0000 7fd15f000640 1 -- 172.21.15.102:0/1493913303 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15819f330 con 0x7fd15810e030 2024-04-25T02:10:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.897+0000 7fd154c00640 1 -- 172.21.15.102:0/1493913303 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150036040 con 0x7fd15810e030 2024-04-25T02:10:56.907 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.898+0000 7fd15f000640 1 -- 172.21.15.102:0/1493913303 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd158101680 con 0x7fd15810e030 2024-04-25T02:10:56.914 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.906+0000 7fd161f98740 1 Processor -- start 2024-04-25T02:10:56.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.907+0000 7fd154c00640 1 -- 172.21.15.102:0/1493913303 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd150044020 con 0x7fd15810e030 2024-04-25T02:10:56.915 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.907+0000 7fd161f98740 1 -- start start 2024-04-25T02:10:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.907+0000 7fd161f98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae4fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.907+0000 7fd161f98740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 0x55faae508aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.907+0000 7fd161f98740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae50b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.907+0000 7fd161f98740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55faae461ab0 con 0x55faae4fe720 2024-04-25T02:10:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.907+0000 7fd161f98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55faae40a640 con 0x55faae4fde00 2024-04-25T02:10:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.907+0000 7fd161f98740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55faae45da70 con 0x55faae508fe0 2024-04-25T02:10:56.916 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.908+0000 7fd157400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae4fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.908+0000 7fd157400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae4fe1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46422/0 (socket says 172.21.15.102:46422) 2024-04-25T02:10:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.908+0000 7fd157400640 1 -- 172.21.15.102:0/909699865 learned_addr learned my addr 172.21.15.102:0/909699865 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.908+0000 7fd157400640 1 --2- 172.21.15.102:0/909699865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae50b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.909+0000 7fd157400640 1 -- 172.21.15.102:0/909699865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 msgr2=0x55faae50b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.909+0000 7fd157400640 1 --2- 172.21.15.102:0/909699865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae50b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.909+0000 7fd157400640 1 -- 172.21.15.102:0/909699865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 msgr2=0x55faae508aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.909+0000 7fd157400640 1 --2- 172.21.15.102:0/909699865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 0x55faae508aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.909+0000 7fd157400640 1 -- 172.21.15.102:0/909699865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faae176210 con 0x55faae4fde00 2024-04-25T02:10:56.918 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.909+0000 7fd154c00640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd13c0a2da0 0x7fd13c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.919 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.909+0000 7fd154c00640 1 -- 172.21.15.102:0/1493913303 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8513+0+0 (secure 0 0 0) 0x7fd1500f0250 con 0x7fd15810e030 2024-04-25T02:10:56.919 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.911+0000 7fd157400640 1 --2- 172.21.15.102:0/909699865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae4fe1e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd144004650 tx=0x7fd14400b470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b02dc0125696ed4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:56.920 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.911+0000 7fd14d600640 1 -- 172.21.15.102:0/909699865 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144019070 con 0x55faae4fde00 2024-04-25T02:10:56.920 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.911+0000 7fd14d600640 1 -- 172.21.15.102:0/909699865 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd14401eb40 con 0x55faae4fde00 2024-04-25T02:10:56.920 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.911+0000 7fd14d600640 1 -- 172.21.15.102:0/909699865 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144010b60 con 0x55faae4fde00 2024-04-25T02:10:56.920 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.912+0000 7fd157e00640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd13c0a2da0 0x7fd13c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.912+0000 7fd161f98740 1 -- 172.21.15.102:0/909699865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 msgr2=0x55faae4fe1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:56.921 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.912+0000 7fd161f98740 1 --2- 172.21.15.102:0/909699865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae4fe1e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd144004650 tx=0x7fd14400b470 comp rx=0 tx=0).stop 2024-04-25T02:10:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.913+0000 7fd157e00640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd13c0a2da0 0x7fd13c0a5260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd1480058c0 tx=0x7fd148005850 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.914+0000 7fd161f98740 1 -- 172.21.15.102:0/909699865 shutdown_connections 2024-04-25T02:10:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.914+0000 7fd161f98740 1 --2- 172.21.15.102:0/909699865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae50b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.914+0000 7fd161f98740 1 --2- 172.21.15.102:0/909699865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 0x55faae508aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.914+0000 7fd161f98740 1 --2- 172.21.15.102:0/909699865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae4fe1e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.914+0000 7fd161f98740 1 -- 172.21.15.102:0/909699865 >> 172.21.15.102:0/909699865 conn(0x55faae4fa310 msgr2=0x55faae4fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.915+0000 7fd161f98740 1 -- 172.21.15.102:0/909699865 shutdown_connections 2024-04-25T02:10:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.915+0000 7fd161f98740 1 -- 172.21.15.102:0/909699865 wait complete. 2024-04-25T02:10:56.924 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.915+0000 7fd161f98740 1 -- start start 2024-04-25T02:10:56.924 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.916+0000 7fd161f98740 1 Processor -- start 2024-04-25T02:10:56.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.917+0000 7fd161f98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae505fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.917+0000 7fd161f98740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 0x55faae5064e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.917+0000 7fd161f98740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae506a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.917+0000 7fd157e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae505fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.917+0000 7fd157e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae505fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46430/0 (socket says 172.21.15.102:46430) 2024-04-25T02:10:56.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.917+0000 7fd157400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 0x55faae5064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.926 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.917+0000 7fd157400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 0x55faae5064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49626/0 (socket says 172.21.15.102:49626) 2024-04-25T02:10:56.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.918+0000 7fd157e00640 1 -- 172.21.15.102:0/3765977372 learned_addr learned my addr 172.21.15.102:0/3765977372 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:56.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.918+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x55faae50bc10 con 0x55faae4fe720 2024-04-25T02:10:56.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.918+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x55faae41d990 con 0x55faae4fde00 2024-04-25T02:10:56.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.918+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55faae50bd90 con 0x55faae508fe0 2024-04-25T02:10:56.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.918+0000 7fd15d000640 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae506a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.927 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.918+0000 7fd161f98740 10 client.0 _ll_register_callbacks cb 0x55faae41f950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.919+0000 7fd157e00640 1 -- 172.21.15.102:0/3765977372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 msgr2=0x55faae506a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.919+0000 7fd157e00640 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae506a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.919+0000 7fd157e00640 1 -- 172.21.15.102:0/3765977372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 msgr2=0x55faae5064e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.919+0000 7fd157e00640 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 0x55faae5064e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.919+0000 7fd157e00640 1 -- 172.21.15.102:0/3765977372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faae242620 con 0x55faae4fde00 2024-04-25T02:10:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.919+0000 7fd15d000640 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae506a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:10:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.919+0000 7fd157400640 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 0x55faae5064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:10:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.920+0000 7fd157e00640 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae505fa0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd14800ee50 tx=0x7fd1480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98afcabe1795929 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.920+0000 7fd142000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:10:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.921+0000 7fd142000640 1 -- 172.21.15.102:0/3765977372 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148021070 con 0x55faae4fde00 2024-04-25T02:10:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.921+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd14401eb40 con 0x55faae4fde00 2024-04-25T02:10:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.922+0000 7fd142000640 1 -- 172.21.15.102:0/3765977372 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8513+0+0 (secure 0 0 0) 0x7fd14801d460 con 0x55faae4fde00 2024-04-25T02:10:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.922+0000 7fd142000640 1 -- 172.21.15.102:0/3765977372 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fd14802d050 con 0x55faae4fde00 2024-04-25T02:10:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.922+0000 7fd161f98740 10 client.27767 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.922+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55faae517bc0 con 0x55faae4fde00 2024-04-25T02:10:56.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.923+0000 7fd142000640 1 -- 172.21.15.102:0/3765977372 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd14401eb40 con 0x55faae4fde00 2024-04-25T02:10:56.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.924+0000 7fd142000640 1 -- 172.21.15.102:0/3765977372 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7fd1480105f0 con 0x55faae4fde00 2024-04-25T02:10:56.932 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.924+0000 7fd161f98740 10 client.27767 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.924+0000 7fd161f98740 10 client.27767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.924+0000 7fd161f98740 20 client.27767 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.924+0000 7fd161f98740 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55faae5190d0 0x55faae51b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.924+0000 7fd161f98740 4 client.27767 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T02:10:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.924+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55faae46eff0 con 0x55faae5190d0 2024-04-25T02:10:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.924+0000 7fd157400640 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55faae5190d0 0x55faae51b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.933 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.925+0000 7fd157400640 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55faae5190d0 0x55faae51b560 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0114ae81e581709 server_cookie=38e23dac36f5d703 in_seq=0 out_seq=0 2024-04-25T02:10:56.934 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.925+0000 7fd142000640 10 client.27767 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:56.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.926+0000 7fd142000640 1 -- 172.21.15.102:0/3765977372 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55faae46eff0 con 0x55faae5190d0 2024-04-25T02:10:56.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.926+0000 7fd142000640 10 client.27767 handle_command_reply: tid=0 2024-04-25T02:10:56.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.926+0000 7fd161f98740 1 client.27767 shutdown 2024-04-25T02:10:56.935 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.927+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 msgr2=0x55faae505fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:56.936 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.927+0000 7fd161f98740 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae505fa0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd14800ee50 tx=0x7fd1480046c0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.928+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 shutdown_connections 2024-04-25T02:10:56.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.928+0000 7fd161f98740 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55faae5190d0 0x55faae51b560 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.929+0000 7fd161f98740 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae506a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.929+0000 7fd142000640 0 client.27767 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.929+0000 7fd161f98740 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fe720 0x55faae5064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.929+0000 7fd161f98740 1 --2- 172.21.15.102:0/3765977372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fde00 0x55faae505fa0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.929+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 >> 172.21.15.102:0/3765977372 conn(0x55faae4fa310 msgr2=0x55faae500090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.932+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 shutdown_connections 2024-04-25T02:10:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.932+0000 7fd161f98740 1 -- 172.21.15.102:0/3765977372 wait complete. 2024-04-25T02:10:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.932+0000 7fd161f98740 20 client.27767 trim_cache size 0 max 16384 2024-04-25T02:10:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.962+0000 7fd161f98740 1 Processor -- start 2024-04-25T02:10:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.962+0000 7fd161f98740 1 -- start start 2024-04-25T02:10:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd161f98740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fde00 0x55faae514ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd161f98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae506840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd161f98740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae4f9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd161f98740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581adb60 con 0x55faae4fde00 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd161f98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581ab3d0 con 0x55faae4fe720 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd161f98740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581a9280 con 0x55faae508fe0 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd15d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae4f9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd15d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae4f9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51464/0 (socket says 172.21.15.102:51464) 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd15d000640 1 -- 172.21.15.102:0/3534054919 learned_addr learned my addr 172.21.15.102:0/3534054919 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd157e00640 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fde00 0x55faae514ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.963+0000 7fd157400640 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae506840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.964+0000 7fd15d000640 1 -- 172.21.15.102:0/3534054919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 msgr2=0x55faae506840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.964+0000 7fd15d000640 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae506840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.964+0000 7fd15d000640 1 -- 172.21.15.102:0/3534054919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fde00 msgr2=0x55faae514ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.965+0000 7fd15d000640 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fde00 0x55faae514ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.967+0000 7fd15d000640 1 -- 172.21.15.102:0/3534054919 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faae4f5620 con 0x55faae508fe0 2024-04-25T02:10:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.967+0000 7fd157400640 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae506840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T02:10:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.967+0000 7fd157e00640 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fde00 0x55faae514ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T02:10:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.967+0000 7fd15d000640 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae4f9300 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd1500eda50 tx=0x7fd150031440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a87ecc8b97553cd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.968+0000 7fd14cc00640 1 -- 172.21.15.102:0/3534054919 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15004e350 con 0x55faae508fe0 2024-04-25T02:10:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.968+0000 7fd14cc00640 1 -- 172.21.15.102:0/3534054919 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1500d5070 con 0x55faae508fe0 2024-04-25T02:10:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.968+0000 7fd14cc00640 1 -- 172.21.15.102:0/3534054919 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1500edc20 con 0x55faae508fe0 2024-04-25T02:10:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.969+0000 7fd161f98740 1 -- 172.21.15.102:0/3534054919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 msgr2=0x55faae4f9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:10:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.969+0000 7fd161f98740 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae4f9300 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd1500eda50 tx=0x7fd150031440 comp rx=0 tx=0).stop 2024-04-25T02:10:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.970+0000 7fd161f98740 1 -- 172.21.15.102:0/3534054919 shutdown_connections 2024-04-25T02:10:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.970+0000 7fd161f98740 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae508fe0 0x55faae4f9300 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.970+0000 7fd161f98740 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae506840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.970+0000 7fd161f98740 1 --2- 172.21.15.102:0/3534054919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae4fde00 0x55faae514ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.970+0000 7fd161f98740 1 -- 172.21.15.102:0/3534054919 >> 172.21.15.102:0/3534054919 conn(0x55faae4fa310 msgr2=0x55faae4f8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:10:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.970+0000 7fd161f98740 1 -- 172.21.15.102:0/3534054919 shutdown_connections 2024-04-25T02:10:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.970+0000 7fd161f98740 1 -- 172.21.15.102:0/3534054919 wait complete. 2024-04-25T02:10:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.970+0000 7fd161f98740 1 -- start start 2024-04-25T02:10:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.971+0000 7fd161f98740 1 Processor -- start 2024-04-25T02:10:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.975+0000 7fd161f98740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae4fde00 0x55faae502e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.975+0000 7fd161f98740 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae503390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.975+0000 7fd161f98740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae508fe0 0x55faae5038d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.975+0000 7fd161f98740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581a98d0 con 0x55faae508fe0 2024-04-25T02:10:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.975+0000 7fd161f98740 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581aad00 con 0x55faae4fe720 2024-04-25T02:10:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.975+0000 7fd161f98740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581a9f60 con 0x55faae4fde00 2024-04-25T02:10:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.975+0000 7fd161f98740 10 client.0 _ll_register_callbacks cb 0x7fd148014050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T02:10:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.976+0000 7fd157400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae503390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.976+0000 7fd157e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae4fde00 0x55faae502e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.976+0000 7fd157e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae4fde00 0x55faae502e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:51468/0 (socket says 172.21.15.102:51468) 2024-04-25T02:10:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.976+0000 7fd157400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae503390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:46440/0 (socket says 172.21.15.102:46440) 2024-04-25T02:10:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.976+0000 7fd157400640 1 -- 172.21.15.102:0/2560455752 learned_addr learned my addr 172.21.15.102:0/2560455752 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:10:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.977+0000 7fd157400640 1 -- 172.21.15.102:0/2560455752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae4fde00 msgr2=0x55faae502e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.977+0000 7fd157400640 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae4fde00 0x55faae502e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.977+0000 7fd157400640 1 -- 172.21.15.102:0/2560455752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae508fe0 msgr2=0x55faae5038d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T02:10:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.977+0000 7fd157400640 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae508fe0 0x55faae5038d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:10:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.977+0000 7fd157400640 1 -- 172.21.15.102:0/2560455752 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faae58f8b0 con 0x55faae4fe720 2024-04-25T02:10:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.978+0000 7fd157400640 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae503390 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd144020010 tx=0x7fd14400bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=627470c60b5f9878 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:10:56.987 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.978+0000 7fd142000640 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2024-04-25T02:10:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.978+0000 7fd142000640 1 -- 172.21.15.102:0/2560455752 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14402f070 con 0x55faae4fe720 2024-04-25T02:10:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.979+0000 7fd142000640 1 -- 172.21.15.102:0/2560455752 <== mon.2 v2:172.21.15.110:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8513+0+0 (secure 0 0 0) 0x7fd144004220 con 0x55faae4fe720 2024-04-25T02:10:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.979+0000 7fd161f98740 1 -- 172.21.15.102:0/2560455752 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1500d5070 con 0x55faae4fe720 2024-04-25T02:10:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.979+0000 7fd142000640 1 -- 172.21.15.102:0/2560455752 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fd144038050 con 0x55faae4fe720 2024-04-25T02:10:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.979+0000 7fd161f98740 10 client.27773 fetch_fsmap learned FSMap version 54 2024-04-25T02:10:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.979+0000 7fd161f98740 1 -- 172.21.15.102:0/2560455752 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55faae590af0 con 0x55faae4fe720 2024-04-25T02:10:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.980+0000 7fd142000640 1 -- 172.21.15.102:0/2560455752 <== mon.2 v2:172.21.15.110:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd144010680 con 0x55faae4fe720 2024-04-25T02:10:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.981+0000 7fd142000640 1 -- 172.21.15.102:0/2560455752 <== mon.2 v2:172.21.15.110:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7fd144028070 con 0x55faae4fe720 2024-04-25T02:10:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.981+0000 7fd161f98740 10 client.27773 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T02:10:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.981+0000 7fd161f98740 10 client.27773 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T02:10:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.981+0000 7fd161f98740 20 client.27773 populate_metadata read hostname 'smithi102' 2024-04-25T02:10:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.981+0000 7fd161f98740 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55faae5190d0 0x55faae593120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:10:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.981+0000 7fd161f98740 4 client.27773 mds_command: new command op to 24587 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T02:10:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.981+0000 7fd161f98740 1 -- 172.21.15.102:0/2560455752 --> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd1500d5070 con 0x55faae5190d0 2024-04-25T02:10:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.982+0000 7fd157e00640 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55faae5190d0 0x55faae593120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:10:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.983+0000 7fd157e00640 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55faae5190d0 0x55faae593120 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a91b8ccbfb2bc5a server_cookie=452ada96ec3fc232 in_seq=0 out_seq=0 2024-04-25T02:10:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:10:56.983+0000 7fd142000640 10 client.27773 ms_handle_connect on v2:172.21.15.110:6838/4063392981 2024-04-25T02:10:57.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:57 smithi102 ceph-mon[27796]: pgmap v489: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.1 MiB/s wr, 643 op/s 2024-04-25T02:10:57.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:57 smithi110 ceph-mon[32848]: pgmap v489: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.1 MiB/s wr, 643 op/s 2024-04-25T02:10:58.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:57 smithi193 ceph-mon[32981]: pgmap v489: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.1 MiB/s wr, 643 op/s 2024-04-25T02:10:58.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:58 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:10:58.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:58 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:10:59.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:58 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T02:10:59.934 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:10:59 smithi102 ceph-mon[27796]: pgmap v490: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.7 MiB/s wr, 848 op/s 2024-04-25T02:10:59.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:10:59 smithi110 ceph-mon[32848]: pgmap v490: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.7 MiB/s wr, 848 op/s 2024-04-25T02:11:00.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:10:59 smithi193 ceph-mon[32981]: pgmap v490: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7.7 MiB/s wr, 848 op/s 2024-04-25T02:11:00.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:00 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:11:00.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:00 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:11:00.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:00 smithi102 ceph-mon[27796]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:11:00.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:00 smithi102 ceph-mon[27796]: pgmap v491: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.3 MiB/s wr, 780 op/s 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.929+0000 7fd142000640 1 -- 172.21.15.102:0/2560455752 <== mds.0 v2:172.21.15.110:6838/4063392981 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd1500d5070 con 0x55faae5190d0 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.929+0000 7fd142000640 10 client.27773 handle_command_reply: tid=0 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.929+0000 7fd161f98740 1 client.27773 shutdown 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.929+0000 7fd161f98740 1 -- 172.21.15.102:0/2560455752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 msgr2=0x55faae503390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.929+0000 7fd161f98740 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae503390 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd144020010 tx=0x7fd14400bfa0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd161f98740 1 -- 172.21.15.102:0/2560455752 shutdown_connections 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd161f98740 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.110:6838/4063392981,v1:172.21.15.110:6839/4063392981] conn(0x55faae5190d0 0x55faae593120 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd161f98740 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55faae508fe0 0x55faae5038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd161f98740 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55faae4fe720 0x55faae503390 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd161f98740 1 --2- 172.21.15.102:0/2560455752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55faae4fde00 0x55faae502e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd142000640 0 client.27773 ms_handle_reset on v2:172.21.15.110:6838/4063392981 2024-04-25T02:11:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd161f98740 1 -- 172.21.15.102:0/2560455752 >> 172.21.15.102:0/2560455752 conn(0x55faae4fa310 msgr2=0x55faae4f9480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:{ 2024-04-25T02:11:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "message": "", 2024-04-25T02:11:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout: "return_code": 0 2024-04-25T02:11:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stdout:} 2024-04-25T02:11:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd161f98740 1 -- 172.21.15.102:0/2560455752 shutdown_connections 2024-04-25T02:11:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd161f98740 1 -- 172.21.15.102:0/2560455752 wait complete. 2024-04-25T02:11:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.930+0000 7fd161f98740 20 client.27773 trim_cache size 0 max 16384 2024-04-25T02:11:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 -- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd13c0a2da0 msgr2=0x7fd13c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd13c0a2da0 0x7fd13c0a5260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd1480058c0 tx=0x7fd148005850 comp rx=0 tx=0).stop 2024-04-25T02:11:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 -- 172.21.15.102:0/1493913303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15810e030 msgr2=0x7fd15819e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15810e030 0x7fd15819e920 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd150009b30 tx=0x7fd1500305d0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 -- 172.21.15.102:0/1493913303 shutdown_connections 2024-04-25T02:11:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fd13c0a2da0 0x7fd13c0a5260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd15810e030 0x7fd15819e920 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd158074360 0x7fd15819a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 --2- 172.21.15.102:0/1493913303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd158073a40 0x7fd15819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.931+0000 7fd15f000640 1 -- 172.21.15.102:0/1493913303 >> 172.21.15.102:0/1493913303 conn(0x7fd158078bd0 msgr2=0x7fd1580fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.932+0000 7fd15f000640 1 -- 172.21.15.102:0/1493913303 shutdown_connections 2024-04-25T02:11:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi102.front.sepia.ceph.com.smithi102.stderr:2024-04-25T02:11:00.932+0000 7fd15f000640 1 -- 172.21.15.102:0/1493913303 wait complete. 2024-04-25T02:11:00.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:00 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:11:00.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:00 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:11:00.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:00 smithi110 ceph-mon[32848]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:11:00.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:00 smithi110 ceph-mon[32848]: pgmap v491: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.3 MiB/s wr, 780 op/s 2024-04-25T02:11:01.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:00 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T02:11:01.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:00 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T02:11:01.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:00 smithi193 ceph-mon[32981]: from='mgr.25214 ' entity='mgr.x' 2024-04-25T02:11:01.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:00 smithi193 ceph-mon[32981]: pgmap v491: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 7.3 MiB/s wr, 780 op/s 2024-04-25T02:11:03.180 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:02 smithi102 ceph-mon[27796]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T02:11:03.180 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:02 smithi102 ceph-mon[27796]: pgmap v493: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.9 MiB/s wr, 957 op/s 2024-04-25T02:11:03.244 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:02 smithi110 ceph-mon[32848]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T02:11:03.245 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:02 smithi110 ceph-mon[32848]: pgmap v493: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.9 MiB/s wr, 957 op/s 2024-04-25T02:11:03.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:02 smithi193 ceph-mon[32981]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T02:11:03.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:02 smithi193 ceph-mon[32981]: pgmap v493: 97 pgs: 97 active+clean; 5.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.9 MiB/s wr, 957 op/s 2024-04-25T02:11:04.112 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-25T02:11:04.288 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/: 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 .. 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 data 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data: 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:total 36 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 . 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 .. 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 01:55 base 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Apr 25 01:55 current_logfiles 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 25 01:55 global 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 log 2024-04-25T02:11:04.289 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_commit_ts 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_dynshmem 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 25 01:55 pg_hba.conf 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 25 01:55 pg_ident.conf 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 02:11 pg_logical 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 pg_multixact 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_notify 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_replslot 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_serial 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_snapshots 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 02:11 pg_stat 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 02:11 pg_stat_tmp 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 02:11 pg_subtrans 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_tblspc 2024-04-25T02:11:04.290 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pg_twophase 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 PG_VERSION 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 25 02:11 pg_wal 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 02:02 pg_xact 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 25 01:55 postgresql.auto.conf 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 25 01:55 postgresql.conf 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Apr 25 01:55 postmaster.opts 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/base: 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 01:55 . 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 1 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 13493 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 25 01:59 13494 2024-04-25T02:11:04.291 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 pgsql_tmp 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:total 7777 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 . 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 01:55 .. 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 112 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 113 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 1247 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1247_fsm 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1247_vm 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 01:55 1249 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1249_fsm 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1249_vm 2024-04-25T02:11:04.292 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 01:55 1255 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1255_fsm 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1255_vm 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 01:55 1259 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1259_fsm 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1259_vm 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 01:55 13344 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13344_fsm 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13344_vm 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13346 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13348 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13349_fsm 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349_vm 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13351 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13353 2024-04-25T02:11:04.293 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13354_fsm 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354_vm 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13356 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13358 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13359_fsm 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359_vm 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13361 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13363 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1417 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1418 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 174 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 175 2024-04-25T02:11:04.294 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2187 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2224 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2328 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2336 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2337 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2579 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2600 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2600_fsm 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2600_vm 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2601_fsm 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601_vm 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2602 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2602_fsm 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2602_vm 2024-04-25T02:11:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2603 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2603_fsm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2603_vm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2604 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2605 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2605_fsm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2605_vm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2606_fsm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606_vm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2607 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2607_fsm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2607_vm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 01:55 2608 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2608_fsm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2608_vm 2024-04-25T02:11:04.296 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 01:55 2609 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2609_fsm 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2609_vm 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2610 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2610_fsm 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2610_vm 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2611 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2612_fsm 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612_vm 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2613 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2615_fsm 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615_vm 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2616 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2616_fsm 2024-04-25T02:11:04.297 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2616_vm 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 01:55 2617 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2617_fsm 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2617_vm 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 01:55 2618 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2618_fsm 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2618_vm 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 01:55 2619 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2619_fsm 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2619_vm 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2620 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2650 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2651 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2652 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2653 2024-04-25T02:11:04.298 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2654 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2655 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2656 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2657 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 01:55 2658 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 2659 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2660 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2661 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2662 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2663 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2664 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2665 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2666 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2667 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2668 2024-04-25T02:11:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2669 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2670 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 01:55 2673 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 01:55 2674 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 01:55 2675 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2678 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2679 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2680 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2681 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2682 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2683 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2684 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2685 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2686 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2687 2024-04-25T02:11:04.300 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2688 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2689 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 01:55 2690 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 01:55 2691 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2692 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2693 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2696 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2699 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2701 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2702 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2703 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2704 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2753 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2753_fsm 2024-04-25T02:11:04.301 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2753_vm 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2754 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2755 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2756 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2757 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2830 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2831 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2832 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2833 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2834 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2835 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2836 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2837 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 01:55 2838 2024-04-25T02:11:04.302 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2838_fsm 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2838_vm 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2839 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840_fsm 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2840_vm 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2841 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2995 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2996 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3079_fsm 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079_vm 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3080 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3081 2024-04-25T02:11:04.303 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3085 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3118 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3119 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 01:55 3164 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3256 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3257 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3258 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3350 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3351 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3379 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3380 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3381 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3394 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3394_fsm 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3394_vm 2024-04-25T02:11:04.304 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3395 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3429 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3430 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3431 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3433 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3439 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3440 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3455 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 01:55 3456 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3456_fsm 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3456_vm 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3466 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3467 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3468 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3501 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3502 2024-04-25T02:11:04.305 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3503 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3534 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3541_fsm 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541_vm 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3542 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3574 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3575 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3576 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3596 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3597 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3598 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3599 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600 2024-04-25T02:11:04.306 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3600_fsm 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600_vm 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3601_fsm 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601_vm 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3602_fsm 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602_vm 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603_fsm 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3603_vm 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3604 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3605 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3606 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3607 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3608 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 3609 2024-04-25T02:11:04.307 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3712 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3764_fsm 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764_vm 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3766 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3767 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3997 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4143 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4144 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4145 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4146 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4147 2024-04-25T02:11:04.308 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4148 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4149 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4150 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4151 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4152 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4153 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4154 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4155 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4156 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4157 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4158 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4159 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4160 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4161 2024-04-25T02:11:04.309 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4162 2024-04-25T02:11:04.333 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4163 2024-04-25T02:11:04.333 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4164 2024-04-25T02:11:04.333 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4165 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4166 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4167 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4168 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4169 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4170 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4171 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4172 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4173 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4174 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 5002 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 548 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 549 2024-04-25T02:11:04.334 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6102 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6104 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6106 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6110 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6111 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6112 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6113 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6117 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 826 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 827 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 828 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 01:55 pg_filenode.map 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 PG_VERSION 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-25T02:11:04.335 INFO:teuthology.orchestra.run.smithi102.stdout:total 7777 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Apr 25 01:55 . 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 01:55 .. 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 112 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 113 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 1247 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1247_fsm 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1247_vm 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 01:55 1249 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1249_fsm 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1249_vm 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 01:55 1255 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1255_fsm 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1255_vm 2024-04-25T02:11:04.336 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 01:55 1259 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1259_fsm 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1259_vm 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 01:55 13344 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13344_fsm 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13344_vm 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13346 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13348 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13349_fsm 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349_vm 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13351 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13353 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13354_fsm 2024-04-25T02:11:04.337 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354_vm 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13356 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13358 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13359_fsm 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359_vm 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13361 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13363 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1417 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1418 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 174 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 175 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2187 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2224 2024-04-25T02:11:04.338 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2328 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2336 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2337 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2579 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2600 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2600_fsm 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2600_vm 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2601_fsm 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601_vm 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2602 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2602_fsm 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2602_vm 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2603 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2603_fsm 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2603_vm 2024-04-25T02:11:04.339 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2604 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2605 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2605_fsm 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2605_vm 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2606_fsm 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606_vm 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2607 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2607_fsm 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2607_vm 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 01:55 2608 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2608_fsm 2024-04-25T02:11:04.340 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2608_vm 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 01:55 2609 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2609_fsm 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2609_vm 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2610 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2610_fsm 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2610_vm 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2611 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2612_fsm 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612_vm 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2613 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2615_fsm 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615_vm 2024-04-25T02:11:04.341 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2616 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2616_fsm 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2616_vm 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 01:55 2617 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2617_fsm 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2617_vm 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 01:55 2618 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2618_fsm 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2618_vm 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 25 01:55 2619 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2619_fsm 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2619_vm 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2620 2024-04-25T02:11:04.342 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2650 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2651 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2652 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2653 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2654 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2655 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2656 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2657 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 01:55 2658 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 2659 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2660 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2661 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2662 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2663 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2664 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2665 2024-04-25T02:11:04.343 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2666 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2667 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2668 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2669 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2670 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 01:55 2673 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 01:55 2674 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 01:55 2675 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2678 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2679 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2680 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2681 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2682 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2683 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2684 2024-04-25T02:11:04.344 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2685 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2686 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2687 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2688 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2689 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 01:55 2690 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 01:55 2691 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2692 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2693 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2696 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2699 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2701 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2702 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2703 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2704 2024-04-25T02:11:04.345 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2753 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2753_fsm 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2753_vm 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2754 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2755 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2756 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2757 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2830 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2831 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2832 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2833 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2834 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2835 2024-04-25T02:11:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2836 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2837 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 01:55 2838 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2838_fsm 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2838_vm 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2839 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840_fsm 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2840_vm 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2841 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2995 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2996 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3079_fsm 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079_vm 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3080 2024-04-25T02:11:04.347 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3081 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3085 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3118 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3119 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 01:55 3164 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3256 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3257 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3258 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3350 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3351 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3379 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3380 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3381 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3394 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3394_fsm 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3394_vm 2024-04-25T02:11:04.348 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3395 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3429 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3430 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3431 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3433 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3439 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3440 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3455 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 01:55 3456 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3456_fsm 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3456_vm 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3466 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3467 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3468 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3501 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3502 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3503 2024-04-25T02:11:04.349 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3534 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3541_fsm 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541_vm 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3542 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3574 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3575 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3576 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3596 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3597 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3598 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3599 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3600_fsm 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600_vm 2024-04-25T02:11:04.350 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3601_fsm 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601_vm 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3602_fsm 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602_vm 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603_fsm 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3603_vm 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3604 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3605 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3606 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3607 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3608 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 3609 2024-04-25T02:11:04.351 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3712 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3764_fsm 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764_vm 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3766 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3767 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3997 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4143 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4144 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4145 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4146 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4147 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4148 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4149 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4150 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4151 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4152 2024-04-25T02:11:04.352 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4153 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4154 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4155 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4156 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4157 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4158 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4159 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4160 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4161 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4162 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4163 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4164 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4165 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4166 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4167 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4168 2024-04-25T02:11:04.353 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4169 2024-04-25T02:11:04.354 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4170 2024-04-25T02:11:04.354 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4171 2024-04-25T02:11:04.354 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4172 2024-04-25T02:11:04.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4173 2024-04-25T02:11:04.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4174 2024-04-25T02:11:04.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 5002 2024-04-25T02:11:04.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 548 2024-04-25T02:11:04.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 549 2024-04-25T02:11:04.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6102 2024-04-25T02:11:04.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6104 2024-04-25T02:11:04.443 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6106 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6110 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6111 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6112 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6113 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6117 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 826 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 827 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 828 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 01:55 pg_filenode.map 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 PG_VERSION 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:total 610102 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Apr 25 01:59 . 2024-04-25T02:11:04.444 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 01:55 .. 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 112 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 113 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 1247 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1247_fsm 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1247_vm 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 25 01:55 1249 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1249_fsm 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1249_vm 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 25 01:55 1255 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1255_fsm 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1255_vm 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 25 02:10 1259 2024-04-25T02:11:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1259_fsm 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1259_vm 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 25 01:55 13344 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13344_fsm 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13344_vm 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13346 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13348 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13349_fsm 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13349_vm 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13351 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13353 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13354_fsm 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13354_vm 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13356 2024-04-25T02:11:04.446 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13358 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 13359_fsm 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13359_vm 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 13361 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 13363 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1417 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 1418 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 444407808 Apr 25 02:10 16396 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Apr 25 02:10 16396_fsm 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 02:01 16396_vm 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1818624 Apr 25 02:10 16397 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 02:10 16397_fsm 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 02:10 16397_vm 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 2252800 Apr 25 02:10 16399 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 02:10 16399_fsm 2024-04-25T02:11:04.447 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 02:10 16399_vm 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 02:10 16400 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 02:10 16402 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Apr 25 02:10 16404 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 95879168 Apr 25 02:10 16406 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 02:10 16406_fsm 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 02:09 16406_vm 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 174 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 175 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2187 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2224 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2328 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2336 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2337 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2579 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2600 2024-04-25T02:11:04.448 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2600_fsm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2600_vm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2601_fsm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2601_vm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2602 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2602_fsm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2602_vm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2603 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2603_fsm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2603_vm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2604 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2605 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2605_fsm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2605_vm 2024-04-25T02:11:04.449 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2606_fsm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2606_vm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2607 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2607_fsm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2607_vm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 25 01:55 2608 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2608_fsm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2608_vm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 25 01:55 2609 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2609_fsm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2609_vm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2610 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2610_fsm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2610_vm 2024-04-25T02:11:04.450 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2611 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2612_fsm 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2612_vm 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2613 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2615_fsm 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2615_vm 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2616 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2616_fsm 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2616_vm 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 25 01:55 2617 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2617_fsm 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2617_vm 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 25 01:55 2618 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2618_fsm 2024-04-25T02:11:04.451 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2618_vm 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 163840 Apr 25 02:10 2619 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:57 2619_fsm 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2619_vm 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2620 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2650 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2651 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2652 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2653 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2654 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2655 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2656 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2657 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 25 01:55 2658 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 25 01:55 2659 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2660 2024-04-25T02:11:04.452 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2661 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2662 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2663 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2664 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2665 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2666 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2667 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2668 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2669 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2670 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 25 01:55 2673 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 25 01:55 2674 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 01:55 2675 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2678 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2679 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2680 2024-04-25T02:11:04.453 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2681 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2682 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2683 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2684 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2685 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2686 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2687 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2688 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 25 01:55 2689 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 25 01:55 2690 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 25 01:55 2691 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2692 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2693 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 02:01 2696 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2699 2024-04-25T02:11:04.454 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2701 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2702 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2703 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 2704 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2753 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2753_fsm 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2753_vm 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2754 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2755 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2756 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 2757 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2830 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2831 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2832 2024-04-25T02:11:04.455 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2833 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2834 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2835 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2836 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2837 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 25 01:55 2838 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2838_fsm 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2838_vm 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2839 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2840_fsm 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2840_vm 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2841 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2995 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2996 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079 2024-04-25T02:11:04.456 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3079_fsm 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3079_vm 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3080 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3081 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3085 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3118 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3119 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 25 01:55 3164 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3256 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3257 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3258 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3350 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3351 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3379 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3380 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3381 2024-04-25T02:11:04.457 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3394 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3394_fsm 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3394_vm 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3395 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3429 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3430 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3431 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3433 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3439 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3440 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 25 01:55 3455 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 25 01:55 3456 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3456_fsm 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3456_vm 2024-04-25T02:11:04.458 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3466 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3467 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3468 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3501 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3502 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3503 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3534 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3541_fsm 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3541_vm 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3542 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3574 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3575 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3576 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3596 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3597 2024-04-25T02:11:04.459 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3598 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3599 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3600_fsm 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3600_vm 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3601_fsm 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3601_vm 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3602_fsm 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3602_vm 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3603_fsm 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3603_vm 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3604 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3605 2024-04-25T02:11:04.460 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3606 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3607 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3608 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 25 01:55 3609 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3712 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 3764_fsm 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3764_vm 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3766 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 3767 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3997 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4143 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4144 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4145 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4146 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4147 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4148 2024-04-25T02:11:04.461 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4149 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4150 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4151 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4152 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4153 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4154 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4155 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4156 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4157 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4158 2024-04-25T02:11:04.462 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4159 2024-04-25T02:11:04.466 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4160 2024-04-25T02:11:04.466 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4161 2024-04-25T02:11:04.466 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4162 2024-04-25T02:11:04.466 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4163 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4164 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4165 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4166 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4167 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4168 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4169 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4170 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4171 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4172 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4173 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4174 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 5002 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 548 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 549 2024-04-25T02:11:04.467 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6102 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6104 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6106 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6110 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6111 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6112 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6113 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6117 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 826 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 827 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 828 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 01:55 pg_filenode.map 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Apr 25 01:55 pg_internal.init 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 01:55 PG_VERSION 2024-04-25T02:11:04.468 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Apr 25 01:55 .. 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/global: 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:total 560 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Apr 25 01:55 . 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1213 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1213_fsm 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1213_vm 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1214 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1214_fsm 2024-04-25T02:11:04.469 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1214_vm 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 1232 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 1233 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1260 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1260_fsm 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1260_vm 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1261 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1261_fsm 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1261_vm 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1262 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 1262_fsm 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 1262_vm 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2396 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 25 01:55 2396_fsm 2024-04-25T02:11:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2396_vm 2024-04-25T02:11:04.471 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2397 2024-04-25T02:11:04.471 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2671 2024-04-25T02:11:04.471 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2672 2024-04-25T02:11:04.471 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2676 2024-04-25T02:11:04.471 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2677 2024-04-25T02:11:04.471 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2694 2024-04-25T02:11:04.471 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2695 2024-04-25T02:11:04.471 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2697 2024-04-25T02:11:04.471 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 25 01:55 2698 2024-04-25T02:11:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2846 2024-04-25T02:11:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2847 2024-04-25T02:11:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2964 2024-04-25T02:11:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2965 2024-04-25T02:11:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 2966 2024-04-25T02:11:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 2967 2024-04-25T02:11:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 3592 2024-04-25T02:11:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 3593 2024-04-25T02:11:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4060 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4061 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4175 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4176 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4177 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4178 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4181 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4182 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4183 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4184 2024-04-25T02:11:04.473 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 4185 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 4186 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6000 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6001 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6002 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 6100 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6114 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 6115 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 02:11 pg_control 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 25 01:55 pg_filenode.map 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Apr 25 01:55 pg_internal.init 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/log: 2024-04-25T02:11:04.474 INFO:teuthology.orchestra.run.smithi102.stdout:total 2 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1047 Apr 25 02:11 postgresql-Thu.log 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.475 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:total 1 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 02:11 . 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 mappings 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 25 02:11 replorigin_checkpoint 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 snapshots 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 02:11 .. 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.476 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 02:11 .. 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 . 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 members 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 offsets 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:total 8 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 .. 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:55 0000 2024-04-25T02:11:04.477 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:total 8 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 01:55 . 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 01:55 .. 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 25 01:59 0000 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.478 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.494 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.494 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.494 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-25T02:11:04.494 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:total 15 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 25 02:11 . 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.495 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Apr 25 02:11 db_0.stat 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Apr 25 02:11 db_13494.stat 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Apr 25 02:11 global.stat 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 02:11 . 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:total 192 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Apr 25 02:11 . 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 02:10 001B 2024-04-25T02:11:04.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:total 1048576 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 25 02:11 . 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:11 0000000100000000000000A0 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000A1 2024-04-25T02:11:04.497 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000A2 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:07 0000000100000000000000A3 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000A4 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:04 0000000100000000000000A5 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000A6 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000A7 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000A8 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000A9 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000AA 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:04 0000000100000000000000AB 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:04 0000000100000000000000AC 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000AD 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000AE 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000AF 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000B0 2024-04-25T02:11:04.498 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000B1 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000B2 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:04 0000000100000000000000B3 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000B4 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000B5 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000B6 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000B7 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:06 0000000100000000000000B8 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:04 0000000100000000000000B9 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:07 0000000100000000000000BA 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:06 0000000100000000000000BB 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:05 0000000100000000000000BC 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:03 0000000100000000000000BD 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000BE 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000BF 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000C0 2024-04-25T02:11:04.499 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:10 0000000100000000000000C1 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000C2 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:09 0000000100000000000000C3 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:09 0000000100000000000000C4 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000C5 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000C6 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:09 0000000100000000000000C7 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000C8 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000C9 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000CA 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:10 0000000100000000000000CB 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000CC 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:09 0000000100000000000000CD 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000CE 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:09 0000000100000000000000CF 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:09 0000000100000000000000D0 2024-04-25T02:11:04.500 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:10 0000000100000000000000D1 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000D2 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000D3 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000D4 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000D5 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000D6 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000D7 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000D8 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:09 0000000100000000000000D9 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000DA 2024-04-25T02:11:04.501 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:09 0000000100000000000000DB 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000DC 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000DD 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000DE 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 25 02:08 0000000100000000000000DF 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 archive_status 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout:total 0 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 25 01:55 . 2024-04-25T02:11:04.502 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Apr 25 02:11 .. 2024-04-25T02:11:04.503 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:04.503 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-25T02:11:04.503 INFO:teuthology.orchestra.run.smithi102.stdout:total 448 2024-04-25T02:11:04.503 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Apr 25 02:02 . 2024-04-25T02:11:04.503 INFO:teuthology.orchestra.run.smithi102.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Apr 25 02:11 .. 2024-04-25T02:11:04.503 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Apr 25 02:02 0000 2024-04-25T02:11:04.503 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 25 02:10 0001 2024-04-25T02:11:04.505 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-25T02:11:04.516 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T02:11:04.537 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-25T02:11:04.546 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-25T02:11:04.546 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T02:11:04.547 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T02:11:04.547 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-25T02:11:04.661 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-25T02:11:04.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T02:11:04.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.915+0000 7f1122800640 1 -- 172.21.15.102:0/2561290793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c108950 msgr2=0x7f111c10c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:04.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.915+0000 7f1122800640 1 --2- 172.21.15.102:0/2561290793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c108950 0x7f111c10c990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1104009860 tx=0x7f110402a340 comp rx=0 tx=0).stop 2024-04-25T02:11:04.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.915+0000 7f1122800640 1 -- 172.21.15.102:0/2561290793 shutdown_connections 2024-04-25T02:11:04.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.915+0000 7f1122800640 1 --2- 172.21.15.102:0/2561290793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111c10d070 0x7f111c114c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:04.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.915+0000 7f1122800640 1 --2- 172.21.15.102:0/2561290793 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c108950 0x7f111c10c990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:04.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.915+0000 7f1122800640 1 --2- 172.21.15.102:0/2561290793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f111c107fa0 0x7f111c108380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:04.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.915+0000 7f1122800640 1 -- 172.21.15.102:0/2561290793 >> 172.21.15.102:0/2561290793 conn(0x7f111c075cd0 msgr2=0x7f111c0780f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:04.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.915+0000 7f1122800640 1 -- 172.21.15.102:0/2561290793 shutdown_connections 2024-04-25T02:11:04.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.915+0000 7f1122800640 1 -- 172.21.15.102:0/2561290793 wait complete. 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.916+0000 7f1122800640 1 Processor -- start 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f1122800640 1 -- start start 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f1122800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f111c107fa0 0x7f111c19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f1122800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111c10d070 0x7f111c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f1122800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c1a2e40 0x7f111c1a32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f1122800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f111c1180e0 con 0x7f111c10d070 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f1122800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f111c117f60 con 0x7f111c1a2e40 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f1122800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f111c118260 con 0x7f111c107fa0 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f111b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f111c107fa0 0x7f111c19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f1113e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111c10d070 0x7f111c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f111be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c1a2e40 0x7f111c1a32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f111b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f111c107fa0 0x7f111c19e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50846/0 (socket says 172.21.15.102:50846) 2024-04-25T02:11:04.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f111be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c1a2e40 0x7f111c1a32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58710/0 (socket says 172.21.15.102:58710) 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.917+0000 7f111b400640 1 -- 172.21.15.102:0/1316667740 learned_addr learned my addr 172.21.15.102:0/1316667740 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f111be00640 1 -- 172.21.15.102:0/1316667740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f111c107fa0 msgr2=0x7f111c19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f111be00640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f111c107fa0 0x7f111c19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f111be00640 1 -- 172.21.15.102:0/1316667740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111c10d070 msgr2=0x7f111c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f111be00640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111c10d070 0x7f111c19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f111be00640 1 -- 172.21.15.102:0/1316667740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111c1a3930 con 0x7f111c1a2e40 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f111be00640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c1a2e40 0x7f111c1a32f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1114009fd0 tx=0x7f111400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f1118c00640 1 -- 172.21.15.102:0/1316667740 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111400ce20 con 0x7f111c1a2e40 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f111c1a3c20 con 0x7f111c1a2e40 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f1118c00640 1 -- 172.21.15.102:0/1316667740 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1114005020 con 0x7f111c1a2e40 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f1118c00640 1 -- 172.21.15.102:0/1316667740 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114010720 con 0x7f111c1a2e40 2024-04-25T02:11:04.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.918+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f111c1ab390 con 0x7f111c1a2e40 2024-04-25T02:11:04.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.920+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f111c04f500 con 0x7f111c1a2e40 2024-04-25T02:11:04.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.921+0000 7f1118c00640 1 -- 172.21.15.102:0/1316667740 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1114005540 con 0x7f111c1a2e40 2024-04-25T02:11:04.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.923+0000 7f1118c00640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11000a2da0 0x7f11000a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:04.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.923+0000 7f1118c00640 1 -- 172.21.15.102:0/1316667740 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f11140cfb50 con 0x7f111c1a2e40 2024-04-25T02:11:04.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.923+0000 7f111b400640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11000a2da0 0x7f11000a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:04.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.924+0000 7f111b400640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11000a2da0 0x7f11000a5260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f110c0058c0 tx=0x7f110c009260 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:04.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:04.927+0000 7f1118c00640 1 -- 172.21.15.102:0/1316667740 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f111409c6f0 con 0x7f111c1a2e40 2024-04-25T02:11:05.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.164+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 --> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f111c108950 con 0x7f11000a2da0 2024-04-25T02:11:05.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.181+0000 7f1118c00640 1 -- 172.21.15.102:0/1316667740 <== mgr.25214 v2:172.21.15.102:6800/855543702 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f111c108950 con 0x7f11000a2da0 2024-04-25T02:11:05.188 INFO:teuthology.orchestra.run.smithi102.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T01:55:24", "first": "2024-04-25T01:56:00", "last": "2024-04-25T02:11:00", "last_pruned": "2024-04-25T02:11:00", "created_count": 16, "pruned_count": 9, "active": true} 2024-04-25T02:11:05.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11000a2da0 msgr2=0x7f11000a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11000a2da0 0x7f11000a5260 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f110c0058c0 tx=0x7f110c009260 comp rx=0 tx=0).stop 2024-04-25T02:11:05.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c1a2e40 msgr2=0x7f111c1a32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c1a2e40 0x7f111c1a32f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f1114009fd0 tx=0x7f111400ef10 comp rx=0 tx=0).stop 2024-04-25T02:11:05.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 shutdown_connections 2024-04-25T02:11:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f11000a2da0 0x7f11000a5260 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c1a2e40 0x7f111c1a32f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111c10d070 0x7f111c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 --2- 172.21.15.102:0/1316667740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f111c107fa0 0x7f111c19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.183+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 >> 172.21.15.102:0/1316667740 conn(0x7f111c075cd0 msgr2=0x7f111c077f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.184+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 shutdown_connections 2024-04-25T02:11:05.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.184+0000 7f1122800640 1 -- 172.21.15.102:0/1316667740 wait complete. 2024-04-25T02:11:05.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T02:11:05.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.387+0000 7f49b8600640 1 -- 172.21.15.102:0/965521601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49b0106630 msgr2=0x7f49b0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.387+0000 7f49b8600640 1 --2- 172.21.15.102:0/965521601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49b0106630 0x7f49b0106a10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009900 tx=0x7f49a402a3f0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.387+0000 7f49b8600640 1 -- 172.21.15.102:0/965521601 shutdown_connections 2024-04-25T02:11:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.387+0000 7f49b8600640 1 --2- 172.21.15.102:0/965521601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49b0100fe0 0x7f49b010d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.387+0000 7f49b8600640 1 --2- 172.21.15.102:0/965521601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49b0100620 0x7f49b0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.387+0000 7f49b8600640 1 --2- 172.21.15.102:0/965521601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49b0106630 0x7f49b0106a10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.387+0000 7f49b8600640 1 -- 172.21.15.102:0/965521601 >> 172.21.15.102:0/965521601 conn(0x7f49b00fc110 msgr2=0x7f49b00fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.387+0000 7f49b8600640 1 -- 172.21.15.102:0/965521601 shutdown_connections 2024-04-25T02:11:05.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.387+0000 7f49b8600640 1 -- 172.21.15.102:0/965521601 wait complete. 2024-04-25T02:11:05.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.388+0000 7f49b8600640 1 Processor -- start 2024-04-25T02:11:05.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.388+0000 7f49b8600640 1 -- start start 2024-04-25T02:11:05.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.388+0000 7f49b8600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49b0100620 0x7f49b019a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:05.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.388+0000 7f49b8600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49b0100fe0 0x7f49b019a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.388+0000 7f49b8600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49b0106630 0x7f49b019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.388+0000 7f49b8600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f49b0110a50 con 0x7f49b0100fe0 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.388+0000 7f49b8600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f49b01108d0 con 0x7f49b0106630 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.388+0000 7f49b8600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f49b0110bd0 con 0x7f49b0100620 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49b0100620 0x7f49b019a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49b0100620 0x7f49b019a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50868/0 (socket says 172.21.15.102:50868) 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5c00640 1 -- 172.21.15.102:0/162756688 learned_addr learned my addr 172.21.15.102:0/162756688 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5200640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49b0100fe0 0x7f49b019a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b6600640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49b0106630 0x7f49b019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5c00640 1 -- 172.21.15.102:0/162756688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49b0106630 msgr2=0x7f49b019ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5c00640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49b0106630 0x7f49b019ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5c00640 1 -- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49b0100fe0 msgr2=0x7f49b019a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5c00640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49b0100fe0 0x7f49b019a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5c00640 1 -- 172.21.15.102:0/162756688 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b019f130 con 0x7f49b0100620 2024-04-25T02:11:05.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.389+0000 7f49b5c00640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49b0100620 0x7f49b019a140 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f49a402a9e0 tx=0x7f49a4003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.390+0000 7f49ae000640 1 -- 172.21.15.102:0/162756688 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a403f070 con 0x7f49b0100620 2024-04-25T02:11:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.390+0000 7f49ae000640 1 -- 172.21.15.102:0/162756688 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a4031c30 con 0x7f49b0100620 2024-04-25T02:11:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.390+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b019f3c0 con 0x7f49b0100620 2024-04-25T02:11:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.390+0000 7f49ae000640 1 -- 172.21.15.102:0/162756688 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a4035030 con 0x7f49b0100620 2024-04-25T02:11:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.390+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b01a6cb0 con 0x7f49b0100620 2024-04-25T02:11:05.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.391+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b0101d60 con 0x7f49b0100620 2024-04-25T02:11:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.393+0000 7f49ae000640 1 -- 172.21.15.102:0/162756688 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f49a4042070 con 0x7f49b0100620 2024-04-25T02:11:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.394+0000 7f49ae000640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f49980a2da0 0x7f49980a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.394+0000 7f49ae000640 1 -- 172.21.15.102:0/162756688 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f49a40efa80 con 0x7f49b0100620 2024-04-25T02:11:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.394+0000 7f49b5200640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f49980a2da0 0x7f49980a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:05.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.395+0000 7f49b5200640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f49980a2da0 0x7f49980a5260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f49b010d220 tx=0x7f499c005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:05.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.399+0000 7f49ae000640 1 -- 172.21.15.102:0/162756688 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f49a40f1e20 con 0x7f49b0100620 2024-04-25T02:11:05.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:05 smithi102 ceph-mon[27796]: pgmap v494: 97 pgs: 16 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 72 active+clean; 5.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 19 MiB/s wr, 1.80k op/s 2024-04-25T02:11:05.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:05 smithi110 ceph-mon[32848]: pgmap v494: 97 pgs: 16 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 72 active+clean; 5.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 19 MiB/s wr, 1.80k op/s 2024-04-25T02:11:05.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:05 smithi193 ceph-mon[32981]: pgmap v494: 97 pgs: 16 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 72 active+clean; 5.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 19 MiB/s wr, 1.80k op/s 2024-04-25T02:11:05.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.622+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 --> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f49b006a620 con 0x7f49980a2da0 2024-04-25T02:11:05.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.633+0000 7f49ae000640 1 -- 172.21.15.102:0/162756688 <== mgr.25214 v2:172.21.15.102:6800/855543702 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f49b006a620 con 0x7f49980a2da0 2024-04-25T02:11:05.641 INFO:teuthology.orchestra.run.smithi102.stdout:/ 1m 6m3h 2024-04-25T02:11:05.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f49980a2da0 msgr2=0x7f49980a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f49980a2da0 0x7f49980a5260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f49b010d220 tx=0x7f499c005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:05.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49b0100620 msgr2=0x7f49b019a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49b0100620 0x7f49b019a140 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f49a402a9e0 tx=0x7f49a4003c10 comp rx=0 tx=0).stop 2024-04-25T02:11:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 shutdown_connections 2024-04-25T02:11:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f49980a2da0 0x7f49980a5260 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49b0106630 0x7f49b019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49b0100fe0 0x7f49b019a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 --2- 172.21.15.102:0/162756688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49b0100620 0x7f49b019a140 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.636+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 >> 172.21.15.102:0/162756688 conn(0x7f49b00fc110 msgr2=0x7f49b0069f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.637+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 shutdown_connections 2024-04-25T02:11:05.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.637+0000 7f49b8600640 1 -- 172.21.15.102:0/162756688 wait complete. 2024-04-25T02:11:05.663 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-25T02:11:05.674 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-25T02:11:05.674 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-04-25T02:11:05.675 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T02:11:05.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T02:11:05.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.875+0000 7f20a7e00640 1 -- 172.21.15.102:0/547395837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8107640 msgr2=0x7f20a8107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.875+0000 7f20a7e00640 1 --2- 172.21.15.102:0/547395837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8107640 0x7f20a8107a20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2098009930 tx=0x7f209802a420 comp rx=0 tx=0).stop 2024-04-25T02:11:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.875+0000 7f20a7e00640 1 -- 172.21.15.102:0/547395837 shutdown_connections 2024-04-25T02:11:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.875+0000 7f20a7e00640 1 --2- 172.21.15.102:0/547395837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20a8101f40 0x7f20a810e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.875+0000 7f20a7e00640 1 --2- 172.21.15.102:0/547395837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a8101580 0x7f20a8101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.875+0000 7f20a7e00640 1 --2- 172.21.15.102:0/547395837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8107640 0x7f20a8107a20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.875+0000 7f20a7e00640 1 -- 172.21.15.102:0/547395837 >> 172.21.15.102:0/547395837 conn(0x7f20a80fd070 msgr2=0x7f20a80ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.875+0000 7f20a7e00640 1 -- 172.21.15.102:0/547395837 shutdown_connections 2024-04-25T02:11:05.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.875+0000 7f20a7e00640 1 -- 172.21.15.102:0/547395837 wait complete. 2024-04-25T02:11:05.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.876+0000 7f20a7e00640 1 Processor -- start 2024-04-25T02:11:05.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.876+0000 7f20a7e00640 1 -- start start 2024-04-25T02:11:05.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.876+0000 7f20a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20a8101580 0x7f20a8073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:05.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.876+0000 7f20a7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8101f40 0x7f20a8073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:05.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.876+0000 7f20a7e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a8107640 0x7f20a80744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.876+0000 7f20a7e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f20a8111920 con 0x7f20a8101580 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.876+0000 7f20a7e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f20a81117a0 con 0x7f20a8107640 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.876+0000 7f20a7e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f20a8111aa0 con 0x7f20a8101f40 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8101f40 0x7f20a8073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20a8101580 0x7f20a8073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8101f40 0x7f20a8073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50894/0 (socket says 172.21.15.102:50894) 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6000640 1 -- 172.21.15.102:0/2209759749 learned_addr learned my addr 172.21.15.102:0/2209759749 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6a00640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20a8101580 0x7f20a8073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38718/0 (socket says 172.21.15.102:38718) 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a7400640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a8107640 0x7f20a80744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6000640 1 -- 172.21.15.102:0/2209759749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a8107640 msgr2=0x7f20a80744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6000640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a8107640 0x7f20a80744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6000640 1 -- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20a8101580 msgr2=0x7f20a8073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:05.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6000640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20a8101580 0x7f20a8073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:05.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6000640 1 -- 172.21.15.102:0/2209759749 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a81a62a0 con 0x7f20a8101f40 2024-04-25T02:11:05.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f20a6000640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8101f40 0x7f20a8073f80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f209400e9f0 tx=0x7f209400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:05.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f209f400640 1 -- 172.21.15.102:0/2209759749 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209400cd80 con 0x7f20a8101f40 2024-04-25T02:11:05.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.877+0000 7f209f400640 1 -- 172.21.15.102:0/2209759749 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2094005500 con 0x7f20a8101f40 2024-04-25T02:11:05.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.878+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a81a6590 con 0x7f20a8101f40 2024-04-25T02:11:05.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.878+0000 7f209f400640 1 -- 172.21.15.102:0/2209759749 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2094010690 con 0x7f20a8101f40 2024-04-25T02:11:05.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.878+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a81a6ae0 con 0x7f20a8101f40 2024-04-25T02:11:05.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.879+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a8068480 con 0x7f20a8101f40 2024-04-25T02:11:05.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.881+0000 7f209f400640 1 -- 172.21.15.102:0/2209759749 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f20940046c0 con 0x7f20a8101f40 2024-04-25T02:11:05.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.882+0000 7f209f400640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f208c0a2d80 0x7f208c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:05.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.883+0000 7f209f400640 1 -- 172.21.15.102:0/2209759749 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f20940cf850 con 0x7f20a8101f40 2024-04-25T02:11:05.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.883+0000 7f20a6a00640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f208c0a2d80 0x7f208c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:05.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.884+0000 7f20a6a00640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f208c0a2d80 0x7f208c0a5240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2098009900 tx=0x7f20980058d0 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:05.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:05.886+0000 7f209f400640 1 -- 172.21.15.102:0/2209759749 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f209409c3f0 con 0x7f20a8101f40 2024-04-25T02:11:06.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.096+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20a802d170 con 0x7f20a8101f40 2024-04-25T02:11:06.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.098+0000 7f209f400640 1 -- 172.21.15.102:0/2209759749 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+23226 (secure 0 0 0) 0x7f209409bcf0 con 0x7f20a8101f40 2024-04-25T02:11:06.106 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:06.107 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":107,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T02:11:01.842886+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":26,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6801/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/459072323":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/2875896366":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1835499087":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/511834001":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":26,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":26,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T02:11:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.100+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f208c0a2d80 msgr2=0x7f208c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.100+0000 7f20a7e00640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f208c0a2d80 0x7f208c0a5240 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2098009900 tx=0x7f20980058d0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.100+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8101f40 msgr2=0x7f20a8073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.100+0000 7f20a7e00640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8101f40 0x7f20a8073f80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f209400e9f0 tx=0x7f209400edf0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.101+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 shutdown_connections 2024-04-25T02:11:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.101+0000 7f20a7e00640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f208c0a2d80 0x7f208c0a5240 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.101+0000 7f20a7e00640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a8107640 0x7f20a80744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.101+0000 7f20a7e00640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20a8101f40 0x7f20a8073f80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.101+0000 7f20a7e00640 1 --2- 172.21.15.102:0/2209759749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f20a8101580 0x7f20a8073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.101+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 >> 172.21.15.102:0/2209759749 conn(0x7f20a80fd070 msgr2=0x7f20a80fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.101+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 shutdown_connections 2024-04-25T02:11:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.101+0000 7f20a7e00640 1 -- 172.21.15.102:0/2209759749 wait complete. 2024-04-25T02:11:06.127 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:41.585137+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:43.022666+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 26, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T02:11:06.128 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T02:11:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.331+0000 7fc812e00640 1 -- 172.21.15.102:0/4218871272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc80c103920 msgr2=0x7fc80c103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.331+0000 7fc812e00640 1 --2- 172.21.15.102:0/4218871272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc80c103920 0x7fc80c103da0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009930 tx=0x7fc7f402a420 comp rx=0 tx=0).stop 2024-04-25T02:11:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.331+0000 7fc812e00640 1 -- 172.21.15.102:0/4218871272 shutdown_connections 2024-04-25T02:11:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.331+0000 7fc812e00640 1 --2- 172.21.15.102:0/4218871272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc80c1042e0 0x7fc80c1107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.331+0000 7fc812e00640 1 --2- 172.21.15.102:0/4218871272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc80c103920 0x7fc80c103da0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.331+0000 7fc812e00640 1 --2- 172.21.15.102:0/4218871272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc80c109930 0x7fc80c109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.331+0000 7fc812e00640 1 -- 172.21.15.102:0/4218871272 >> 172.21.15.102:0/4218871272 conn(0x7fc80c0ff3b0 msgr2=0x7fc80c1017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.331+0000 7fc812e00640 1 -- 172.21.15.102:0/4218871272 shutdown_connections 2024-04-25T02:11:06.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.331+0000 7fc812e00640 1 -- 172.21.15.102:0/4218871272 wait complete. 2024-04-25T02:11:06.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.332+0000 7fc812e00640 1 Processor -- start 2024-04-25T02:11:06.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.332+0000 7fc812e00640 1 -- start start 2024-04-25T02:11:06.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc812e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc80c103920 0x7fc80c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:06.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc812e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc80c1042e0 0x7fc80c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:06.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc812e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc80c109930 0x7fc80c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:06.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc812e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c113d90 con 0x7fc80c1042e0 2024-04-25T02:11:06.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc812e00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c113c10 con 0x7fc80c103920 2024-04-25T02:11:06.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc812e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c113f10 con 0x7fc80c109930 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc80be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc80c103920 0x7fc80c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc80be00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc80c103920 0x7fc80c19a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:58748/0 (socket says 172.21.15.102:58748) 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc80be00640 1 -- 172.21.15.102:0/1576440017 learned_addr learned my addr 172.21.15.102:0/1576440017 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc803e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc80c1042e0 0x7fc80c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.333+0000 7fc810e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc80c109930 0x7fc80c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc810e00640 1 -- 172.21.15.102:0/1576440017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc80c103920 msgr2=0x7fc80c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc810e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc80c103920 0x7fc80c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc810e00640 1 -- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc80c1042e0 msgr2=0x7fc80c19a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc810e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc80c1042e0 0x7fc80c19a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc810e00640 1 -- 172.21.15.102:0/1576440017 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c19f240 con 0x7fc80c109930 2024-04-25T02:11:06.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc80be00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc80c103920 0x7fc80c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:06.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc810e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc80c109930 0x7fc80c19eb10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc804009fd0 tx=0x7fc80400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:06.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc809600640 1 -- 172.21.15.102:0/1576440017 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80400ce20 con 0x7fc80c109930 2024-04-25T02:11:06.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc809600640 1 -- 172.21.15.102:0/1576440017 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc804005020 con 0x7fc80c109930 2024-04-25T02:11:06.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80c19f530 con 0x7fc80c109930 2024-04-25T02:11:06.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc809600640 1 -- 172.21.15.102:0/1576440017 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804010720 con 0x7fc80c109930 2024-04-25T02:11:06.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.334+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80c1a6dc0 con 0x7fc80c109930 2024-04-25T02:11:06.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.336+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc80c068560 con 0x7fc80c109930 2024-04-25T02:11:06.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.338+0000 7fc809600640 1 -- 172.21.15.102:0/1576440017 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc804005540 con 0x7fc80c109930 2024-04-25T02:11:06.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.339+0000 7fc809600640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc7ec0a2d00 0x7fc7ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:06.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.340+0000 7fc809600640 1 -- 172.21.15.102:0/1576440017 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7fc8040cfa80 con 0x7fc80c109930 2024-04-25T02:11:06.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.340+0000 7fc80be00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc7ec0a2d00 0x7fc7ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:06.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.341+0000 7fc80be00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc7ec0a2d00 0x7fc7ec0a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc011440 tx=0x7fc7fc005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:06.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.344+0000 7fc809600640 1 -- 172.21.15.102:0/1576440017 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc80409c680 con 0x7fc80c109930 2024-04-25T02:11:06.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:06 smithi102 ceph-mon[27796]: from='client.27779 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:11:06.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:06 smithi110 ceph-mon[32848]: from='client.27779 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:11:06.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:06 smithi193 ceph-mon[32981]: from='client.27779 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:11:06.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.545+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc80c02d170 con 0x7fc80c109930 2024-04-25T02:11:06.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.546+0000 7fc809600640 1 -- 172.21.15.102:0/1576440017 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) ==== 93+0+10 (secure 0 0 0) 0x7fc80409bf80 con 0x7fc80c109930 2024-04-25T02:11:06.554 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T02:11:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc7ec0a2d00 msgr2=0x7fc7ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc7ec0a2d00 0x7fc7ec0a51c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc011440 tx=0x7fc7fc005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc80c109930 msgr2=0x7fc80c19eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc80c109930 0x7fc80c19eb10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc804009fd0 tx=0x7fc80400ef10 comp rx=0 tx=0).stop 2024-04-25T02:11:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 shutdown_connections 2024-04-25T02:11:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fc7ec0a2d00 0x7fc7ec0a51c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc80c109930 0x7fc80c19eb10 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc80c1042e0 0x7fc80c19a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 --2- 172.21.15.102:0/1576440017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc80c103920 0x7fc80c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 >> 172.21.15.102:0/1576440017 conn(0x7fc80c0ff3b0 msgr2=0x7fc80c0ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 shutdown_connections 2024-04-25T02:11:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.549+0000 7fc812e00640 1 -- 172.21.15.102:0/1576440017 wait complete. 2024-04-25T02:11:06.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T02:11:06.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.771+0000 7fe912400640 1 -- 172.21.15.102:0/2492975957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c074490 msgr2=0x7fe90c10d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.771+0000 7fe912400640 1 --2- 172.21.15.102:0/2492975957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c074490 0x7fe90c10d8d0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009860 tx=0x7fe8f402a3d0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.771+0000 7fe912400640 1 -- 172.21.15.102:0/2492975957 shutdown_connections 2024-04-25T02:11:06.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.771+0000 7fe912400640 1 --2- 172.21.15.102:0/2492975957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe90c10de10 0x7fe90c110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.771+0000 7fe912400640 1 --2- 172.21.15.102:0/2492975957 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c074490 0x7fe90c10d8d0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.771+0000 7fe912400640 1 --2- 172.21.15.102:0/2492975957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe90c073b70 0x7fe90c073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.771+0000 7fe912400640 1 -- 172.21.15.102:0/2492975957 >> 172.21.15.102:0/2492975957 conn(0x7fe90c0fd0e0 msgr2=0x7fe90c0ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:06.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.771+0000 7fe912400640 1 -- 172.21.15.102:0/2492975957 shutdown_connections 2024-04-25T02:11:06.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.771+0000 7fe912400640 1 -- 172.21.15.102:0/2492975957 wait complete. 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.772+0000 7fe912400640 1 Processor -- start 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.772+0000 7fe912400640 1 -- start start 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe912400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe90c073b70 0x7fe90c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe912400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c074490 0x7fe90c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe912400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe90c10de10 0x7fe90c19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe912400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c113bb0 con 0x7fe90c073b70 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe912400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c113a30 con 0x7fe90c074490 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe912400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c113d30 con 0x7fe90c10de10 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe911000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe90c073b70 0x7fe90c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe911000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe90c073b70 0x7fe90c19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38756/0 (socket says 172.21.15.102:38756) 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe911000640 1 -- 172.21.15.102:0/2616013888 learned_addr learned my addr 172.21.15.102:0/2616013888 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe903e00640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c074490 0x7fe90c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:06.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.773+0000 7fe911a00640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe90c10de10 0x7fe90c19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe911000640 1 -- 172.21.15.102:0/2616013888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe90c10de10 msgr2=0x7fe90c19e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe911000640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe90c10de10 0x7fe90c19e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe911000640 1 -- 172.21.15.102:0/2616013888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c074490 msgr2=0x7fe90c19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe911000640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c074490 0x7fe90c19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe911000640 1 -- 172.21.15.102:0/2616013888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c19f120 con 0x7fe90c073b70 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe903e00640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c074490 0x7fe90c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe911000640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe90c073b70 0x7fe90c19a130 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00e9f0 tx=0x7fe8fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe90a000640 1 -- 172.21.15.102:0/2616013888 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc00ccc0 con 0x7fe90c073b70 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90c19f410 con 0x7fe90c073b70 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe90a000640 1 -- 172.21.15.102:0/2616013888 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8fc005500 con 0x7fe90c073b70 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.774+0000 7fe90a000640 1 -- 172.21.15.102:0/2616013888 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc010690 con 0x7fe90c073b70 2024-04-25T02:11:06.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.775+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe90c1a6ca0 con 0x7fe90c073b70 2024-04-25T02:11:06.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.776+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe90c067bf0 con 0x7fe90c073b70 2024-04-25T02:11:06.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.779+0000 7fe90a000640 1 -- 172.21.15.102:0/2616013888 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe8fc0046c0 con 0x7fe90c073b70 2024-04-25T02:11:06.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.780+0000 7fe90a000640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe8f00a2d80 0x7fe8f00a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:06.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.781+0000 7fe90a000640 1 -- 172.21.15.102:0/2616013888 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7fe8fc0cf900 con 0x7fe90c073b70 2024-04-25T02:11:06.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.781+0000 7fe903e00640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe8f00a2d80 0x7fe8f00a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:06.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.781+0000 7fe903e00640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe8f00a2d80 0x7fe8f00a5240 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0ff7e0 tx=0x7fe8f4009530 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:06.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.783+0000 7fe90a000640 1 -- 172.21.15.102:0/2616013888 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe8fc053740 con 0x7fe90c073b70 2024-04-25T02:11:06.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.981+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe90c108f90 con 0x7fe90c073b70 2024-04-25T02:11:06.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.982+0000 7fe90a000640 1 -- 172.21.15.102:0/2616013888 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) ==== 104+0+11 (secure 0 0 0) 0x7fe8fc09c4a0 con 0x7fe90c073b70 2024-04-25T02:11:06.990 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-25T02:11:06.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.984+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe8f00a2d80 msgr2=0x7fe8f00a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.984+0000 7fe912400640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe8f00a2d80 0x7fe8f00a5240 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe90c0ff7e0 tx=0x7fe8f4009530 comp rx=0 tx=0).stop 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe90c073b70 msgr2=0x7fe90c19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe90c073b70 0x7fe90c19a130 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00e9f0 tx=0x7fe8fc00eeb0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 shutdown_connections 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe8f00a2d80 0x7fe8f00a5240 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe90c10de10 0x7fe90c19e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90c074490 0x7fe90c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 --2- 172.21.15.102:0/2616013888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe90c073b70 0x7fe90c19a130 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 >> 172.21.15.102:0/2616013888 conn(0x7fe90c0fd0e0 msgr2=0x7fe90c0fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 shutdown_connections 2024-04-25T02:11:06.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:06.985+0000 7fe912400640 1 -- 172.21.15.102:0/2616013888 wait complete. 2024-04-25T02:11:07.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T02:11:07.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.208+0000 7f17ef200640 1 -- 172.21.15.102:0/1395594222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 msgr2=0x7f17f010e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:07.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.208+0000 7f17ef200640 1 --2- 172.21.15.102:0/1395594222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 0x7f17f010e060 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009ca0 tx=0x7f17e802a950 comp rx=0 tx=0).stop 2024-04-25T02:11:07.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.208+0000 7f17ef200640 1 -- 172.21.15.102:0/1395594222 shutdown_connections 2024-04-25T02:11:07.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.208+0000 7f17ef200640 1 --2- 172.21.15.102:0/1395594222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 0x7f17f010e060 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.208+0000 7f17ef200640 1 --2- 172.21.15.102:0/1395594222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f01030c0 0x7f17f010b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.208+0000 7f17ef200640 1 --2- 172.21.15.102:0/1395594222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f01027a0 0x7f17f0102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.208+0000 7f17ef200640 1 -- 172.21.15.102:0/1395594222 >> 172.21.15.102:0/1395594222 conn(0x7f17f00fbfc0 msgr2=0x7f17f00fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:07.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.208+0000 7f17ef200640 1 -- 172.21.15.102:0/1395594222 shutdown_connections 2024-04-25T02:11:07.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.208+0000 7f17ef200640 1 -- 172.21.15.102:0/1395594222 wait complete. 2024-04-25T02:11:07.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.209+0000 7f17ef200640 1 Processor -- start 2024-04-25T02:11:07.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.209+0000 7f17ef200640 1 -- start start 2024-04-25T02:11:07.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ef200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f01027a0 0x7f17f0197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:07.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ef200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f01030c0 0x7f17f01983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:07.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ef200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 0x7f17f019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:07.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ef200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0111a10 con 0x7f17f010bbc0 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ef200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0111890 con 0x7f17f01030c0 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ef200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0111b90 con 0x7f17f01027a0 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ee800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 0x7f17f019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ede00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f01027a0 0x7f17f0197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ee800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 0x7f17f019c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38776/0 (socket says 172.21.15.102:38776) 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ee800640 1 -- 172.21.15.102:0/3446762559 learned_addr learned my addr 172.21.15.102:0/3446762559 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ed400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f01030c0 0x7f17f01983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ee800640 1 -- 172.21.15.102:0/3446762559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f01027a0 msgr2=0x7f17f0197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ee800640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f01027a0 0x7f17f0197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ee800640 1 -- 172.21.15.102:0/3446762559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f01030c0 msgr2=0x7f17f01983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ee800640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f01030c0 0x7f17f01983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ee800640 1 -- 172.21.15.102:0/3446762559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f019ce60 con 0x7f17f010bbc0 2024-04-25T02:11:07.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ede00640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f01027a0 0x7f17f0197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:07.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.210+0000 7f17ed400640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f01030c0 0x7f17f01983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:07.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.211+0000 7f17ee800640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 0x7f17f019c730 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009dc0 tx=0x7f17e8003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:07.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.211+0000 7f17e6a00640 1 -- 172.21.15.102:0/3446762559 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8034040 con 0x7f17f010bbc0 2024-04-25T02:11:07.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.211+0000 7f17e6a00640 1 -- 172.21.15.102:0/3446762559 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e802f940 con 0x7f17f010bbc0 2024-04-25T02:11:07.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.211+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f019d0f0 con 0x7f17f010bbc0 2024-04-25T02:11:07.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.211+0000 7f17e6a00640 1 -- 172.21.15.102:0/3446762559 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e80046a0 con 0x7f17f010bbc0 2024-04-25T02:11:07.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.211+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f0100320 con 0x7f17f010bbc0 2024-04-25T02:11:07.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.212+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f00691b0 con 0x7f17f010bbc0 2024-04-25T02:11:07.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.214+0000 7f17e6a00640 1 -- 172.21.15.102:0/3446762559 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f17e8037020 con 0x7f17f010bbc0 2024-04-25T02:11:07.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.216+0000 7f17e6a00640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17d40a2d00 0x7f17d40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:07.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.216+0000 7f17e6a00640 1 -- 172.21.15.102:0/3446762559 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f17e80f00d0 con 0x7f17f010bbc0 2024-04-25T02:11:07.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.216+0000 7f17ede00640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17d40a2d00 0x7f17d40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:07.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.217+0000 7f17ede00640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17d40a2d00 0x7f17d40a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f17e00096a0 tx=0x7f17e0012040 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:07.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.220+0000 7f17e6a00640 1 -- 172.21.15.102:0/3446762559 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f17e80bcd80 con 0x7f17f010bbc0 2024-04-25T02:11:07.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.422+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f17f0100580 con 0x7f17f010bbc0 2024-04-25T02:11:07.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:07 smithi102 ceph-mon[27796]: from='client.27727 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:11:07.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:07 smithi102 ceph-mon[27796]: pgmap v495: 97 pgs: 16 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 72 active+clean; 5.3 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 24 MiB/s wr, 2.18k op/s 2024-04-25T02:11:07.431 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:07 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2209759749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:11:07.431 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:07 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1576440017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:11:07.431 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:07 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2616013888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:11:07.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.423+0000 7f17e6a00640 1 -- 172.21.15.102:0/3446762559 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) ==== 100+0+11 (secure 0 0 0) 0x7f17e80bc680 con 0x7f17f010bbc0 2024-04-25T02:11:07.431 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-25T02:11:07.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.425+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17d40a2d00 msgr2=0x7f17d40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:07.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17d40a2d00 0x7f17d40a51c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f17e00096a0 tx=0x7f17e0012040 comp rx=0 tx=0).stop 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 msgr2=0x7f17f019c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 0x7f17f019c730 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009dc0 tx=0x7f17e8003c10 comp rx=0 tx=0).stop 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 shutdown_connections 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f17d40a2d00 0x7f17d40a51c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17f010bbc0 0x7f17f019c730 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f01030c0 0x7f17f01983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 --2- 172.21.15.102:0/3446762559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17f01027a0 0x7f17f0197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 >> 172.21.15.102:0/3446762559 conn(0x7f17f00fbfc0 msgr2=0x7f17f00fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 shutdown_connections 2024-04-25T02:11:07.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.426+0000 7f17ef200640 1 -- 172.21.15.102:0/3446762559 wait complete. 2024-04-25T02:11:07.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T02:11:07.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:07 smithi110 ceph-mon[32848]: from='client.27727 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:11:07.495 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:07 smithi110 ceph-mon[32848]: pgmap v495: 97 pgs: 16 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 72 active+clean; 5.3 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 24 MiB/s wr, 2.18k op/s 2024-04-25T02:11:07.495 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:07 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2209759749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:11:07.495 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:07 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1576440017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:11:07.495 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:07 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2616013888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:11:07.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:07 smithi193 ceph-mon[32981]: from='client.27727 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T02:11:07.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:07 smithi193 ceph-mon[32981]: pgmap v495: 97 pgs: 16 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 72 active+clean; 5.3 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 24 MiB/s wr, 2.18k op/s 2024-04-25T02:11:07.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:07 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2209759749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:11:07.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:07 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1576440017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:11:07.513 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:07 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2616013888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:11:07.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.638+0000 7f366d400640 1 -- 172.21.15.102:0/2067026394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 msgr2=0x7f3668076050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:07.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.638+0000 7f366d400640 1 --2- 172.21.15.102:0/2067026394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 0x7f3668076050 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3658009890 tx=0x7f365802a370 comp rx=0 tx=0).stop 2024-04-25T02:11:07.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.638+0000 7f366d400640 1 -- 172.21.15.102:0/2067026394 shutdown_connections 2024-04-25T02:11:07.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.638+0000 7f366d400640 1 --2- 172.21.15.102:0/2067026394 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36681125b0 0x7f36681149a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.638+0000 7f366d400640 1 --2- 172.21.15.102:0/2067026394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668076590 0x7f3668112070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.638+0000 7f366d400640 1 --2- 172.21.15.102:0/2067026394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 0x7f3668076050 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.638+0000 7f366d400640 1 -- 172.21.15.102:0/2067026394 >> 172.21.15.102:0/2067026394 conn(0x7f36680ff400 msgr2=0x7f3668101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.638+0000 7f366d400640 1 -- 172.21.15.102:0/2067026394 shutdown_connections 2024-04-25T02:11:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.638+0000 7f366d400640 1 -- 172.21.15.102:0/2067026394 wait complete. 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.639+0000 7f366d400640 1 Processor -- start 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.639+0000 7f366d400640 1 -- start start 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f366d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 0x7f366819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f366d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668076590 0x7f366819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f366d400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36681125b0 0x7f36681a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f366d400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3668118120 con 0x7f3668076590 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f366d400640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3668117fa0 con 0x7f36681125b0 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f366d400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f36681182a0 con 0x7f3668075c70 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3665e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668076590 0x7f366819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3666800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 0x7f366819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3667200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36681125b0 0x7f36681a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:07.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3666800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 0x7f366819e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:50980/0 (socket says 172.21.15.102:50980) 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3666800640 1 -- 172.21.15.102:0/337566671 learned_addr learned my addr 172.21.15.102:0/337566671 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3666800640 1 -- 172.21.15.102:0/337566671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36681125b0 msgr2=0x7f36681a2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3666800640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36681125b0 0x7f36681a2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3666800640 1 -- 172.21.15.102:0/337566671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668076590 msgr2=0x7f366819ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3666800640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668076590 0x7f366819ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3666800640 1 -- 172.21.15.102:0/337566671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36681a3710 con 0x7f3668075c70 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.640+0000 7f3667200640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36681125b0 0x7f36681a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.641+0000 7f3666800640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 0x7f366819e720 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3658030510 tx=0x7f3658033ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.641+0000 7f365f400640 1 -- 172.21.15.102:0/337566671 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3658037040 con 0x7f3668075c70 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.641+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36681a39a0 con 0x7f3668075c70 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.641+0000 7f365f400640 1 -- 172.21.15.102:0/337566671 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3658040070 con 0x7f3668075c70 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.641+0000 7f365f400640 1 -- 172.21.15.102:0/337566671 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36580038c0 con 0x7f3668075c70 2024-04-25T02:11:07.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.641+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36681ab290 con 0x7f3668075c70 2024-04-25T02:11:07.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.642+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36680689b0 con 0x7f3668075c70 2024-04-25T02:11:07.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.644+0000 7f365f400640 1 -- 172.21.15.102:0/337566671 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3658004050 con 0x7f3668075c70 2024-04-25T02:11:07.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.646+0000 7f365f400640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f364c0a2d00 0x7f364c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:07.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.646+0000 7f365f400640 1 -- 172.21.15.102:0/337566671 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8557+0+0 (secure 0 0 0) 0x7f36580efdd0 con 0x7f3668075c70 2024-04-25T02:11:07.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.646+0000 7f3665e00640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f364c0a2d00 0x7f364c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:07.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.647+0000 7f3665e00640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f364c0a2d00 0x7f364c0a51c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3668101840 tx=0x7f3654009260 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:07.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.648+0000 7f365f400640 1 -- 172.21.15.102:0/337566671 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f36580bca80 con 0x7f3668075c70 2024-04-25T02:11:07.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.902+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f366810d740 con 0x7f3668075c70 2024-04-25T02:11:07.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.915+0000 7f365f400640 1 -- 172.21.15.102:0/337566671 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740908 (secure 0 0 0) 0x7f36580bc380 con 0x7f3668075c70 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "epoch": 26, 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "active_gid": 25214, 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "active_name": "x", 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "active_addrs": { 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v2", 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6800", 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 855543702 2024-04-25T02:11:07.932 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:6801", 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 855543702 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "active_addr": "172.21.15.102:6801/855543702", 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "active_change": "2024-04-25T01:55:15.567969+0000", 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "available": true, 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: "standbys": [ 2024-04-25T02:11:07.933 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "gid": 25201, 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "z", 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_features": 4540701547738038271, 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "available_modules": [ 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "alerts", 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.934 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.935 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.936 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.937 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:07.938 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.939 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_destination": { 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_destination", 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.940 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Email address to send alerts to", 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_from_name": { 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_from_name", 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.941 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "Ceph", 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Email From: name", 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_host": { 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_host", 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.942 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SMTP server", 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_password": { 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_password", 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:07.943 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Password to authenticate with", 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_port": { 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_port", 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "465", 2024-04-25T02:11:07.944 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SMTP port", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_sender": { 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_sender", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.945 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SMTP envelope sender", 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_ssl": { 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_ssl", 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.946 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_user": { 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_user", 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.947 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "User to authenticate as", 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "balancer", 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "active": { 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "active", 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.948 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "begin_time": { 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "begin_time", 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.949 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0000", 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "begin_weekday": { 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "begin_weekday", 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.950 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "0", 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "6", 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_compat_max_iterations": { 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crush_compat_max_iterations", 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.951 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "25", 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "250", 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_compat_metrics": { 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crush_compat_metrics", 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.952 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_compat_step": { 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crush_compat_step", 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.953 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0.5", 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "0.001", 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "0.999", 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "aggressiveness of optimization", 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "end_time": { 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "end_time", 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.954 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "2359", 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "end_weekday": { 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "end_weekday", 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.955 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "0", 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "6", 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:07.956 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.957 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.958 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:07.959 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.960 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "min_score": { 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "min_score", 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.961 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": { 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mode", 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "upmap", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "crush-compat", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "none", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "read", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap-read" 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Balancer mode", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.963 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "pool_ids": { 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pool_ids", 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.964 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap_max_deviation": { 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "upmap_max_deviation", 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.966 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap_max_optimizations": { 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "upmap_max_optimizations", 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.967 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cephadm", 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "agent_down_multiplier": { 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "agent_down_multiplier", 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "3.0", 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.968 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "agent_refresh_rate": { 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "agent_refresh_rate", 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "20", 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.969 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "agent_starting_port": { 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "agent_starting_port", 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "4721", 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T02:11:07.970 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "allow_ptrace": { 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "allow_ptrace", 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T02:11:07.971 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "autotune_interval": { 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "autotune_interval", 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.972 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "autotune_memory_target_ratio": { 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0.7", 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.973 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm_log_destination": { 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cephadm_log_destination", 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "file", 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "file,syslog", 2024-04-25T02:11:07.974 INFO:teuthology.orchestra.run.smithi102.stdout: "syslog" 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "cgroups_split": { 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cgroups_split", 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.975 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "config_checks_enabled": { 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "config_checks_enabled", 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T02:11:07.976 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "config_dashboard": { 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "config_dashboard", 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.977 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_alertmanager": { 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_alertmanager", 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:07.978 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_base": { 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_base", 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Container image name, without the tag", 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.979 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_elasticsearch": { 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_elasticsearch", 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "elasticsearch container image", 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.980 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_grafana": { 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_grafana", 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.981 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_haproxy": { 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_haproxy", 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.982 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "HAproxy container image", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_jaeger_agent": { 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_jaeger_agent", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Jaeger agent container image", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.983 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_jaeger_collector": { 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_jaeger_collector", 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Jaeger collector container image", 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.984 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_jaeger_query": { 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_jaeger_query", 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Jaeger query container image", 2024-04-25T02:11:07.985 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_keepalived": { 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_keepalived", 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Keepalived container image", 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.986 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_loki": { 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_loki", 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Loki container image", 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.987 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_node_exporter": { 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_node_exporter", 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:07.988 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_nvmeof": { 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_nvmeof", 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Nvme-of container image", 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.989 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_prometheus": { 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_prometheus", 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.990 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_promtail": { 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_promtail", 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.991 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Promtail container image", 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_samba": { 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_samba", 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.992 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Samba/SMB container image", 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_snmp_gateway": { 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_snmp_gateway", 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SNMP Gateway container image", 2024-04-25T02:11:07.993 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "container_init": { 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_init", 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T02:11:07.994 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "daemon_cache_timeout": { 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "daemon_cache_timeout", 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.995 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "default_cephadm_command_timeout": { 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "900", 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.996 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "default_registry": { 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "default_registry", 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io", 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.997 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "device_cache_timeout": { 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "device_cache_timeout", 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "1800", 2024-04-25T02:11:07.998 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "seconds to cache device inventory", 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "device_enhanced_scan": { 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "device_enhanced_scan", 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:07.999 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "facts_cache_timeout": { 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "facts_cache_timeout", 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.000 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "seconds to cache host facts data", 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "grafana_dashboards_path": { 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "grafana_dashboards_path", 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.001 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "host_check_interval": { 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "host_check_interval", 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to perform a host check", 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "hw_monitoring": { 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "hw_monitoring", 2024-04-25T02:11:08.003 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "inventory_list_all": { 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "inventory_list_all", 2024-04-25T02:11:08.004 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.005 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.006 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "log_refresh_metadata": { 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_refresh_metadata", 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.007 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.008 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.009 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.010 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T02:11:08.011 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.012 INFO:teuthology.orchestra.run.smithi102.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "*", 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "max_count_per_host": { 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_count_per_host", 2024-04-25T02:11:08.013 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "max number of daemons per service per host", 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "max_osd_draining_count": { 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_osd_draining_count", 2024-04-25T02:11:08.014 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "migration_current": { 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "migration_current", 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.015 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "internal - do not modify", 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": { 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mode", 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.016 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "root", 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm-package", 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "root" 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "oob_default_addr": { 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "oob_default_addr", 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.017 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "169.254.1.1", 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "prometheus_alerts_path": { 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "prometheus_alerts_path", 2024-04-25T02:11:08.018 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_insecure": { 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_insecure", 2024-04-25T02:11:08.019 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_password": { 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_password", 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.020 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_url": { 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_url", 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.021 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_username": { 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_username", 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.022 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "secure_monitoring_stack": { 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "secure_monitoring_stack", 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.023 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "service_discovery_port": { 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "service_discovery_port", 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8765", 2024-04-25T02:11:08.024 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "cephadm service discovery port", 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "ssh_config_file": { 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssh_config_file", 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "use_agent": { 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "use_agent", 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.026 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "use_repo_digest": { 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "use_repo_digest", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.027 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_on_failed_host_check": { 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_on_failed_host_check", 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T02:11:08.028 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_on_stray_daemons": { 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_on_stray_daemons", 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.029 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_on_stray_hosts": { 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_on_stray_hosts", 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.030 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crash", 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.031 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.032 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.033 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.034 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.035 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "retain_interval": { 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "retain_interval", 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "31536000", 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.036 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_recent_interval": { 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_recent_interval", 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "1209600", 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.037 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "dashboard", 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.038 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.039 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "AUDIT_API_ENABLED": { 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.041 INFO:teuthology.orchestra.run.smithi102.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T02:11:08.042 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T02:11:08.043 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.044 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.045 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.046 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.047 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.048 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T02:11:08.050 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "admin", 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T02:11:08.051 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_URL": { 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_URL", 2024-04-25T02:11:08.052 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.053 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "admin", 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.054 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.055 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.056 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.057 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.058 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.059 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.060 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T02:11:08.061 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T02:11:08.062 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.063 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.064 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.065 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.066 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.067 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8", 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "45", 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T02:11:08.070 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T02:11:08.071 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "admin", 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T02:11:08.072 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T02:11:08.073 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.074 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.075 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.076 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "cross_origin_url": { 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cross_origin_url", 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.077 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "crt_file": { 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crt_file", 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.078 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "debug": { 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "debug", 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.079 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Enable/disable debug options", 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "jwt_token_ttl": { 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "jwt_token_ttl", 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "28800", 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.080 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "key_file": { 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "key_file", 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.081 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.082 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.083 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.084 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.085 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.086 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "motd": { 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "motd", 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "The message of the day", 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.087 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "redirect_resolve_ip_addr": { 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "server_addr": { 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_addr", 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "::", 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "server_port": { 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_port", 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8080", 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "ssl": { 2024-04-25T02:11:08.090 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssl", 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "ssl_server_port": { 2024-04-25T02:11:08.091 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssl_server_port", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8443", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_behaviour": { 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_behaviour", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.092 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "redirect", 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "redirect" 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.093 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_error_status_code": { 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_error_status_code", 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "500", 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "400", 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "599", 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: "url_prefix": { 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "url_prefix", 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.095 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "devicehealth", 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "enable_monitoring": { 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enable_monitoring", 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.096 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "monitor device health metrics", 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.097 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.098 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.100 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.101 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "mark_out_threshold": { 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mark_out_threshold", 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.102 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "2419200", 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "pool_name": { 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pool_name", 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.103 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "device_health_metrics", 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "retention_period": { 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "retention_period", 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.104 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "15552000", 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how long to retain device health metrics", 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "scrape_frequency": { 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "scrape_frequency", 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.105 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "86400", 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "self_heal": { 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "self_heal", 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.106 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.107 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_threshold": { 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_threshold", 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.108 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "7257600", 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "diskprediction_local", 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.109 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.110 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.111 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.112 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.113 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "predict_interval": { 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "predict_interval", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.114 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "86400", 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "predictor_model": { 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "predictor_model", 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.115 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "prophetstor", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.116 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "influx", 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": false, 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "influxdb python module not found", 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "batch_size": { 2024-04-25T02:11:08.117 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "batch_size", 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5000", 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.118 INFO:teuthology.orchestra.run.smithi102.stdout: "database": { 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "database", 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "ceph", 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.119 INFO:teuthology.orchestra.run.smithi102.stdout: "hostname": { 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "hostname", 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "InfluxDB server hostname", 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "30", 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "5", 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.121 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.122 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.123 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.124 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.125 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "password": { 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "password", 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "password of InfluxDB server user", 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.127 INFO:teuthology.orchestra.run.smithi102.stdout: "port": { 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "port", 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8086", 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "InfluxDB server port", 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "ssl": { 2024-04-25T02:11:08.128 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssl", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "false", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "threads": { 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "threads", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.129 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "32", 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "username": { 2024-04-25T02:11:08.130 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "username", 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "username of InfluxDB server user", 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.131 INFO:teuthology.orchestra.run.smithi102.stdout: "verify_ssl": { 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "verify_ssl", 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "true", 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.132 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "insights", 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.133 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.134 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.135 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.136 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.137 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "iostat", 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.138 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.139 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.140 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.141 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.142 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "k8sevents", 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_event_retention_days": { 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ceph_event_retention_days", 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.143 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "7", 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "config_check_secs": { 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "config_check_secs", 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.144 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "10", 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.145 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.146 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.148 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.149 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "localpool", 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "failure_domain": { 2024-04-25T02:11:08.150 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "failure_domain", 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "host", 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "failure domain for any created local pool", 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.151 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.152 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.153 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.154 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.155 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.156 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "min_size": { 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "min_size", 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.157 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "default min_size for any created local pool", 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "num_rep": { 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "num_rep", 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "3", 2024-04-25T02:11:08.158 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "default replica count for any created local pool", 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "pg_num": { 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pg_num", 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.159 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "128", 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "prefix": { 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "prefix", 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.160 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "name prefix for any created local pool", 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "subtree": { 2024-04-25T02:11:08.161 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "subtree", 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "rack", 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.162 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mds_autoscaler", 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.163 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.164 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.165 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.166 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.167 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mirroring", 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.168 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.169 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.170 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.171 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.172 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "nfs", 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.173 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.174 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.175 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.176 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.177 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "orchestrator", 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.178 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "fail_fs": { 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "fail_fs", 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.179 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.180 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.181 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.182 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.183 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "orchestrator": { 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "orchestrator", 2024-04-25T02:11:08.184 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm", 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "rook", 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "test_orchestrator" 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Orchestrator backend", 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.185 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "osd_perf_query", 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.186 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.187 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.188 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.189 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.190 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "osd_support", 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.191 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.192 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.193 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.194 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.195 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.196 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pg_autoscaler", 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.197 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.198 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.199 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.200 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.201 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.202 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.203 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "threshold": { 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "threshold", 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "3.0", 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1.0", 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "scaling threshold", 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.204 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "progress", 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "allow_pg_recovery_event": { 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "allow_pg_recovery_event", 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.205 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "enabled": { 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enabled", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.206 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.207 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.208 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.209 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.210 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.210 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.210 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.210 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.210 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.210 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.210 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.210 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.210 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.211 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "max_completed_events": { 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_completed_events", 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.212 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "50", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "number of past completed events to remember", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.213 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how long the module is going to sleep", 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "prometheus", 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.214 INFO:teuthology.orchestra.run.smithi102.stdout: "cache": { 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cache", 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "exclude_perf_counters": { 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "exclude_perf_counters", 2024-04-25T02:11:08.215 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.216 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.217 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.218 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.219 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.220 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_stats_pools": { 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_stats_pools", 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.221 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "300", 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.222 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "scrape_interval": { 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "scrape_interval", 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "15.0", 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.223 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "server_addr": { 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_addr", 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "::", 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.224 INFO:teuthology.orchestra.run.smithi102.stdout: "server_port": { 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_port", 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "9283", 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "stale_cache_strategy": { 2024-04-25T02:11:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "stale_cache_strategy", 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "log", 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_behaviour": { 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_behaviour", 2024-04-25T02:11:08.226 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "default", 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "default", 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "error" 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_error_status_code": { 2024-04-25T02:11:08.227 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_error_status_code", 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "500", 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "400", 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "599", 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.228 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_support", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.229 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.230 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.231 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.232 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "max_concurrent_snap_create": { 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_concurrent_snap_create", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.233 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "mirror_snapshot_schedule": { 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.234 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "trash_purge_schedule": { 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "trash_purge_schedule", 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.235 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "restful", 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.236 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "enable_auth": { 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enable_auth", 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.237 INFO:teuthology.orchestra.run.smithi102.stdout: "key_file": { 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "key_file", 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.238 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.239 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.240 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.242 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.243 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "server_addr": { 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_addr", 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.244 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "server_port": { 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_port", 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.245 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rgw", 2024-04-25T02:11:08.246 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.247 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.248 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.249 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.250 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.251 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rook", 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.252 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.253 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.254 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.256 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.257 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "storage_class": { 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "storage_class", 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "local", 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.258 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "selftest", 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.259 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.260 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.261 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.262 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.263 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "roption1": { 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "roption1", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.264 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "roption2": { 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "roption2", 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "xyz", 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.265 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption1": { 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption1", 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.266 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption2": { 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption2", 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.267 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption3": { 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption3", 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.268 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption4": { 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption4", 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.269 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption5": { 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption5", 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.270 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.271 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption6": { 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption6", 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.272 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption7": { 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption7", 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "42", 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.273 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: "testkey": { 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "testkey", 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.274 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "testlkey": { 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "testlkey", 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.275 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "testnewline": { 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "testnewline", 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.276 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snap_schedule", 2024-04-25T02:11:08.277 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "allow_m_granularity": { 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "allow_m_granularity", 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.278 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "dump_on_update": { 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "dump_on_update", 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "dump database to debug log on update", 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.279 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.280 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.282 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.283 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.284 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "stats", 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.285 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.286 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.287 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.289 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.290 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "status", 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.291 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.293 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.294 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.295 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "telegraf", 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: "address": { 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "address", 2024-04-25T02:11:08.296 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.297 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "15", 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.298 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.299 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.300 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.301 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.302 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "telemetry", 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.303 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_basic": { 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_basic", 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.304 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_crash": { 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_crash", 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.305 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_device": { 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_device", 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T02:11:08.306 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_ident": { 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_ident", 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T02:11:08.307 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_perf": { 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_perf", 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.308 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "contact": { 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "contact", 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.309 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "description": { 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "description", 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.310 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "device_url": { 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "device_url", 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T02:11:08.311 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "enabled": { 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enabled", 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.312 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.313 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "24", 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "8", 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "last_opt_revision": { 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "last_opt_revision", 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.314 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "1", 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "leaderboard": { 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "leaderboard", 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.315 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "leaderboard_description": { 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "leaderboard_description", 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.316 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.317 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.318 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.319 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.320 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "organization": { 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "organization", 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.322 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "proxy": { 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "proxy", 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.323 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "url": { 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "url", 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.324 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "test_orchestrator", 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.325 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.326 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.327 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.328 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.329 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "volumes", 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.330 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.331 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.332 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.333 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.334 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "max_concurrent_clones": { 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_concurrent_clones", 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "4", 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.335 INFO:teuthology.orchestra.run.smithi102.stdout: "periodic_async_work": { 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "periodic_async_work", 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Periodically check for async work", 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "snapshot_clone_delay": { 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snapshot_clone_delay", 2024-04-25T02:11:08.336 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "snapshot_clone_no_wait": { 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.337 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix", 2024-04-25T02:11:08.338 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "discovery_interval": { 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "discovery_interval", 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "100", 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "identifier": { 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "identifier", 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.340 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.341 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.342 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.343 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.344 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "zabbix_host": { 2024-04-25T02:11:08.346 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix_host", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "zabbix_port": { 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix_port", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.347 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10051", 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "zabbix_sender": { 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix_sender", 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.348 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: "gid": 25211, 2024-04-25T02:11:08.349 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "y", 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr_features": 4540701547738038271, 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "available_modules": [ 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "alerts", 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.350 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.351 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.352 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.353 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.354 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.355 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_destination": { 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_destination", 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Email address to send alerts to", 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_from_name": { 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_from_name", 2024-04-25T02:11:08.356 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "Ceph", 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Email From: name", 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_host": { 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_host", 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.357 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SMTP server", 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_password": { 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_password", 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.358 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Password to authenticate with", 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_port": { 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_port", 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "465", 2024-04-25T02:11:08.359 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SMTP port", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_sender": { 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_sender", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.360 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SMTP envelope sender", 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_ssl": { 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_ssl", 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.361 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_user": { 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_user", 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.362 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "User to authenticate as", 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "balancer", 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "active": { 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "active", 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.363 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "begin_time": { 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "begin_time", 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.364 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0000", 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "begin_weekday": { 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "begin_weekday", 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.365 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "0", 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "6", 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_compat_max_iterations": { 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crush_compat_max_iterations", 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "25", 2024-04-25T02:11:08.366 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "250", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_compat_metrics": { 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crush_compat_metrics", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.367 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_compat_step": { 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crush_compat_step", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0.5", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "0.001", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "0.999", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "aggressiveness of optimization", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T02:11:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "end_time": { 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "end_time", 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "2359", 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:11:08.369 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "end_weekday": { 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "end_weekday", 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "0", 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "6", 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.370 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.371 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.372 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.373 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.374 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.375 INFO:teuthology.orchestra.run.smithi102.stdout: "min_score": { 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "min_score", 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": { 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mode", 2024-04-25T02:11:08.376 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "upmap", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "crush-compat", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "none", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "read", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap-read" 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Balancer mode", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.377 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "pool_ids": { 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pool_ids", 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap_max_deviation": { 2024-04-25T02:11:08.379 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "upmap_max_deviation", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap_max_optimizations": { 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "upmap_max_optimizations", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.380 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cephadm", 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "agent_down_multiplier": { 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "agent_down_multiplier", 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "3.0", 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.382 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "agent_refresh_rate": { 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "agent_refresh_rate", 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "20", 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.383 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "agent_starting_port": { 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "agent_starting_port", 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "4721", 2024-04-25T02:11:08.384 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "allow_ptrace": { 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "allow_ptrace", 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.385 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "autotune_interval": { 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "autotune_interval", 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.386 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "autotune_memory_target_ratio": { 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T02:11:08.387 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0.7", 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm_log_destination": { 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cephadm_log_destination", 2024-04-25T02:11:08.388 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "file", 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "file,syslog", 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "syslog" 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.389 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "cgroups_split": { 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cgroups_split", 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.390 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "config_checks_enabled": { 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "config_checks_enabled", 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.391 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "config_dashboard": { 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "config_dashboard", 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.392 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_alertmanager": { 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_alertmanager", 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.393 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_base": { 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_base", 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Container image name, without the tag", 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.394 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_elasticsearch": { 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_elasticsearch", 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "elasticsearch container image", 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.395 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_grafana": { 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_grafana", 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_haproxy": { 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_haproxy", 2024-04-25T02:11:08.396 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "HAproxy container image", 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_jaeger_agent": { 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_jaeger_agent", 2024-04-25T02:11:08.397 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Jaeger agent container image", 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_jaeger_collector": { 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_jaeger_collector", 2024-04-25T02:11:08.398 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Jaeger collector container image", 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_jaeger_query": { 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_jaeger_query", 2024-04-25T02:11:08.399 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Jaeger query container image", 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_keepalived": { 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_keepalived", 2024-04-25T02:11:08.400 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Keepalived container image", 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_loki": { 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_loki", 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.401 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Loki container image", 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_node_exporter": { 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_node_exporter", 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.402 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_nvmeof": { 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_nvmeof", 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.403 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Nvme-of container image", 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_prometheus": { 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_prometheus", 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.404 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_promtail": { 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_promtail", 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.405 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Promtail container image", 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_samba": { 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_samba", 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.406 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Samba/SMB container image", 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_snmp_gateway": { 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_snmp_gateway", 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T02:11:08.407 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SNMP Gateway container image", 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "container_init": { 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_init", 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.408 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "daemon_cache_timeout": { 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "daemon_cache_timeout", 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.409 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "default_cephadm_command_timeout": { 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "900", 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.410 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "default_registry": { 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "default_registry", 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io", 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.411 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "device_cache_timeout": { 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "device_cache_timeout", 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "1800", 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "seconds to cache device inventory", 2024-04-25T02:11:08.412 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "device_enhanced_scan": { 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "device_enhanced_scan", 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.413 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "facts_cache_timeout": { 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "facts_cache_timeout", 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "seconds to cache host facts data", 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "grafana_dashboards_path": { 2024-04-25T02:11:08.414 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "grafana_dashboards_path", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "host_check_interval": { 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "host_check_interval", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.415 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to perform a host check", 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "hw_monitoring": { 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "hw_monitoring", 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.416 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "inventory_list_all": { 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "inventory_list_all", 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.417 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "log_refresh_metadata": { 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_refresh_metadata", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.419 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.420 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.421 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.422 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.423 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.424 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "*", 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "max_count_per_host": { 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_count_per_host", 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "max number of daemons per service per host", 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "max_osd_draining_count": { 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_osd_draining_count", 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.426 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "migration_current": { 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "migration_current", 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.427 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "internal - do not modify", 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": { 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mode", 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "root", 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.428 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm-package", 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "root" 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "oob_default_addr": { 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "oob_default_addr", 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "169.254.1.1", 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.429 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "prometheus_alerts_path": { 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "prometheus_alerts_path", 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.430 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_insecure": { 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_insecure", 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T02:11:08.431 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_password": { 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_password", 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.432 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_url": { 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_url", 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.433 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_username": { 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_username", 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "secure_monitoring_stack": { 2024-04-25T02:11:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "secure_monitoring_stack", 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "service_discovery_port": { 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "service_discovery_port", 2024-04-25T02:11:08.435 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8765", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "cephadm service discovery port", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "ssh_config_file": { 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssh_config_file", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.436 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "use_agent": { 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "use_agent", 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.437 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "use_repo_digest": { 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "use_repo_digest", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.438 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_on_failed_host_check": { 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_on_failed_host_check", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.439 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_on_stray_daemons": { 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_on_stray_daemons", 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.440 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_on_stray_hosts": { 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_on_stray_hosts", 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.441 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crash", 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.442 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.443 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.444 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.445 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.446 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "retain_interval": { 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "retain_interval", 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "31536000", 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.447 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_recent_interval": { 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_recent_interval", 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "1209600", 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.448 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "dashboard", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.449 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.450 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.451 INFO:teuthology.orchestra.run.smithi102.stdout: "AUDIT_API_ENABLED": { 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T02:11:08.452 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.453 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.454 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.455 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.456 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.457 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.458 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.459 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.460 INFO:teuthology.orchestra.run.smithi102.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.461 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "admin", 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T02:11:08.462 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_URL": { 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_URL", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.463 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.464 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "admin", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.465 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.466 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.467 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.468 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T02:11:08.469 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T02:11:08.470 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.471 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.472 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.473 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.474 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.475 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.476 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T02:11:08.477 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T02:11:08.478 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.479 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8", 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.480 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "45", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.481 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "admin", 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.482 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.483 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.484 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.485 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T02:11:08.486 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "cross_origin_url": { 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cross_origin_url", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.487 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "crt_file": { 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crt_file", 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.488 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "debug": { 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "debug", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.489 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Enable/disable debug options", 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "jwt_token_ttl": { 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "jwt_token_ttl", 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "28800", 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.490 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "key_file": { 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "key_file", 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.491 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.492 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.493 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.494 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.495 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "motd": { 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "motd", 2024-04-25T02:11:08.496 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "The message of the day", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "redirect_resolve_ip_addr": { 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.497 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "server_addr": { 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_addr", 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.498 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "::", 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "server_port": { 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_port", 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.499 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8080", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "ssl": { 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssl", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.500 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "ssl_server_port": { 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssl_server_port", 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8443", 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.501 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_behaviour": { 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_behaviour", 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "redirect", 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.502 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "redirect" 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_error_status_code": { 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_error_status_code", 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "500", 2024-04-25T02:11:08.503 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "400", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "599", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "url_prefix": { 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "url_prefix", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.504 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "devicehealth", 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "enable_monitoring": { 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enable_monitoring", 2024-04-25T02:11:08.505 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "monitor device health metrics", 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.506 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.507 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.508 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.509 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "mark_out_threshold": { 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mark_out_threshold", 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "2419200", 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.511 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "pool_name": { 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pool_name", 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "device_health_metrics", 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.512 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "retention_period": { 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "retention_period", 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "15552000", 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how long to retain device health metrics", 2024-04-25T02:11:08.513 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "scrape_frequency": { 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "scrape_frequency", 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "86400", 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.514 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "self_heal": { 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "self_heal", 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.515 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_threshold": { 2024-04-25T02:11:08.516 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_threshold", 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "7257600", 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.517 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "diskprediction_local", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.519 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.520 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.521 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.522 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "predict_interval": { 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "predict_interval", 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "86400", 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "predictor_model": { 2024-04-25T02:11:08.523 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "predictor_model", 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "prophetstor", 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.524 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.525 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "influx", 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": false, 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "influxdb python module not found", 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "batch_size": { 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "batch_size", 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5000", 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.526 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "database": { 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "database", 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "ceph", 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.527 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "hostname": { 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "hostname", 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.528 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "InfluxDB server hostname", 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.529 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.530 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.530 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "30", 2024-04-25T02:11:08.530 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "5", 2024-04-25T02:11:08.530 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.530 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.530 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T02:11:08.530 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.530 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.530 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.531 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.531 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.531 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.531 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.531 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.531 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.531 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.531 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.531 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.532 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.532 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.532 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.532 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.532 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.532 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.532 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.532 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.533 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.533 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.533 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.533 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.533 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.533 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.533 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.534 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.536 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "password": { 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "password", 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "password of InfluxDB server user", 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.537 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "port": { 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "port", 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8086", 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "InfluxDB server port", 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.538 INFO:teuthology.orchestra.run.smithi102.stdout: "ssl": { 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssl", 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "false", 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "threads": { 2024-04-25T02:11:08.539 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "threads", 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "32", 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "username": { 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "username", 2024-04-25T02:11:08.540 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "username of InfluxDB server user", 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "verify_ssl": { 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "verify_ssl", 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.541 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "true", 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.542 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "insights", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.543 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.544 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.545 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.546 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.547 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "iostat", 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.549 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.550 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.551 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.552 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "k8sevents", 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_event_retention_days": { 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ceph_event_retention_days", 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "7", 2024-04-25T02:11:08.553 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "config_check_secs": { 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "config_check_secs", 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "10", 2024-04-25T02:11:08.554 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.555 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.556 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.557 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.558 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.559 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "localpool", 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "failure_domain": { 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "failure_domain", 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "host", 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.560 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "failure domain for any created local pool", 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.562 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.563 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.564 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.565 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "min_size": { 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "min_size", 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "default min_size for any created local pool", 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.566 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "num_rep": { 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "num_rep", 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "3", 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "default replica count for any created local pool", 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.567 INFO:teuthology.orchestra.run.smithi102.stdout: "pg_num": { 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pg_num", 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "128", 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.568 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "prefix": { 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "prefix", 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "name prefix for any created local pool", 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.569 INFO:teuthology.orchestra.run.smithi102.stdout: "subtree": { 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "subtree", 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "rack", 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.570 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mds_autoscaler", 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.571 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.572 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.573 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.574 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.575 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mirroring", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.576 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.577 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.578 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.579 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "nfs", 2024-04-25T02:11:08.580 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.581 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.582 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.583 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.584 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "orchestrator", 2024-04-25T02:11:08.585 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "fail_fs": { 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "fail_fs", 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.586 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.587 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.588 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.589 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.590 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "orchestrator": { 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "orchestrator", 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.591 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm", 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "rook", 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "test_orchestrator" 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Orchestrator backend", 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.592 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "osd_perf_query", 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.593 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.594 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.595 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.596 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.597 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "osd_support", 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.598 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.599 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.600 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.601 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.602 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.603 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pg_autoscaler", 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.604 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.605 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.606 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.607 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.609 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.610 INFO:teuthology.orchestra.run.smithi102.stdout: "threshold": { 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "threshold", 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "3.0", 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1.0", 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "scaling threshold", 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.611 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "progress", 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "allow_pg_recovery_event": { 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "allow_pg_recovery_event", 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.612 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "enabled": { 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enabled", 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.613 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.614 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.615 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.616 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.617 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "max_completed_events": { 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_completed_events", 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.618 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "50", 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "number of past completed events to remember", 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.619 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how long the module is going to sleep", 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "prometheus", 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.620 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "cache": { 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cache", 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.621 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "exclude_perf_counters": { 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "exclude_perf_counters", 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T02:11:08.622 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.623 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.624 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.625 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.626 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.627 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_stats_pools": { 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_stats_pools", 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.628 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "300", 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.629 INFO:teuthology.orchestra.run.smithi102.stdout: "scrape_interval": { 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "scrape_interval", 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "15.0", 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "server_addr": { 2024-04-25T02:11:08.630 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_addr", 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "::", 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "server_port": { 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_port", 2024-04-25T02:11:08.631 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "9283", 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "stale_cache_strategy": { 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "stale_cache_strategy", 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.632 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "log", 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_behaviour": { 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_behaviour", 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.633 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "default", 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "default", 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "error" 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_error_status_code": { 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_error_status_code", 2024-04-25T02:11:08.634 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "500", 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "400", 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "599", 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_support", 2024-04-25T02:11:08.635 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.636 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.637 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.638 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.639 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "max_concurrent_snap_create": { 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_concurrent_snap_create", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.640 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "mirror_snapshot_schedule": { 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.641 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "trash_purge_schedule": { 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "trash_purge_schedule", 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.642 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "restful", 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "enable_auth": { 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enable_auth", 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.643 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "key_file": { 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "key_file", 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.644 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.645 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.646 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.647 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.648 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "server_addr": { 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_addr", 2024-04-25T02:11:08.649 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "server_port": { 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_port", 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.650 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rgw", 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.651 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.652 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.653 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.654 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.655 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rook", 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.656 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.657 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.658 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.659 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.660 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "storage_class": { 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "storage_class", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.661 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "local", 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.662 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "selftest", 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.663 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.664 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.665 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.666 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.667 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "roption1": { 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "roption1", 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.668 INFO:teuthology.orchestra.run.smithi102.stdout: "roption2": { 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "roption2", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "xyz", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption1": { 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption1", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.669 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption2": { 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption2", 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.670 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption3": { 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption3", 2024-04-25T02:11:08.671 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.672 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:08 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3446762559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:11:08.672 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:08 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/337566671' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T02:11:08.672 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:08 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3446762559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:11:08.673 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:08 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/337566671' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T02:11:08.675 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.675 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.675 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.675 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.675 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.675 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption4": { 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption4", 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.676 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption5": { 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption5", 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.677 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption6": { 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption6", 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption7": { 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption7", 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "42", 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.679 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "testkey": { 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "testkey", 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "testlkey": { 2024-04-25T02:11:08.680 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "testlkey", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "testnewline": { 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "testnewline", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.681 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.682 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snap_schedule", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "allow_m_granularity": { 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "allow_m_granularity", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.683 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "dump_on_update": { 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "dump_on_update", 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "dump database to debug log on update", 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.684 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.685 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.686 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.687 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.688 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "stats", 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.689 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.690 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.691 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.692 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.693 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "status", 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.694 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.695 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.696 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.697 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.698 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "telegraf", 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.699 INFO:teuthology.orchestra.run.smithi102.stdout: "address": { 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "address", 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.700 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "15", 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.701 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.702 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.703 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.704 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.705 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.706 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "telemetry", 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_basic": { 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_basic", 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.707 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_crash": { 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_crash", 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.708 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_device": { 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_device", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.709 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_ident": { 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_ident", 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.710 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_perf": { 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_perf", 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.711 INFO:teuthology.orchestra.run.smithi102.stdout: "contact": { 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "contact", 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "description": { 2024-04-25T02:11:08.712 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "description", 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "device_url": { 2024-04-25T02:11:08.713 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "device_url", 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "enabled": { 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enabled", 2024-04-25T02:11:08.714 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.715 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "24", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "8", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "last_opt_revision": { 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "last_opt_revision", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "1", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.716 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "leaderboard": { 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "leaderboard", 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.717 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "leaderboard_description": { 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "leaderboard_description", 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.718 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.719 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.720 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.721 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.722 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.723 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "organization": { 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "organization", 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.724 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "proxy": { 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "proxy", 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.725 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "url": { 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "url", 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T02:11:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "test_orchestrator", 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.727 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.728 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.729 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.730 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.731 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.732 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.733 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "volumes", 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.735 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.736 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.737 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.738 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "max_concurrent_clones": { 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_concurrent_clones", 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "4", 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.739 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "periodic_async_work": { 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "periodic_async_work", 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.740 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Periodically check for async work", 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "snapshot_clone_delay": { 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snapshot_clone_delay", 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T02:11:08.741 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "snapshot_clone_no_wait": { 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix", 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "discovery_interval": { 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "discovery_interval", 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.743 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "100", 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "identifier": { 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "identifier", 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.744 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.745 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.746 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.747 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.748 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.749 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.750 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "zabbix_host": { 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix_host", 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.751 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "zabbix_port": { 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix_port", 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10051", 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.752 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "zabbix_sender": { 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix_sender", 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.753 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.754 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: "modules": [ 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm", 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: "dashboard", 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: "iostat", 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: "nfs", 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: "restful", 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: "snap_schedule" 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.755 INFO:teuthology.orchestra.run.smithi102.stdout: "available_modules": [ 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "alerts", 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.756 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T02:11:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.758 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.759 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.760 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.761 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.762 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_destination": { 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_destination", 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.763 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Email address to send alerts to", 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_from_name": { 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_from_name", 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "Ceph", 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.764 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Email From: name", 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_host": { 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_host", 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.765 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SMTP server", 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_password": { 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_password", 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.766 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Password to authenticate with", 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_port": { 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_port", 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "465", 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.767 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SMTP port", 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_sender": { 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_sender", 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.768 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SMTP envelope sender", 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_ssl": { 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_ssl", 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.769 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "smtp_user": { 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "smtp_user", 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.770 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "User to authenticate as", 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "balancer", 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.771 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "active": { 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "active", 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.772 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "begin_time": { 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "begin_time", 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0000", 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.773 INFO:teuthology.orchestra.run.smithi102.stdout: "begin_weekday": { 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "begin_weekday", 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "0", 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "6", 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_compat_max_iterations": { 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crush_compat_max_iterations", 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "25", 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "250", 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.775 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_compat_metrics": { 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crush_compat_metrics", 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.776 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "crush_compat_step": { 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crush_compat_step", 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0.5", 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "0.001", 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "0.999", 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "aggressiveness of optimization", 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.777 INFO:teuthology.orchestra.run.smithi102.stdout: "end_time": { 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "end_time", 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "2359", 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.778 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "end_weekday": { 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "end_weekday", 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "0", 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "6", 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.779 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.780 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.781 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.782 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "min_score": { 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "min_score", 2024-04-25T02:11:08.784 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": { 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mode", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.785 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "upmap", 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "crush-compat", 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "none", 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "read", 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap", 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap-read" 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Balancer mode", 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.786 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "pool_ids": { 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pool_ids", 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.787 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap_max_deviation": { 2024-04-25T02:11:08.788 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "upmap_max_deviation", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "upmap_max_optimizations": { 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "upmap_max_optimizations", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:08.789 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.790 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cephadm", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "agent_down_multiplier": { 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "agent_down_multiplier", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "3.0", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.791 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "agent_refresh_rate": { 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "agent_refresh_rate", 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "20", 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.792 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "agent_starting_port": { 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "agent_starting_port", 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "4721", 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.793 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "allow_ptrace": { 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "allow_ptrace", 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.794 INFO:teuthology.orchestra.run.smithi102.stdout: "autotune_interval": { 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "autotune_interval", 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "autotune_memory_target_ratio": { 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T02:11:08.795 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0.7", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm_log_destination": { 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cephadm_log_destination", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.796 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "file", 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "file,syslog", 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "syslog" 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "cgroups_split": { 2024-04-25T02:11:08.797 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cgroups_split", 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "config_checks_enabled": { 2024-04-25T02:11:08.798 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "config_checks_enabled", 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "config_dashboard": { 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "config_dashboard", 2024-04-25T02:11:08.799 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_alertmanager": { 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_alertmanager", 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.800 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_base": { 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_base", 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.801 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Container image name, without the tag", 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_elasticsearch": { 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_elasticsearch", 2024-04-25T02:11:08.802 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "elasticsearch container image", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_grafana": { 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_grafana", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.803 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_haproxy": { 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_haproxy", 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T02:11:08.804 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "HAproxy container image", 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_jaeger_agent": { 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_jaeger_agent", 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.805 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Jaeger agent container image", 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_jaeger_collector": { 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_jaeger_collector", 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.806 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Jaeger collector container image", 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_jaeger_query": { 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_jaeger_query", 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.807 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Jaeger query container image", 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_keepalived": { 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_keepalived", 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.808 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Keepalived container image", 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_loki": { 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_loki", 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.809 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Loki container image", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_node_exporter": { 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_node_exporter", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.810 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_nvmeof": { 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_nvmeof", 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Nvme-of container image", 2024-04-25T02:11:08.811 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_prometheus": { 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_prometheus", 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Prometheus container image", 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.812 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_promtail": { 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_promtail", 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Promtail container image", 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.813 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_samba": { 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_samba", 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Samba/SMB container image", 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.814 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "container_image_snmp_gateway": { 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_image_snmp_gateway", 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "SNMP Gateway container image", 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.815 INFO:teuthology.orchestra.run.smithi102.stdout: "container_init": { 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "container_init", 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "daemon_cache_timeout": { 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "daemon_cache_timeout", 2024-04-25T02:11:08.816 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "default_cephadm_command_timeout": { 2024-04-25T02:11:08.817 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "900", 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "default_registry": { 2024-04-25T02:11:08.818 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "default_registry", 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "docker.io", 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "device_cache_timeout": { 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "device_cache_timeout", 2024-04-25T02:11:08.819 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "1800", 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "seconds to cache device inventory", 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "device_enhanced_scan": { 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "device_enhanced_scan", 2024-04-25T02:11:08.820 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "facts_cache_timeout": { 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "facts_cache_timeout", 2024-04-25T02:11:08.821 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "seconds to cache host facts data", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "grafana_dashboards_path": { 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "grafana_dashboards_path", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.822 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "host_check_interval": { 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "host_check_interval", 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.823 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to perform a host check", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "hw_monitoring": { 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "hw_monitoring", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.824 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "inventory_list_all": { 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "inventory_list_all", 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T02:11:08.825 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.826 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "log_refresh_metadata": { 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_refresh_metadata", 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.827 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.828 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.829 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.830 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.831 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.832 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "*", 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "max_count_per_host": { 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_count_per_host", 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.833 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "max number of daemons per service per host", 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "max_osd_draining_count": { 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_osd_draining_count", 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.834 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "migration_current": { 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "migration_current", 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.835 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "internal - do not modify", 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "mode": { 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mode", 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "root", 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.836 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm-package", 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "root" 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "oob_default_addr": { 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "oob_default_addr", 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "169.254.1.1", 2024-04-25T02:11:08.837 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "prometheus_alerts_path": { 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "prometheus_alerts_path", 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.838 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_insecure": { 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_insecure", 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.839 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_password": { 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_password", 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.840 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_url": { 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_url", 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.841 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "registry_username": { 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "registry_username", 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T02:11:08.842 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "secure_monitoring_stack": { 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "secure_monitoring_stack", 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.843 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "service_discovery_port": { 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "service_discovery_port", 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8765", 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "cephadm service discovery port", 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.844 INFO:teuthology.orchestra.run.smithi102.stdout: "ssh_config_file": { 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssh_config_file", 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "use_agent": { 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "use_agent", 2024-04-25T02:11:08.845 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "use_repo_digest": { 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "use_repo_digest", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.846 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_on_failed_host_check": { 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_on_failed_host_check", 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.847 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_on_stray_daemons": { 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_on_stray_daemons", 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.848 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_on_stray_hosts": { 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_on_stray_hosts", 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.849 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crash", 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.850 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.851 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.852 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.853 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.854 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "retain_interval": { 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "retain_interval", 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "31536000", 2024-04-25T02:11:08.855 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_recent_interval": { 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_recent_interval", 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "1209600", 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.856 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "dashboard", 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T02:11:08.857 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.858 INFO:teuthology.orchestra.run.smithi102.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T02:11:08.859 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "AUDIT_API_ENABLED": { 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.860 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.861 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T02:11:08.862 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.863 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.864 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.865 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.866 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.867 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.868 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.869 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.870 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "admin", 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.871 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.872 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_URL": { 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_URL", 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.873 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "admin", 2024-04-25T02:11:08.874 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.875 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.876 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.877 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.878 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.879 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T02:11:08.880 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.881 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.882 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.883 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.884 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.885 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.886 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.887 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.888 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.889 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.890 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.891 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8", 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.892 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "45", 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.893 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.894 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "admin", 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T02:11:08.895 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.896 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.899 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "cross_origin_url": { 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cross_origin_url", 2024-04-25T02:11:08.900 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.901 INFO:teuthology.orchestra.run.smithi102.stdout: "crt_file": { 2024-04-25T02:11:08.902 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "crt_file", 2024-04-25T02:11:08.902 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.902 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.902 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.902 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.902 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.902 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.902 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.903 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.903 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.903 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.903 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.903 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.903 INFO:teuthology.orchestra.run.smithi102.stdout: "debug": { 2024-04-25T02:11:08.903 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "debug", 2024-04-25T02:11:08.903 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.903 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Enable/disable debug options", 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "jwt_token_ttl": { 2024-04-25T02:11:08.904 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "jwt_token_ttl", 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "28800", 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.905 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "key_file": { 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "key_file", 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.906 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.907 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.908 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.909 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.910 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "motd": { 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "motd", 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.911 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "The message of the day", 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "redirect_resolve_ip_addr": { 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.912 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "server_addr": { 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_addr", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "::", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.913 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "server_port": { 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_port", 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8080", 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.914 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "ssl": { 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssl", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.915 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "ssl_server_port": { 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssl_server_port", 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8443", 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.916 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_behaviour": { 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_behaviour", 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "redirect", 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "redirect" 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.917 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_error_status_code": { 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_error_status_code", 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "500", 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "400", 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "599", 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.918 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "url_prefix": { 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "url_prefix", 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.919 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "devicehealth", 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "enable_monitoring": { 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enable_monitoring", 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.920 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "monitor device health metrics", 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.921 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.922 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.923 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.924 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "mark_out_threshold": { 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mark_out_threshold", 2024-04-25T02:11:08.925 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "2419200", 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "pool_name": { 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pool_name", 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.926 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "device_health_metrics", 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "retention_period": { 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "retention_period", 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.927 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "15552000", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how long to retain device health metrics", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "scrape_frequency": { 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "scrape_frequency", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "86400", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.928 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "self_heal": { 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "self_heal", 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.929 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T02:11:08.930 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "warn_threshold": { 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "warn_threshold", 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "7257600", 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.931 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "diskprediction_local", 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.932 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.933 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.934 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.935 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.936 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "predict_interval": { 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "predict_interval", 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.937 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "86400", 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "predictor_model": { 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "predictor_model", 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.938 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "prophetstor", 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.939 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "600", 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.940 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "influx", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": false, 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "influxdb python module not found", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "batch_size": { 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "batch_size", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5000", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.941 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "database": { 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "database", 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "ceph", 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.942 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "hostname": { 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "hostname", 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "InfluxDB server hostname", 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.943 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "30", 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "5", 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.944 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.945 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.946 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.947 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.948 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "password": { 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "password", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "password of InfluxDB server user", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.949 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "port": { 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "port", 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "8086", 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "InfluxDB server port", 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.950 INFO:teuthology.orchestra.run.smithi102.stdout: "ssl": { 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ssl", 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "false", 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.951 INFO:teuthology.orchestra.run.smithi102.stdout: "threads": { 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "threads", 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "32", 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.952 INFO:teuthology.orchestra.run.smithi102.stdout: "username": { 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "username", 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "username of InfluxDB server user", 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.953 INFO:teuthology.orchestra.run.smithi102.stdout: "verify_ssl": { 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "verify_ssl", 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "true", 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.954 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "insights", 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.955 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.956 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.957 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.958 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.959 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "iostat", 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.960 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.961 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.962 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.963 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.964 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "k8sevents", 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_event_retention_days": { 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "ceph_event_retention_days", 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "7", 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.965 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "config_check_secs": { 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "config_check_secs", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "10", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.966 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.967 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.969 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.970 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "localpool", 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.971 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "failure_domain": { 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "failure_domain", 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "host", 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "failure domain for any created local pool", 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.972 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.973 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.974 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.975 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.976 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "min_size": { 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "min_size", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.977 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "default min_size for any created local pool", 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "num_rep": { 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "num_rep", 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "3", 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "default replica count for any created local pool", 2024-04-25T02:11:08.978 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "pg_num": { 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pg_num", 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "128", 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.979 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "prefix": { 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "prefix", 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "name prefix for any created local pool", 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.980 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "subtree": { 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "subtree", 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "rack", 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.981 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mds_autoscaler", 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.982 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.983 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.984 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.985 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.986 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mirroring", 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.987 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.988 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.989 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.990 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.991 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "nfs", 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.992 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.993 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:08.994 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.995 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.996 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "orchestrator", 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "fail_fs": { 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "fail_fs", 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:08.997 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:08.998 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:08.999 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.000 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.001 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.002 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "orchestrator": { 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "orchestrator", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.003 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "cephadm", 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "rook", 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "test_orchestrator" 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Orchestrator backend", 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "osd_perf_query", 2024-04-25T02:11:09.004 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.005 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.006 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.007 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.008 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.009 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "osd_support", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.010 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.011 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.012 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.013 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.014 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "pg_autoscaler", 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.015 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.016 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.017 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.018 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.019 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.020 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "threshold": { 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "threshold", 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "3.0", 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1.0", 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "scaling threshold", 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T02:11:09.021 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "progress", 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: "allow_pg_recovery_event": { 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "allow_pg_recovery_event", 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.022 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "enabled": { 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enabled", 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.023 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.024 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.025 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.026 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.027 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.028 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.029 INFO:teuthology.orchestra.run.smithi102.stdout: "max_completed_events": { 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_completed_events", 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "50", 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "number of past completed events to remember", 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.030 INFO:teuthology.orchestra.run.smithi102.stdout: "sleep_interval": { 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "sleep_interval", 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "5", 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "how long the module is going to sleep", 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.031 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "prometheus", 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "cache": { 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "cache", 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.032 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "exclude_perf_counters": { 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "exclude_perf_counters", 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.033 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.034 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.035 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.036 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.037 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.038 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_stats_pools": { 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_stats_pools", 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.039 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "300", 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.040 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "scrape_interval": { 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "scrape_interval", 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "15.0", 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.041 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "server_addr": { 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_addr", 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "::", 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.042 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "server_port": { 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_port", 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "9283", 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.043 INFO:teuthology.orchestra.run.smithi102.stdout: "stale_cache_strategy": { 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "stale_cache_strategy", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "log", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_behaviour": { 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_behaviour", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.044 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "default", 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "default", 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "error" 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "standby_error_status_code": { 2024-04-25T02:11:09.045 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "standby_error_status_code", 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "500", 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "400", 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "599", 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.046 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_support", 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.047 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.048 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.049 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.050 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.051 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "max_concurrent_snap_create": { 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_concurrent_snap_create", 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10", 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.052 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "mirror_snapshot_schedule": { 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.053 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "trash_purge_schedule": { 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "trash_purge_schedule", 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.054 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "restful", 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "enable_auth": { 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enable_auth", 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:09.055 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "key_file": { 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "key_file", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.056 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.057 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.058 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.059 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.060 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "server_addr": { 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_addr", 2024-04-25T02:11:09.061 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "server_port": { 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "server_port", 2024-04-25T02:11:09.062 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.063 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rgw", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.064 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.065 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.066 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.067 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.068 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rook", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.069 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.070 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.073 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.073 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.073 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.073 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.073 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.073 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.074 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.075 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "storage_class": { 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "storage_class", 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.076 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "local", 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "selftest", 2024-04-25T02:11:09.077 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.078 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.079 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.080 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.081 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "roption1": { 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "roption1", 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.082 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "roption2": { 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "roption2", 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "xyz", 2024-04-25T02:11:09.083 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption1": { 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption1", 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.084 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption2": { 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption2", 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.085 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption3": { 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption3", 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "float", 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.086 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption4": { 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption4", 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.087 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption5": { 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption5", 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.088 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption6": { 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption6", 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: "rwoption7": { 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rwoption7", 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "1", 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "42", 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "testkey": { 2024-04-25T02:11:09.090 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "testkey", 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "testlkey": { 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "testlkey", 2024-04-25T02:11:09.091 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "testnewline": { 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "testnewline", 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.092 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snap_schedule", 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.093 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "allow_m_granularity": { 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "allow_m_granularity", 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.094 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "dump_on_update": { 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "dump_on_update", 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "dump database to debug log on update", 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.095 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.096 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.097 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.098 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.099 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.100 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "stats", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.101 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.102 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.103 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.104 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.105 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "status", 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.106 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.107 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.108 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.109 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.110 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "telegraf", 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "address": { 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "address", 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.111 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "15", 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.112 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.113 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.114 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.115 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.116 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "telemetry", 2024-04-25T02:11:09.117 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_basic": { 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_basic", 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.118 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_crash": { 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_crash", 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.119 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_device": { 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_device", 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.120 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_ident": { 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_ident", 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "channel_perf": { 2024-04-25T02:11:09.121 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "channel_perf", 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "contact": { 2024-04-25T02:11:09.122 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "contact", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "description": { 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "description", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.123 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "device_url": { 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "device_url", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T02:11:09.124 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "enabled": { 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "enabled", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.125 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "24", 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "8", 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.126 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "last_opt_revision": { 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "last_opt_revision", 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "1", 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.127 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "leaderboard": { 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "leaderboard", 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.128 INFO:teuthology.orchestra.run.smithi102.stdout: "leaderboard_description": { 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "leaderboard_description", 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.129 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.130 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.131 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.132 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.133 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "organization": { 2024-04-25T02:11:09.134 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "organization", 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "proxy": { 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "proxy", 2024-04-25T02:11:09.135 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "url": { 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "url", 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.136 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.137 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "test_orchestrator", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.138 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.139 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.140 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.141 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.142 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "volumes", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.143 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.144 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.145 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.146 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "max_concurrent_clones": { 2024-04-25T02:11:09.147 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "max_concurrent_clones", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "4", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "periodic_async_work": { 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "periodic_async_work", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.148 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Periodically check for async work", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "snapshot_clone_delay": { 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snapshot_clone_delay", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "0", 2024-04-25T02:11:09.149 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "snapshot_clone_no_wait": { 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "True", 2024-04-25T02:11:09.150 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix", 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "can_run": true, 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "error_string": "", 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "module_options": { 2024-04-25T02:11:09.151 INFO:teuthology.orchestra.run.smithi102.stdout: "discovery_interval": { 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "discovery_interval", 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "uint", 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "100", 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "identifier": { 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "identifier", 2024-04-25T02:11:09.152 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "interval": { 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "interval", 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "secs", 2024-04-25T02:11:09.153 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "60", 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "log_level": { 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_level", 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.154 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.155 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster": { 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster", 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.156 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_cluster_level": { 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_cluster_level", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "info", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [ 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "critical", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "debug", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "error", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "info", 2024-04-25T02:11:09.157 INFO:teuthology.orchestra.run.smithi102.stdout: "warning" 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "log_to_file": { 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "log_to_file", 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "bool", 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 1, 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "False", 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.158 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "zabbix_host": { 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix_host", 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "", 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "zabbix_port": { 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix_port", 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "int", 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "10051", 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.160 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "zabbix_sender": { 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "zabbix_sender", 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "str", 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "level": "advanced", 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "flags": 0, 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "min": "", 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "max": "", 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "enum_allowed": [], 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "desc": "", 2024-04-25T02:11:09.161 INFO:teuthology.orchestra.run.smithi102.stdout: "long_desc": "", 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "tags": [], 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "see_also": [] 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "services": { 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "dashboard": "https://172.21.15.102:8443/" 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "always_on_modules": { 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "octopus": [ 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "balancer", 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "devicehealth", 2024-04-25T02:11:09.162 INFO:teuthology.orchestra.run.smithi102.stdout: "orchestrator", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "pg_autoscaler", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "progress", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_support", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "status", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "telemetry", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "volumes" 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "pacific": [ 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "balancer", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "devicehealth", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "orchestrator", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "pg_autoscaler", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "progress", 2024-04-25T02:11:09.163 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_support", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "status", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "telemetry", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "volumes" 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "quincy": [ 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "balancer", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "devicehealth", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "orchestrator", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "pg_autoscaler", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "progress", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_support", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "status", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "telemetry", 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: "volumes" 2024-04-25T02:11:09.164 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "reef": [ 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "balancer", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "devicehealth", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "orchestrator", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "pg_autoscaler", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "progress", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_support", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "status", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "telemetry", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "volumes" 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: ], 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "squid": [ 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "balancer", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "crash", 2024-04-25T02:11:09.165 INFO:teuthology.orchestra.run.smithi102.stdout: "devicehealth", 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "orchestrator", 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "pg_autoscaler", 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "progress", 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "rbd_support", 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "status", 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "telemetry", 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "volumes" 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "last_failure_osd_epoch": 90, 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "active_clients": [ 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "devicehealth", 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2024-04-25T02:11:09.166 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v2", 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 747799522 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "libcephsqlite", 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v2", 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 1435486184 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:09.167 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "rbd_support", 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v2", 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 173723776 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snap_schedule", 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v2", 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-04-25T02:11:09.168 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 1870414423 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "snap_schedule", 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v2", 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 1849939896 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: "name": "volumes", 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: "addrvec": [ 2024-04-25T02:11:09.169 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-04-25T02:11:09.170 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v2", 2024-04-25T02:11:09.170 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-04-25T02:11:09.170 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 4223698199 2024-04-25T02:11:09.170 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.170 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:09.170 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:09.170 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-04-25T02:11:09.170 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:11:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f364c0a2d00 msgr2=0x7f364c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f364c0a2d00 0x7f364c0a51c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3668101840 tx=0x7f3654009260 comp rx=0 tx=0).stop 2024-04-25T02:11:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 msgr2=0x7f366819e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 0x7f366819e720 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3658030510 tx=0x7f3658033ae0 comp rx=0 tx=0).stop 2024-04-25T02:11:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 shutdown_connections 2024-04-25T02:11:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f364c0a2d00 0x7f364c0a51c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36681125b0 0x7f36681a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3668076590 0x7f366819ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:09.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 --2- 172.21.15.102:0/337566671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3668075c70 0x7f366819e720 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:09.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 >> 172.21.15.102:0/337566671 conn(0x7f36680ff400 msgr2=0x7f3668113d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:09.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 shutdown_connections 2024-04-25T02:11:09.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:07.930+0000 7f366d400640 1 -- 172.21.15.102:0/337566671 wait complete. 2024-04-25T02:11:09.177 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T02:11:09.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T02:11:09.177 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:08 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3446762559' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:11:09.178 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:08 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/337566671' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T02:11:09.396 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:09 smithi110 ceph-mon[32848]: pgmap v496: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 21 MiB/s wr, 1.85k op/s 2024-04-25T02:11:09.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:09 smithi102 ceph-mon[27796]: pgmap v496: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 21 MiB/s wr, 1.85k op/s 2024-04-25T02:11:09.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:09 smithi193 ceph-mon[32981]: pgmap v496: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 21 MiB/s wr, 1.85k op/s 2024-04-25T02:11:09.624 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T02:11:11.430 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:11 smithi102 ceph-mon[27796]: pgmap v497: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 20 MiB/s wr, 1.76k op/s 2024-04-25T02:11:11.494 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:11 smithi110 ceph-mon[32848]: pgmap v497: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 20 MiB/s wr, 1.76k op/s 2024-04-25T02:11:11.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:11 smithi193 ceph-mon[32981]: pgmap v497: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 20 MiB/s wr, 1.76k op/s 2024-04-25T02:11:12.041 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 0, 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 2, 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 0, 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 0, 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 4, 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 0, 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.004101601, 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.000954618, 2024-04-25T02:11:12.042 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:12.043 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 7, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 231, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 3504, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 274128, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 6, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 3, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.041930680, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.032798913, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.004262020, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 229, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.049531388, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000216294 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.044 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 274128 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 309, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 298, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 1605856, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 184080, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 2, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 1, 2024-04-25T02:11:12.045 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.102969956, 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.046403632, 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.054881545, 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.011446850, 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 297, 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.049124244, 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000165401 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:12.046 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 1605856, 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 184080 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "cct": { 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "total_workers": 0, 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "unhealthy_workers": 0 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "mempool": { 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:12.047 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:12.048 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_items": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:12.049 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_bytes": 149745, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_items": 31, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_items": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bytes": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_items": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_bytes": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_items": 0, 2024-04-25T02:11:12.050 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_items": 0, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_bytes": 928, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_items": 2, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_items": 0, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_items": 0 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 104857600, 2024-04-25T02:11:12.051 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 312, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 1562286, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 312, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 312, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 1562286, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:12.052 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:12.053 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:11:13.034 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-25T02:11:13.034 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-25T02:11:13.034 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-25T02:11:13.034 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-25T02:11:13.034 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-25T02:11:13.034 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-25T02:11:13.034 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-25T02:11:13.034 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T02:11:13.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T02:11:13.464 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T02:11:13.725 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:13 smithi102 ceph-mon[27796]: pgmap v498: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 18 MiB/s wr, 1.56k op/s 2024-04-25T02:11:13.725 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:13 smithi102 ceph-mon[27796]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T02:11:13.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:13 smithi110 ceph-mon[32848]: pgmap v498: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 18 MiB/s wr, 1.56k op/s 2024-04-25T02:11:13.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:13 smithi110 ceph-mon[32848]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T02:11:13.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:13 smithi193 ceph-mon[32981]: pgmap v498: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 18 MiB/s wr, 1.56k op/s 2024-04-25T02:11:13.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:13 smithi193 ceph-mon[32981]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T02:11:15.495 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:15 smithi102 ceph-mon[27796]: pgmap v500: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 6.6 MiB/s wr, 556 op/s 2024-04-25T02:11:15.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:15 smithi110 ceph-mon[32848]: pgmap v500: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 6.6 MiB/s wr, 556 op/s 2024-04-25T02:11:15.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:15 smithi193 ceph-mon[32981]: pgmap v500: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 6.6 MiB/s wr, 556 op/s 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 0, 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 2, 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 0, 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 0, 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 4, 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 0, 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.004036709, 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.000874975, 2024-04-25T02:11:15.802 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:15.803 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 320, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 307, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 1610784, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 189008, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 2, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 1, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.113602356, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.051089635, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.065294790, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.012527907, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 307, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.055561152, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000180980 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.804 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 1610784, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 189008 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 2, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 232, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 320, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 196240, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 5, 2024-04-25T02:11:15.805 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 3, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.037699824, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.029295446, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.002862914, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 229, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.052495193, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000229236 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.806 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 196240 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "cct": { 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "total_workers": 0, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "unhealthy_workers": 0 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "mempool": { 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:15.807 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:15.808 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_items": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_bytes": 149745, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_items": 31, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:11:15.809 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_items": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bytes": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_items": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_bytes": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_items": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_items": 0, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_bytes": 928, 2024-04-25T02:11:15.810 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_items": 2, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_items": 0, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_items": 0 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 104857600, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 318, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 1563169, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 318, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-04-25T02:11:15.811 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-04-25T02:11:15.812 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 318, 2024-04-25T02:11:15.812 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 1563169, 2024-04-25T02:11:15.812 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-04-25T02:11:15.812 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:15.812 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:15.812 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:15.812 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:15.812 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:15.812 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:11:16.835 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T02:11:16.835 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T02:11:16.835 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-25T02:11:16.836 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-25T02:11:16.836 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-25T02:11:16.836 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-25T02:11:16.836 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-25T02:11:16.836 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T02:11:16.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T02:11:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T02:11:17.728 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:17 smithi102 ceph-mon[27796]: pgmap v501: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 618 KiB/s wr, 58 op/s 2024-04-25T02:11:17.728 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:17 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:11:17.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:17 smithi110 ceph-mon[32848]: pgmap v501: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 618 KiB/s wr, 58 op/s 2024-04-25T02:11:17.744 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:17 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:11:17.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:17 smithi193 ceph-mon[32981]: pgmap v501: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 618 KiB/s wr, 58 op/s 2024-04-25T02:11:17.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:17 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:11:19.694 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 0, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 2, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 0, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 0, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 4, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 0, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.005015140, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.000812887, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:19.695 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 312, 2024-04-25T02:11:19.696 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 301, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 1606672, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 186000, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 2, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 1, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.104257619, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.041227569, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.066275635, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.012286416, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 300, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.056211352, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000187371 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:19.697 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 1606672, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 186000 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 7, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 233, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 3504, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 274480, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 6, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 3, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.039869179, 2024-04-25T02:11:19.698 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.029423466, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.005210321, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 231, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.064539605, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000279392 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 274480 2024-04-25T02:11:19.699 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "cct": { 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "total_workers": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "unhealthy_workers": 0 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "mempool": { 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:19.700 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:19.701 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_items": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_bytes": 148714, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_items": 30, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:11:19.702 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_items": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bytes": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_items": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_bytes": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_items": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_items": 0, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_bytes": 928, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_items": 2, 2024-04-25T02:11:19.703 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_items": 0, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_items": 0 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 104857600, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 315, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 1562676, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 315, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-04-25T02:11:19.704 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 315, 2024-04-25T02:11:19.705 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 1562676, 2024-04-25T02:11:19.705 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-04-25T02:11:19.705 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:19.705 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:19.705 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:19.705 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:19.705 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:19.705 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:11:19.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:19 smithi193 ceph-mon[32981]: pgmap v502: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 66 KiB/s wr, 16 op/s 2024-04-25T02:11:19.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:19 smithi102 ceph-mon[27796]: pgmap v502: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 66 KiB/s wr, 16 op/s 2024-04-25T02:11:19.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:19 smithi110 ceph-mon[32848]: pgmap v502: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 66 KiB/s wr, 16 op/s 2024-04-25T02:11:20.550 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-25T02:11:20.550 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-25T02:11:20.550 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-04-25T02:11:20.551 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-04-25T02:11:20.551 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-25T02:11:20.551 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-25T02:11:20.551 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-25T02:11:20.551 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T02:11:20.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T02:11:21.001 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T02:11:21.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:21 smithi193 ceph-mon[32981]: pgmap v503: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 66 KiB/s wr, 12 op/s 2024-04-25T02:11:21.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:21 smithi102 ceph-mon[27796]: pgmap v503: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 66 KiB/s wr, 12 op/s 2024-04-25T02:11:21.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:21 smithi110 ceph-mon[32848]: pgmap v503: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 66 KiB/s wr, 12 op/s 2024-04-25T02:11:22.797 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:22 smithi102 ceph-mon[27796]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:11:22.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:22 smithi110 ceph-mon[32848]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:11:23.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:22 smithi193 ceph-mon[32981]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T02:11:23.345 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-04-25T02:11:23.345 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:23.345 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 4, 2024-04-25T02:11:23.345 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 4, 2024-04-25T02:11:23.345 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 3024, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 448, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 4, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 0, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.004326709, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.000070698, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.001335718, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 3, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.001467085, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000489028 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.346 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 318, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 307, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 1608304, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 189840, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 2, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 1, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.116814512, 2024-04-25T02:11:23.347 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.047287179, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.070470666, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.014126710, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 306, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.053298756, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000174178 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 1608304, 2024-04-25T02:11:23.348 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 189840 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 3, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 236, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 480, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 274912, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 6, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 3, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.036105975, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.027181749, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.003710818, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:23.349 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 233, 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.050918100, 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000218532 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 274912 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "cct": { 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "total_workers": 0, 2024-04-25T02:11:23.350 INFO:teuthology.orchestra.run.smithi102.stdout: "unhealthy_workers": 0 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "mempool": { 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:23.351 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:23.352 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_items": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_bytes": 147947, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_items": 29, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:11:23.353 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_items": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bytes": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_items": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_bytes": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_items": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_items": 0, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_bytes": 928, 2024-04-25T02:11:23.354 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_items": 2, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_items": 0, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_items": 0 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 104857600, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 321, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 1563456, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 321, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-04-25T02:11:23.355 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-04-25T02:11:23.356 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 321, 2024-04-25T02:11:23.356 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 1563456, 2024-04-25T02:11:23.356 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-04-25T02:11:23.356 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-04-25T02:11:23.356 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-04-25T02:11:23.356 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-04-25T02:11:23.356 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:23.356 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-04-25T02:11:23.356 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-04-25T02:11:23.889 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:23 smithi102 ceph-mon[27796]: pgmap v504: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 80 KiB/s wr, 13 op/s 2024-04-25T02:11:23.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:23 smithi110 ceph-mon[32848]: pgmap v504: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 80 KiB/s wr, 13 op/s 2024-04-25T02:11:24.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:23 smithi193 ceph-mon[32981]: pgmap v504: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 80 KiB/s wr, 13 op/s 2024-04-25T02:11:24.325 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T02:11:24.325 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T02:11:24.325 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-25T02:11:24.326 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-25T02:11:24.326 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-25T02:11:24.326 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-25T02:11:24.326 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-25T02:11:24.326 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T02:11:24.326 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T02:11:24.576 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T02:11:25.682 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:25 smithi110 ceph-mon[32848]: pgmap v505: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 63 KiB/s wr, 12 op/s 2024-04-25T02:11:25.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:25 smithi102 ceph-mon[27796]: pgmap v505: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 63 KiB/s wr, 12 op/s 2024-04-25T02:11:26.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:25 smithi193 ceph-mon[32981]: pgmap v505: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 63 KiB/s wr, 12 op/s 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 9969, 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 9272, 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 5770707, 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 19944556, 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 230, 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 7, 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.681744478, 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.225098633, 2024-04-25T02:11:26.221 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.358593713, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.109328102, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 9273, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 1.150991254, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000124122 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 12155, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.002001392, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000164 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 42224, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 976 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:26.222 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 7771, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 8090, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 5461710, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 10951115, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 8, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 6, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.527685826, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.223491181, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.264650502, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.076587973, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 8092, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.818128922, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000101103 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.223 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 9986, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.001540618, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000154 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1806816, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 654368 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 8405, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 9649, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 3130090, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 10223727, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 34, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 8, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.624091055, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.217218698, 2024-04-25T02:11:26.224 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.273650812, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.159674621, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 9641, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 1.661513823, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000172338 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7020, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.001949245, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000277 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 594720, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 329648 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.225 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 1, 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-MDSRank": { 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 6559, 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 2.535548302, 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000386575 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-PurgeQueue": { 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-25T02:11:26.226 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 153, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.005613407, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000036688 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": { 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "request": 2275, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "reply": 2050, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "reply_latency": { 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 2050, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 7.462858622, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.003640418 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_reply": 0, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "forward": 225, 2024-04-25T02:11:26.227 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_complete": 16, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_keys": 1, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_commit": 982, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_split": 3, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_merge": 0, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes": 1314, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_top": 448, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_bottom": 850, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pin_tail": 16, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pinned": 673, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_expired": 2956, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_with_caps": 654, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": 654, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "subtrees": 49, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse": 3204, 2024-04-25T02:11:26.228 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_hit": 2445, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_forward": 205, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_discover": 95, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_dir_fetch": 2, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_remote_ino": 0, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_lock": 15, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "load_cent": 25227, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "q": 0, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "exported": 103, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "exported_inodes": 2721, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "imported": 58, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "imported_inodes": 2611, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_dir_fetch": 0, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_backtrace_fetch": 2745, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_peer_discover": 0, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rfiles": 1006, 2024-04-25T02:11:26.229 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rbytes": 1378262209, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rsnaps": 7, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_fetch": 2745, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_set_tag": 0, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_inodes": 317, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_base_inodes": 12, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dirfrag_rstats": 305, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_file_inodes": 2428, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_inode_file_caps": 24, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_revoke": 518, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_grant": 1581, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:11:26.230 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps": 1682, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps_dirty": 913, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_cap_release": 49, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "process_request_cap_release": 244 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_cache": { 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update": 100, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update_receipt": 64, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_try_discover": 36, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_send_discover": 137, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_handle_discover": 242, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays": 0, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_delayed": 0, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_created": 43, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_enqueued": 33, 2024-04-25T02:11:26.231 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_reintegrated": 6, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_migrated": 6, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_processing": 0, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_started": 0, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_completed": 0, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_enqueue_scrub": 12, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_exportdir": 103, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_flush": 0, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragmentdir": 3, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragstats": 0, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_inodestats": 0 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_log": { 2024-04-25T02:11:26.232 INFO:teuthology.orchestra.run.smithi110.stdout: "evlrg": 1, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "evadd": 3565, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "evex": 3545, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "evtrm": 3545, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "ev": 20, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "evexg": 0, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "evexd": 7, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "segadd": 78, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "segex": 77, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "segtrm": 77, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "seg": 1, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "segmjr": 1, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "segexg": 0, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "segexd": 1, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "expos": 26066930, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "wrpos": 26352720, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "rdpos": 0, 2024-04-25T02:11:26.233 INFO:teuthology.orchestra.run.smithi110.stdout: "jlat": { 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1647, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 15.078734861, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.009155273 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "replayed": 0 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_mem": { 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "ino": 1317, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "ino+": 4032, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "ino-": 2715, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "dir": 117, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "dir+": 197, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "dir-": 80, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "dn": 1314, 2024-04-25T02:11:26.234 INFO:teuthology.orchestra.run.smithi110.stdout: "dn+": 4536, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "dn-": 3222, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "cap": 654, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "cap+": 1848, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "cap-": 1194, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "rss": 77044, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "heap": 231804 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_server": { 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_client_request": 2744, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_server_request": 914, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_request": 2275, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_session": 214, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_peer_request": 1627, 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "req_create_latency": { 2024-04-25T02:11:26.235 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 986, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 5.668882711, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.005749373 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getattr_latency": { 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 211, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.578098426, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.002739802 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getfilelock_latency": { 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "req_link_latency": { 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 15, 2024-04-25T02:11:26.236 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.093182116, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.006212141 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookup_latency": { 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 235, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.141606293, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000602579 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookuphash_latency": { 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupino_latency": { 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000227243, 2024-04-25T02:11:26.237 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000227243 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupname_latency": { 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupparent_latency": { 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupsnap_latency": { 2024-04-25T02:11:26.238 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.239 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.239 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.239 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.239 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lssnap_latency": { 2024-04-25T02:11:26.239 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 16, 2024-04-25T02:11:26.239 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.003204312, 2024-04-25T02:11:26.239 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000200269 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mkdir_latency": { 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 37, 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.039942158, 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001079517 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mknod_latency": { 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mksnap_latency": { 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 16, 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.182467825, 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.011404239 2024-04-25T02:11:26.240 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.241 INFO:teuthology.orchestra.run.smithi110.stdout: "req_open_latency": { 2024-04-25T02:11:26.241 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 385, 2024-04-25T02:11:26.241 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.137066341, 2024-04-25T02:11:26.241 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000356016 2024-04-25T02:11:26.241 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.241 INFO:teuthology.orchestra.run.smithi110.stdout: "req_readdir_latency": { 2024-04-25T02:11:26.241 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 72, 2024-04-25T02:11:26.241 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.184490890, 2024-04-25T02:11:26.241 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.002562373 2024-04-25T02:11:26.242 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.242 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rename_latency": { 2024-04-25T02:11:26.242 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 27, 2024-04-25T02:11:26.242 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.257725139, 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.009545375 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "req_renamesnap_latency": { 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "req_snapdiff_latency": { 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmdir_latency": { 2024-04-25T02:11:26.243 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 2, 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.014423547, 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.007211773 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmsnap_latency": { 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 9, 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.041230770, 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.004581196 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmxattr_latency": { 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.244 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setattr_latency": { 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1, 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000995843, 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000995843 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setdirlayout_latency": { 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setfilelock_latency": { 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.245 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setlayout_latency": { 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setxattr_latency": { 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7, 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.019772586, 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.002824655 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "req_symlink_latency": { 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.246 INFO:teuthology.orchestra.run.smithi110.stdout: "req_unlink_latency": { 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 30, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.099542422, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.003318080 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getvxattr_latency": { 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_sessions": { 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "session_count": 2, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "session_add": 5, 2024-04-25T02:11:26.247 INFO:teuthology.orchestra.run.smithi110.stdout: "session_remove": 3, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_open": 2, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_stale": 0, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "total_load": 77, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "average_load": 38, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "avg_session_uptime": 944, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "mdthresh_evicted": 0 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:26.248 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:26.249 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:26.250 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 452606, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 1458, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 88, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 1, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 8032, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 167, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:26.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 4493327, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 41752, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "objecter": { 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "op_active": 0, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "op_laggy": 0, 2024-04-25T02:11:26.252 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send": 7565, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send_bytes": 26240105, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "op_resend": 37, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "op_reply": 7565, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "op_latency": { 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7565, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 118.698927210, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.015690538 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "op_inflight": 0, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "oplen_avg": { 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7565, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 11275 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "op": 7565, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "op_r": 2803, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "op_w": 4762, 2024-04-25T02:11:26.253 INFO:teuthology.orchestra.run.smithi110.stdout: "op_rmw": 0, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "op_pg": 0, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_stat": 914, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_create": 1025, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_read": 29, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_write": 1664, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writefull": 276, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writesame": 0, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_append": 0, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_zero": 0, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_truncate": 3, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_delete": 569, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_mapext": 0, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_sparse_read": 0, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_clonerange": 0, 2024-04-25T02:11:26.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_getxattr": 2767, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_setxattr": 2050, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_rmxattr": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_call": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_watch": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_notify": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_other": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_active": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_send": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_resend": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_ping": 0, 2024-04-25T02:11:26.255 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_active": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_send": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_resend": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_active": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_send": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_resend": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_active": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_send": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_resend": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "command_active": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "command_send": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "command_resend": 0, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "map_epoch": 108, 2024-04-25T02:11:26.256 INFO:teuthology.orchestra.run.smithi110.stdout: "map_full": 0, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "map_inc": 22, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_sessions": 12, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_open": 12, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_close": 0, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_laggy": 0, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_wr": 1839, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_rd": 34, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_del": 105 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "oft": { 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_objs": 1, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:11:26.257 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_updates": 0, 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_removes": 0 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "osd-slow-ops": { 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_ops_count": 0 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "purge_queue": { 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops": 0, 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops_high_water": 17, 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing": 0, 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_high_water": 9, 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed_ops": 47, 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed": 33, 2024-04-25T02:11:26.258 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_item_in_journal": 0 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 26141, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 11997711, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 26141, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 26141, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 11997711, 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_bytes": { 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 7565, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 26243346, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 7565, 2024-04-25T02:11:26.260 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 26243346, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_ops": { 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 1024, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:26.261 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 7565, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 7565, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 7565, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 7565, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:26.262 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 33, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 3501, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 33, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 17, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 3501, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle-0x5621fd8b2500": { 2024-04-25T02:11:26.263 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 3565, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 22158416, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 3565, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 1647, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 22158416, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:26.264 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:26.265 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-25T02:11:27.025 INFO:tasks.check_counter:Daemon mds.b mds.exported=103 2024-04-25T02:11:27.025 INFO:tasks.check_counter:Daemon mds.b mds.imported=58 2024-04-25T02:11:27.025 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=100 2024-04-25T02:11:27.025 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=64 2024-04-25T02:11:27.025 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-04-25T02:11:27.025 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=16 2024-04-25T02:11:27.025 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=9 2024-04-25T02:11:27.025 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T02:11:27.026 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T02:11:27.271 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T02:11:27.669 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:27 smithi110 ceph-mon[32848]: pgmap v506: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 63 KiB/s wr, 11 op/s 2024-04-25T02:11:27.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:27 smithi102 ceph-mon[27796]: pgmap v506: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 63 KiB/s wr, 11 op/s 2024-04-25T02:11:28.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:27 smithi193 ceph-mon[32981]: pgmap v506: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 63 KiB/s wr, 11 op/s 2024-04-25T02:11:28.886 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-25T02:11:28.887 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:28.887 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 822, 2024-04-25T02:11:28.887 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 824, 2024-04-25T02:11:28.887 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 4407715, 2024-04-25T02:11:28.887 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 261460, 2024-04-25T02:11:28.887 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 7, 2024-04-25T02:11:28.887 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 2, 2024-04-25T02:11:28.887 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.061098215, 2024-04-25T02:11:28.887 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.021769829, 2024-04-25T02:11:28.888 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.016467201, 2024-04-25T02:11:28.888 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.016696798, 2024-04-25T02:11:28.888 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:28.888 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 822, 2024-04-25T02:11:28.888 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.028921365, 2024-04-25T02:11:28.888 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000035184 2024-04-25T02:11:28.888 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.888 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:28.888 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 1098, 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 1329, 2024-04-25T02:11:28.889 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 4473486, 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 923405, 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 7, 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 5, 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.095170908, 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.029037324, 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.026285338, 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.030341239, 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1327, 2024-04-25T02:11:28.890 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.074049418, 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000055802 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 576144 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.891 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 1910, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 1864, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 2267536, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 681226, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 4, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 3, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.192465885, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.057420598, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.075393747, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.057496452, 2024-04-25T02:11:28.892 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1863, 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.090578828, 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000048619 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1783328, 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 194368 2024-04-25T02:11:28.893 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 1, 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-MDSRank": { 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 5931, 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.145477593, 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000024528 2024-04-25T02:11:28.894 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-PurgeQueue": { 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": { 2024-04-25T02:11:28.895 INFO:teuthology.orchestra.run.smithi110.stdout: "request": 0, 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "reply": 0, 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "reply_latency": { 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_reply": 0, 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "forward": 0, 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_complete": 0, 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_keys": 0, 2024-04-25T02:11:28.896 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_commit": 0, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_split": 0, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_merge": 0, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes": 814, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_top": 248, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_bottom": 566, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pin_tail": 0, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pinned": 459, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_expired": 0, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_with_caps": 0, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": 0, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "subtrees": 14, 2024-04-25T02:11:28.897 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_hit": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_forward": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_discover": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_dir_fetch": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_remote_ino": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_lock": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "load_cent": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "q": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "exported": 0, 2024-04-25T02:11:28.898 INFO:teuthology.orchestra.run.smithi110.stdout: "exported_inodes": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "imported": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "imported_inodes": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_dir_fetch": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_peer_discover": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rfiles": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rbytes": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rsnaps": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_set_tag": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:11:28.899 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_file_inodes": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_grant": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:11:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps": 0, 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps_dirty": 0, 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_cap_release": 0, 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "process_request_cap_release": 0 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_cache": { 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update": 0, 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update_receipt": 0, 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_try_discover": 0, 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_send_discover": 0, 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_handle_discover": 0, 2024-04-25T02:11:28.901 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_delayed": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_created": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_enqueued": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_reintegrated": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_migrated": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_processing": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_started": 0, 2024-04-25T02:11:28.902 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_completed": 0, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_exportdir": 0, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_flush": 0, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragmentdir": 0, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragstats": 0, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_inodestats": 0 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_log": { 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "evlrg": 0, 2024-04-25T02:11:28.903 INFO:teuthology.orchestra.run.smithi110.stdout: "evadd": 0, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "evex": 0, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "evtrm": 0, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "ev": 3626, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "evexg": 0, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "evexd": 0, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "segadd": 0, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "segex": 0, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "segtrm": 0, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "seg": 6, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "segmjr": 2, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "segexg": 0, 2024-04-25T02:11:28.904 INFO:teuthology.orchestra.run.smithi110.stdout: "segexd": 0, 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "expos": 4194304, 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "wrpos": 12591215, 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "rdpos": 12590616, 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "jlat": { 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "replayed": 3626 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_mem": { 2024-04-25T02:11:28.905 INFO:teuthology.orchestra.run.smithi110.stdout: "ino": 433, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "ino+": 2798, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "ino-": 2365, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "dir": 26, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "dir+": 118, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "dir-": 92, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "dn": 814, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "dn+": 3183, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "dn-": 2369, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "cap": 0, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "cap+": 0, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "cap-": 0, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "rss": 60060, 2024-04-25T02:11:28.906 INFO:teuthology.orchestra.run.smithi110.stdout: "heap": 215420 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_server": { 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_client_request": 0, 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_server_request": 0, 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_request": 0, 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_session": 0, 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_peer_request": 0, 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "req_create_latency": { 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.907 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getattr_latency": { 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getfilelock_latency": { 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "req_link_latency": { 2024-04-25T02:11:28.908 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookup_latency": { 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookuphash_latency": { 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.909 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupino_latency": { 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupname_latency": { 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupparent_latency": { 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.910 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupsnap_latency": { 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lssnap_latency": { 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.911 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.912 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mkdir_latency": { 2024-04-25T02:11:28.912 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.912 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.912 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.912 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.912 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mknod_latency": { 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mksnap_latency": { 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "req_open_latency": { 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.913 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: "req_readdir_latency": { 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rename_latency": { 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.914 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: "req_renamesnap_latency": { 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: "req_snapdiff_latency": { 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.915 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmdir_latency": { 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmsnap_latency": { 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.916 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmxattr_latency": { 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setattr_latency": { 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setdirlayout_latency": { 2024-04-25T02:11:28.917 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setfilelock_latency": { 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setlayout_latency": { 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.918 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setxattr_latency": { 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: "req_symlink_latency": { 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.919 INFO:teuthology.orchestra.run.smithi110.stdout: "req_unlink_latency": { 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getvxattr_latency": { 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:28.920 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_sessions": { 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "session_count": 2, 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "session_add": 4, 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "session_remove": 2, 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_open": 4, 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_stale": 0, 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "total_load": 0, 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "average_load": 0, 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "avg_session_uptime": 3859, 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: "mdthresh_evicted": 0 2024-04-25T02:11:28.921 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:28.922 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:28.923 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:28.924 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 219120, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 583, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 8032, 2024-04-25T02:11:28.925 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 167, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 1727221, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 38467, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.926 INFO:teuthology.orchestra.run.smithi110.stdout: "objecter": { 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "op_active": 0, 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "op_laggy": 0, 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send": 3445, 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send_bytes": 21, 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "op_resend": 3, 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "op_reply": 3445, 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "op_latency": { 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3445, 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 27.724529641, 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.008047758 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.927 INFO:teuthology.orchestra.run.smithi110.stdout: "op_inflight": 0, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "oplen_avg": { 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3445, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 3446 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "op": 3445, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "op_r": 3445, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "op_w": 0, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "op_rmw": 0, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "op_pg": 0, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_stat": 2039, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_create": 0, 2024-04-25T02:11:28.928 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_read": 1404, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_write": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writefull": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writesame": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_append": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_zero": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_truncate": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_delete": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_mapext": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_sparse_read": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_clonerange": 0, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_getxattr": 1, 2024-04-25T02:11:28.929 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_setxattr": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_rmxattr": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_call": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_watch": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_notify": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_other": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_active": 0, 2024-04-25T02:11:28.930 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_send": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_resend": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_ping": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_active": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_send": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_resend": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_active": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_send": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_resend": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_active": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_send": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_resend": 0, 2024-04-25T02:11:28.931 INFO:teuthology.orchestra.run.smithi110.stdout: "command_active": 0, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "command_send": 0, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "command_resend": 0, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "map_epoch": 108, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "map_full": 0, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "map_inc": 21, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_sessions": 6, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_open": 7, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_close": 1, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_laggy": 0, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_wr": 0, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_rd": 2, 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_del": 0 2024-04-25T02:11:28.932 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "oft": { 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_objs": 0, 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_updates": 0, 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_removes": 0 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "osd-slow-ops": { 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_ops_count": 0 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "purge_queue": { 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops": 0, 2024-04-25T02:11:28.933 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing": 0, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_high_water": 0, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed_ops": 0, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed": 0, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_item_in_journal": 0 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 3826, 2024-04-25T02:11:28.934 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 10782145, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 3826, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3826, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 10782145, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_bytes": { 2024-04-25T02:11:28.935 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 3445, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 8396916, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3445, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 8396916, 2024-04-25T02:11:28.936 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_ops": { 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 1024, 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:28.937 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 3445, 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 3445, 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 3445, 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 3445, 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:28.938 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.939 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle-0x55fbbdd406e0": { 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:28.940 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:28.941 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 0, 2024-04-25T02:11:28.941 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 0, 2024-04-25T02:11:28.941 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:28.941 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:28.941 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:28.941 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:28.941 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:28.941 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:28.941 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-25T02:11:29.641 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:29 smithi110 ceph-mon[32848]: pgmap v507: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 94 KiB/s wr, 14 op/s 2024-04-25T02:11:29.669 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T02:11:29.669 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T02:11:29.669 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-25T02:11:29.669 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-25T02:11:29.670 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-25T02:11:29.670 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:11:29.670 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:11:29.670 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T02:11:29.670 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T02:11:29.907 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T02:11:29.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:29 smithi102 ceph-mon[27796]: pgmap v507: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 94 KiB/s wr, 14 op/s 2024-04-25T02:11:30.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:29 smithi193 ceph-mon[32981]: pgmap v507: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 94 KiB/s wr, 14 op/s 2024-04-25T02:11:31.649 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-25T02:11:31.649 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:31.649 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 8793, 2024-04-25T02:11:31.649 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 8476, 2024-04-25T02:11:31.649 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 3985376, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 7815284, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 37, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 2, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.418396519, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.162000271, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.247172758, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.037393298, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 8477, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.224875931, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000026527 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.650 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 12728, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.001651678, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000129 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 3952, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 1424 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 3604, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 3838, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 1807650, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 6166440, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 14, 2024-04-25T02:11:31.651 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 6, 2024-04-25T02:11:31.652 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.205490479, 2024-04-25T02:11:31.652 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.070083870, 2024-04-25T02:11:31.652 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.050973473, 2024-04-25T02:11:31.652 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.065039118, 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3834, 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.173010288, 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000045125 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 18, 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000002499, 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000138 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.653 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 2624, 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 657184 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 10770, 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 12353, 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 7359677, 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 6088854, 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 9, 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 8, 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.639594867, 2024-04-25T02:11:31.654 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.287298841, 2024-04-25T02:11:31.655 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.389448007, 2024-04-25T02:11:31.655 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.044469836, 2024-04-25T02:11:31.655 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:31.655 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 12350, 2024-04-25T02:11:31.655 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.337641608, 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000027339 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 13870, 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.003088002, 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000222 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1799552, 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 230080 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.656 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 1, 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-MDSRank": { 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7253, 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 1.566681368, 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000216004 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.657 INFO:teuthology.orchestra.run.smithi110.stdout: "finisher-PurgeQueue": { 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "queue_len": 0, 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "complete_latency": { 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 492, 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.018132287, 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000036854 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "mds": { 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "request": 1450, 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "reply": 1359, 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "reply_latency": { 2024-04-25T02:11:31.658 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1359, 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 13.972598487, 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.010281529 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_reply": 0, 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: "forward": 91, 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_complete": 5, 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_fetch_keys": 0, 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_commit": 19, 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_split": 1, 2024-04-25T02:11:31.659 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_merge": 0, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes": 705, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_top": 189, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_bottom": 397, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pin_tail": 119, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_pinned": 435, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_expired": 2541, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "inodes_with_caps": 134, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "caps": 135, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "subtrees": 47, 2024-04-25T02:11:31.660 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse": 4890, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_hit": 3997, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_forward": 86, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_discover": 131, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_dir_fetch": 1, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_remote_ino": 8, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "traverse_lock": 147, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "load_cent": 2848, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "q": 0, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "exported": 48, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "exported_inodes": 2273, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "imported": 72, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "imported_inodes": 2126, 2024-04-25T02:11:31.661 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_dir_fetch": 0, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_backtrace_fetch": 5801, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "openino_peer_discover": 26, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rfiles": 1005, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rbytes": 1277170734, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "root_rsnaps": 6, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_fetch": 5801, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_set_tag": 0, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_inodes": 30, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:11:31.662 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_dirfrag_rstats": 30, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "scrub_file_inodes": 5771, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_inode_file_caps": 103, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_revoke": 529, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_grant": 1097, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps": 1407, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_caps_dirty": 556, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_cap_release": 124, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "process_request_cap_release": 729 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_cache": { 2024-04-25T02:11:31.663 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update": 15, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_update_receipt": 274, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_try_discover": 151, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_send_discover": 340, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "dir_handle_discover": 135, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays": 10, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_delayed": 0, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_created": 205, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_enqueued": 125, 2024-04-25T02:11:31.664 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_reintegrated": 75, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "strays_migrated": 24, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_processing": 0, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_started": 0, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "recovery_completed": 0, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_exportdir": 48, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_flush": 0, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragmentdir": 1, 2024-04-25T02:11:31.665 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_fragstats": 0, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "ireq_inodestats": 0 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_log": { 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "evlrg": 0, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "evadd": 3626, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "evex": 0, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "evtrm": 0, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "ev": 3626, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "evexg": 0, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "evexd": 0, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "segadd": 6, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "segex": 0, 2024-04-25T02:11:31.666 INFO:teuthology.orchestra.run.smithi110.stdout: "segtrm": 0, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "seg": 6, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "segmjr": 2, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "segexg": 0, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "segexd": 0, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "expos": 4194304, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "wrpos": 12591215, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "rdpos": 0, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "jlat": { 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1085, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 10.519476597, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.009695370 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.667 INFO:teuthology.orchestra.run.smithi110.stdout: "replayed": 0 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_mem": { 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "ino": 465, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "ino+": 2867, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "ino-": 2402, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "dir": 112, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "dir+": 233, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "dir-": 121, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "dn": 716, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "dn+": 3412, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "dn-": 2696, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "cap": 135, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "cap+": 925, 2024-04-25T02:11:31.668 INFO:teuthology.orchestra.run.smithi110.stdout: "cap-": 790, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "rss": 69844, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "heap": 223612 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_server": { 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_client_request": 3555, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "dispatch_server_request": 827, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_request": 1450, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_client_session": 188, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "handle_peer_request": 1898, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "req_create_latency": { 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 120, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.176916182, 2024-04-25T02:11:31.669 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001474301 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getattr_latency": { 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 62, 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.077540431, 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.001250652 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getfilelock_latency": { 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "req_link_latency": { 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 78, 2024-04-25T02:11:31.670 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.631483332, 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.008095940 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookup_latency": { 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 623, 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 1.727972869, 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.002773632 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookuphash_latency": { 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.671 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupino_latency": { 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 1, 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000129941, 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000129941 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupname_latency": { 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupparent_latency": { 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.672 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.673 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.673 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lookupsnap_latency": { 2024-04-25T02:11:31.673 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.673 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.673 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.673 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "req_lssnap_latency": { 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mkdir_latency": { 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mknod_latency": { 2024-04-25T02:11:31.674 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "req_mksnap_latency": { 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "req_open_latency": { 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 137, 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.108612354, 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000792790 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "req_readdir_latency": { 2024-04-25T02:11:31.675 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 29, 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 2.883668680, 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.099436851 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rename_latency": { 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 216, 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 7.973855334, 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.036915996 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "req_renamesnap_latency": { 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.676 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "req_snapdiff_latency": { 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmdir_latency": { 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmsnap_latency": { 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.677 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "req_rmxattr_latency": { 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setattr_latency": { 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setdirlayout_latency": { 2024-04-25T02:11:31.678 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setfilelock_latency": { 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setlayout_latency": { 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.679 INFO:teuthology.orchestra.run.smithi110.stdout: "req_setxattr_latency": { 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "req_symlink_latency": { 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "req_unlink_latency": { 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 93, 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.392419364, 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.004219563 2024-04-25T02:11:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "req_getvxattr_latency": { 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_sessions": { 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "session_count": 2, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "session_add": 4, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "session_remove": 2, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_open": 2, 2024-04-25T02:11:31.681 INFO:teuthology.orchestra.run.smithi110.stdout: "sessions_stale": 0, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "total_load": 226, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "average_load": 113, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "avg_session_uptime": 944, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "mdthresh_evicted": 0 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:31.682 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:31.683 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 838613, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 782, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 88, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 1, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 8032, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 167, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:31.685 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 2054049, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 52618, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "objecter": { 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "op_active": 0, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "op_laggy": 0, 2024-04-25T02:11:31.686 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send": 7517, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "op_send_bytes": 8612346, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "op_resend": 28, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "op_reply": 7517, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "op_latency": { 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7517, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 20.731644994, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.002757967 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "op_inflight": 0, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "oplen_avg": { 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 7517, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 7795 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.687 INFO:teuthology.orchestra.run.smithi110.stdout: "op": 7517, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "op_r": 5869, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "op_w": 1648, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "op_rmw": 0, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "op_pg": 0, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_stat": 0, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_create": 10, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_read": 63, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_write": 1148, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writefull": 117, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_writesame": 0, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_append": 0, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_zero": 0, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_truncate": 0, 2024-04-25T02:11:31.688 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_delete": 140, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_mapext": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_sparse_read": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_clonerange": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_getxattr": 5801, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_setxattr": 20, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_rmxattr": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_call": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_watch": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_notify": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls": 0, 2024-04-25T02:11:31.689 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "osdop_other": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_active": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_send": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_resend": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "linger_ping": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_active": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_send": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "poolop_resend": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_active": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_send": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "poolstat_resend": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_active": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_send": 0, 2024-04-25T02:11:31.690 INFO:teuthology.orchestra.run.smithi110.stdout: "statfs_resend": 0, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "command_active": 0, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "command_send": 0, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "command_resend": 0, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "map_epoch": 108, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "map_full": 0, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "map_inc": 21, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_sessions": 12, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_open": 12, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_session_close": 0, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_laggy": 0, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_wr": 375, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_rd": 10, 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_del": 111 2024-04-25T02:11:31.691 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "oft": { 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_objs": 1, 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_updates": 0, 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "omap_total_removes": 0 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "osd-slow-ops": { 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "slow_ops_count": 0 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "purge_queue": { 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops": 0, 2024-04-25T02:11:31.692 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_ops_high_water": 9, 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing": 0, 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executing_high_water": 9, 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed_ops": 125, 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_executed": 125, 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "pq_item_in_journal": 0 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:31.693 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 23163, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 11093777, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 23163, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 23163, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 11093777, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.694 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_bytes": { 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 7517, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 8625323, 2024-04-25T02:11:31.695 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 7517, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 8625323, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-objecter_ops": { 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 1024, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 0, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 0, 2024-04-25T02:11:31.696 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 7517, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 7517, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 7517, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 7517, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-25T02:11:31.697 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 125, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 13057, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 125, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 63, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 13057, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:31.698 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-write_buf_throttle-0x564100d86500": { 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 3758096384, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 3626, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 8396911, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 3626, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 1085, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 8396911, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:31.699 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:31.700 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:31.700 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:31.700 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:31.700 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-25T02:11:31.700 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:31 smithi110 ceph-mon[32848]: pgmap v508: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 50 KiB/s wr, 10 op/s 2024-04-25T02:11:31.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:31 smithi102 ceph-mon[27796]: pgmap v508: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 50 KiB/s wr, 10 op/s 2024-04-25T02:11:32.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:31 smithi193 ceph-mon[32981]: pgmap v508: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 50 KiB/s wr, 10 op/s 2024-04-25T02:11:32.497 INFO:tasks.check_counter:Daemon mds.h mds.exported=48 2024-04-25T02:11:32.497 INFO:tasks.check_counter:Daemon mds.h mds.imported=72 2024-04-25T02:11:32.497 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=15 2024-04-25T02:11:32.497 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=274 2024-04-25T02:11:32.497 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-04-25T02:11:32.498 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:11:32.498 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:11:32.498 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T02:11:32.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T02:11:32.734 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T02:11:33.814 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:33 smithi193 ceph-mon[32981]: pgmap v509: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 51 KiB/s wr, 10 op/s 2024-04-25T02:11:33.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:33 smithi102 ceph-mon[27796]: pgmap v509: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 51 KiB/s wr, 10 op/s 2024-04-25T02:11:33.934 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:33 smithi110 ceph-mon[32848]: pgmap v509: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 51 KiB/s wr, 10 op/s 2024-04-25T02:11:34.473 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-04-25T02:11:34.473 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:34.473 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 322, 2024-04-25T02:11:34.473 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 311, 2024-04-25T02:11:34.473 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 1609392, 2024-04-25T02:11:34.473 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 192400, 2024-04-25T02:11:34.473 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 4, 2024-04-25T02:11:34.473 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 0, 2024-04-25T02:11:34.473 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.051172517, 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.015415667, 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.036475144, 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.006507242, 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 310, 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.013212175, 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000042619 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:34.474 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 1609392, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 192400 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 3, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 240, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 480, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 353584, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 7, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 3, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.014662919, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.008660850, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.003641336, 2024-04-25T02:11:34.475 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 237, 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.026720065, 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000112742 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 353584 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:34.476 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_messages": 4, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages": 4, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_bytes": 3024, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_bytes": 448, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_created_connections": 2, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_active_connections": 1, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_total_time": 0.000586788, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_send_time": 0.000035731, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_recv_time": 0.000556907, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 3, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000724728, 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000241576 2024-04-25T02:11:34.477 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "cct": { 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "total_workers": 0, 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "unhealthy_workers": 0 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "mempool": { 2024-04-25T02:11:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:34.479 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:34.480 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_items": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_bytes": 147908, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_anon_items": 28, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "buffer_meta_items": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_bytes": 0, 2024-04-25T02:11:34.481 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_items": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_bytes": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_items": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "pgmap_items": 0, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_bytes": 928, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_co_items": 2, 2024-04-25T02:11:34.482 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_1_items": 0, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "unittest_2_items": 0 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "val": 0, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "max": 104857600, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "get_started": 0, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "get": 325, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "get_sum": 1563976, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "get_or_fail_success": 325, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "take": 0, 2024-04-25T02:11:34.483 INFO:teuthology.orchestra.run.smithi110.stdout: "take_sum": 0, 2024-04-25T02:11:34.484 INFO:teuthology.orchestra.run.smithi110.stdout: "put": 325, 2024-04-25T02:11:34.484 INFO:teuthology.orchestra.run.smithi110.stdout: "put_sum": 1563976, 2024-04-25T02:11:34.484 INFO:teuthology.orchestra.run.smithi110.stdout: "wait": { 2024-04-25T02:11:34.484 INFO:teuthology.orchestra.run.smithi110.stdout: "avgcount": 0, 2024-04-25T02:11:34.484 INFO:teuthology.orchestra.run.smithi110.stdout: "sum": 0.000000000, 2024-04-25T02:11:34.484 INFO:teuthology.orchestra.run.smithi110.stdout: "avgtime": 0.000000000 2024-04-25T02:11:34.484 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:34.484 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-04-25T02:11:34.484 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-04-25T02:11:35.140 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-25T02:11:35.140 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-25T02:11:35.140 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-25T02:11:35.140 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-25T02:11:35.140 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-25T02:11:35.141 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-25T02:11:35.141 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-25T02:11:35.141 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T02:11:35.141 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T02:11:35.394 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T02:11:35.765 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:35 smithi193 ceph-mon[32981]: pgmap v510: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 11 op/s 2024-04-25T02:11:35.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:35 smithi102 ceph-mon[27796]: pgmap v510: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 11 op/s 2024-04-25T02:11:35.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:35 smithi110 ceph-mon[32848]: pgmap v510: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 43 KiB/s wr, 11 op/s 2024-04-25T02:11:37.150 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-25T02:11:37.150 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:37.150 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2143, 2024-04-25T02:11:37.150 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2097, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 10713046, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 755488, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 1, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.218324600, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.061820678, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.088427621, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.063938265, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2096, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.112618394, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000053730 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.151 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1783760, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 195168 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1177, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1409, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 8687868, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 1115454, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 9, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-04-25T02:11:37.152 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.099264370, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.032022248, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.028080377, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.030645478, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1407, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.078023959, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000055454 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 743328 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 762, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 764, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 5583228, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 242323, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.067270877, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.019106846, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.019960550, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.022310973, 2024-04-25T02:11:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 762, 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.043037895, 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000056480 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-04-25T02:11:37.155 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 6245, 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.183654776, 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000029408 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.156 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-04-25T02:11:37.157 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 1358, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 939, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 419, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 16, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 42, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-04-25T02:11:37.158 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-04-25T02:11:37.159 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 1006, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 1379950092, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 7, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:11:37.160 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:11:37.161 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-04-25T02:11:37.162 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 3567, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-04-25T02:11:37.163 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 2, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 72, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 26066930, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 26362630, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 26353294, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 3567 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 1311, 2024-04-25T02:11:37.164 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 3886, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 2575, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 81, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 171, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 90, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 1358, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 4041, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 2683, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 59344, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215420 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-04-25T02:11:37.165 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-04-25T02:11:37.166 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-04-25T02:11:37.167 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-04-25T02:11:37.168 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-04-25T02:11:37.169 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.170 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.171 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.172 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.173 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-04-25T02:11:37.176 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 5, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 3, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 4, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 3857, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:37.177 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:37.178 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:37.179 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 248216, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 1388, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:37.180 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 8032, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 167, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 4107988, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 33935, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-04-25T02:11:37.181 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 3694, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 26, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 4, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 3694, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3694, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 18.085715926, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.004895970 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3694, 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3695 2024-04-25T02:11:37.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 3694, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 3694, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 2204, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 1487, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-04-25T02:11:37.183 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 2, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-04-25T02:11:37.184 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-04-25T02:11:37.185 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 108, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 21, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 6, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 6, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-04-25T02:11:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-04-25T02:11:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 4078, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 24594729, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 4078, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 4078, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 24594729, 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3694, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 22168336, 2024-04-25T02:11:37.189 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3694, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 22168336, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:37.190 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3694, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 3694, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3694, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3694, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:37.191 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x55dd0c39e6e0": { 2024-04-25T02:11:37.192 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:37.193 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:37.194 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-25T02:11:37.650 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:37 smithi193 ceph-mon[32981]: pgmap v511: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 44 KiB/s wr, 10 op/s 2024-04-25T02:11:37.925 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-25T02:11:37.925 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-25T02:11:37.925 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-25T02:11:37.925 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-04-25T02:11:37.925 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=7 2024-04-25T02:11:37.925 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:11:37.925 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:11:37.926 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T02:11:37.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T02:11:37.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:37 smithi102 ceph-mon[27796]: pgmap v511: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 44 KiB/s wr, 10 op/s 2024-04-25T02:11:37.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:37 smithi110 ceph-mon[32848]: pgmap v511: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 44 KiB/s wr, 10 op/s 2024-04-25T02:11:38.170 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T02:11:39.694 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:39 smithi193 ceph-mon[32981]: pgmap v512: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 45 KiB/s wr, 11 op/s 2024-04-25T02:11:39.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:39 smithi102 ceph-mon[27796]: pgmap v512: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 45 KiB/s wr, 11 op/s 2024-04-25T02:11:39.957 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-25T02:11:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1507, 2024-04-25T02:11:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1462, 2024-04-25T02:11:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 2161824, 2024-04-25T02:11:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 554034, 2024-04-25T02:11:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-04-25T02:11:39.957 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 1, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.167341664, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.050308722, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.070687921, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.046011391, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1461, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.081711775, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000055928 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.958 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1783872, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 195648 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 418, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 651, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 4451727, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 874264, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 9, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.050873980, 2024-04-25T02:11:39.959 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.016241079, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.015912347, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.013667907, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 649, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.084734916, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000130562 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-04-25T02:11:39.960 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 743504 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2154, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2155, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 21528936, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 684661, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.158470550, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.043064016, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.049355967, 2024-04-25T02:11:39.961 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.055677573, 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2153, 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.096225232, 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000044693 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:11:39.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 6278, 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.156268568, 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000024891 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.963 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.964 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 1194, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 345, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 849, 2024-04-25T02:11:39.965 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 702, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 38, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-04-25T02:11:39.966 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:11:39.967 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-04-25T02:11:39.968 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-04-25T02:11:39.969 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-04-25T02:11:39.970 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 4735, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 10, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 2, 2024-04-25T02:11:39.971 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 29510477, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 29507874, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 4735 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 752, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 3226, 2024-04-25T02:11:39.972 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 2474, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 81, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 170, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 89, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 1194, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 3714, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 2520, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 59384, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215420 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-04-25T02:11:39.973 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.974 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.975 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.977 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-04-25T02:11:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.980 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-04-25T02:11:39.981 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-04-25T02:11:39.982 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 5, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 3, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 4, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-04-25T02:11:39.985 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 3860, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:39.986 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:39.987 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 235518, 2024-04-25T02:11:39.988 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 1046, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 8032, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 167, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-04-25T02:11:39.989 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 2972924, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 46575, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 3691, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 21, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 3, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 3691, 2024-04-25T02:11:39.990 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3691, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 12.973796103, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.003514981 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3691, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3692 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 3691, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 3691, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-04-25T02:11:39.991 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 2154, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 1535, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 1, 2024-04-25T02:11:39.992 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-04-25T02:11:39.993 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 108, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-04-25T02:11:39.994 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 21, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 6, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 7, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 1, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-04-25T02:11:39.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:39.996 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 4075, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 27753338, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 4075, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 4075, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 27753338, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.997 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3691, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 25326585, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3691, 2024-04-25T02:11:39.998 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 25326585, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:39.999 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3691, 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 3691, 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3691, 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3691, 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-25T02:11:40.000 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:40.001 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x55d5907ec6e0": { 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:40.002 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:40.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:40.003 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-25T02:11:40.003 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:39 smithi110 ceph-mon[32848]: pgmap v512: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 45 KiB/s wr, 11 op/s 2024-04-25T02:11:40.782 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T02:11:40.782 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T02:11:40.782 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-25T02:11:40.782 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-25T02:11:40.782 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-25T02:11:40.782 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:11:40.782 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:11:40.782 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T02:11:40.782 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T02:11:41.020 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T02:11:41.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:41 smithi193 ceph-mon[32981]: pgmap v513: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-04-25T02:11:41.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:41 smithi102 ceph-mon[27796]: pgmap v513: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-04-25T02:11:41.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:41 smithi110 ceph-mon[32848]: pgmap v513: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2024-04-25T02:11:42.633 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-25T02:11:42.633 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:42.633 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 13262, 2024-04-25T02:11:42.633 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 15780, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 5410224, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 24088386, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 13, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.758585992, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.319836133, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.433321334, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.075048258, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 15778, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.526139976, 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000033346 2024-04-25T02:11:42.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 16789, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.003685197, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000219 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 7643, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 7996, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 4335365, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 8139396, 2024-04-25T02:11:42.635 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 37, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.379710085, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.144068251, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.253718772, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.022859761, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 7993, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.223163384, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000027919 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 12936, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.001900172, 2024-04-25T02:11:42.636 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000146 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3872, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 141264 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1977, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2161, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 2427805, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 5904412, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 10, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 7, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.175985692, 2024-04-25T02:11:42.637 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.065290934, 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.079040501, 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.042915264, 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2161, 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.104156883, 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000048198 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 49, 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000008134, 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000166 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.638 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1806000, 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 675968 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 6278, 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.786431456, 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000284554 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:42.639 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 462, 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.017264859, 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000037369 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 1930, 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 1832, 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1832, 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 38.482911138, 2024-04-25T02:11:42.640 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.021005955 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 98, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 22, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 69, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 7, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 990, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 227, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 588, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 175, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 665, 2024-04-25T02:11:42.641 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 2707, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 277, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 277, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 47, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 5582, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 4356, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 92, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 107, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 3, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 222, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 1476, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-04-25T02:11:42.642 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 32, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 2461, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 53, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 2718, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 2919, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 5, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 1005, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 1277170734, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 6, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 2919, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-04-25T02:11:42.643 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 56, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 56, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 2863, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 86, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 656, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 1562, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 2079, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 988, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 127, 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 1048 2024-04-25T02:11:42.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 89, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 120, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 67, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 167, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 267, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 10, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 243, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 110, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 122, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 25, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-04-25T02:11:42.645 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 33, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 7, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-04-25T02:11:42.646 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 1, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 4770, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 4770, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 10, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 10, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 2, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-04-25T02:11:42.647 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 29782839, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 0, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1859, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 15.649478767, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.008418224 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 0 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 751, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 3291, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 2540, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 122, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 177, 2024-04-25T02:11:42.648 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 55, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 1019, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 3935, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 2916, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 215, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 917, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 702, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 70244, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 223612 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 4068, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 1236, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 1930, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 371, 2024-04-25T02:11:42.649 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 2520, 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 187, 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.303673191, 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001623920 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 65, 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.167141101, 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.002571401 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.650 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 130, 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.648532210, 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.004988709 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 805, 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 33.673197031, 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.041830058 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.651 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.652 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.653 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.653 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-04-25T02:11:42.653 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.653 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.653 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.654 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 167, 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.095291036, 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000570605 2024-04-25T02:11:42.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 83, 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.098048881, 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001181311 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 249, 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3.039348185, 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.012206217 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.657 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-04-25T02:11:42.658 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 146, 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.457679503, 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.003134791 2024-04-25T02:11:42.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 5, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 3, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 2, 2024-04-25T02:11:42.661 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 285, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 142, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 962, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:42.662 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:42.663 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:42.664 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 865682, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 1042, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 440, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 5, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 8032, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 167, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 3098662, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 63595, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.666 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 5535, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 26096340, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 31, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 5535, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 5535, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 23.740625987, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.004289182 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 5535, 2024-04-25T02:11:42.667 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 5965 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 5535, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 3001, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 2534, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 0, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 10, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 60, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 1919, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 120, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-04-25T02:11:42.668 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 2, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 133, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 2919, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 20, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-04-25T02:11:42.669 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-04-25T02:11:42.670 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 108, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 21, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 12, 2024-04-25T02:11:42.671 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 12, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 604, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 44, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 134 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 1, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-04-25T02:11:42.672 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 3 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 10, 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 10, 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 110, 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 110, 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.673 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 22878, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 10162520, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 22878, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 22878, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 10162520, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.674 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 5535, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 26107666, 2024-04-25T02:11:42.675 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 5535, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 26107666, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-04-25T02:11:42.676 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 5535, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 5535, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 5535, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 5535, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:42.677 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 110, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 11678, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 110, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 60, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 11678, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.678 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x56336f58a500": { 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 223338, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 4770, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 25588535, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 4770, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 1859, 2024-04-25T02:11:42.679 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 25365197, 2024-04-25T02:11:42.680 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:42.680 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:42.680 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:42.680 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:42.680 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:42.680 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:42.680 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-25T02:11:43.422 INFO:tasks.check_counter:Daemon mds.i mds.exported=32 2024-04-25T02:11:43.422 INFO:tasks.check_counter:Daemon mds.i mds.imported=53 2024-04-25T02:11:43.422 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=89 2024-04-25T02:11:43.422 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=120 2024-04-25T02:11:43.422 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-25T02:11:43.422 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-25T02:11:43.423 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T02:11:43.423 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T02:11:43.423 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551 shell --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T02:11:43.663 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T02:11:43.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:43 smithi193 ceph-mon[32981]: pgmap v514: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-04-25T02:11:43.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:43 smithi102 ceph-mon[27796]: pgmap v514: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-04-25T02:11:43.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:43 smithi110 ceph-mon[32848]: pgmap v514: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 326, 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 315, 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 1610480, 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 194960, 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 4, 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 0, 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.055837242, 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.016830497, 2024-04-25T02:11:45.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.039487122, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.007488907, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 314, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.011953814, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000038069 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1610480, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 194960 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.379 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 8, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 245, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 3664, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 354560, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 7, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 3, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.013384856, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.008721565, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.002208798, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 243, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.037658625, 2024-04-25T02:11:45.380 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000154973 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3664, 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 354560 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 0, 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2, 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 0, 2024-04-25T02:11:45.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 0, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 2, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 1, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.000213581, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.000142235, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:45.382 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 0, 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T02:11:45.383 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-04-25T02:11:45.384 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-04-25T02:11:45.385 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 147930, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 28, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-04-25T02:11:45.386 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 928, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 2, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-04-25T02:11:45.387 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 330, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 1564523, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 330, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 330, 2024-04-25T02:11:45.388 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 1564523, 2024-04-25T02:11:45.389 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-04-25T02:11:45.389 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-04-25T02:11:45.389 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-04-25T02:11:45.389 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-04-25T02:11:45.389 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:45.389 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-04-25T02:11:45.389 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-04-25T02:11:45.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:45 smithi193 ceph-mon[32981]: pgmap v515: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T02:11:45.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:45 smithi102 ceph-mon[27796]: pgmap v515: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T02:11:45.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:45 smithi110 ceph-mon[32848]: pgmap v515: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-04-25T02:11:46.007 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-25T02:11:46.007 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-25T02:11:46.007 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-25T02:11:46.007 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-25T02:11:46.007 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-25T02:11:46.007 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-25T02:11:46.007 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-25T02:11:46.008 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T02:11:46.019 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T02:11:46.019 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T02:11:46.019 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T02:11:46.080 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T02:11:46.080 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T02:11:46.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T02:11:46.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.322+0000 7f716ac00640 1 -- 172.21.15.102:0/474858662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7164108580 msgr2=0x7f71641149f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.322+0000 7f716ac00640 1 --2- 172.21.15.102:0/474858662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7164108580 0x7f71641149f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f715c009ca0 tx=0x7f715c02a950 comp rx=0 tx=0).stop 2024-04-25T02:11:46.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.322+0000 7f716ac00640 1 -- 172.21.15.102:0/474858662 shutdown_connections 2024-04-25T02:11:46.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.322+0000 7f716ac00640 1 --2- 172.21.15.102:0/474858662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7164108580 0x7f71641149f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.322+0000 7f716ac00640 1 --2- 172.21.15.102:0/474858662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7164107be0 0x7f7164108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.322+0000 7f716ac00640 1 --2- 172.21.15.102:0/474858662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f716410dc50 0x7f716410e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.322+0000 7f716ac00640 1 -- 172.21.15.102:0/474858662 >> 172.21.15.102:0/474858662 conn(0x7f71640772c0 msgr2=0x7f71640796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:46.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.322+0000 7f716ac00640 1 -- 172.21.15.102:0/474858662 shutdown_connections 2024-04-25T02:11:46.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.322+0000 7f716ac00640 1 -- 172.21.15.102:0/474858662 wait complete. 2024-04-25T02:11:46.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.323+0000 7f716ac00640 1 Processor -- start 2024-04-25T02:11:46.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f716ac00640 1 -- start start 2024-04-25T02:11:46.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f716ac00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7164107be0 0x7f716419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:46.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f716ac00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7164108580 0x7f716419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:46.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f716ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f716410dc50 0x7f71641a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:46.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f716ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7164117f70 con 0x7f716410dc50 2024-04-25T02:11:46.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f716ac00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f7164117df0 con 0x7f7164107be0 2024-04-25T02:11:46.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f716ac00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f71641180f0 con 0x7f7164108580 2024-04-25T02:11:46.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f7169800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7164107be0 0x7f716419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:46.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f716a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f716410dc50 0x7f71641a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:46.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f7168e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7164108580 0x7f716419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:46.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f7169800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7164107be0 0x7f716419e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42834/0 (socket says 172.21.15.102:42834) 2024-04-25T02:11:46.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.324+0000 7f7169800640 1 -- 172.21.15.102:0/2165469108 learned_addr learned my addr 172.21.15.102:0/2165469108 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:46.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.325+0000 7f7168e00640 1 -- 172.21.15.102:0/2165469108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7164107be0 msgr2=0x7f716419e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.325+0000 7f7168e00640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7164107be0 0x7f716419e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.325+0000 7f7168e00640 1 -- 172.21.15.102:0/2165469108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f716410dc50 msgr2=0x7f71641a2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.325+0000 7f7168e00640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f716410dc50 0x7f71641a2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.325+0000 7f7168e00640 1 -- 172.21.15.102:0/2165469108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71641a3370 con 0x7f7164108580 2024-04-25T02:11:46.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.325+0000 7f7168e00640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7164108580 0x7f716419e910 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f715000e9f0 tx=0x7f715000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:46.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.325+0000 7f7162000640 1 -- 172.21.15.102:0/2165469108 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715000ccc0 con 0x7f7164108580 2024-04-25T02:11:46.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.326+0000 7f7162000640 1 -- 172.21.15.102:0/2165469108 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7150010910 con 0x7f7164108580 2024-04-25T02:11:46.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.326+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71641a3660 con 0x7f7164108580 2024-04-25T02:11:46.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.326+0000 7f7162000640 1 -- 172.21.15.102:0/2165469108 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150018960 con 0x7f7164108580 2024-04-25T02:11:46.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.326+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71641aaef0 con 0x7f7164108580 2024-04-25T02:11:46.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.327+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71640688d0 con 0x7f7164108580 2024-04-25T02:11:46.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.329+0000 7f7162000640 1 -- 172.21.15.102:0/2165469108 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7150010ac0 con 0x7f7164108580 2024-04-25T02:11:46.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.331+0000 7f7162000640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f714c0a2d80 0x7f714c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:46.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.331+0000 7f7162000640 1 -- 172.21.15.102:0/2165469108 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f71500cfa80 con 0x7f7164108580 2024-04-25T02:11:46.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.331+0000 7f7169800640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f714c0a2d80 0x7f714c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:46.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.332+0000 7f7169800640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f714c0a2d80 0x7f714c0a5240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7158009c10 tx=0x7f7158005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:46.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.332+0000 7f7162000640 1 -- 172.21.15.102:0/2165469108 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f715009bff0 con 0x7f7164108580 2024-04-25T02:11:46.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.534+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f716410fab0 con 0x7f7164108580 2024-04-25T02:11:46.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.536+0000 7f7162000640 1 -- 172.21.15.102:0/2165469108 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+23158 (secure 0 0 0) 0x7f71500a02f0 con 0x7f7164108580 2024-04-25T02:11:46.545 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:46.545 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":108,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T02:11:12.458705+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":26,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6801/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/459072323":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/2875896366":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1835499087":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/511834001":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":26,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T02:11:46.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.540+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f714c0a2d80 msgr2=0x7f714c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.540+0000 7f716ac00640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f714c0a2d80 0x7f714c0a5240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7158009c10 tx=0x7f7158005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:46.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.540+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7164108580 msgr2=0x7f716419e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.540+0000 7f716ac00640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7164108580 0x7f716419e910 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f715000e9f0 tx=0x7f715000eeb0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.541+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 shutdown_connections 2024-04-25T02:11:46.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.541+0000 7f716ac00640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f714c0a2d80 0x7f714c0a5240 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.541+0000 7f716ac00640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f716410dc50 0x7f71641a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.541+0000 7f716ac00640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7164108580 0x7f716419e910 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.541+0000 7f716ac00640 1 --2- 172.21.15.102:0/2165469108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7164107be0 0x7f716419e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.541+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 >> 172.21.15.102:0/2165469108 conn(0x7f71640772c0 msgr2=0x7f71640789b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:46.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.541+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 shutdown_connections 2024-04-25T02:11:46.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.541+0000 7f716ac00640 1 -- 172.21.15.102:0/2165469108 wait complete. 2024-04-25T02:11:46.569 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T01:47:31.284676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T01:52:41.585137+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T01:52:43.022666+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 26, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T02:11:46.569 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T02:11:46.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.770+0000 7f095f200640 1 -- 172.21.15.102:0/632645769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0960075c70 msgr2=0x7f0960076050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.770+0000 7f095f200640 1 --2- 172.21.15.102:0/632645769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0960075c70 0x7f0960076050 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0950009900 tx=0x7f095002a3f0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.771+0000 7f095f200640 1 -- 172.21.15.102:0/632645769 shutdown_connections 2024-04-25T02:11:46.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.771+0000 7f095f200640 1 --2- 172.21.15.102:0/632645769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0960112480 0x7f0960114870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.771+0000 7f095f200640 1 --2- 172.21.15.102:0/632645769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0960076590 0x7f0960111f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.771+0000 7f095f200640 1 --2- 172.21.15.102:0/632645769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0960075c70 0x7f0960076050 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.771+0000 7f095f200640 1 -- 172.21.15.102:0/632645769 >> 172.21.15.102:0/632645769 conn(0x7f09600ff250 msgr2=0x7f0960101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:46.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.771+0000 7f095f200640 1 -- 172.21.15.102:0/632645769 shutdown_connections 2024-04-25T02:11:46.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.771+0000 7f095f200640 1 -- 172.21.15.102:0/632645769 wait complete. 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095f200640 1 Processor -- start 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095f200640 1 -- start start 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095f200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0960075c70 0x7f096019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0960076590 0x7f096019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0960112480 0x7f09601a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0960118030 con 0x7f0960112480 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095f200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f0960117eb0 con 0x7f0960075c70 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095f200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f09601181b0 con 0x7f0960076590 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095de00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0960075c70 0x7f096019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0960112480 0x7f09601a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095de00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0960075c70 0x7f096019e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42860/0 (socket says 172.21.15.102:42860) 2024-04-25T02:11:46.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095de00640 1 -- 172.21.15.102:0/1164140264 learned_addr learned my addr 172.21.15.102:0/1164140264 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.772+0000 7f095d400640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0960076590 0x7f096019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f095de00640 1 -- 172.21.15.102:0/1164140264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0960076590 msgr2=0x7f096019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f095de00640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0960076590 0x7f096019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f095de00640 1 -- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0960112480 msgr2=0x7f09601a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f095de00640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0960112480 0x7f09601a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f095de00640 1 -- 172.21.15.102:0/1164140264 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09601a34f0 con 0x7f0960075c70 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f095e800640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0960112480 0x7f09601a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f095de00640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0960075c70 0x7f096019e550 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f095002a9e0 tx=0x7f0950003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f0956a00640 1 -- 172.21.15.102:0/1164140264 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095003f070 con 0x7f0960075c70 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f0956a00640 1 -- 172.21.15.102:0/1164140264 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0950031c30 con 0x7f0960075c70 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.773+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09601a3780 con 0x7f0960075c70 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.774+0000 7f0956a00640 1 -- 172.21.15.102:0/1164140264 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950035030 con 0x7f0960075c70 2024-04-25T02:11:46.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.774+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09601ab070 con 0x7f0960075c70 2024-04-25T02:11:46.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.775+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09600688d0 con 0x7f0960075c70 2024-04-25T02:11:46.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.777+0000 7f0956a00640 1 -- 172.21.15.102:0/1164140264 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0950042070 con 0x7f0960075c70 2024-04-25T02:11:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.778+0000 7f0956a00640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09440a2d00 0x7f09440a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:46.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.778+0000 7f0956a00640 1 -- 172.21.15.102:0/1164140264 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f09500f01b0 con 0x7f0960075c70 2024-04-25T02:11:46.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.779+0000 7f095d400640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09440a2d00 0x7f09440a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:46.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.779+0000 7f095d400640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09440a2d00 0x7f09440a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0960101720 tx=0x7f094c005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:46.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.781+0000 7f0956a00640 1 -- 172.21.15.102:0/1164140264 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0950031de0 con 0x7f0960075c70 2024-04-25T02:11:46.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:46 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2165469108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:11:46.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.984+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f096010d610 con 0x7f0960075c70 2024-04-25T02:11:46.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.985+0000 7f0956a00640 1 -- 172.21.15.102:0/1164140264 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) ==== 93+0+10 (secure 0 0 0) 0x7f09500bc720 con 0x7f0960075c70 2024-04-25T02:11:46.993 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-04-25T02:11:46.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:46 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2165469108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:11:46.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09440a2d00 msgr2=0x7f09440a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09440a2d00 0x7f09440a51c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0960101720 tx=0x7f094c005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0960075c70 msgr2=0x7f096019e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0960075c70 0x7f096019e550 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f095002a9e0 tx=0x7f0950003c10 comp rx=0 tx=0).stop 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 shutdown_connections 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f09440a2d00 0x7f09440a51c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0960112480 0x7f09601a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0960076590 0x7f096019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 --2- 172.21.15.102:0/1164140264 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0960075c70 0x7f096019e550 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 >> 172.21.15.102:0/1164140264 conn(0x7f09600ff250 msgr2=0x7f0960113ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.988+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 shutdown_connections 2024-04-25T02:11:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:46.989+0000 7f095f200640 1 -- 172.21.15.102:0/1164140264 wait complete. 2024-04-25T02:11:47.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:46 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2165469108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:11:47.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T02:11:47.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.210+0000 7f70aa800640 1 -- 172.21.15.102:0/46615027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a4103920 msgr2=0x7f70a4103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.210+0000 7f70aa800640 1 --2- 172.21.15.102:0/46615027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a4103920 0x7f70a4103da0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7090009930 tx=0x7f709002a420 comp rx=0 tx=0).stop 2024-04-25T02:11:47.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.210+0000 7f70aa800640 1 -- 172.21.15.102:0/46615027 shutdown_connections 2024-04-25T02:11:47.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.210+0000 7f70aa800640 1 --2- 172.21.15.102:0/46615027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a41042e0 0x7f70a41107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.210+0000 7f70aa800640 1 --2- 172.21.15.102:0/46615027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a4103920 0x7f70a4103da0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.210+0000 7f70aa800640 1 --2- 172.21.15.102:0/46615027 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4109930 0x7f70a4109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.210+0000 7f70aa800640 1 -- 172.21.15.102:0/46615027 >> 172.21.15.102:0/46615027 conn(0x7f70a40ff3b0 msgr2=0x7f70a41017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:47.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.210+0000 7f70aa800640 1 -- 172.21.15.102:0/46615027 shutdown_connections 2024-04-25T02:11:47.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.210+0000 7f70aa800640 1 -- 172.21.15.102:0/46615027 wait complete. 2024-04-25T02:11:47.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.211+0000 7f70aa800640 1 Processor -- start 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.211+0000 7f70aa800640 1 -- start start 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70aa800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a4103920 0x7f70a419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70aa800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a41042e0 0x7f70a419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70aa800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4109930 0x7f70a419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70aa800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4113d90 con 0x7f70a4103920 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70aa800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4113c10 con 0x7f70a4109930 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70aa800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4113f10 con 0x7f70a41042e0 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a41042e0 0x7f70a419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a3e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4109930 0x7f70a419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a41042e0 0x7f70a419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:48316/0 (socket says 172.21.15.102:48316) 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a4103920 0x7f70a419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a2a00640 1 -- 172.21.15.102:0/4261154156 learned_addr learned my addr 172.21.15.102:0/4261154156 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a3e00640 1 -- 172.21.15.102:0/4261154156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a41042e0 msgr2=0x7f70a419a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a3e00640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a41042e0 0x7f70a419a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a3e00640 1 -- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a4103920 msgr2=0x7f70a419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a3e00640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a4103920 0x7f70a419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a3e00640 1 -- 172.21.15.102:0/4261154156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a419f260 con 0x7f70a4109930 2024-04-25T02:11:47.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.212+0000 7f70a3400640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a4103920 0x7f70a419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:47.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.213+0000 7f70a3e00640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4109930 0x7f70a419eb30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f709c009fd0 tx=0x7f709c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.213+0000 7f709be00640 1 -- 172.21.15.102:0/4261154156 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c00ce20 con 0x7f70a4109930 2024-04-25T02:11:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.213+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a419f550 con 0x7f70a4109930 2024-04-25T02:11:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.213+0000 7f709be00640 1 -- 172.21.15.102:0/4261154156 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f709c005020 con 0x7f70a4109930 2024-04-25T02:11:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.213+0000 7f709be00640 1 -- 172.21.15.102:0/4261154156 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c010720 con 0x7f70a4109930 2024-04-25T02:11:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.213+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a41a6de0 con 0x7f70a4109930 2024-04-25T02:11:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.214+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a4068560 con 0x7f70a4109930 2024-04-25T02:11:47.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.216+0000 7f709be00640 1 -- 172.21.15.102:0/4261154156 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f709c005540 con 0x7f70a4109930 2024-04-25T02:11:47.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.218+0000 7f709be00640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70880a2d00 0x7f70880a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:47.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.218+0000 7f709be00640 1 -- 172.21.15.102:0/4261154156 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f709c0cfa70 con 0x7f70a4109930 2024-04-25T02:11:47.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.218+0000 7f70a3400640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70880a2d00 0x7f70880a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:47.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.219+0000 7f70a3400640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70880a2d00 0x7f70880a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f70940095f0 tx=0x7f7094005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:47.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.220+0000 7f709be00640 1 -- 172.21.15.102:0/4261154156 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f709c09c630 con 0x7f70a4109930 2024-04-25T02:11:47.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.421+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f70a410bef0 con 0x7f70a4109930 2024-04-25T02:11:47.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.422+0000 7f709be00640 1 -- 172.21.15.102:0/4261154156 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) ==== 104+0+11 (secure 0 0 0) 0x7f709c09bf30 con 0x7f70a4109930 2024-04-25T02:11:47.430 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-04-25T02:11:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70880a2d00 msgr2=0x7f70880a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70880a2d00 0x7f70880a51c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f70940095f0 tx=0x7f7094005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4109930 msgr2=0x7f70a419eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4109930 0x7f70a419eb30 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f709c009fd0 tx=0x7f709c00ef10 comp rx=0 tx=0).stop 2024-04-25T02:11:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 shutdown_connections 2024-04-25T02:11:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f70880a2d00 0x7f70880a51c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4109930 0x7f70a419eb30 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a41042e0 0x7f70a419a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 --2- 172.21.15.102:0/4261154156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70a4103920 0x7f70a419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 >> 172.21.15.102:0/4261154156 conn(0x7f70a40ff3b0 msgr2=0x7f70a40ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 shutdown_connections 2024-04-25T02:11:47.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.425+0000 7f70aa800640 1 -- 172.21.15.102:0/4261154156 wait complete. 2024-04-25T02:11:47.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T02:11:47.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.647+0000 7f97abe00640 1 -- 172.21.15.102:0/1572092518 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac074490 msgr2=0x7f97ac10d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.647+0000 7f97abe00640 1 --2- 172.21.15.102:0/1572092518 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac074490 0x7f97ac10d8f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9794009900 tx=0x7f979402a480 comp rx=0 tx=0).stop 2024-04-25T02:11:47.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.647+0000 7f97abe00640 1 -- 172.21.15.102:0/1572092518 shutdown_connections 2024-04-25T02:11:47.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.647+0000 7f97abe00640 1 --2- 172.21.15.102:0/1572092518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ac10de30 0x7f97ac110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.647+0000 7f97abe00640 1 --2- 172.21.15.102:0/1572092518 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac074490 0x7f97ac10d8f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.647+0000 7f97abe00640 1 --2- 172.21.15.102:0/1572092518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97ac073b70 0x7f97ac073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.647+0000 7f97abe00640 1 -- 172.21.15.102:0/1572092518 >> 172.21.15.102:0/1572092518 conn(0x7f97ac0fd0e0 msgr2=0x7f97ac0ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:47.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.647+0000 7f97abe00640 1 -- 172.21.15.102:0/1572092518 shutdown_connections 2024-04-25T02:11:47.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.647+0000 7f97abe00640 1 -- 172.21.15.102:0/1572092518 wait complete. 2024-04-25T02:11:47.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.648+0000 7f97abe00640 1 Processor -- start 2024-04-25T02:11:47.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.648+0000 7f97abe00640 1 -- start start 2024-04-25T02:11:47.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.648+0000 7f97abe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac073b70 0x7f97ac19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:47.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.648+0000 7f97abe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ac074490 0x7f97ac19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:47.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.648+0000 7f97abe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97ac10de30 0x7f97ac19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:47.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97abe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac113c20 con 0x7f97ac073b70 2024-04-25T02:11:47.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97abe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac113aa0 con 0x7f97ac10de30 2024-04-25T02:11:47.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97abe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac113da0 con 0x7f97ac074490 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac073b70 0x7f97ac19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac073b70 0x7f97ac19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48348/0 (socket says 172.21.15.102:48348) 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97aaa00640 1 -- 172.21.15.102:0/2303348939 learned_addr learned my addr 172.21.15.102:0/2303348939 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97a2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ac074490 0x7f97ac19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97ab400640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97ac10de30 0x7f97ac19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97aaa00640 1 -- 172.21.15.102:0/2303348939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ac074490 msgr2=0x7f97ac19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97aaa00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ac074490 0x7f97ac19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97aaa00640 1 -- 172.21.15.102:0/2303348939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97ac10de30 msgr2=0x7f97ac19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97aaa00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97ac10de30 0x7f97ac19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.649+0000 7f97aaa00640 1 -- 172.21.15.102:0/2303348939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ac19f100 con 0x7f97ac073b70 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.650+0000 7f97aaa00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac073b70 0x7f97ac19a160 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f979c00ebc0 tx=0x7f979c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.650+0000 7f97a3e00640 1 -- 172.21.15.102:0/2303348939 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c010630 con 0x7f97ac073b70 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.650+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ac19f3f0 con 0x7f97ac073b70 2024-04-25T02:11:47.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.650+0000 7f97a3e00640 1 -- 172.21.15.102:0/2303348939 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f979c010cb0 con 0x7f97ac073b70 2024-04-25T02:11:47.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.650+0000 7f97a3e00640 1 -- 172.21.15.102:0/2303348939 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c018bd0 con 0x7f97ac073b70 2024-04-25T02:11:47.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.650+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ac1a6c80 con 0x7f97ac073b70 2024-04-25T02:11:47.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.651+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97ac0688d0 con 0x7f97ac073b70 2024-04-25T02:11:47.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.653+0000 7f97a3e00640 1 -- 172.21.15.102:0/2303348939 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f979c0107d0 con 0x7f97ac073b70 2024-04-25T02:11:47.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.655+0000 7f97a3e00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f97900a2d00 0x7f97900a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:47.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.655+0000 7f97a3e00640 1 -- 172.21.15.102:0/2303348939 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f979c0cfad0 con 0x7f97ac073b70 2024-04-25T02:11:47.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.656+0000 7f97a2000640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f97900a2d00 0x7f97900a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:47.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.656+0000 7f97a2000640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f97900a2d00 0x7f97900a51c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9794000c00 tx=0x7f9794005b50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:47.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.660+0000 7f97a3e00640 1 -- 172.21.15.102:0/2303348939 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f979c09c040 con 0x7f97ac073b70 2024-04-25T02:11:47.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.865+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f97ac109000 con 0x7f97ac073b70 2024-04-25T02:11:47.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.866+0000 7f97a3e00640 1 -- 172.21.15.102:0/2303348939 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) ==== 100+0+11 (secure 0 0 0) 0x7f97ac109000 con 0x7f97ac073b70 2024-04-25T02:11:47.874 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-04-25T02:11:47.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.868+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f97900a2d00 msgr2=0x7f97900a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.868+0000 7f97abe00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f97900a2d00 0x7f97900a51c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9794000c00 tx=0x7f9794005b50 comp rx=0 tx=0).stop 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.868+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac073b70 msgr2=0x7f97ac19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.868+0000 7f97abe00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac073b70 0x7f97ac19a160 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f979c00ebc0 tx=0x7f979c00c6a0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.869+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 shutdown_connections 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.869+0000 7f97abe00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f97900a2d00 0x7f97900a51c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.869+0000 7f97abe00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97ac10de30 0x7f97ac19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.869+0000 7f97abe00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97ac074490 0x7f97ac19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.869+0000 7f97abe00640 1 --2- 172.21.15.102:0/2303348939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97ac073b70 0x7f97ac19a160 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.869+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 >> 172.21.15.102:0/2303348939 conn(0x7f97ac0fd0e0 msgr2=0x7f97ac0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.869+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 shutdown_connections 2024-04-25T02:11:47.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:47.869+0000 7f97abe00640 1 -- 172.21.15.102:0/2303348939 wait complete. 2024-04-25T02:11:47.894 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:47 smithi102 ceph-mon[27796]: pgmap v516: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-25T02:11:47.894 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:47 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1164140264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:11:47.894 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:47 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/4261154156' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:11:47.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T02:11:47.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:47 smithi110 ceph-mon[32848]: pgmap v516: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-25T02:11:47.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:47 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1164140264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:11:47.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:47 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/4261154156' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:11:48.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:47 smithi193 ceph-mon[32981]: pgmap v516: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 8 op/s 2024-04-25T02:11:48.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:47 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1164140264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T02:11:48.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:47 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/4261154156' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T02:11:48.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.087+0000 7f6877000640 1 -- 172.21.15.102:0/138151808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6870109800 msgr2=0x7f6870109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:48.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.087+0000 7f6877000640 1 --2- 172.21.15.102:0/138151808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6870109800 0x7f6870109be0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6864009900 tx=0x7f686402a3f0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.087+0000 7f6877000640 1 -- 172.21.15.102:0/138151808 shutdown_connections 2024-04-25T02:11:48.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.087+0000 7f6877000640 1 --2- 172.21.15.102:0/138151808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68701041b0 0x7f68701106c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.087+0000 7f6877000640 1 --2- 172.21.15.102:0/138151808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68701037f0 0x7f6870103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.087+0000 7f6877000640 1 --2- 172.21.15.102:0/138151808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6870109800 0x7f6870109be0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.087+0000 7f6877000640 1 -- 172.21.15.102:0/138151808 >> 172.21.15.102:0/138151808 conn(0x7f68700ff280 msgr2=0x7f68701016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:48.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.087+0000 7f6877000640 1 -- 172.21.15.102:0/138151808 shutdown_connections 2024-04-25T02:11:48.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.087+0000 7f6877000640 1 -- 172.21.15.102:0/138151808 wait complete. 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.088+0000 7f6877000640 1 Processor -- start 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.088+0000 7f6877000640 1 -- start start 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.088+0000 7f6877000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68701037f0 0x7f687019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.088+0000 7f6877000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68701041b0 0x7f687019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.088+0000 7f6877000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6870109800 0x7f687019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.088+0000 7f6877000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6870113c10 con 0x7f68701037f0 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.088+0000 7f6877000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f6870113a90 con 0x7f6870109800 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.088+0000 7f6877000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6870113d90 con 0x7f68701041b0 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.088+0000 7f6875c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68701037f0 0x7f687019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6875c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68701037f0 0x7f687019a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48378/0 (socket says 172.21.15.102:48378) 2024-04-25T02:11:48.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6875c00640 1 -- 172.21.15.102:0/3872721436 learned_addr learned my addr 172.21.15.102:0/3872721436 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6876600640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6870109800 0x7f687019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6875200640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68701041b0 0x7f687019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6875c00640 1 -- 172.21.15.102:0/3872721436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68701041b0 msgr2=0x7f687019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6875c00640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68701041b0 0x7f687019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6875c00640 1 -- 172.21.15.102:0/3872721436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6870109800 msgr2=0x7f687019e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6875c00640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6870109800 0x7f687019e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6875c00640 1 -- 172.21.15.102:0/3872721436 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687019f040 con 0x7f68701037f0 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.089+0000 7f6875c00640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68701037f0 0x7f687019a050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6864006010 tx=0x7f6864003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.090+0000 7f686e000640 1 -- 172.21.15.102:0/3872721436 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686403f070 con 0x7f68701037f0 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.090+0000 7f686e000640 1 -- 172.21.15.102:0/3872721436 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6864031c30 con 0x7f68701037f0 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.090+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f687019f2d0 con 0x7f68701037f0 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.090+0000 7f686e000640 1 -- 172.21.15.102:0/3872721436 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6864035030 con 0x7f68701037f0 2024-04-25T02:11:48.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.090+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68701a6bc0 con 0x7f68701037f0 2024-04-25T02:11:48.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.091+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68700688d0 con 0x7f68701037f0 2024-04-25T02:11:48.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.093+0000 7f686e000640 1 -- 172.21.15.102:0/3872721436 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6864042070 con 0x7f68701037f0 2024-04-25T02:11:48.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.095+0000 7f686e000640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f68580a2d00 0x7f68580a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:48.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.096+0000 7f686e000640 1 -- 172.21.15.102:0/3872721436 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f68640ef770 con 0x7f68701037f0 2024-04-25T02:11:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.096+0000 7f6875200640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f68580a2d00 0x7f68580a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:48.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.096+0000 7f6875200640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f68580a2d00 0x7f68580a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f685c0095f0 tx=0x7f685c005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:48.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.101+0000 7f686e000640 1 -- 172.21.15.102:0/3872721436 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6864031de0 con 0x7f68701037f0 2024-04-25T02:11:48.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.317+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f687010bd60 con 0x7f68701037f0 2024-04-25T02:11:48.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.318+0000 7f686e000640 1 -- 172.21.15.102:0/3872721436 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v108) ==== 81+0+954 (secure 0 0 0) 0x7f68640bbce0 con 0x7f68701037f0 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/83904525 2024-04-26T01:55:15.567762+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/459072323 2024-04-26T01:55:15.567762+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/83904525 2024-04-26T01:55:15.567762+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2875896366 2024-04-26T01:55:15.567762+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/3113170519 2024-04-26T01:44:42.581235+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/1124203809 2024-04-26T01:44:42.581235+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/1124203809 2024-04-26T01:44:42.581235+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1835499087 2024-04-26T01:55:15.567762+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6801/2757023783 2024-04-26T01:44:08.232557+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/511834001 2024-04-26T01:55:15.567762+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:6800/2757023783 2024-04-26T01:44:08.232557+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/411628693 2024-04-26T01:44:42.581235+0000 2024-04-25T02:11:48.326 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/1716050795 2024-04-26T01:44:08.232557+0000 2024-04-25T02:11:48.327 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/2344465952 2024-04-26T01:44:42.581235+0000 2024-04-25T02:11:48.327 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/4034021546 2024-04-26T01:44:08.232557+0000 2024-04-25T02:11:48.327 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102:0/316635575 2024-04-26T01:44:08.232557+0000 2024-04-25T02:11:48.329 INFO:teuthology.orchestra.run.smithi102.stderr:listed 16 entries 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f68580a2d00 msgr2=0x7f68580a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f68580a2d00 0x7f68580a51c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f685c0095f0 tx=0x7f685c005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68701037f0 msgr2=0x7f687019a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68701037f0 0x7f687019a050 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6864006010 tx=0x7f6864003c10 comp rx=0 tx=0).stop 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 shutdown_connections 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f68580a2d00 0x7f68580a51c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6870109800 0x7f687019e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68701041b0 0x7f687019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 --2- 172.21.15.102:0/3872721436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f68701037f0 0x7f687019a050 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.321+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 >> 172.21.15.102:0/3872721436 conn(0x7f68700ff280 msgr2=0x7f68700ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.322+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 shutdown_connections 2024-04-25T02:11:48.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:48.322+0000 7f6877000640 1 -- 172.21.15.102:0/3872721436 wait complete. 2024-04-25T02:11:48.348 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T02:11:48.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:11:48.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /tmp/cephfs 2024-04-25T02:11:48.717 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T02:11:48.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:11:48.717 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-04-25T02:11:48.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:48 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2303348939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:11:48.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:48 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3872721436' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T02:11:48.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:11:48.812 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-04-25T02:11:48.890 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-04-25T02:11:48.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:11:48.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-04-25T02:11:48.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:48 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2303348939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:11:48.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:48 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3872721436' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T02:11:48.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:48 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2303348939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T02:11:48.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:48 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3872721436' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T02:11:49.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:11:49.008 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-04-25T02:11:49.210 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T02:11:49.210 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T02:11:49.210 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T02:11:49.286 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052731,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/a0bb9eebb358e2065de62ea3abf6b0c921b31e8f2f01fc9755e4ee96ab9c6ad1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/a0bb9eebb358e2065de62ea3abf6b0c921b31e8f2f01fc9755e4ee96ab9c6ad1/diff,workdir=/var/lib/containers/storage/overlay/a0bb9eebb358e2065de62ea3abf6b0c921b31e8f2f01fc9755e4ee96ab9c6ad1/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f42eb45711db6c114cd24ef22fb3ba0eccec3132b82833a2a7368e0380a84884/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/f42eb45711db6c114cd24ef22fb3ba0eccec3132b82833a2a7368e0380a84884/diff,workdir=/var/lib/containers/storage/overlay/f42eb45711db6c114cd24ef22fb3ba0eccec3132b82833a2a7368e0380a84884/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248996k,nr_inodes=812249,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/b096d31276fb6ab0f0660a7559672f151801db3155e26cf73664dd39cc0ae000/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/b096d31276fb6ab0f0660a7559672f151801db3155e26cf73664dd39cc0ae000/diff,workdir=/var/lib/containers/storage/overlay/b096d31276fb6ab0f0660a7559672f151801db3155e26cf73664dd39cc0ae000/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b8417f7ff474397314b36a09ffc2116e63ad57be25c2cd6cf4bd8f680a9a4722/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/b8417f7ff474397314b36a09ffc2116e63ad57be25c2cd6cf4bd8f680a9a4722/diff,workdir=/var/lib/containers/storage/overlay/b8417f7ff474397314b36a09ffc2116e63ad57be25c2cd6cf4bd8f680a9a4722/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e2b6e11696d7e9922308cf8c327b9e1c388791c212b12bdba7fa373a1f0ae9a2/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/e2b6e11696d7e9922308cf8c327b9e1c388791c212b12bdba7fa373a1f0ae9a2/diff,workdir=/var/lib/containers/storage/overlay/e2b6e11696d7e9922308cf8c327b9e1c388791c212b12bdba7fa373a1f0ae9a2/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b716ec00bc34d9b2a30721ca2722c5d04c8a458f2e023a035f1a7228f5919914/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/b716ec00bc34d9b2a30721ca2722c5d04c8a458f2e023a035f1a7228f5919914/diff,workdir=/var/lib/containers/storage/overlay/b716ec00bc34d9b2a30721ca2722c5d04c8a458f2e023a035f1a7228f5919914/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/186b020356a282fb5f11608c746ba9463c191c7a9525124ac2c9c82d5f20a47e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/186b020356a282fb5f11608c746ba9463c191c7a9525124ac2c9c82d5f20a47e/diff,workdir=/var/lib/containers/storage/overlay/186b020356a282fb5f11608c746ba9463c191c7a9525124ac2c9c82d5f20a47e/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/088796a3c07f127933b99ed572f0c87e1db0cde482fbba69780e4293f6a49b69/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/088796a3c07f127933b99ed572f0c87e1db0cde482fbba69780e4293f6a49b69/diff,workdir=/var/lib/containers/storage/overlay/088796a3c07f127933b99ed572f0c87e1db0cde482fbba69780e4293f6a49b69/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/bb00fb7951c3971d58f7c3dac0ba26f5ebc9e83eb5dcb567d9b92b9a175b69ad/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/bb00fb7951c3971d58f7c3dac0ba26f5ebc9e83eb5dcb567d9b92b9a175b69ad/diff,workdir=/var/lib/containers/storage/overlay/bb00fb7951c3971d58f7c3dac0ba26f5ebc9e83eb5dcb567d9b92b9a175b69ad/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/94c2cb6426eeb3e94a2c0b223f85c11ab3753de7cb7f9820f07fd58f562a86d9/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/75W6Y6AXX6E7CGU5FSCHLYIQLM:/var/lib/containers/storage/overlay/l/X34ZA7FAM4UMXHAKO4IUBEREG6,upperdir=/var/lib/containers/storage/overlay/94c2cb6426eeb3e94a2c0b223f85c11ab3753de7cb7f9820f07fd58f562a86d9/diff,workdir=/var/lib/containers/storage/overlay/94c2cb6426eeb3e94a2c0b223f85c11ab3753de7cb7f9820f07fd58f562a86d9/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498000k,nr_inodes=819200,mode=755 0 0 2024-04-25T02:11:49.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:11:49.286 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-04-25T02:11:49.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T02:11:49.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-04-25T02:11:49.430 INFO:teuthology.orchestra.run.smithi102.stderr:Cannot find device "ceph-brx" 2024-04-25T02:11:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:11:49.433 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T02:11:49.444 INFO:tasks.cephadm:Teardown begin 2024-04-25T02:11:49.444 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T02:11:49.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T02:11:49.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.599+0000 7f187ce00640 1 -- 172.21.15.102:0/949121829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f187810ced0 msgr2=0x7f1878114aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:49.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.599+0000 7f187ce00640 1 --2- 172.21.15.102:0/949121829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f187810ced0 0x7f1878114aa0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f1870009ca0 tx=0x7f187002a8c0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.599+0000 7f187ce00640 1 -- 172.21.15.102:0/949121829 shutdown_connections 2024-04-25T02:11:49.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.599+0000 7f187ce00640 1 --2- 172.21.15.102:0/949121829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f187810ced0 0x7f1878114aa0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.599+0000 7f187ce00640 1 --2- 172.21.15.102:0/949121829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18781088a0 0x7f187810c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.599+0000 7f187ce00640 1 --2- 172.21.15.102:0/949121829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1878107ef0 0x7f18781082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.599+0000 7f187ce00640 1 -- 172.21.15.102:0/949121829 >> 172.21.15.102:0/949121829 conn(0x7f1878077560 msgr2=0x7f1878106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:49.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.600+0000 7f187ce00640 1 -- 172.21.15.102:0/949121829 shutdown_connections 2024-04-25T02:11:49.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.600+0000 7f187ce00640 1 -- 172.21.15.102:0/949121829 wait complete. 2024-04-25T02:11:49.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.600+0000 7f187ce00640 1 Processor -- start 2024-04-25T02:11:49.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f187ce00640 1 -- start start 2024-04-25T02:11:49.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f187ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1878107ef0 0x7f187819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:49.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f187ce00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f187810ced0 0x7f187819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:49.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f187ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18781a2cd0 0x7f18781a3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f187ce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1878117ff0 con 0x7f1878107ef0 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f187ce00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f1878117e70 con 0x7f187810ced0 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f187ce00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1878118170 con 0x7f18781a2cd0 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f1877400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1878107ef0 0x7f187819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f1877400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1878107ef0 0x7f187819e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48390/0 (socket says 172.21.15.102:48390) 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f1877400640 1 -- 172.21.15.102:0/741255434 learned_addr learned my addr 172.21.15.102:0/741255434 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f1876a00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f187810ced0 0x7f187819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.601+0000 7f1877e00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18781a2cd0 0x7f18781a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f1877400640 1 -- 172.21.15.102:0/741255434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18781a2cd0 msgr2=0x7f18781a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f1877400640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18781a2cd0 0x7f18781a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f1877400640 1 -- 172.21.15.102:0/741255434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f187810ced0 msgr2=0x7f187819e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f1877400640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f187810ced0 0x7f187819e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f1877400640 1 -- 172.21.15.102:0/741255434 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18781a37c0 con 0x7f1878107ef0 2024-04-25T02:11:49.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f1877400640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1878107ef0 0x7f187819e410 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f186800e9f0 tx=0x7f186800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:49.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f186fe00640 1 -- 172.21.15.102:0/741255434 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186800cd80 con 0x7f1878107ef0 2024-04-25T02:11:49.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18781a3ab0 con 0x7f1878107ef0 2024-04-25T02:11:49.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f186fe00640 1 -- 172.21.15.102:0/741255434 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1868005500 con 0x7f1878107ef0 2024-04-25T02:11:49.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f186fe00640 1 -- 172.21.15.102:0/741255434 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1868018630 con 0x7f1878107ef0 2024-04-25T02:11:49.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.602+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18781ab1f0 con 0x7f1878107ef0 2024-04-25T02:11:49.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.603+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1878068480 con 0x7f1878107ef0 2024-04-25T02:11:49.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.605+0000 7f186fe00640 1 -- 172.21.15.102:0/741255434 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f18680046c0 con 0x7f1878107ef0 2024-04-25T02:11:49.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.606+0000 7f186fe00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f185c0a2d00 0x7f185c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:49.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.607+0000 7f186fe00640 1 -- 172.21.15.102:0/741255434 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f18680cf990 con 0x7f1878107ef0 2024-04-25T02:11:49.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.607+0000 7f1876a00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f185c0a2d00 0x7f185c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:49.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.607+0000 7f1876a00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f185c0a2d00 0x7f185c0a51c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f18640095f0 tx=0x7f1864005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:49.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.610+0000 7f186fe00640 1 -- 172.21.15.102:0/741255434 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f186809c550 con 0x7f1878107ef0 2024-04-25T02:11:49.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.834+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1878108ab0 con 0x7f1878107ef0 2024-04-25T02:11:49.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.836+0000 7f186fe00640 1 -- 172.21.15.102:0/741255434 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11907 (secure 0 0 0) 0x7f1878108ab0 con 0x7f1878107ef0 2024-04-25T02:11:49.844 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:49.844 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T02:11:20.955431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":141,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587,24533,24541]},"id":1}]} 2024-04-25T02:11:49.844 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 56 2024-04-25T02:11:49.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f185c0a2d00 msgr2=0x7f185c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:49.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f185c0a2d00 0x7f185c0a51c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f18640095f0 tx=0x7f1864005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:49.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1878107ef0 msgr2=0x7f187819e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:49.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1878107ef0 0x7f187819e410 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f186800e9f0 tx=0x7f186800edf0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 shutdown_connections 2024-04-25T02:11:49.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f185c0a2d00 0x7f185c0a51c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18781a2cd0 0x7f18781a3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f187810ced0 0x7f187819e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 --2- 172.21.15.102:0/741255434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1878107ef0 0x7f187819e410 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:49.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.840+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 >> 172.21.15.102:0/741255434 conn(0x7f1878077560 msgr2=0x7f1878077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:49.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.841+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 shutdown_connections 2024-04-25T02:11:49.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:49.841+0000 7f187ce00640 1 -- 172.21.15.102:0/741255434 wait complete. 2024-04-25T02:11:49.863 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:49 smithi102 ceph-mon[27796]: pgmap v517: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 10 op/s 2024-04-25T02:11:49.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T02:11:49.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:49 smithi110 ceph-mon[32848]: pgmap v517: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 10 op/s 2024-04-25T02:11:50.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:49 smithi193 ceph-mon[32981]: pgmap v517: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 10 op/s 2024-04-25T02:11:50.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.041+0000 7f33bfe00640 1 -- 172.21.15.102:0/3326101741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 msgr2=0x7f33c010e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.041+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3326101741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 0x7f33c010e030 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f33b0009900 tx=0x7f33b002a480 comp rx=0 tx=0).stop 2024-04-25T02:11:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.041+0000 7f33bfe00640 1 -- 172.21.15.102:0/3326101741 shutdown_connections 2024-04-25T02:11:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.041+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3326101741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c0108580 0x7f33c01149f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.041+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3326101741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c0107be0 0x7f33c0108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.041+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3326101741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 0x7f33c010e030 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.041+0000 7f33bfe00640 1 -- 172.21.15.102:0/3326101741 >> 172.21.15.102:0/3326101741 conn(0x7f33c00772c0 msgr2=0x7f33c00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.041+0000 7f33bfe00640 1 -- 172.21.15.102:0/3326101741 shutdown_connections 2024-04-25T02:11:50.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.041+0000 7f33bfe00640 1 -- 172.21.15.102:0/3326101741 wait complete. 2024-04-25T02:11:50.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.042+0000 7f33bfe00640 1 Processor -- start 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.042+0000 7f33bfe00640 1 -- start start 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c0107be0 0x7f33c019e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bfe00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c0108580 0x7f33c019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 0x7f33c01a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bfe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f33c0117fc0 con 0x7f33c010dc50 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bfe00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f33c0117e40 con 0x7f33c0108580 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bfe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f33c0118140 con 0x7f33c0107be0 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bf400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 0x7f33c01a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c0107be0 0x7f33c019e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bf400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 0x7f33c01a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48406/0 (socket says 172.21.15.102:48406) 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bf400640 1 -- 172.21.15.102:0/3767857317 learned_addr learned my addr 172.21.15.102:0/3767857317 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:50.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33be000640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c0108580 0x7f33c019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bf400640 1 -- 172.21.15.102:0/3767857317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c0107be0 msgr2=0x7f33c019e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bf400640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c0107be0 0x7f33c019e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bf400640 1 -- 172.21.15.102:0/3767857317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c0108580 msgr2=0x7f33c019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bf400640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c0108580 0x7f33c019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33bf400640 1 -- 172.21.15.102:0/3767857317 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c01a3410 con 0x7f33c010dc50 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.043+0000 7f33be000640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c0108580 0x7f33c019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.044+0000 7f33bf400640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 0x7f33c01a2ce0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f33b800d950 tx=0x7f33b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.044+0000 7f33b7400640 1 -- 172.21.15.102:0/3767857317 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8017030 con 0x7f33c010dc50 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.044+0000 7f33b7400640 1 -- 172.21.15.102:0/3767857317 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33b8004020 con 0x7f33c010dc50 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.044+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c01a3700 con 0x7f33c010dc50 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.044+0000 7f33b7400640 1 -- 172.21.15.102:0/3767857317 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8003c00 con 0x7f33c010dc50 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.044+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c01aaf90 con 0x7f33c010dc50 2024-04-25T02:11:50.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.045+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33c00688d0 con 0x7f33c010dc50 2024-04-25T02:11:50.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.047+0000 7f33b7400640 1 -- 172.21.15.102:0/3767857317 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f33b801a070 con 0x7f33c010dc50 2024-04-25T02:11:50.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.048+0000 7f33b7400640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f33a40a2d00 0x7f33a40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.048+0000 7f33b7400640 1 -- 172.21.15.102:0/3767857317 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7f33b80cecd0 con 0x7f33c010dc50 2024-04-25T02:11:50.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.049+0000 7f33bea00640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f33a40a2d00 0x7f33a40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.049+0000 7f33bea00640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f33a40a2d00 0x7f33a40a51c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f33b0000c00 tx=0x7f33b0005b50 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:50.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.052+0000 7f33b7400640 1 -- 172.21.15.102:0/3767857317 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f33b8021070 con 0x7f33c010dc50 2024-04-25T02:11:50.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.277+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33c002d170 con 0x7f33c010dc50 2024-04-25T02:11:50.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.278+0000 7f33b7400640 1 -- 172.21.15.102:0/3767857317 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11907 (secure 0 0 0) 0x7f33b809b890 con 0x7f33c010dc50 2024-04-25T02:11:50.286 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:50.286 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14394,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2866248981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2866248981},{"type":"v1","addr":"172.21.15.102:6835","nonce":2866248981}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3271665645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3271665645},{"type":"v1","addr":"172.21.15.102:6837","nonce":3271665645}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14457,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/4134663811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":4134663811},{"type":"v1","addr":"172.21.15.102:6839","nonce":4134663811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14475,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6841/3724889216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6840","nonce":3724889216},{"type":"v1","addr":"172.21.15.102:6841","nonce":3724889216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3303248011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3303248011},{"type":"v1","addr":"172.21.15.193:6833","nonce":3303248011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/3320624567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3320624567},{"type":"v1","addr":"172.21.15.110:6833","nonce":3320624567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T01:52:43.526145+0000","modified":"2024-04-25T02:11:20.955431+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24587,"mds_1":24541,"mds_2":24533},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4057730311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4057730311},{"type":"v1","addr":"172.21.15.110:6835","nonce":4057730311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/1885428891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":1885428891},{"type":"v1","addr":"172.21.15.193:6835","nonce":1885428891}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/3613019309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":3613019309},{"type":"v1","addr":"172.21.15.193:6837","nonce":3613019309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.110:6837/1533256041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1533256041},{"type":"v1","addr":"172.21.15.110:6837","nonce":1533256041}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":141,"addr":"172.21.15.193:6839/2484219231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2484219231},{"type":"v1","addr":"172.21.15.193:6839","nonce":2484219231}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24587":{"gid":24587,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6839/4063392981","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":4063392981},{"type":"v1","addr":"172.21.15.110:6839","nonce":4063392981}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24587,"qdb_cluster":[24587,24533,24541]},"id":1}]} 2024-04-25T02:11:50.286 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 56 2024-04-25T02:11:50.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f33a40a2d00 msgr2=0x7f33a40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f33a40a2d00 0x7f33a40a51c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f33b0000c00 tx=0x7f33b0005b50 comp rx=0 tx=0).stop 2024-04-25T02:11:50.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 msgr2=0x7f33c01a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 0x7f33c01a2ce0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f33b800d950 tx=0x7f33b800de10 comp rx=0 tx=0).stop 2024-04-25T02:11:50.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 shutdown_connections 2024-04-25T02:11:50.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f33a40a2d00 0x7f33a40a51c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c010dc50 0x7f33c01a2ce0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c0108580 0x7f33c019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 --2- 172.21.15.102:0/3767857317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33c0107be0 0x7f33c019e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.281+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 >> 172.21.15.102:0/3767857317 conn(0x7f33c00772c0 msgr2=0x7f33c00789c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:50.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.282+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 shutdown_connections 2024-04-25T02:11:50.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.282+0000 7f33bfe00640 1 -- 172.21.15.102:0/3767857317 wait complete. 2024-04-25T02:11:50.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T02:11:50.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.493+0000 7fe937200640 1 -- 172.21.15.102:0/4203665915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe938106480 msgr2=0x7fe9380688d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.493+0000 7fe937200640 1 --2- 172.21.15.102:0/4203665915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe938106480 0x7fe9380688d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe928009900 tx=0x7fe92802a410 comp rx=0 tx=0).stop 2024-04-25T02:11:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.493+0000 7fe937200640 1 -- 172.21.15.102:0/4203665915 shutdown_connections 2024-04-25T02:11:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.493+0000 7fe937200640 1 --2- 172.21.15.102:0/4203665915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938103fd0 0x7fe938104480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.493+0000 7fe937200640 1 --2- 172.21.15.102:0/4203665915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9381068f0 0x7fe938068e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.493+0000 7fe937200640 1 --2- 172.21.15.102:0/4203665915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe938106480 0x7fe9380688d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.493+0000 7fe937200640 1 -- 172.21.15.102:0/4203665915 >> 172.21.15.102:0/4203665915 conn(0x7fe938075f80 msgr2=0x7fe938076390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.493+0000 7fe937200640 1 -- 172.21.15.102:0/4203665915 shutdown_connections 2024-04-25T02:11:50.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.493+0000 7fe937200640 1 -- 172.21.15.102:0/4203665915 wait complete. 2024-04-25T02:11:50.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.494+0000 7fe937200640 1 Processor -- start 2024-04-25T02:11:50.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe937200640 1 -- start start 2024-04-25T02:11:50.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe937200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938103fd0 0x7fe93819e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe937200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe938106480 0x7fe93819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe937200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9381068f0 0x7fe9381a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe937200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe938117eb0 con 0x7fe9381068f0 2024-04-25T02:11:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe937200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7fe938117d30 con 0x7fe938103fd0 2024-04-25T02:11:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe937200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe938118030 con 0x7fe938106480 2024-04-25T02:11:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe935e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938103fd0 0x7fe93819e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe935e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938103fd0 0x7fe93819e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42956/0 (socket says 172.21.15.102:42956) 2024-04-25T02:11:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe936800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9381068f0 0x7fe9381a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe935e00640 1 -- 172.21.15.102:0/2274210068 learned_addr learned my addr 172.21.15.102:0/2274210068 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:50.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.495+0000 7fe92d400640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe938106480 0x7fe93819e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.496+0000 7fe935e00640 1 -- 172.21.15.102:0/2274210068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe938106480 msgr2=0x7fe93819e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.496+0000 7fe935e00640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe938106480 0x7fe93819e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.496+0000 7fe935e00640 1 -- 172.21.15.102:0/2274210068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9381068f0 msgr2=0x7fe9381a2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.496+0000 7fe935e00640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9381068f0 0x7fe9381a2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.496+0000 7fe935e00640 1 -- 172.21.15.102:0/2274210068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9381a33a0 con 0x7fe938103fd0 2024-04-25T02:11:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.496+0000 7fe92d400640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe938106480 0x7fe93819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.496+0000 7fe935e00640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938103fd0 0x7fe93819e3b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe928004770 tx=0x7fe928003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.497+0000 7fe92f400640 1 -- 172.21.15.102:0/2274210068 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe92803f070 con 0x7fe938103fd0 2024-04-25T02:11:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.497+0000 7fe92f400640 1 -- 172.21.15.102:0/2274210068 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe928031ca0 con 0x7fe938103fd0 2024-04-25T02:11:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.497+0000 7fe92f400640 1 -- 172.21.15.102:0/2274210068 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe928035030 con 0x7fe938103fd0 2024-04-25T02:11:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.497+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9381a3690 con 0x7fe938103fd0 2024-04-25T02:11:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.497+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9381aaf20 con 0x7fe938103fd0 2024-04-25T02:11:50.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.498+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe938068480 con 0x7fe938103fd0 2024-04-25T02:11:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.500+0000 7fe92f400640 1 -- 172.21.15.102:0/2274210068 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe928042070 con 0x7fe938103fd0 2024-04-25T02:11:50.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.501+0000 7fe92f400640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe91c0a2d80 0x7fe91c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.501+0000 7fe92f400640 1 -- 172.21.15.102:0/2274210068 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7fe9280efc20 con 0x7fe938103fd0 2024-04-25T02:11:50.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.501+0000 7fe92d400640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe91c0a2d80 0x7fe91c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.502+0000 7fe92d400640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe91c0a2d80 0x7fe91c0a5240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe938076790 tx=0x7fe92000b410 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:50.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.503+0000 7fe92f400640 1 -- 172.21.15.102:0/2274210068 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe9280bc190 con 0x7fe938103fd0 2024-04-25T02:11:50.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.696+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9381ab220 con 0x7fe938103fd0 2024-04-25T02:11:50.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.698+0000 7fe92f400640 1 -- 172.21.15.102:0/2274210068 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+23158 (secure 0 0 0) 0x7fe9280c0490 con 0x7fe938103fd0 2024-04-25T02:11:50.706 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-04-25T02:11:50.706 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":108,"fsid":"145ca8d0-02a5-11ef-bc93-c7b262605968","created":"2024-04-25T01:43:34.661160+0000","modified":"2024-04-25T02:11:12.458705+0000","last_up_change":"2024-04-25T01:51:42.254833+0000","last_in_change":"2024-04-25T01:51:21.715540+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T01:47:31.284676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T01:52:41.585137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T01:52:43.022666+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":26,"snap_epoch":107,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"caf505e1-b2d5-43da-848f-89b616616084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6803","nonce":417549417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6805","nonce":417549417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6809","nonce":417549417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":417549417},{"type":"v1","addr":"172.21.15.102:6807","nonce":417549417}]},"public_addr":"172.21.15.102:6803/417549417","cluster_addr":"172.21.15.102:6805/417549417","heartbeat_back_addr":"172.21.15.102:6809/417549417","heartbeat_front_addr":"172.21.15.102:6807/417549417","state":["exists","up"]},{"osd":1,"uuid":"437ccdd6-3c94-47c1-9350-b252f8aec22f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6801","nonce":2162810099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6803","nonce":2162810099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6807","nonce":2162810099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":2162810099},{"type":"v1","addr":"172.21.15.110:6805","nonce":2162810099}]},"public_addr":"172.21.15.110:6801/2162810099","cluster_addr":"172.21.15.110:6803/2162810099","heartbeat_back_addr":"172.21.15.110:6807/2162810099","heartbeat_front_addr":"172.21.15.110:6805/2162810099","state":["exists","up"]},{"osd":2,"uuid":"c730312e-3b4f-4839-bcc4-587ad4c5c576","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6801","nonce":602865964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6803","nonce":602865964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6807","nonce":602865964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":602865964},{"type":"v1","addr":"172.21.15.193:6805","nonce":602865964}]},"public_addr":"172.21.15.193:6801/602865964","cluster_addr":"172.21.15.193:6803/602865964","heartbeat_back_addr":"172.21.15.193:6807/602865964","heartbeat_front_addr":"172.21.15.193:6805/602865964","state":["exists","up"]},{"osd":3,"uuid":"240b29df-2fe9-4354-83c2-6a319aecefac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6811","nonce":3993205287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6813","nonce":3993205287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6817","nonce":3993205287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3993205287},{"type":"v1","addr":"172.21.15.102:6815","nonce":3993205287}]},"public_addr":"172.21.15.102:6811/3993205287","cluster_addr":"172.21.15.102:6813/3993205287","heartbeat_back_addr":"172.21.15.102:6817/3993205287","heartbeat_front_addr":"172.21.15.102:6815/3993205287","state":["exists","up"]},{"osd":4,"uuid":"63705793-8fc3-4a74-80d0-41e70f84c5bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6809","nonce":4200899007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6811","nonce":4200899007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6815","nonce":4200899007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":4200899007},{"type":"v1","addr":"172.21.15.110:6813","nonce":4200899007}]},"public_addr":"172.21.15.110:6809/4200899007","cluster_addr":"172.21.15.110:6811/4200899007","heartbeat_back_addr":"172.21.15.110:6815/4200899007","heartbeat_front_addr":"172.21.15.110:6813/4200899007","state":["exists","up"]},{"osd":5,"uuid":"8109f99d-f1f9-424a-8d2e-d1ddb0626f0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6809","nonce":3543273137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6811","nonce":3543273137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6815","nonce":3543273137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3543273137},{"type":"v1","addr":"172.21.15.193:6813","nonce":3543273137}]},"public_addr":"172.21.15.193:6809/3543273137","cluster_addr":"172.21.15.193:6811/3543273137","heartbeat_back_addr":"172.21.15.193:6815/3543273137","heartbeat_front_addr":"172.21.15.193:6813/3543273137","state":["exists","up"]},{"osd":6,"uuid":"92fb32af-539d-474a-87ff-2305e626f89a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6819","nonce":2856826689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6821","nonce":2856826689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6825","nonce":2856826689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":2856826689},{"type":"v1","addr":"172.21.15.102:6823","nonce":2856826689}]},"public_addr":"172.21.15.102:6819/2856826689","cluster_addr":"172.21.15.102:6821/2856826689","heartbeat_back_addr":"172.21.15.102:6825/2856826689","heartbeat_front_addr":"172.21.15.102:6823/2856826689","state":["exists","up"]},{"osd":7,"uuid":"99d65788-2491-4fc8-b15c-1861814a488a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6817","nonce":2471551369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6819","nonce":2471551369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6823","nonce":2471551369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":2471551369},{"type":"v1","addr":"172.21.15.110:6821","nonce":2471551369}]},"public_addr":"172.21.15.110:6817/2471551369","cluster_addr":"172.21.15.110:6819/2471551369","heartbeat_back_addr":"172.21.15.110:6823/2471551369","heartbeat_front_addr":"172.21.15.110:6821/2471551369","state":["exists","up"]},{"osd":8,"uuid":"bc7180db-d4e2-468d-a845-3846caa7a043","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6817","nonce":3634064328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6819","nonce":3634064328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6823","nonce":3634064328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3634064328},{"type":"v1","addr":"172.21.15.193:6821","nonce":3634064328}]},"public_addr":"172.21.15.193:6817/3634064328","cluster_addr":"172.21.15.193:6819/3634064328","heartbeat_back_addr":"172.21.15.193:6823/3634064328","heartbeat_front_addr":"172.21.15.193:6821/3634064328","state":["exists","up"]},{"osd":9,"uuid":"d15d150e-08b4-45c2-8f9d-cd64dba163fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6827","nonce":1305468681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6829","nonce":1305468681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6833","nonce":1305468681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1305468681},{"type":"v1","addr":"172.21.15.102:6831","nonce":1305468681}]},"public_addr":"172.21.15.102:6827/1305468681","cluster_addr":"172.21.15.102:6829/1305468681","heartbeat_back_addr":"172.21.15.102:6833/1305468681","heartbeat_front_addr":"172.21.15.102:6831/1305468681","state":["exists","up"]},{"osd":10,"uuid":"eb0f9f59-1254-4fbb-9f87-566e43a05d06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6825","nonce":851854931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6827","nonce":851854931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6831","nonce":851854931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":851854931},{"type":"v1","addr":"172.21.15.110:6829","nonce":851854931}]},"public_addr":"172.21.15.110:6825/851854931","cluster_addr":"172.21.15.110:6827/851854931","heartbeat_back_addr":"172.21.15.110:6831/851854931","heartbeat_front_addr":"172.21.15.110:6829/851854931","state":["exists","up"]},{"osd":11,"uuid":"0e8f1ff1-80f8-459e-8fea-733a1b7e54ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6825","nonce":40469709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6827","nonce":40469709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6831","nonce":40469709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":40469709},{"type":"v1","addr":"172.21.15.193:6829","nonce":40469709}]},"public_addr":"172.21.15.193:6825/40469709","cluster_addr":"172.21.15.193:6827/40469709","heartbeat_back_addr":"172.21.15.193:6831/40469709","heartbeat_front_addr":"172.21.15.193:6829/40469709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:46:37.817445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:02.442086+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:26.700035+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:47:59.927709+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:25.329853+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:48:50.462318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:23.905802+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:49:49.686633+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:15.055488+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:50:47.907770+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:14.296549+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T01:51:39.716744+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6801/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/459072323":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/83904525":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/2875896366":"2024-04-26T01:55:15.567762+0000","172.21.15.102:0/3113170519":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6801/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:6800/1124203809":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1835499087":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6801/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/511834001":"2024-04-26T01:55:15.567762+0000","172.21.15.102:6800/2757023783":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/411628693":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/1716050795":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/2344465952":"2024-04-26T01:44:42.581235+0000","172.21.15.102:0/4034021546":"2024-04-26T01:44:08.232557+0000","172.21.15.102:0/316635575":"2024-04-26T01:44:08.232557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":14,"length":1},{"begin":26,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T02:11:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe91c0a2d80 msgr2=0x7fe91c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe91c0a2d80 0x7fe91c0a5240 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe938076790 tx=0x7fe92000b410 comp rx=0 tx=0).stop 2024-04-25T02:11:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938103fd0 msgr2=0x7fe93819e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938103fd0 0x7fe93819e3b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe928004770 tx=0x7fe928003c10 comp rx=0 tx=0).stop 2024-04-25T02:11:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 shutdown_connections 2024-04-25T02:11:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7fe91c0a2d80 0x7fe91c0a5240 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9381068f0 0x7fe9381a2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe938106480 0x7fe93819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 --2- 172.21.15.102:0/2274210068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938103fd0 0x7fe93819e3b0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 >> 172.21.15.102:0/2274210068 conn(0x7fe938075f80 msgr2=0x7fe9380ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:50.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 shutdown_connections 2024-04-25T02:11:50.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.701+0000 7fe937200640 1 -- 172.21.15.102:0/2274210068 wait complete. 2024-04-25T02:11:50.727 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T02:11:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.915+0000 7efe5f800640 1 -- 172.21.15.102:0/114504265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe58074330 msgr2=0x7efe581122b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.915+0000 7efe5f800640 1 --2- 172.21.15.102:0/114504265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe58074330 0x7efe581122b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efe40009860 tx=0x7efe4002a3d0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.916+0000 7efe5f800640 1 -- 172.21.15.102:0/114504265 shutdown_connections 2024-04-25T02:11:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.916+0000 7efe5f800640 1 --2- 172.21.15.102:0/114504265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe581127f0 0x7efe58114be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.916+0000 7efe5f800640 1 --2- 172.21.15.102:0/114504265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe58074330 0x7efe581122b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.916+0000 7efe5f800640 1 --2- 172.21.15.102:0/114504265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe58073a10 0x7efe58073df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.916+0000 7efe5f800640 1 -- 172.21.15.102:0/114504265 >> 172.21.15.102:0/114504265 conn(0x7efe58078b90 msgr2=0x7efe58078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:50.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.916+0000 7efe5f800640 1 -- 172.21.15.102:0/114504265 shutdown_connections 2024-04-25T02:11:50.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.916+0000 7efe5f800640 1 -- 172.21.15.102:0/114504265 wait complete. 2024-04-25T02:11:50.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.917+0000 7efe5f800640 1 Processor -- start 2024-04-25T02:11:50.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.917+0000 7efe5f800640 1 -- start start 2024-04-25T02:11:50.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.917+0000 7efe5f800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe58073a10 0x7efe5819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.917+0000 7efe5f800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe58074330 0x7efe5819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.917+0000 7efe5f800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe581127f0 0x7efe581a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5f800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efe58118150 con 0x7efe58073a10 2024-04-25T02:11:50.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5f800640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7efe58117fd0 con 0x7efe58074330 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5f800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efe581182d0 con 0x7efe581127f0 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe58073a10 0x7efe5819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5d800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe581127f0 0x7efe581a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe57e00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe58074330 0x7efe5819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5d800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe581127f0 0x7efe581a2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:48456/0 (socket says 172.21.15.102:48456) 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5d800640 1 -- 172.21.15.102:0/483530171 learned_addr learned my addr 172.21.15.102:0/483530171 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5d800640 1 -- 172.21.15.102:0/483530171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe58074330 msgr2=0x7efe5819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5d800640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe58074330 0x7efe5819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5d800640 1 -- 172.21.15.102:0/483530171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe58073a10 msgr2=0x7efe5819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5d800640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe58073a10 0x7efe5819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe5d800640 1 -- 172.21.15.102:0/483530171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe581a3640 con 0x7efe581127f0 2024-04-25T02:11:50.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.918+0000 7efe57e00640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe58074330 0x7efe5819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:50.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.919+0000 7efe5d800640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe581127f0 0x7efe581a2f10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efe5000eb40 tx=0x7efe5000ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:50.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.919+0000 7efe55600640 1 -- 172.21.15.102:0/483530171 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5000ce40 con 0x7efe581127f0 2024-04-25T02:11:50.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.919+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe581a3930 con 0x7efe581127f0 2024-04-25T02:11:50.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.919+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe581ab1c0 con 0x7efe581127f0 2024-04-25T02:11:50.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.919+0000 7efe55600640 1 -- 172.21.15.102:0/483530171 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe50005020 con 0x7efe581127f0 2024-04-25T02:11:50.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.919+0000 7efe55600640 1 -- 172.21.15.102:0/483530171 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe50010670 con 0x7efe581127f0 2024-04-25T02:11:50.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.920+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe34005290 con 0x7efe581127f0 2024-04-25T02:11:50.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:50 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/741255434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:11:50.929 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:50 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/3767857317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:11:50.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.922+0000 7efe55600640 1 -- 172.21.15.102:0/483530171 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7efe50004a40 con 0x7efe581127f0 2024-04-25T02:11:50.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.924+0000 7efe55600640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efe3c0a2d80 0x7efe3c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:50.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.924+0000 7efe55600640 1 -- 172.21.15.102:0/483530171 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8513+0+0 (secure 0 0 0) 0x7efe500cfa10 con 0x7efe581127f0 2024-04-25T02:11:50.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.924+0000 7efe5ce00640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efe3c0a2d80 0x7efe3c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:50.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.925+0000 7efe5ce00640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efe3c0a2d80 0x7efe3c0a5240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efe48005fd0 tx=0x7efe48009260 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:50.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:50.926+0000 7efe55600640 1 -- 172.21.15.102:0/483530171 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efe5009c630 con 0x7efe581127f0 2024-04-25T02:11:50.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:50 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/741255434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:11:50.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:50 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/3767857317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:11:51.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:50 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/741255434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:11:51.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:50 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/3767857317' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T02:11:51.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.173+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7efe34005850 con 0x7efe581127f0 2024-04-25T02:11:51.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.187+0000 7efe55600640 1 -- 172.21.15.102:0/483530171 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) ==== 161+0+0 (secure 0 0 0) 0x7efe5009bf30 con 0x7efe581127f0 2024-04-25T02:11:51.194 INFO:teuthology.orchestra.run.smithi102.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T02:11:51.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.189+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efe3c0a2d80 msgr2=0x7efe3c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.189+0000 7efe5f800640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efe3c0a2d80 0x7efe3c0a5240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efe48005fd0 tx=0x7efe48009260 comp rx=0 tx=0).stop 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.189+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe581127f0 msgr2=0x7efe581a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.189+0000 7efe5f800640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe581127f0 0x7efe581a2f10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efe5000eb40 tx=0x7efe5000ef40 comp rx=0 tx=0).stop 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.189+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 shutdown_connections 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.189+0000 7efe5f800640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7efe3c0a2d80 0x7efe3c0a5240 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.189+0000 7efe5f800640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe581127f0 0x7efe581a2f10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.190+0000 7efe5f800640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe58074330 0x7efe5819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.190+0000 7efe5f800640 1 --2- 172.21.15.102:0/483530171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe58073a10 0x7efe5819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.190+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 >> 172.21.15.102:0/483530171 conn(0x7efe58078b90 msgr2=0x7efe580fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.190+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 shutdown_connections 2024-04-25T02:11:51.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.190+0000 7efe5f800640 1 -- 172.21.15.102:0/483530171 wait complete. 2024-04-25T02:11:51.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T02:11:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.396+0000 7f9a72200640 1 -- 172.21.15.102:0/2101092809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a6c1015e0 msgr2=0x7f9a6c101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.396+0000 7f9a72200640 1 --2- 172.21.15.102:0/2101092809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a6c1015e0 0x7f9a6c101a60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9a58009890 tx=0x7f9a5802a360 comp rx=0 tx=0).stop 2024-04-25T02:11:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.396+0000 7f9a72200640 1 -- 172.21.15.102:0/2101092809 shutdown_connections 2024-04-25T02:11:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.396+0000 7f9a72200640 1 --2- 172.21.15.102:0/2101092809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6c101fa0 0x7f9a6c10e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.396+0000 7f9a72200640 1 --2- 172.21.15.102:0/2101092809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a6c1015e0 0x7f9a6c101a60 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:51.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.396+0000 7f9a72200640 1 --2- 172.21.15.102:0/2101092809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a6c107680 0x7f9a6c107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.396+0000 7f9a72200640 1 -- 172.21.15.102:0/2101092809 >> 172.21.15.102:0/2101092809 conn(0x7f9a6c0fd110 msgr2=0x7f9a6c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.396+0000 7f9a72200640 1 -- 172.21.15.102:0/2101092809 shutdown_connections 2024-04-25T02:11:51.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.396+0000 7f9a72200640 1 -- 172.21.15.102:0/2101092809 wait complete. 2024-04-25T02:11:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.397+0000 7f9a72200640 1 Processor -- start 2024-04-25T02:11:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.397+0000 7f9a72200640 1 -- start start 2024-04-25T02:11:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.397+0000 7f9a72200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a6c1015e0 0x7f9a6c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a72200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6c101fa0 0x7f9a6c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a72200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a6c107680 0x7f9a6c19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a72200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c1119c0 con 0x7f9a6c101fa0 2024-04-25T02:11:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a72200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c111840 con 0x7f9a6c107680 2024-04-25T02:11:51.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a72200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c111b40 con 0x7f9a6c1015e0 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a70e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a6c1015e0 0x7f9a6c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a6be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6c101fa0 0x7f9a6c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a71800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a6c107680 0x7f9a6c19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a71800640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a6c107680 0x7f9a6c19e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:42998/0 (socket says 172.21.15.102:42998) 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a70e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a6c1015e0 0x7f9a6c19a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.102:48478/0 (socket says 172.21.15.102:48478) 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a71800640 1 -- 172.21.15.102:0/1276579961 learned_addr learned my addr 172.21.15.102:0/1276579961 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a71800640 1 -- 172.21.15.102:0/1276579961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a6c1015e0 msgr2=0x7f9a6c19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a71800640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a6c1015e0 0x7f9a6c19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a71800640 1 -- 172.21.15.102:0/1276579961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6c101fa0 msgr2=0x7f9a6c19a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a71800640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6c101fa0 0x7f9a6c19a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.398+0000 7f9a71800640 1 -- 172.21.15.102:0/1276579961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c19f060 con 0x7f9a6c107680 2024-04-25T02:11:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.399+0000 7f9a71800640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a6c107680 0x7f9a6c19e930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009fd0 tx=0x7f9a6400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.399+0000 7f9a69600640 1 -- 172.21.15.102:0/1276579961 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64004b20 con 0x7f9a6c107680 2024-04-25T02:11:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.399+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6c19f350 con 0x7f9a6c107680 2024-04-25T02:11:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.399+0000 7f9a69600640 1 -- 172.21.15.102:0/1276579961 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a64003e50 con 0x7f9a6c107680 2024-04-25T02:11:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.399+0000 7f9a69600640 1 -- 172.21.15.102:0/1276579961 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6400b630 con 0x7f9a6c107680 2024-04-25T02:11:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.399+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6c1a6be0 con 0x7f9a6c107680 2024-04-25T02:11:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.400+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a6c068480 con 0x7f9a6c107680 2024-04-25T02:11:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.402+0000 7f9a69600640 1 -- 172.21.15.102:0/1276579961 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9a6401d020 con 0x7f9a6c107680 2024-04-25T02:11:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.403+0000 7f9a69600640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a540a2d00 0x7f9a540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.404+0000 7f9a69600640 1 -- 172.21.15.102:0/1276579961 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8727+0+0 (secure 0 0 0) 0x7f9a640cf530 con 0x7f9a6c107680 2024-04-25T02:11:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.404+0000 7f9a70e00640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a540a2d00 0x7f9a540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:51.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.405+0000 7f9a70e00640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a540a2d00 0x7f9a540a51c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c102720 tx=0x7f9a60005f90 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:51.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.408+0000 7f9a69600640 1 -- 172.21.15.102:0/1276579961 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9a6409c020 con 0x7f9a6c107680 2024-04-25T02:11:51.494 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e[63484]: ignoring --setuser ceph since I am not root 2024-04-25T02:11:51.494 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e[63484]: ignoring --setgroup ceph since I am not root 2024-04-25T02:11:51.494 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e[63484]: starting mds.e at 2024-04-25T02:11:51.495 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b[64106]: ignoring --setuser ceph since I am not root 2024-04-25T02:11:51.495 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b[64106]: ignoring --setgroup ceph since I am not root 2024-04-25T02:11:51.495 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mds[64110]: -- 172.21.15.110:0/1650950860 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ffc581ba40 con 0x55ffc57eec00 2024-04-25T02:11:51.495 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b[64106]: starting mds.b at 2024-04-25T02:11:51.495 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h[63799]: ignoring --setuser ceph since I am not root 2024-04-25T02:11:51.495 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h[63799]: ignoring --setgroup ceph since I am not root 2024-04-25T02:11:51.495 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mds[63803]: -- 172.21.15.110:0/3690039824 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619d5b29a40 con 0x5619d5afcc00 2024-04-25T02:11:51.495 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h[63799]: starting mds.h at 2024-04-25T02:11:51.512 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f[63361]: ignoring --setuser ceph since I am not root 2024-04-25T02:11:51.512 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f[63361]: ignoring --setgroup ceph since I am not root 2024-04-25T02:11:51.512 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mds[63365]: -- 172.21.15.193:0/2044170928 <== mon.2 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561f3e907a40 con 0x561f3e8db000 2024-04-25T02:11:51.512 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f[63361]: starting mds.f at 2024-04-25T02:11:51.512 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c[63666]: ignoring --setuser ceph since I am not root 2024-04-25T02:11:51.512 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c[63666]: ignoring --setgroup ceph since I am not root 2024-04-25T02:11:51.512 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mds[63670]: -- 172.21.15.193:0/3749544387 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55db208d84e0 con 0x55db208ae800 2024-04-25T02:11:51.513 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mds[63670]: -- 172.21.15.193:0/3749544387 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55db208dba40 con 0x55db208ae800 2024-04-25T02:11:51.513 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c[63666]: starting mds.c at 2024-04-25T02:11:51.513 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: ignoring --setuser ceph since I am not root 2024-04-25T02:11:51.513 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: ignoring --setgroup ceph since I am not root 2024-04-25T02:11:51.513 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mds[63977]: -- 172.21.15.193:0/3873817941 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c75f77a40 con 0x564c75f4a800 2024-04-25T02:11:51.513 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: starting mds.i at 2024-04-25T02:11:51.573 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a[27792]: 2024-04-25T02:11:51.176+0000 7f4d15e00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T02:11:51.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:51.650+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f9a6c1a6db0 con 0x7f9a6c107680 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: pgmap v518: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2274210068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/483530171' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T02:11:51.885 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:51 smithi102 ceph-mon[27796]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T02:11:51.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: pgmap v518: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2024-04-25T02:11:51.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2274210068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:11:51.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/483530171' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T02:11:51.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T02:11:51.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T02:11:51.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T02:11:51.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-04-25T02:11:51.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T02:11:51.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T02:11:51.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:51 smithi110 ceph-mon[32848]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T02:11:52.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: pgmap v518: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2024-04-25T02:11:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2274210068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T02:11:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/483530171' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T02:11:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T02:11:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T02:11:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T02:11:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-04-25T02:11:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T02:11:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T02:11:52.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:51 smithi193 ceph-mon[32981]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T02:11:52.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.185+0000 7f9a69600640 1 -- 172.21.15.102:0/1276579961 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) ==== 104+0+0 (secure 0 0 0) 0x7f9a6409b920 con 0x7f9a6c107680 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.187+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a540a2d00 msgr2=0x7f9a540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.187+0000 7f9a72200640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a540a2d00 0x7f9a540a51c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c102720 tx=0x7f9a60005f90 comp rx=0 tx=0).stop 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.187+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a6c107680 msgr2=0x7f9a6c19e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.187+0000 7f9a72200640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a6c107680 0x7f9a6c19e930 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009fd0 tx=0x7f9a6400c800 comp rx=0 tx=0).stop 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.188+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 shutdown_connections 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.188+0000 7f9a72200640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f9a540a2d00 0x7f9a540a51c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.188+0000 7f9a72200640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9a6c107680 0x7f9a6c19e930 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.188+0000 7f9a72200640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a6c101fa0 0x7f9a6c19a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.188+0000 7f9a72200640 1 --2- 172.21.15.102:0/1276579961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a6c1015e0 0x7f9a6c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:52.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.188+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 >> 172.21.15.102:0/1276579961 conn(0x7f9a6c0fd110 msgr2=0x7f9a6c10d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:52.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.188+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 shutdown_connections 2024-04-25T02:11:52.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.188+0000 7f9a72200640 1 -- 172.21.15.102:0/1276579961 wait complete. 2024-04-25T02:11:52.215 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T02:11:52.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.398+0000 7f3c19000640 1 -- 172.21.15.102:0/2788752838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c1410de20 msgr2=0x7f3c14110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:52.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.398+0000 7f3c19000640 1 --2- 172.21.15.102:0/2788752838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c1410de20 0x7f3c14110270 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009be0 tx=0x7f3c0c02a750 comp rx=0 tx=0).stop 2024-04-25T02:11:52.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.398+0000 7f3c19000640 1 -- 172.21.15.102:0/2788752838 shutdown_connections 2024-04-25T02:11:52.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.398+0000 7f3c19000640 1 --2- 172.21.15.102:0/2788752838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c1410de20 0x7f3c14110270 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:52.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.398+0000 7f3c19000640 1 --2- 172.21.15.102:0/2788752838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c14074490 0x7f3c1410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:52.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.398+0000 7f3c19000640 1 --2- 172.21.15.102:0/2788752838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c14073b70 0x7f3c14073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:52.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.398+0000 7f3c19000640 1 -- 172.21.15.102:0/2788752838 >> 172.21.15.102:0/2788752838 conn(0x7f3c140fd110 msgr2=0x7f3c140ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:52.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.398+0000 7f3c19000640 1 -- 172.21.15.102:0/2788752838 shutdown_connections 2024-04-25T02:11:52.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.398+0000 7f3c19000640 1 -- 172.21.15.102:0/2788752838 wait complete. 2024-04-25T02:11:52.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.399+0000 7f3c19000640 1 Processor -- start 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c19000640 1 -- start start 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c19000640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c14073b70 0x7f3c1419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c19000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c14074490 0x7f3c1419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c19000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1410de20 0x7f3c1419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c19000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14113bc0 con 0x7f3c14074490 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c19000640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14113a40 con 0x7f3c14073b70 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c19000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14113d40 con 0x7f3c1410de20 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c12a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c14074490 0x7f3c1419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c13400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c14073b70 0x7f3c1419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c13400640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c14073b70 0x7f3c1419a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.102:43018/0 (socket says 172.21.15.102:43018) 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c12a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c14074490 0x7f3c1419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48486/0 (socket says 172.21.15.102:48486) 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.400+0000 7f3c13400640 1 -- 172.21.15.102:0/2412161192 learned_addr learned my addr 172.21.15.102:0/2412161192 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c13e00640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1410de20 0x7f3c1419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:52.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c13400640 1 -- 172.21.15.102:0/2412161192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1410de20 msgr2=0x7f3c1419e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c13400640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1410de20 0x7f3c1419e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c13400640 1 -- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c14074490 msgr2=0x7f3c1419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c13400640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c14074490 0x7f3c1419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c13400640 1 -- 172.21.15.102:0/2412161192 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1419f040 con 0x7f3c14073b70 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c12a00640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c14074490 0x7f3c1419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c13400640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c14073b70 0x7f3c1419a0a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400ea60 tx=0x7f3c0400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c0be00640 1 -- 172.21.15.102:0/2412161192 <== mon.2 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0400cd20 con 0x7f3c14073b70 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1419f330 con 0x7f3c14073b70 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c0be00640 1 -- 172.21.15.102:0/2412161192 <== mon.2 v2:172.21.15.110:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c040035f0 con 0x7f3c14073b70 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.401+0000 7f3c0be00640 1 -- 172.21.15.102:0/2412161192 <== mon.2 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c04010690 con 0x7f3c14073b70 2024-04-25T02:11:52.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.402+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c141a6bc0 con 0x7f3c14073b70 2024-04-25T02:11:52.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.403+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c14068480 con 0x7f3c14073b70 2024-04-25T02:11:52.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.405+0000 7f3c0be00640 1 -- 172.21.15.102:0/2412161192 <== mon.2 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3c040037a0 con 0x7f3c14073b70 2024-04-25T02:11:52.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.406+0000 7f3c0be00640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf80a2d00 0x7f3bf80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:52.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.406+0000 7f3c0be00640 1 -- 172.21.15.102:0/2412161192 <== mon.2 v2:172.21.15.110:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8727+0+0 (secure 0 0 0) 0x7f3c040cf900 con 0x7f3c14073b70 2024-04-25T02:11:52.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.406+0000 7f3c12a00640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf80a2d00 0x7f3bf80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:52.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.407+0000 7f3c12a00640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf80a2d00 0x7f3bf80a51c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3c1410f240 tx=0x7f3c00009260 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:52.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.408+0000 7f3c0be00640 1 -- 172.21.15.102:0/2412161192 <== mon.2 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3c0409c480 con 0x7f3c14073b70 2024-04-25T02:11:52.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:52.606+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3c14109000 con 0x7f3c14073b70 2024-04-25T02:11:52.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/1276579961' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: Cluster is now healthy 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.110:6835/1245884723,v1:172.21.15.110:6837/1245884723] up:boot 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.110:6836/978395058,v1:172.21.15.110:6839/978395058] up:boot 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.110:6834/4123152894,v1:172.21.15.110:6838/4123152894] up:boot 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.193:6835/1145035256,v1:172.21.15.193:6837/1145035256] up:boot 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.193:6834/203335690,v1:172.21.15.193:6838/203335690] up:boot 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: mds.? [v2:172.21.15.193:6836/1291264885,v1:172.21.15.193:6839/1291264885] up:boot 2024-04-25T02:11:52.931 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: fsmap 12 up:standby 2024-04-25T02:11:52.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T02:11:52.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T02:11:52.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T02:11:52.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T02:11:52.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T02:11:52.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T02:11:52.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2412161192' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T02:11:52.932 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:52 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T02:11:52.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/1276579961' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T02:11:52.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T02:11:52.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: Cluster is now healthy 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.110:6835/1245884723,v1:172.21.15.110:6837/1245884723] up:boot 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.110:6836/978395058,v1:172.21.15.110:6839/978395058] up:boot 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.110:6834/4123152894,v1:172.21.15.110:6838/4123152894] up:boot 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.193:6835/1145035256,v1:172.21.15.193:6837/1145035256] up:boot 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.193:6834/203335690,v1:172.21.15.193:6838/203335690] up:boot 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: mds.? [v2:172.21.15.193:6836/1291264885,v1:172.21.15.193:6839/1291264885] up:boot 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: fsmap 12 up:standby 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T02:11:52.995 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T02:11:52.996 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T02:11:52.996 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T02:11:52.996 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T02:11:52.996 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2412161192' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T02:11:52.996 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:52 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T02:11:53.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/1276579961' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T02:11:53.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T02:11:53.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T02:11:53.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T02:11:53.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: Cluster is now healthy 2024-04-25T02:11:53.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T02:11:53.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.110:6835/1245884723,v1:172.21.15.110:6837/1245884723] up:boot 2024-04-25T02:11:53.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.110:6836/978395058,v1:172.21.15.110:6839/978395058] up:boot 2024-04-25T02:11:53.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.110:6834/4123152894,v1:172.21.15.110:6838/4123152894] up:boot 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.193:6835/1145035256,v1:172.21.15.193:6837/1145035256] up:boot 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.193:6834/203335690,v1:172.21.15.193:6838/203335690] up:boot 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: mds.? [v2:172.21.15.193:6836/1291264885,v1:172.21.15.193:6839/1291264885] up:boot 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: fsmap 12 up:standby 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='mgr.25214 172.21.15.102:0/1236176321' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2412161192' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T02:11:53.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:52 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T02:11:53.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.187+0000 7f3c0be00640 1 -- 172.21.15.102:0/2412161192 <== mon.2 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v110) ==== 181+0+0 (secure 0 0 0) 0x7f3c0409bd80 con 0x7f3c14073b70 2024-04-25T02:11:53.195 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' removed 2024-04-25T02:11:53.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.193+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf80a2d00 msgr2=0x7f3bf80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:53.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.193+0000 7f3c19000640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf80a2d00 0x7f3bf80a51c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3c1410f240 tx=0x7f3c00009260 comp rx=0 tx=0).stop 2024-04-25T02:11:53.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.193+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c14073b70 msgr2=0x7f3c1419a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:53.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.193+0000 7f3c19000640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c14073b70 0x7f3c1419a0a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400ea60 tx=0x7f3c0400ee60 comp rx=0 tx=0).stop 2024-04-25T02:11:53.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.194+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 shutdown_connections 2024-04-25T02:11:53.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.194+0000 7f3c19000640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f3bf80a2d00 0x7f3bf80a51c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.194+0000 7f3c19000640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1410de20 0x7f3c1419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.194+0000 7f3c19000640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c14074490 0x7f3c1419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.194+0000 7f3c19000640 1 --2- 172.21.15.102:0/2412161192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3c14073b70 0x7f3c1419a0a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.194+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 >> 172.21.15.102:0/2412161192 conn(0x7f3c140fd110 msgr2=0x7f3c140fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:53.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.195+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 shutdown_connections 2024-04-25T02:11:53.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.196+0000 7f3c19000640 1 -- 172.21.15.102:0/2412161192 wait complete. 2024-04-25T02:11:53.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T02:11:53.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.401+0000 7f805f200640 1 -- 172.21.15.102:0/994556615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805810df60 msgr2=0x7f8058110350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.401+0000 7f805f200640 1 --2- 172.21.15.102:0/994556615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805810df60 0x7f8058110350 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8054009be0 tx=0x7f805402a6d0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.401+0000 7f805f200640 1 -- 172.21.15.102:0/994556615 shutdown_connections 2024-04-25T02:11:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.401+0000 7f805f200640 1 --2- 172.21.15.102:0/994556615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805810df60 0x7f8058110350 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.401+0000 7f805f200640 1 --2- 172.21.15.102:0/994556615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8058100d60 0x7f805810d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.401+0000 7f805f200640 1 --2- 172.21.15.102:0/994556615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80581003b0 0x7f8058100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.401+0000 7f805f200640 1 -- 172.21.15.102:0/994556615 >> 172.21.15.102:0/994556615 conn(0x7f80580fbfc0 msgr2=0x7f80580fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.401+0000 7f805f200640 1 -- 172.21.15.102:0/994556615 shutdown_connections 2024-04-25T02:11:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.401+0000 7f805f200640 1 -- 172.21.15.102:0/994556615 wait complete. 2024-04-25T02:11:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.403+0000 7f805f200640 1 Processor -- start 2024-04-25T02:11:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.403+0000 7f805f200640 1 -- start start 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.403+0000 7f805f200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80581003b0 0x7f805819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.403+0000 7f805f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058100d60 0x7f805819a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.403+0000 7f805f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805810df60 0x7f805819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.403+0000 7f805f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8058113b80 con 0x7f805810df60 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.403+0000 7f805f200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 -- 0x7f8058113a00 con 0x7f80581003b0 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.403+0000 7f805f200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8058113d00 con 0x7f8058100d60 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.403+0000 7f805e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805810df60 0x7f805819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805810df60 0x7f805819e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45066/0 (socket says 172.21.15.102:45066) 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805e800640 1 -- 172.21.15.102:0/2114508365 learned_addr learned my addr 172.21.15.102:0/2114508365 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805de00640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80581003b0 0x7f805819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805d400640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058100d60 0x7f805819a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:53.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805e800640 1 -- 172.21.15.102:0/2114508365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058100d60 msgr2=0x7f805819a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:53.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805e800640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058100d60 0x7f805819a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805e800640 1 -- 172.21.15.102:0/2114508365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80581003b0 msgr2=0x7f805819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805e800640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80581003b0 0x7f805819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805e800640 1 -- 172.21.15.102:0/2114508365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805819f050 con 0x7f805810df60 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805de00640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80581003b0 0x7f805819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.404+0000 7f805e800640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805810df60 0x7f805819e920 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8054034040 tx=0x7f80540305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.405+0000 7f8052a00640 1 -- 172.21.15.102:0/2114508365 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054041020 con 0x7f805810df60 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.405+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805819f2e0 con 0x7f805810df60 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.405+0000 7f8052a00640 1 -- 172.21.15.102:0/2114508365 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8054037040 con 0x7f805810df60 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.405+0000 7f8052a00640 1 -- 172.21.15.102:0/2114508365 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054036070 con 0x7f805810df60 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.405+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80581a6bd0 con 0x7f805810df60 2024-04-25T02:11:53.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.406+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8058068480 con 0x7f805810df60 2024-04-25T02:11:53.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.408+0000 7f8052a00640 1 -- 172.21.15.102:0/2114508365 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8054044020 con 0x7f805810df60 2024-04-25T02:11:53.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.409+0000 7f8052a00640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f80400a2d00 0x7f80400a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T02:11:53.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.410+0000 7f8052a00640 1 -- 172.21.15.102:0/2114508365 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8173+0+0 (secure 0 0 0) 0x7f80540efff0 con 0x7f805810df60 2024-04-25T02:11:53.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.410+0000 7f805de00640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f80400a2d00 0x7f80400a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T02:11:53.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.410+0000 7f805de00640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f80400a2d00 0x7f80400a51c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f804c0096c0 tx=0x7f804c009260 comp rx=0 tx=0).ready entity=mgr.25214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T02:11:53.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.412+0000 7f8052a00640 1 -- 172.21.15.102:0/2114508365 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f805404b020 con 0x7f805810df60 2024-04-25T02:11:53.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:53.617+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8058109060 con 0x7f805810df60 2024-04-25T02:11:53.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:53 smithi102 ceph-mon[27796]: pgmap v520: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 99 KiB/s wr, 9 op/s 2024-04-25T02:11:53.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:53 smithi102 ceph-mon[27796]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T02:11:53.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:53 smithi102 ceph-mon[27796]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T02:11:53.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:53 smithi102 ceph-mon[27796]: from='client.? 172.21.15.102:0/2114508365' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T02:11:53.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:53 smithi110 ceph-mon[32848]: pgmap v520: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 99 KiB/s wr, 9 op/s 2024-04-25T02:11:53.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:53 smithi110 ceph-mon[32848]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T02:11:53.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:53 smithi110 ceph-mon[32848]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T02:11:53.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:53 smithi110 ceph-mon[32848]: from='client.? 172.21.15.102:0/2114508365' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T02:11:54.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:53 smithi193 ceph-mon[32981]: pgmap v520: 97 pgs: 97 active+clean; 5.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 99 KiB/s wr, 9 op/s 2024-04-25T02:11:54.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:53 smithi193 ceph-mon[32981]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T02:11:54.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:53 smithi193 ceph-mon[32981]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T02:11:54.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:53 smithi193 ceph-mon[32981]: from='client.? 172.21.15.102:0/2114508365' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T02:11:54.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.187+0000 7f8052a00640 1 -- 172.21.15.102:0/2114508365 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v111) ==== 169+0+0 (secure 0 0 0) 0x7f80540bce90 con 0x7f805810df60 2024-04-25T02:11:54.195 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' removed 2024-04-25T02:11:54.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.191+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f80400a2d00 msgr2=0x7f80400a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.191+0000 7f805f200640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f80400a2d00 0x7f80400a51c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f804c0096c0 tx=0x7f804c009260 comp rx=0 tx=0).stop 2024-04-25T02:11:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.191+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805810df60 msgr2=0x7f805819e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T02:11:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.191+0000 7f805f200640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805810df60 0x7f805819e920 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8054034040 tx=0x7f80540305d0 comp rx=0 tx=0).stop 2024-04-25T02:11:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.192+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 shutdown_connections 2024-04-25T02:11:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.192+0000 7f805f200640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:6800/855543702,v1:172.21.15.102:6801/855543702] conn(0x7f80400a2d00 0x7f80400a51c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.192+0000 7f805f200640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805810df60 0x7f805819e920 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.192+0000 7f805f200640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058100d60 0x7f805819a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.192+0000 7f805f200640 1 --2- 172.21.15.102:0/2114508365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80581003b0 0x7f805819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T02:11:54.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.192+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 >> 172.21.15.102:0/2114508365 conn(0x7f80580fbfc0 msgr2=0x7f80580fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T02:11:54.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.193+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 shutdown_connections 2024-04-25T02:11:54.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-04-25T02:11:54.193+0000 7f805f200640 1 -- 172.21.15.102:0/2114508365 wait complete. 2024-04-25T02:11:54.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:11:54.338 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:11:54.377 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:11:54.415 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T02:11:54.415 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T02:11:54.453 DEBUG:teuthology.orchestra.run.smithi110:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T02:11:54.469 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T02:11:54.487 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T02:11:54.487 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T02:11:54.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.a 2024-04-25T02:11:54.930 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:54 smithi102 systemd[1]: Stopping Ceph mon.a for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:11:55.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:54 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a[27792]: 2024-04-25T02:11:54.986+0000 7f4d1cc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:11:55.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:54 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a[27792]: 2024-04-25T02:11:54.986+0000 7f4d1cc00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T02:11:55.366 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:55 smithi102 podman[95772]: 2024-04-25 02:11:55.018398844 +0000 UTC m=+0.169397630 container died 7e663022620a73e032edbc75ede968d589e57846b0437e3a481d4a7dfccba870 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T02:11:55.679 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:55 smithi102 podman[95772]: 2024-04-25 02:11:55.491286636 +0000 UTC m=+0.642285424 container cleanup 7e663022620a73e032edbc75ede968d589e57846b0437e3a481d4a7dfccba870 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T02:11:55.680 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:55 smithi102 bash[95772]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a 2024-04-25T02:11:56.120 INFO:journalctl@ceph.mon.a.smithi102.stdout:Apr 25 02:11:55 smithi102 podman[95786]: 2024-04-25 02:11:55.749632362 +0000 UTC m=+0.722393065 container remove 7e663022620a73e032edbc75ede968d589e57846b0437e3a481d4a7dfccba870 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:11:56.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T02:11:56.261 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:11:56.261 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T02:11:56.262 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T02:11:56.262 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.b 2024-04-25T02:11:56.627 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:56 smithi110 systemd[1]: Stopping Ceph mon.b for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:11:56.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:56 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-b[32844]: 2024-04-25T02:11:56.622+0000 7fe9ec220700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:11:56.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:56 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-b[32844]: 2024-04-25T02:11:56.622+0000 7fe9ec220700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T02:11:56.994 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:56 smithi110 podman[70508]: 2024-04-25 02:11:56.703722463 +0000 UTC m=+0.231762340 container died b5efaa94d32a25aecb916be99cb717878be4328379788bba6117289998654c66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T02:11:57.363 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:57 smithi110 podman[70508]: 2024-04-25 02:11:57.088469612 +0000 UTC m=+0.616509483 container cleanup b5efaa94d32a25aecb916be99cb717878be4328379788bba6117289998654c66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T02:11:57.363 INFO:journalctl@ceph.mon.b.smithi110.stdout:Apr 25 02:11:57 smithi110 bash[70508]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-b 2024-04-25T02:11:57.525 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T02:11:57.589 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:11:57.589 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T02:11:57.589 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T02:11:57.589 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.c 2024-04-25T02:11:57.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:57 smithi193 systemd[1]: Stopping Ceph mon.c for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:11:58.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:57 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-c[32977]: 2024-04-25T02:11:57.900+0000 7f25f7350700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:11:58.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:57 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-c[32977]: 2024-04-25T02:11:57.900+0000 7f25f7350700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T02:11:58.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:58 smithi193 podman[70837]: 2024-04-25 02:11:58.084921736 +0000 UTC m=+0.312735330 container died 0ded0404a1ee47d12e26f343884593cd2a83266418e9ee2108596e7443c3b6dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T02:11:58.626 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:58 smithi193 podman[70837]: 2024-04-25 02:11:58.426790044 +0000 UTC m=+0.654603640 container cleanup 0ded0404a1ee47d12e26f343884593cd2a83266418e9ee2108596e7443c3b6dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T02:11:58.626 INFO:journalctl@ceph.mon.c.smithi193.stdout:Apr 25 02:11:58 smithi193 bash[70837]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mon-c 2024-04-25T02:11:58.845 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T02:11:58.891 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:11:58.892 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T02:11:58.892 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T02:11:58.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.x 2024-04-25T02:11:59.280 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 02:11:58 smithi102 systemd[1]: Stopping Ceph mgr.x for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:11:59.535 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 02:11:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T02:11:59.409+0000 7f50dde00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:11:59.535 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 02:11:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x[28038]: 2024-04-25T02:11:59.409+0000 7f50dde00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-25T02:11:59.535 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 02:11:59 smithi102 podman[95889]: 2024-04-25 02:11:59.512136891 +0000 UTC m=+0.248217702 container died 5b9d2d220f12f87c864763b3123735ccc0f3f14ef7ba2321fe2ac2fd72b719a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:12:00.180 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 02:12:00 smithi102 podman[95889]: 2024-04-25 02:12:00.011966282 +0000 UTC m=+0.748047105 container cleanup 5b9d2d220f12f87c864763b3123735ccc0f3f14ef7ba2321fe2ac2fd72b719a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T02:12:00.180 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 02:12:00 smithi102 bash[95889]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x 2024-04-25T02:12:00.679 INFO:journalctl@ceph.mgr.x.smithi102.stdout:Apr 25 02:12:00 smithi102 podman[95903]: 2024-04-25 02:12:00.379845638 +0000 UTC m=+0.854215275 container remove 5b9d2d220f12f87c864763b3123735ccc0f3f14ef7ba2321fe2ac2fd72b719a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d2c45d977142e024a59b68a8f1dce8f0b9840551, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T02:12:00.872 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T02:12:00.969 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:12:00.969 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T02:12:00.969 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T02:12:00.969 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.y 2024-04-25T02:12:01.345 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 02:12:01 smithi110 systemd[1]: Stopping Ceph mgr.y for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:12:01.608 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 02:12:01 smithi110 podman[70627]: 2024-04-25 02:12:01.338609555 +0000 UTC m=+0.181739861 container died 9c2040968d4de02d200a1af591d19e8a03b23f7529c8c058822387b8d7968c89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T02:12:01.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 02:12:01 smithi110 podman[70627]: 2024-04-25 02:12:01.847735831 +0000 UTC m=+0.690866137 container cleanup 9c2040968d4de02d200a1af591d19e8a03b23f7529c8c058822387b8d7968c89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD) 2024-04-25T02:12:01.871 INFO:journalctl@ceph.mgr.y.smithi110.stdout:Apr 25 02:12:01 smithi110 bash[70627]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-y 2024-04-25T02:12:02.182 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T02:12:02.219 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:12:02.219 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T02:12:02.219 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T02:12:02.219 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.z 2024-04-25T02:12:02.511 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 02:12:02 smithi193 systemd[1]: Stopping Ceph mgr.z for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:12:02.821 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 02:12:02 smithi193 podman[70956]: 2024-04-25 02:12:02.526617524 +0000 UTC m=+0.126460607 container died 5416e1fa7c8039606b4c0171ac2dbb79985fd7ba38134e0a220c6be9946b8bf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:12:03.147 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 02:12:02 smithi193 podman[70956]: 2024-04-25 02:12:02.914026019 +0000 UTC m=+0.513869102 container cleanup 5416e1fa7c8039606b4c0171ac2dbb79985fd7ba38134e0a220c6be9946b8bf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:12:03.147 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Apr 25 02:12:02 smithi193 bash[70956]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mgr-z 2024-04-25T02:12:03.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T02:12:03.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:12:03.319 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T02:12:03.319 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T02:12:03.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.0 2024-04-25T02:12:03.679 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:03 smithi102 systemd[1]: Stopping Ceph osd.0 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:12:04.180 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0[42920]: 2024-04-25T02:12:03.807+0000 7f2303200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:12:04.180 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0[42920]: 2024-04-25T02:12:03.807+0000 7f2303200700 -1 osd.0 111 *** Got signal Terminated *** 2024-04-25T02:12:04.180 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0[42920]: 2024-04-25T02:12:03.807+0000 7f2303200700 -1 osd.0 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:12:09.931 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:09 smithi102 podman[96010]: 2024-04-25 02:12:09.719910071 +0000 UTC m=+6.050359850 container died 5bb9e2e67e68c904385a704e10d4262b23481764569ec03b5d285201d48acf30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T02:12:10.430 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:10 smithi102 podman[96010]: 2024-04-25 02:12:10.225248159 +0000 UTC m=+6.555697979 container cleanup 5bb9e2e67e68c904385a704e10d4262b23481764569ec03b5d285201d48acf30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424) 2024-04-25T02:12:10.430 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:10 smithi102 bash[96010]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0 2024-04-25T02:12:10.930 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:10 smithi102 podman[96030]: 2024-04-25 02:12:10.45974161 +0000 UTC m=+0.732143468 container remove 5bb9e2e67e68c904385a704e10d4262b23481764569ec03b5d285201d48acf30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T02:12:11.640 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:11 smithi102 podman[96119]: 2024-04-25 02:12:11.184056418 +0000 UTC m=+0.032859751 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:12:11.640 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:11 smithi102 podman[96119]: 2024-04-25 02:12:11.344034354 +0000 UTC m=+0.192837724 container create 8aa4ee984ddd1662d6e69a8563d6bb517208a3d3b6ed2dbe03ff75f20f2d0ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-25T02:12:11.930 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:11 smithi102 podman[96119]: 2024-04-25 02:12:11.632538178 +0000 UTC m=+0.481341585 container init 8aa4ee984ddd1662d6e69a8563d6bb517208a3d3b6ed2dbe03ff75f20f2d0ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T02:12:11.931 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:11 smithi102 podman[96119]: 2024-04-25 02:12:11.645218085 +0000 UTC m=+0.494021431 container start 8aa4ee984ddd1662d6e69a8563d6bb517208a3d3b6ed2dbe03ff75f20f2d0ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T02:12:11.931 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:11 smithi102 podman[96119]: 2024-04-25 02:12:11.721319124 +0000 UTC m=+0.570122541 container attach 8aa4ee984ddd1662d6e69a8563d6bb517208a3d3b6ed2dbe03ff75f20f2d0ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-25T02:12:12.430 INFO:journalctl@ceph.osd.0.smithi102.stdout:Apr 25 02:12:12 smithi102 podman[96119]: 2024-04-25 02:12:12.096547911 +0000 UTC m=+0.945351245 container died 8aa4ee984ddd1662d6e69a8563d6bb517208a3d3b6ed2dbe03ff75f20f2d0ddb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T02:12:12.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T02:12:12.984 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:12:12.984 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T02:12:12.985 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T02:12:12.985 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.1 2024-04-25T02:12:13.287 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:13 smithi110 systemd[1]: Stopping Ceph osd.1 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:12:13.744 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:13 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1[37452]: 2024-04-25T02:12:13.280+0000 7fb84bb0a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:12:13.744 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:13 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1[37452]: 2024-04-25T02:12:13.280+0000 7fb84bb0a700 -1 osd.1 111 *** Got signal Terminated *** 2024-04-25T02:12:13.744 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:13 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1[37452]: 2024-04-25T02:12:13.280+0000 7fb84bb0a700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:12:19.187 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:18 smithi110 podman[70764]: 2024-04-25 02:12:18.906845406 +0000 UTC m=+5.747877277 container died ed52a3260d4f684b3ceb07a43bb0125519b99999dcbea531a0d5e47ee32923f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T02:12:19.494 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:19 smithi110 podman[70764]: 2024-04-25 02:12:19.255276537 +0000 UTC m=+6.096308408 container cleanup ed52a3260d4f684b3ceb07a43bb0125519b99999dcbea531a0d5e47ee32923f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=squid-d2c45d9) 2024-04-25T02:12:19.494 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:19 smithi110 bash[70764]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1 2024-04-25T02:12:19.872 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:19 smithi110 podman[70781]: 2024-04-25 02:12:19.514765005 +0000 UTC m=+0.610389205 container remove ed52a3260d4f684b3ceb07a43bb0125519b99999dcbea531a0d5e47ee32923f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:12:20.157 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:19 smithi110 podman[70872]: 2024-04-25 02:12:19.768329557 +0000 UTC m=+0.017673783 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:12:20.158 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:19 smithi110 podman[70872]: 2024-04-25 02:12:19.884403528 +0000 UTC m=+0.133747735 container create 49004c6c77a5f422d2e9822082fe87ee0df4eb9317dc4026ec5c72387992d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T02:12:20.158 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:20 smithi110 podman[70872]: 2024-04-25 02:12:20.085915571 +0000 UTC m=+0.335259778 container init 49004c6c77a5f422d2e9822082fe87ee0df4eb9317dc4026ec5c72387992d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:12:20.158 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:20 smithi110 podman[70872]: 2024-04-25 02:12:20.089109751 +0000 UTC m=+0.338453967 container start 49004c6c77a5f422d2e9822082fe87ee0df4eb9317dc4026ec5c72387992d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T02:12:20.158 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:20 smithi110 podman[70872]: 2024-04-25 02:12:20.151357045 +0000 UTC m=+0.400701252 container attach 49004c6c77a5f422d2e9822082fe87ee0df4eb9317dc4026ec5c72387992d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T02:12:20.494 INFO:journalctl@ceph.osd.1.smithi110.stdout:Apr 25 02:12:20 smithi110 podman[70872]: 2024-04-25 02:12:20.37333403 +0000 UTC m=+0.622678232 container died 49004c6c77a5f422d2e9822082fe87ee0df4eb9317dc4026ec5c72387992d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T02:12:20.947 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T02:12:20.986 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:12:20.986 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T02:12:20.986 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T02:12:20.986 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.2 2024-04-25T02:12:21.282 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:21 smithi193 systemd[1]: Stopping Ceph osd.2 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:12:21.761 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:21 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2[38151]: 2024-04-25T02:12:21.274+0000 7f17cd137700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:12:21.761 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:21 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2[38151]: 2024-04-25T02:12:21.274+0000 7f17cd137700 -1 osd.2 111 *** Got signal Terminated *** 2024-04-25T02:12:21.761 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:21 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2[38151]: 2024-04-25T02:12:21.274+0000 7f17cd137700 -1 osd.2 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:12:27.261 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:27 smithi193 podman[71094]: 2024-04-25 02:12:27.003070193 +0000 UTC m=+5.828409304 container died d57a39582c2333fc4f8ba6af4871e0e825c5995f6ee60c81c61257175af17c24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:12:27.556 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:27 smithi193 podman[71094]: 2024-04-25 02:12:27.340986449 +0000 UTC m=+6.166325569 container cleanup d57a39582c2333fc4f8ba6af4871e0e825c5995f6ee60c81c61257175af17c24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:12:27.556 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:27 smithi193 bash[71094]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2 2024-04-25T02:12:27.957 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:27 smithi193 podman[71111]: 2024-04-25 02:12:27.550602287 +0000 UTC m=+0.547335798 container remove d57a39582c2333fc4f8ba6af4871e0e825c5995f6ee60c81c61257175af17c24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:12:28.261 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:27 smithi193 podman[71202]: 2024-04-25 02:12:27.951477956 +0000 UTC m=+0.113790118 container create 90fc8caee4ee2246922e12f945c5bb2fecc231c1b9903df7ffe6570d1d8d0c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9) 2024-04-25T02:12:28.261 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:27 smithi193 podman[71202]: 2024-04-25 02:12:27.856224855 +0000 UTC m=+0.018536990 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:12:28.261 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:28 smithi193 podman[71202]: 2024-04-25 02:12:28.187191289 +0000 UTC m=+0.349503409 container init 90fc8caee4ee2246922e12f945c5bb2fecc231c1b9903df7ffe6570d1d8d0c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-25T02:12:28.261 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:28 smithi193 podman[71202]: 2024-04-25 02:12:28.190492063 +0000 UTC m=+0.352804187 container start 90fc8caee4ee2246922e12f945c5bb2fecc231c1b9903df7ffe6570d1d8d0c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T02:12:28.761 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:28 smithi193 podman[71202]: 2024-04-25 02:12:28.278960746 +0000 UTC m=+0.441272872 container attach 90fc8caee4ee2246922e12f945c5bb2fecc231c1b9903df7ffe6570d1d8d0c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T02:12:28.761 INFO:journalctl@ceph.osd.2.smithi193.stdout:Apr 25 02:12:28 smithi193 podman[71202]: 2024-04-25 02:12:28.482403201 +0000 UTC m=+0.644715314 container died 90fc8caee4ee2246922e12f945c5bb2fecc231c1b9903df7ffe6570d1d8d0c4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T02:12:29.048 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T02:12:29.087 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:12:29.087 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T02:12:29.087 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T02:12:29.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.3 2024-04-25T02:12:29.429 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:29 smithi102 systemd[1]: Stopping Ceph osd.3 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:12:29.930 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:29 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3[49716]: 2024-04-25T02:12:29.623+0000 7f3fb6c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:12:29.930 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:29 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3[49716]: 2024-04-25T02:12:29.623+0000 7f3fb6c00700 -1 osd.3 111 *** Got signal Terminated *** 2024-04-25T02:12:29.930 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:29 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3[49716]: 2024-04-25T02:12:29.623+0000 7f3fb6c00700 -1 osd.3 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:12:31.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:30 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:30.518+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:32.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:31 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:31.513+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:32.179 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:31 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3[49716]: 2024-04-25T02:12:31.842+0000 7f3fade00700 -1 osd.3 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.595012+0000 front 2024-04-25T02:12:08.595379+0000 (oldest deadline 2024-04-25T02:12:30.895248+0000) 2024-04-25T02:12:32.743 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:32 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:32.302+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:32.929 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:32 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3[49716]: 2024-04-25T02:12:32.806+0000 7f3fade00700 -1 osd.3 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.595012+0000 front 2024-04-25T02:12:08.595379+0000 (oldest deadline 2024-04-25T02:12:30.895248+0000) 2024-04-25T02:12:32.930 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:32 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:32.563+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:33.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:32 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:32.543+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:33.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:32 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:32.772+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:33.743 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:33 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:33.319+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:33.912 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:33 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:33.577+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:33.929 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:33 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3[49716]: 2024-04-25T02:12:33.781+0000 7f3fade00700 -1 osd.3 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.595012+0000 front 2024-04-25T02:12:08.595379+0000 (oldest deadline 2024-04-25T02:12:30.895248+0000) 2024-04-25T02:12:33.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:33 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:33.536+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:34.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:33 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:33.753+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:34.573 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:34 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:34.308+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:34.713 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:34 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:34.290+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:34.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:34 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:34.565+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:34.993 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:34 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:34.706+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:35.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:34 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:34.580+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:35.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:35 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:35.518+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:35.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:35 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:35.270+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:35.742 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:35 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:35.282+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:35.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:35 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:35.549+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:35.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:35 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:35.485+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:35.993 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:35 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:35.734+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:36.031 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:35 smithi102 podman[96224]: 2024-04-25 02:12:35.675669684 +0000 UTC m=+6.173036012 container died 39cbdd876e58274be83b190b4c74228508b7124745e5bc78efb807a07b0a2486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:12:36.402 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:36 smithi102 podman[96224]: 2024-04-25 02:12:36.14414958 +0000 UTC m=+6.641515901 container cleanup 39cbdd876e58274be83b190b4c74228508b7124745e5bc78efb807a07b0a2486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T02:12:36.402 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:36 smithi102 bash[96224]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3 2024-04-25T02:12:36.402 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:36 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:36.267+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:36.679 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:36 smithi102 podman[96257]: 2024-04-25 02:12:36.395170304 +0000 UTC m=+0.703594252 container remove 39cbdd876e58274be83b190b4c74228508b7124745e5bc78efb807a07b0a2486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T02:12:36.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:36 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:36.520+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:36.743 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:36 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:36.265+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:37.011 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:36.508+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:37.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:36.584+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:37.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:36 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:36.778+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:37.537 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:37 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:37.261+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:37.537 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:37 smithi102 podman[96346]: 2024-04-25 02:12:37.129482189 +0000 UTC m=+0.042552901 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:12:37.537 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:37 smithi102 podman[96346]: 2024-04-25 02:12:37.237894731 +0000 UTC m=+0.150965449 container create 1644c8af61ee46a2e04466ef950cb24574183fb7facf4767d0477d774c7146b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T02:12:37.743 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:37 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:37.277+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:37.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:37 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:37.529+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:37.930 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:37 smithi102 podman[96346]: 2024-04-25 02:12:37.549968148 +0000 UTC m=+0.463038873 container init 1644c8af61ee46a2e04466ef950cb24574183fb7facf4767d0477d774c7146b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:12:37.930 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:37 smithi102 podman[96346]: 2024-04-25 02:12:37.560904561 +0000 UTC m=+0.473975285 container start 1644c8af61ee46a2e04466ef950cb24574183fb7facf4767d0477d774c7146b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T02:12:37.930 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:37 smithi102 podman[96346]: 2024-04-25 02:12:37.690420663 +0000 UTC m=+0.603491444 container attach 1644c8af61ee46a2e04466ef950cb24574183fb7facf4767d0477d774c7146b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T02:12:38.011 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:37 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:37.532+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:38.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:37 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:37.626+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:38.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:37 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:37.813+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:38.269 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:38 smithi102 podman[96346]: 2024-04-25 02:12:38.009289121 +0000 UTC m=+0.922359802 container died 1644c8af61ee46a2e04466ef950cb24574183fb7facf4767d0477d774c7146b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:12:38.549 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:38 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:38.262+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:38.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T02:12:38.743 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:38 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:38.253+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:38.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:38.489+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:38.576+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:38.803 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:38 smithi102 podman[96372]: 2024-04-25 02:12:38.668549987 +0000 UTC m=+0.635140889 container remove 1644c8af61ee46a2e04466ef950cb24574183fb7facf4767d0477d774c7146b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-25T02:12:38.803 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:38 smithi102 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-25T02:12:38.803 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:38 smithi102 systemd[1]: Stopped Ceph osd.3 for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T02:12:38.803 INFO:journalctl@ceph.osd.3.smithi102.stdout:Apr 25 02:12:38 smithi102 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.3.service: Consumed 13min 28.226s CPU time. 2024-04-25T02:12:38.803 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:38 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:38.541+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:38.837 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:12:38.837 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T02:12:38.837 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T02:12:38.837 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.4 2024-04-25T02:12:39.155 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:38 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:38.786+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:39.156 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:38 smithi110 systemd[1]: Stopping Ceph osd.4 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:12:39.493 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:39 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:39.147+0000 7f74deaeb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:12:39.494 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:39 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:39.147+0000 7f74deaeb700 -1 osd.4 111 *** Got signal Terminated *** 2024-04-25T02:12:39.494 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:39 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:39.147+0000 7f74deaeb700 -1 osd.4 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:12:39.494 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:39 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:39.265+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:39.568 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:39.309+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:39.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:39.446+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:39.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:39.616+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:39.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:39.561+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:39.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:39 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:39.561+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:40.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:39 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:39.816+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:40.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:39 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:39.816+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:40.244 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:39 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:39.791+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:40.518 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:40 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:40.281+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:40.618 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:40 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:40.332+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:40.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:40.401+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:40.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:40.606+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:40.835 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:40 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:40.758+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:40.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:40 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:40.611+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:40.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:40 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:40.611+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:41.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:40 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:40.827+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:41.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:40 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:40.827+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:41.643 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:41 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:41.326+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:41.649 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:41.372+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:41.743 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:41 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:41.246+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:41.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:41 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:41.635+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:41.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:41 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:41.635+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:42.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:41.643+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:42.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:41.643+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:42.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:41 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:41.827+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:42.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:41 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:41.827+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:42.244 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:41 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:41.778+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:42.429 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:42 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:42.345+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:42.430 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:42 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:42.345+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:42.668 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:42.395+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:42.743 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:42 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:42.288+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:42.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:42 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:42.736+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:42.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:42 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:42.786+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:42.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:42 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:42.786+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:43.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:42.661+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:43.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:42.661+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:43.179 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:42 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:42.678+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:43.179 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:42 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:42.678+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:43.663 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:43 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:43.367+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:43.663 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:43 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:43.367+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:43.740 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:43 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4[44468]: 2024-04-25T02:12:43.265+0000 7f74db8e0700 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.988785+0000 front 2024-04-25T02:12:05.988715+0000 (oldest deadline 2024-04-25T02:12:31.888505+0000) 2024-04-25T02:12:43.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:43 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:43.436+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:43.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:43 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:43.436+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:18.130249+0000 front 2024-04-25T02:12:18.130235+0000 (oldest deadline 2024-04-25T02:12:42.830259+0000) 2024-04-25T02:12:43.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:43 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:43.615+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:43.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:43 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:43.615+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:43.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:43 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:43.656+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:43.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:43 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:43.656+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:43.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:43 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:43.733+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:43.993 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:43 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:43.817+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:43.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:43 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:43.817+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:44.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:43 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:43.767+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:44.647 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:44 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:44.367+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:44.647 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:44 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:44.367+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:44.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:44 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:44.479+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:44.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:44 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:44.479+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:18.130249+0000 front 2024-04-25T02:12:18.130235+0000 (oldest deadline 2024-04-25T02:12:42.830259+0000) 2024-04-25T02:12:44.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:44 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:44.578+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:44.762 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:44 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:44.578+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:44.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:44 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:44.639+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:44.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:44 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:44.639+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:44.972 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:44 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:44.695+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:44.973 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:44 smithi110 podman[70977]: 2024-04-25 02:12:44.708452386 +0000 UTC m=+5.689005426 container died 277050e002be5567d2c54f9486ff68eb15b5626bb1c47422d83d80675235ec1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-25T02:12:44.973 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:44 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:44.821+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:44.973 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:44 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:44.821+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:45.239 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:45 smithi110 podman[70977]: 2024-04-25 02:12:45.040264607 +0000 UTC m=+6.020817656 container cleanup 277050e002be5567d2c54f9486ff68eb15b5626bb1c47422d83d80675235ec1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-25T02:12:45.239 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:45 smithi110 bash[70977]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4 2024-04-25T02:12:45.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:44 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:44.784+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:45.493 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:45 smithi110 podman[71008]: 2024-04-25 02:12:45.233101821 +0000 UTC m=+0.523268748 container remove 277050e002be5567d2c54f9486ff68eb15b5626bb1c47422d83d80675235ec1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:12:45.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:45 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:45.613+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:45.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:45 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:45.613+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:45.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:45 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:45.383+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:45.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:45 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:45.383+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:45.756 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:45 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:45.449+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:45.757 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:45 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:45.449+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:18.130249+0000 front 2024-04-25T02:12:18.130235+0000 (oldest deadline 2024-04-25T02:12:42.830259+0000) 2024-04-25T02:12:45.757 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:45 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:45.586+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:45.757 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:45 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:45.586+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:45.975 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:45 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:45.672+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:45.975 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:45 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:45.863+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:45.975 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:45 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:45.863+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:45.975 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:45 smithi110 podman[71104]: 2024-04-25 02:12:45.552143267 +0000 UTC m=+0.018030140 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:12:45.975 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:45 smithi110 podman[71104]: 2024-04-25 02:12:45.658936489 +0000 UTC m=+0.124823350 container create bca3db6962bbf99538f030356eb594a18c5bce6a5677c105f173b59133bdad1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T02:12:45.976 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:45 smithi110 podman[71104]: 2024-04-25 02:12:45.894524887 +0000 UTC m=+0.360411748 container init bca3db6962bbf99538f030356eb594a18c5bce6a5677c105f173b59133bdad1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-25T02:12:45.976 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:45 smithi110 podman[71104]: 2024-04-25 02:12:45.89752157 +0000 UTC m=+0.363408430 container start bca3db6962bbf99538f030356eb594a18c5bce6a5677c105f173b59133bdad1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T02:12:46.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:45 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:45.750+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:46.243 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:45 smithi110 podman[71104]: 2024-04-25 02:12:45.969270396 +0000 UTC m=+0.435157256 container attach bca3db6962bbf99538f030356eb594a18c5bce6a5677c105f173b59133bdad1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T02:12:46.244 INFO:journalctl@ceph.osd.4.smithi110.stdout:Apr 25 02:12:46 smithi110 podman[71104]: 2024-04-25 02:12:46.186419349 +0000 UTC m=+0.652306211 container died bca3db6962bbf99538f030356eb594a18c5bce6a5677c105f173b59133bdad1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T02:12:46.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:46 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:46.573+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:46.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:46 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:46.573+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:46.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:46 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:46.346+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:46.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:46 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:46.346+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:46.720 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:46 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:46.427+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:46.720 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:46 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:46.427+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:18.130249+0000 front 2024-04-25T02:12:18.130235+0000 (oldest deadline 2024-04-25T02:12:42.830259+0000) 2024-04-25T02:12:46.720 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:46 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:46.612+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:46.720 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:46 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:46.612+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:46.795 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:46 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:46.657+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:46.823 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T02:12:46.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:12:46.862 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T02:12:46.862 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T02:12:46.863 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.5 2024-04-25T02:12:47.011 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:46 smithi193 systemd[1]: Stopping Ceph osd.5 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:12:47.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:46 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:46.714+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:47.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:46 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:46.816+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:47.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:46 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:46.816+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:47.452 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:47 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:47.175+0000 7fccda24d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:12:47.452 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:47 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:47.175+0000 7fccda24d700 -1 osd.5 111 *** Got signal Terminated *** 2024-04-25T02:12:47.453 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:47 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:47.175+0000 7fccda24d700 -1 osd.5 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:12:47.599 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:47 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:47.329+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:47.600 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:47 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:47.329+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:47.712 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:47 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:47.446+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:47.712 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:47 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:47.446+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:18.130249+0000 front 2024-04-25T02:12:18.130235+0000 (oldest deadline 2024-04-25T02:12:42.830259+0000) 2024-04-25T02:12:47.713 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:47 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:47.609+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:47.713 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:47 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:47.609+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:47.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:47 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:47.592+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:47.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:47 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:47.592+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:47.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:47 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:47.619+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:47.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:47 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:47.619+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:47.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:47 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:47.788+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:47.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:47 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:47.788+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:48.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:47 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:47.706+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:48.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:48.557+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:48.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:48.557+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:48.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:48.331+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:48.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:48 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:48.331+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:48.761 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:48 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:48.667+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:48.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:48 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:48.445+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:48.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:48 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:48.445+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:18.130249+0000 front 2024-04-25T02:12:18.130235+0000 (oldest deadline 2024-04-25T02:12:42.830259+0000) 2024-04-25T02:12:48.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:48 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:48.651+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:48.762 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:48 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:48.651+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:48.762 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:48 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:48.651+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:48.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:48 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:48.591+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:48.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:48 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:48.591+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:48.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:48 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:48.768+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:48.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:48 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:48.768+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:48.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:48 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:48.768+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:49.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:49.522+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:49.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:49.522+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:49.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:49.296+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:49.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:49 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:49.296+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:49.761 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:49 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:49.711+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:49.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:49 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:49.470+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:49.761 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:49 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:49.470+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:18.130249+0000 front 2024-04-25T02:12:18.130235+0000 (oldest deadline 2024-04-25T02:12:42.830259+0000) 2024-04-25T02:12:49.762 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:49 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:49.633+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:49.762 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:49 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:49.633+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:49.762 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:49 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:49.633+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:49.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:49 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:49.603+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:49.994 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:49 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:49.603+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:49.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:49 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:49.809+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:49.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:49 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:49.809+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:49.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:49 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:49.809+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:50.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:50 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:50.490+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:50.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:50 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:50.490+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:50.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:50 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:50.490+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:50.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:50 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:50.309+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:50.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:50 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:50.309+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:50.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:50 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:50.309+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:50.956 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:50 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:50.633+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:50.956 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:50 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:50.633+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:50.956 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:50 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:50.800+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:50.956 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:50 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:50.800+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:50.956 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:50 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:50.800+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:51.011 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:50 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:50.518+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:51.011 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:50 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:50.518+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:18.130249+0000 front 2024-04-25T02:12:18.130235+0000 (oldest deadline 2024-04-25T02:12:42.830259+0000) 2024-04-25T02:12:51.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:50 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:50.680+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:51.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:50 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:50.680+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:12:51.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:50 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:50.609+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:51.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:50 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:50.609+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:51.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:50 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:50.609+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:51.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:51.521+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:51.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:51.521+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:51.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:51.521+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:51.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:51.349+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:51.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:51.349+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:51.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:51 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:51.349+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:51.824 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:51.595+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:51.824 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:51.595+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:52.011 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:51.512+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:05.927911+0000 front 2024-04-25T02:12:05.927989+0000 (oldest deadline 2024-04-25T02:12:35.328459+0000) 2024-04-25T02:12:52.011 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:51.512+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:18.130249+0000 front 2024-04-25T02:12:18.130235+0000 (oldest deadline 2024-04-25T02:12:42.830259+0000) 2024-04-25T02:12:52.011 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5[44694]: 2024-04-25T02:12:51.512+0000 7fccd7042700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.131200+0000 front 2024-04-25T02:12:25.131112+0000 (oldest deadline 2024-04-25T02:12:51.031301+0000) 2024-04-25T02:12:52.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:51.672+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:52.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:51.672+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:12:52.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:51.590+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:52.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:51.590+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:52.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:51 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:51.590+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:52.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:51.817+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:52.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:51.817+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:52.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:51 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:51.817+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:52.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:52.557+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:52.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:52.557+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:52.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:52.557+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:52.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:52.374+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:52.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:52.374+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:52.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:52.374+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:52.860 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:52 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:52.578+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:52.860 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:52 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:52.578+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:53.011 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:52 smithi193 podman[71323]: 2024-04-25 02:12:52.773823649 +0000 UTC m=+5.722394473 container died 3187d04a345abd3278494cd08d1c97d15057a8c23a03e4bb3c1251b5c52c5b42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux ) 2024-04-25T02:12:53.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:52 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:52.655+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:53.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:52 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:52.655+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:12:53.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:52 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:52.577+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:53.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:52 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:52.577+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:53.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:52 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:52.577+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:53.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:52 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:52.853+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:53.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:52 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:52.853+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:53.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:52 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:52.853+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:53.323 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:53 smithi193 podman[71323]: 2024-04-25 02:12:53.108055316 +0000 UTC m=+6.056626130 container cleanup 3187d04a345abd3278494cd08d1c97d15057a8c23a03e4bb3c1251b5c52c5b42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:12:53.323 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:53 smithi193 bash[71323]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5 2024-04-25T02:12:53.668 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:53 smithi193 podman[71342]: 2024-04-25 02:12:53.317930794 +0000 UTC m=+0.537314299 container remove 3187d04a345abd3278494cd08d1c97d15057a8c23a03e4bb3c1251b5c52c5b42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:12:53.668 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:53 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:53.528+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:53.668 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:53 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:53.528+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:53.668 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:53 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:53.528+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:53.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:53 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:53.540+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:53.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:53 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:53.540+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:53.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:53 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:53.540+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:53.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:53 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:53.388+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:53.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:53 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:53.388+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:53.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:53 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:53.388+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:53.897 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:53 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:53.595+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:53.897 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:53 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:53.595+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:53.985 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:53 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:53.661+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:53.985 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:53 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:53.661+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:12:53.985 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:53 smithi193 podman[71428]: 2024-04-25 02:12:53.626504534 +0000 UTC m=+0.018197802 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:12:53.985 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:53 smithi193 podman[71428]: 2024-04-25 02:12:53.752284118 +0000 UTC m=+0.143977424 container create 98b9acbb5e3de3aefc22eb7c01b1c6e3008652ee538233ed13f616e76b7cdfbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:12:54.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:53 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:53.890+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:54.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:53 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:53.890+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:54.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:53 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:53.890+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:54.261 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:53 smithi193 podman[71428]: 2024-04-25 02:12:53.979425559 +0000 UTC m=+0.371118832 container init 98b9acbb5e3de3aefc22eb7c01b1c6e3008652ee538233ed13f616e76b7cdfbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-25T02:12:54.262 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:53 smithi193 podman[71428]: 2024-04-25 02:12:53.983033294 +0000 UTC m=+0.374726551 container start 98b9acbb5e3de3aefc22eb7c01b1c6e3008652ee538233ed13f616e76b7cdfbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:12:54.262 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:54 smithi193 podman[71428]: 2024-04-25 02:12:54.054268724 +0000 UTC m=+0.445962002 container attach 98b9acbb5e3de3aefc22eb7c01b1c6e3008652ee538233ed13f616e76b7cdfbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T02:12:54.575 INFO:journalctl@ceph.osd.5.smithi193.stdout:Apr 25 02:12:54 smithi193 podman[71428]: 2024-04-25 02:12:54.273591233 +0000 UTC m=+0.665284488 container died 98b9acbb5e3de3aefc22eb7c01b1c6e3008652ee538233ed13f616e76b7cdfbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9) 2024-04-25T02:12:54.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:54 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:54.558+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:54.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:54 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:54.558+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:54.679 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:54 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:54.558+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:54.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:54 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:54.386+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:54.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:54 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:54.386+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:54.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:54 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:54.386+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:54.838 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:54 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:54.654+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:54.838 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:54 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:54.654+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:12:54.839 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:54 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:54.567+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:54.839 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:54 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:54.567+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:54.839 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:54 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:54.567+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:54.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T02:12:54.888 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:54 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:54.561+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:54.889 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:54 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:54.561+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:54.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:12:54.893 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T02:12:54.893 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T02:12:54.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.6 2024-04-25T02:12:55.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:54 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:54.882+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:55.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:54 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:54.882+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:55.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:54 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:54.882+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:55.391 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:55 smithi102 systemd[1]: Stopping Ceph osd.6 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:12:55.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:55 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:55.384+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:55.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:55 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:55.384+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:55.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:55 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:55.384+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:55.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:55 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:55.513+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:55.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:55 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:55.513+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:55.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:55 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:55.513+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:55.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:55 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:55.519+0000 7f038ca00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:12:55.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:55 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:55.519+0000 7f038ca00700 -1 osd.6 111 *** Got signal Terminated *** 2024-04-25T02:12:55.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:55 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:55.519+0000 7f038ca00700 -1 osd.6 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:12:55.914 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:55 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:55.573+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:55.914 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:55 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:55.573+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:56.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:55 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:55.703+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:56.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:55 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:55.703+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:12:56.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:55 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:55.523+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:56.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:55 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:55.523+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:56.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:55 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:55.523+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:56.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:55 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:55.907+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:56.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:55 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:55.907+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:56.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:55 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:55.907+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:56.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:56 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:56.406+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:56.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:56 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:56.406+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:56.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:56 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:56.406+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:56.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:56 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:56.543+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:56.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:56 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:56.543+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:56.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:56 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:56.543+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:56.680 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:56 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:56.543+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:30.110318+0000 front 2024-04-25T02:12:30.110356+0000 (oldest deadline 2024-04-25T02:12:56.009767+0000) 2024-04-25T02:12:56.913 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:56 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:56.611+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:56.913 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:56 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:56.611+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:56.913 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:56 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:56.611+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:12:57.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:56 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:56.555+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:57.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:56 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:56.555+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:57.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:56 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:56.555+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:57.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:56 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:56.748+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:57.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:56 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:56.748+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:12:57.012 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:56 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:56.748+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:12:57.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:56 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:56.906+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:57.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:56 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:56.906+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:57.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:56 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:56.906+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:57.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:56 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:56.906+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:12:57.897 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:57 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:57.600+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:57.897 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:57 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:57.600+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:57.897 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:57 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:57.600+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:12:57.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:57 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:57.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:57.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:57 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:57.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:57.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:57 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:57.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:57.930 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:57 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:57.503+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:57.930 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:57 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:57.503+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:57.930 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:57 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:57.503+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:57.930 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:57 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:57.503+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:30.110318+0000 front 2024-04-25T02:12:30.110356+0000 (oldest deadline 2024-04-25T02:12:56.009767+0000) 2024-04-25T02:12:58.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:57 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:57.753+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:58.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:57 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:57.753+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:12:58.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:57 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:57.753+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:12:58.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:57 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:57.516+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:58.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:57 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:57.516+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:58.012 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:57 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:57.516+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:58.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:57 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:57.890+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:58.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:57 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:57.890+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:58.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:57 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:57.890+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:58.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:57 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:57.890+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:12:58.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:58 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:58.492+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:12:58.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:58 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:58.492+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:12:58.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:58 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:58.492+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:12:58.761 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:58 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:58.732+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:12:58.761 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:58 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:58.732+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:12:58.762 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:58 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:58.732+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:12:58.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:58 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:58.546+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:58.929 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:58 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:58.546+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:58.930 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:58 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:58.546+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:58.930 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:58 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:58.546+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:30.110318+0000 front 2024-04-25T02:12:30.110356+0000 (oldest deadline 2024-04-25T02:12:56.009767+0000) 2024-04-25T02:12:58.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:58 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:58.449+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:58.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:58 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:58.449+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:58.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:58 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:58.449+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:58.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:58 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:58.449+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:12:58.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:58 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:58.617+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:58.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:58 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:58.617+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:58.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:58 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:58.617+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:12:58.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:58 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:58.852+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:58.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:58 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:58.852+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:58.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:58 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:58.852+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:58.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:58 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:58.852+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:12:59.832 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:59.565+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:06.005462+0000 front 2024-04-25T02:12:06.005536+0000 (oldest deadline 2024-04-25T02:12:31.905106+0000) 2024-04-25T02:12:59.832 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:59.565+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.806984+0000 front 2024-04-25T02:12:14.807034+0000 (oldest deadline 2024-04-25T02:12:39.507070+0000) 2024-04-25T02:12:59.832 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:59.565+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.209664+0000 front 2024-04-25T02:12:24.209072+0000 (oldest deadline 2024-04-25T02:12:50.108943+0000) 2024-04-25T02:12:59.832 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:12:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6[56292]: 2024-04-25T02:12:59.565+0000 7f0388c00700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:30.110318+0000 front 2024-04-25T02:12:30.110356+0000 (oldest deadline 2024-04-25T02:12:56.009767+0000) 2024-04-25T02:12:59.832 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:59.476+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:12:59.833 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:59.476+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:12:59.833 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:59.476+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:12:59.833 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:12:59 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:12:59.476+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:12:59.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:59 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:59.655+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:12:59.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:59 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:59.655+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:12:59.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:12:59 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:12:59.655+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:12:59.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:59 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:59.881+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:12:59.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:59 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:59.881+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:12:59.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:59 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:59.881+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:12:59.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:12:59 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:12:59.881+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:00.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:59 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:59.737+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:00.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:59 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:59.737+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:00.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:12:59 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:12:59.737+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:00.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:59 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:59.524+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:00.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:59 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:59.524+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:00.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:12:59 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:12:59.524+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:00.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:00 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:00.484+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:00.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:00 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:00.484+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:00.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:00 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:00.484+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:00.862 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:00.490+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:00.862 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:00.490+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:00.862 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:00.490+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:00.862 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:00 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:00.490+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:00.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:00 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:00.668+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:13:00.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:00 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:00.668+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:13:00.994 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:00 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:00.668+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:13:00.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:00 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:00.903+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:00.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:00 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:00.904+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:00.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:00 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:00.904+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:00.994 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:00 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:00.904+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:01.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:00 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:00.758+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:01.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:00 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:00.758+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:01.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:00 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:00.758+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:01.476 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:01 smithi102 podman[96451]: 2024-04-25 02:13:01.416159197 +0000 UTC m=+6.077958667 container died 4658a2f6896d9c90d8311abc670c9ae868d2af8e033f40eaa213077dc132c9d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T02:13:01.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:01 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:01.453+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:01.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:01 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:01.453+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:01.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:01 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:01.453+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:01.766 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:01 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:01.469+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:01.767 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:01 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:01.469+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:01.767 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:01 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:01.469+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:01.767 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:01 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:01.469+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:01.954 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:01 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:01.687+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:13:01.954 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:01 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:01.687+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:13:01.954 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:01 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:01.687+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:13:02.128 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:01 smithi102 podman[96451]: 2024-04-25 02:13:01.871290486 +0000 UTC m=+6.533089999 container cleanup 4658a2f6896d9c90d8311abc670c9ae868d2af8e033f40eaa213077dc132c9d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6, ceph=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T02:13:02.129 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:01 smithi102 bash[96451]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6 2024-04-25T02:13:02.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:01 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:01.947+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:02.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:01 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:01.947+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:02.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:01 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:01.947+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:02.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:01 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:01.947+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:02.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:01 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:01.776+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:02.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:01 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:01.776+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:02.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:01 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:01.776+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:02.429 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:02 smithi102 podman[96473]: 2024-04-25 02:13:02.122216287 +0000 UTC m=+0.686761085 container remove 4658a2f6896d9c90d8311abc670c9ae868d2af8e033f40eaa213077dc132c9d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:13:02.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:02 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:02.484+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:02.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:02 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:02.484+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:02.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:02 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:02.484+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:02.803 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:02 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:02.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:02.803 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:02 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:02.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:02.803 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:02 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:02.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:02.803 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:02 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:02.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:02.964 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:02 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:02.704+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:13:02.964 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:02 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:02.704+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:13:02.964 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:02 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:02.704+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:13:03.099 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:02 smithi102 podman[96569]: 2024-04-25 02:13:02.702530899 +0000 UTC m=+0.032991188 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:13:03.099 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:02 smithi102 podman[96569]: 2024-04-25 02:13:02.823198675 +0000 UTC m=+0.153658995 container create 2ffc473270f1bd9998f06c5a6041d97f0abbf01a98190ae9eb625b5c86f7c520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T02:13:03.099 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:03 smithi102 podman[96569]: 2024-04-25 02:13:03.093208292 +0000 UTC m=+0.423668576 container init 2ffc473270f1bd9998f06c5a6041d97f0abbf01a98190ae9eb625b5c86f7c520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T02:13:03.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:02 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:02.957+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:03.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:02 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:02.957+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:03.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:02 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:02.957+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:03.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:02 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:02.957+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:03.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:02 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:02.760+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:03.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:02 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:02.760+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:03.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:02 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:02.760+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:03.429 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:03 smithi102 podman[96569]: 2024-04-25 02:13:03.103654902 +0000 UTC m=+0.434115192 container start 2ffc473270f1bd9998f06c5a6041d97f0abbf01a98190ae9eb625b5c86f7c520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T02:13:03.429 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:03 smithi102 podman[96569]: 2024-04-25 02:13:03.184084522 +0000 UTC m=+0.514544849 container attach 2ffc473270f1bd9998f06c5a6041d97f0abbf01a98190ae9eb625b5c86f7c520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:13:03.536 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:03 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:03.469+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:03.536 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:03 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:03.469+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:03.537 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:03 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:03.469+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:03.907 INFO:journalctl@ceph.osd.6.smithi102.stdout:Apr 25 02:13:03 smithi102 podman[96569]: 2024-04-25 02:13:03.556784002 +0000 UTC m=+0.887244329 container died 2ffc473270f1bd9998f06c5a6041d97f0abbf01a98190ae9eb625b5c86f7c520 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:13:03.907 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:03.441+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:03.908 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:03.441+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:03.908 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:03.441+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:03.908 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:03.441+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:03.973 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:03.700+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:13:03.973 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:03.700+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:13:03.973 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:03.700+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:13:04.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:03 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:03.761+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:04.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:03 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:03.761+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:04.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:03 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:03.761+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:04.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:03.967+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:04.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:03.967+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:04.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:03.967+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:04.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:03 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:03.967+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:04.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T02:13:04.453 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:13:04.453 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T02:13:04.454 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T02:13:04.454 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.7 2024-04-25T02:13:04.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:04 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:04.459+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:04.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:04 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:04.459+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:04.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:04 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:04.459+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:04.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:04 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:04.459+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:04.743 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:04 smithi110 systemd[1]: Stopping Ceph osd.7 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:13:04.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:04.507+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:04.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:04.507+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:04.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:04.507+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:05.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:04.747+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:05.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:04.747+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:05.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:04 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:04.747+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:05.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:04.978+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:05.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:04.978+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:05.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:04.978+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:05.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:04.978+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:05.244 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:04.737+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:13:05.244 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:04.737+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:13:05.244 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:04.737+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:13:05.244 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:04.776+0000 7f1e63322700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:13:05.244 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:04.776+0000 7f1e63322700 -1 osd.7 111 *** Got signal Terminated *** 2024-04-25T02:13:05.244 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:04 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:04.776+0000 7f1e63322700 -1 osd.7 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:13:05.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:05 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:05.496+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:05.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:05 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:05.496+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:05.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:05 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:05.496+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:05.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:05 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:05.510+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:05.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:05 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:05.510+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:05.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:05 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:05.510+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:05.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:05 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:05.510+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:06.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:05 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:05.778+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:13:06.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:05 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:05.778+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:13:06.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:05 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:05.778+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:13:06.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:05 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:05.961+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:06.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:05 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:05.961+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:06.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:05 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:05.961+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:06.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:05 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:05.961+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:06.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:05 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:05.753+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:06.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:05 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:05.753+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:06.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:05 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:05.753+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:06.752 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:06.449+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:06.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:06.449+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:06.753 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:06.449+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:06.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:06.491+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:06.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:06.491+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:06.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:06.491+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:06.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:06 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:06.491+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:07.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:06.746+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:07.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:06.746+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:07.011 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:06 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:06.746+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:07.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:06.826+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:13:07.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:06.826+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:13:07.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:06.826+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:13:07.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:06.991+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:07.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:06.991+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:07.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:06.991+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:07.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:06 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:06.991+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:07.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:07.416+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:07.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:07.416+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:07.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:07.416+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:07.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:07.416+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:07.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:07 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:07.470+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:07.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:07 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:07.470+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:07.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:07 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:07.470+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:07.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:07 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:07.470+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:07.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:07 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:07.470+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:08.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:07 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:07.824+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:13:08.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:07 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:07.824+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:13:08.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:07 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:07.824+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:13:08.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:08.026+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:08.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:08.026+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:08.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:08.026+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:08.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:08.026+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:08.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:07.767+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:08.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:07.767+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:08.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:07 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:07.767+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:08.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:08.444+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:08.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:08.444+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:08.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:08.444+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:08.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:08.444+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:08.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:08 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:08.429+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:08.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:08 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:08.429+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:08.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:08 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:08.429+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:08.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:08 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:08.429+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:08.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:08 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:08.429+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:09.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:08.847+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:14.840250+0000 front 2024-04-25T02:12:14.840286+0000 (oldest deadline 2024-04-25T02:12:39.540359+0000) 2024-04-25T02:13:09.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:08.847+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.042626+0000 front 2024-04-25T02:12:24.042665+0000 (oldest deadline 2024-04-25T02:12:47.542753+0000) 2024-04-25T02:13:09.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:08.847+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.744753+0000 front 2024-04-25T02:12:32.744782+0000 (oldest deadline 2024-04-25T02:12:56.244640+0000) 2024-04-25T02:13:09.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7[51345]: 2024-04-25T02:13:08.847+0000 7f1e60117700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.646000+0000 front 2024-04-25T02:12:42.646022+0000 (oldest deadline 2024-04-25T02:13:07.946094+0000) 2024-04-25T02:13:09.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:08.996+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:09.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:08.996+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:09.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:08.996+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:09.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:08.996+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:09.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:08 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:08.996+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:09.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:08.785+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:09.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:08.785+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:09.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:08 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:08.785+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:09.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:09.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:09.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:09.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:09.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:09.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:09.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:09.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:09.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:09.454+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:09.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:09.454+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:09.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:09.454+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:09.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:09.454+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:09.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:09 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:09.454+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:10.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:09.952+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:10.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:09.952+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:10.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:09.952+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:10.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:09.952+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:10.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:09.952+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:10.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:09.761+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:10.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:09.761+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:10.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:09.761+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:10.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:09 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:09.761+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:39.136891+0000 front 2024-04-25T02:12:39.136951+0000 (oldest deadline 2024-04-25T02:13:09.137564+0000) 2024-04-25T02:13:10.664 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:10 smithi110 podman[71212]: 2024-04-25 02:13:10.392322423 +0000 UTC m=+5.748079116 container died 18eb98db8ad0b040277105ef40d959b2ee85ec9cf5211c72ed08310138e71d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T02:13:10.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:10.467+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:10.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:10.467+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:10.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:10.467+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:10.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:10.467+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:10.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:10 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:10.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:10.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:10 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:10.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:10.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:10 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:10.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:10.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:10 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:10.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:10.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:10 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:10.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:10.971 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:10 smithi110 podman[71212]: 2024-04-25 02:13:10.758123127 +0000 UTC m=+6.113879823 container cleanup 18eb98db8ad0b040277105ef40d959b2ee85ec9cf5211c72ed08310138e71d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True) 2024-04-25T02:13:10.972 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:10 smithi110 bash[71212]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7 2024-04-25T02:13:11.243 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:11 smithi110 podman[71230]: 2024-04-25 02:13:11.009574498 +0000 UTC m=+0.616773764 container remove 18eb98db8ad0b040277105ef40d959b2ee85ec9cf5211c72ed08310138e71d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:13:11.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:10.965+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:11.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:10.965+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:11.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:10.965+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:11.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:10.965+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:11.244 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:10 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:10.965+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:11.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:10.811+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:11.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:10.811+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:11.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:10.811+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:11.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:10 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:10.811+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:39.136891+0000 front 2024-04-25T02:12:39.136951+0000 (oldest deadline 2024-04-25T02:13:09.137564+0000) 2024-04-25T02:13:11.677 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:11 smithi110 podman[71330]: 2024-04-25 02:13:11.255073433 +0000 UTC m=+0.018400917 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:13:11.677 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:11 smithi110 podman[71330]: 2024-04-25 02:13:11.377417864 +0000 UTC m=+0.140745322 container create 95a9073bc0a4c1f2451a1851b764b56d31d4773514932c1851dd0a72e8458c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T02:13:11.677 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:11 smithi110 podman[71330]: 2024-04-25 02:13:11.587730531 +0000 UTC m=+0.351057991 container init 95a9073bc0a4c1f2451a1851b764b56d31d4773514932c1851dd0a72e8458c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T02:13:11.677 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:11 smithi110 podman[71330]: 2024-04-25 02:13:11.591287757 +0000 UTC m=+0.354615216 container start 95a9073bc0a4c1f2451a1851b764b56d31d4773514932c1851dd0a72e8458c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:13:11.677 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:11 smithi110 podman[71330]: 2024-04-25 02:13:11.671047508 +0000 UTC m=+0.434374974 container attach 95a9073bc0a4c1f2451a1851b764b56d31d4773514932c1851dd0a72e8458c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:13:11.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:11 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:11.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:11.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:11 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:11.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:11.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:11 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:11.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:11.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:11 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:11.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:11.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:11 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:11.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:11.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:11.497+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:11.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:11.497+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:11.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:11.497+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:11.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:11.497+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:11.993 INFO:journalctl@ceph.osd.7.smithi110.stdout:Apr 25 02:13:11 smithi110 podman[71330]: 2024-04-25 02:13:11.883888985 +0000 UTC m=+0.647216444 container died 95a9073bc0a4c1f2451a1851b764b56d31d4773514932c1851dd0a72e8458c80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T02:13:12.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:11.844+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:12.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:11.844+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:12.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:11.844+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:12.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:11 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:11.844+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:39.136891+0000 front 2024-04-25T02:12:39.136951+0000 (oldest deadline 2024-04-25T02:13:09.137564+0000) 2024-04-25T02:13:12.421 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.014+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:12.421 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.014+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:12.421 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.014+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:12.421 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.014+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:12.422 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.014+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:12.450 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T02:13:12.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:13:12.489 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T02:13:12.490 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T02:13:12.490 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.8 2024-04-25T02:13:12.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:12 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:12.355+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:12.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:12 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:12.355+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:12.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:12 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:12.355+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:12.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:12 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:12.355+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:12.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:12 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:12.355+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:12.782 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:12 smithi193 systemd[1]: Stopping Ceph osd.8 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:13:12.782 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:12.775+0000 7fc34de7e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:13:12.783 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:12.532+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:12.783 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:12.532+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:12.783 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:12.532+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:12.783 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:12.532+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:13.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:12.775+0000 7fc34de7e700 -1 osd.8 111 *** Got signal Terminated *** 2024-04-25T02:13:13.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:12.775+0000 7fc34de7e700 -1 osd.8 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:13:13.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:12.876+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:13.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:12.876+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:13.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:12.876+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:13.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:12 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:12.876+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:39.136891+0000 front 2024-04-25T02:12:39.136951+0000 (oldest deadline 2024-04-25T02:13:09.137564+0000) 2024-04-25T02:13:13.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.994+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:13.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.994+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:13.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.994+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:13.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.994+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:13.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:12.994+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:13.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:13.385+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:13.744 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:13.385+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:13.744 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:13.385+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:13.745 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:13.385+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:13.745 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:13 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:13.385+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:13.911 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:13 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:13.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:13.912 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:13 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:13.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:13.912 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:13 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:13.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:13.912 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:13 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:13.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:14.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:13 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:13.905+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:14.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:13 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:13.905+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:14.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:13 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:13.905+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:14.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:13 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:13.905+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:39.136891+0000 front 2024-04-25T02:12:39.136951+0000 (oldest deadline 2024-04-25T02:13:09.137564+0000) 2024-04-25T02:13:14.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:13 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:13.905+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:52.038367+0000 front 2024-04-25T02:12:52.038389+0000 (oldest deadline 2024-04-25T02:13:13.639009+0000) 2024-04-25T02:13:14.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.019+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:14.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.019+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:14.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.019+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:14.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.019+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:14.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.019+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:14.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.019+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:14.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:14.381+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:14.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:14.381+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:14.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:14.381+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:14.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:14.381+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:14.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:14.381+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:14.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:14 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:14.381+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:14.944 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:14 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:14.577+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:14.944 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:14 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:14.577+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:14.944 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:14 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:14.577+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:14.944 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:14 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:14.577+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:15.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:14 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:14.937+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:15.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:14 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:14.937+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:15.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:14 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:14.937+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:15.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:14 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:14.937+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:39.136891+0000 front 2024-04-25T02:12:39.136951+0000 (oldest deadline 2024-04-25T02:13:09.137564+0000) 2024-04-25T02:13:15.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:14 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:14.937+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:52.038367+0000 front 2024-04-25T02:12:52.038389+0000 (oldest deadline 2024-04-25T02:13:13.639009+0000) 2024-04-25T02:13:15.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.992+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:15.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.992+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:15.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.992+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:15.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.992+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:15.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.992+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:15.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:14 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:14.992+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:15.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:15.396+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:15.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:15.396+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:15.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:15.396+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:15.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:15.396+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:15.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:15.396+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:15.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:15 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:15.396+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:15.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:15 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:15.561+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:15.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:15 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:15.561+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:15.962 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:15 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:15.561+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:15.962 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:15 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:15.561+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:16.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:15 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:15.955+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:16.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:15 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:15.955+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:16.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:15 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:15.955+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:16.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:15 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:15.955+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:39.136891+0000 front 2024-04-25T02:12:39.136951+0000 (oldest deadline 2024-04-25T02:13:09.137564+0000) 2024-04-25T02:13:16.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:15 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:15.955+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:52.038367+0000 front 2024-04-25T02:12:52.038389+0000 (oldest deadline 2024-04-25T02:13:13.639009+0000) 2024-04-25T02:13:16.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:16 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:16.040+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:16.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:16 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:16.040+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:16.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:16 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:16.040+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:16.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:16 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:16.040+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:16.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:16 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:16.040+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:16.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:16 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:16.040+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:16.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:16 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:16.415+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:16.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:16 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:16.415+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:16.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:16 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:16.415+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:16.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:16 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:16.415+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:16.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:16 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:16.415+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:16.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:16 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:16.415+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:16.948 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:16.512+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:16.948 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:16.512+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:16.948 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:16.512+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:16.948 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:16.512+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:16.948 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:16.512+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:17.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:16.942+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.633332+0000 front 2024-04-25T02:12:17.633507+0000 (oldest deadline 2024-04-25T02:12:43.533359+0000) 2024-04-25T02:13:17.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:16.942+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:25.834319+0000 front 2024-04-25T02:12:25.834436+0000 (oldest deadline 2024-04-25T02:12:49.934416+0000) 2024-04-25T02:13:17.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:16.942+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.136057+0000 front 2024-04-25T02:12:32.136154+0000 (oldest deadline 2024-04-25T02:12:56.235959+0000) 2024-04-25T02:13:17.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:16.942+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:39.136891+0000 front 2024-04-25T02:12:39.136951+0000 (oldest deadline 2024-04-25T02:13:09.137564+0000) 2024-04-25T02:13:17.261 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:16 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8[51569]: 2024-04-25T02:13:16.942+0000 7fc346c6b700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:52.038367+0000 front 2024-04-25T02:12:52.038389+0000 (oldest deadline 2024-04-25T02:13:13.639009+0000) 2024-04-25T02:13:17.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:17 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:17.062+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:17.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:17 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:17.062+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:17.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:17 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:17.062+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:17.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:17 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:17.062+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:17.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:17 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:17.062+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:17.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:17 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:17.062+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:17.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:17 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:17.406+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:17.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:17 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:17.406+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:17.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:17 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:17.406+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:17.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:17 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:17.406+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:17.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:17 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:17.406+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:17.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:17 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:17.406+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:18.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:17 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:17.513+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:18.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:17 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:17.513+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:18.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:17 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:17.513+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:18.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:17 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:17.513+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:18.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:17 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:17.513+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:18.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:18 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:18.015+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:18.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:18 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:18.015+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:18.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:18 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:18.015+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:18.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:18 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:18.015+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:18.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:18 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:18.015+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:18.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:18 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:18.015+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:18.566 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:18 smithi193 podman[71548]: 2024-04-25 02:13:18.310246827 +0000 UTC m=+5.631306590 container died 24c05019223325cd16bdb217df3f0c934b6ad617eb19e727503c78462b3399e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:13:18.566 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:18 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:18.481+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:18.566 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:18 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:18.481+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:18.567 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:18 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:18.481+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:18.567 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:18 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:18.481+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:18.567 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:18 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:18.481+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:18.824 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:18 smithi193 podman[71548]: 2024-04-25 02:13:18.659036276 +0000 UTC m=+5.980096040 container cleanup 24c05019223325cd16bdb217df3f0c934b6ad617eb19e727503c78462b3399e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T02:13:18.824 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:18 smithi193 bash[71548]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8 2024-04-25T02:13:18.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:18 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:18.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:18.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:18 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:18.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:18.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:18 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:18.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:18.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:18 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:18.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:18.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:18 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:18.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:18.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:18 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:18.427+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:19.224 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:18 smithi193 podman[71565]: 2024-04-25 02:13:18.818470225 +0000 UTC m=+0.501713944 container remove 24c05019223325cd16bdb217df3f0c934b6ad617eb19e727503c78462b3399e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True) 2024-04-25T02:13:19.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:19 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:19.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:19.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:19 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:19.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:19.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:19 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:19.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:19.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:19 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:19.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:19.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:19 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:19.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:19.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:19 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:19.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:19.511 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:19 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:19.449+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:19.511 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:19 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:19.449+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:19.511 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:19 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:19.449+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:19.511 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:19 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:19.449+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:19.511 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:19 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:19.449+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:19.511 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:19 smithi193 podman[71658]: 2024-04-25 02:13:19.121265056 +0000 UTC m=+0.018007426 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:13:19.512 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:19 smithi193 podman[71658]: 2024-04-25 02:13:19.22770022 +0000 UTC m=+0.124442635 container create 1cac1af59663e1440c583664cfb943828074e502b6cbaa151aa1b7e4dd71e47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T02:13:19.512 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:19 smithi193 podman[71658]: 2024-04-25 02:13:19.455612489 +0000 UTC m=+0.352354864 container init 1cac1af59663e1440c583664cfb943828074e502b6cbaa151aa1b7e4dd71e47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T02:13:19.512 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:19 smithi193 podman[71658]: 2024-04-25 02:13:19.459073861 +0000 UTC m=+0.355816234 container start 1cac1af59663e1440c583664cfb943828074e502b6cbaa151aa1b7e4dd71e47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T02:13:19.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:19 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:19.386+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:19.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:19 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:19.386+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:19.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:19 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:19.386+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:19.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:19 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:19.386+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:19.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:19 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:19.386+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:19.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:19 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:19.386+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:19.763 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:19 smithi193 podman[71658]: 2024-04-25 02:13:19.538297581 +0000 UTC m=+0.435039963 container attach 1cac1af59663e1440c583664cfb943828074e502b6cbaa151aa1b7e4dd71e47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:13:20.063 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:19 smithi193 podman[71658]: 2024-04-25 02:13:19.758662951 +0000 UTC m=+0.655405314 container died 1cac1af59663e1440c583664cfb943828074e502b6cbaa151aa1b7e4dd71e47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:13:20.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T02:13:20.325 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:20 smithi193 podman[71685]: 2024-04-25 02:13:20.274760689 +0000 UTC m=+0.510857395 container remove 1cac1af59663e1440c583664cfb943828074e502b6cbaa151aa1b7e4dd71e47f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-25T02:13:20.325 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:20 smithi193 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.8.service: Deactivated successfully. 2024-04-25T02:13:20.325 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:20 smithi193 systemd[1]: Stopped Ceph osd.8 for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T02:13:20.325 INFO:journalctl@ceph.osd.8.smithi193.stdout:Apr 25 02:13:20 smithi193 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.8.service: Consumed 6min 58.612s CPU time. 2024-04-25T02:13:20.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:13:20.341 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T02:13:20.341 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T02:13:20.342 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.9 2024-04-25T02:13:20.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:20 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:20.078+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:20.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:20 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:20.078+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:20.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:20 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:20.078+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:20.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:20 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:20.078+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:20.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:20 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:20.078+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:20.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:20 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:20.078+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:20.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:20.373+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:20.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:20.373+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:20.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:20.373+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:20.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:20.373+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:20.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:20.373+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:20.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:20.373+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:20.680 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 systemd[1]: Stopping Ceph osd.9 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:13:20.724 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:20 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:20.482+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:20.724 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:20 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:20.482+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:20.724 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:20 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:20.482+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:20.725 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:20 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:20.482+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:20.725 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:20 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:20.482+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:21.179 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:20.870+0000 7f5d41e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:13:21.179 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:20.870+0000 7f5d41e00700 -1 osd.9 111 *** Got signal Terminated *** 2024-04-25T02:13:21.179 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:20 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:20.870+0000 7f5d41e00700 -1 osd.9 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:13:21.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:21 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:21.030+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:21.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:21 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:21.030+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:21.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:21 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:21.030+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:21.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:21 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:21.030+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:21.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:21 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:21.030+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:21.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:21 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:21.030+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:21.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:21 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:21.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:21.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:21 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:21.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:21.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:21 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:21.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:21.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:21 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:21.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:21.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:21 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:21.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:21.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:21 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:21.394+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:21.760 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:21 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:21.476+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:21.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:21 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:21.476+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:21.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:21 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:21.476+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:21.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:21 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:21.476+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:21.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:21 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:21.476+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:22.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:22 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:22.031+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:22.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:22 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:22.031+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:22.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:22 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:22.031+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:22.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:22 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:22.032+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:22.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:22 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:22.032+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:22.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:22 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:22.032+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:22.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:22 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:22.409+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:22.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:22 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:22.409+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:22.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:22 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:22.409+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:22.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:22 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:22.409+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:22.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:22 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:22.409+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:22.679 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:22 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:22.409+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:23.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:22 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:22.525+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:23.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:22 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:22.525+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:23.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:22 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:22.525+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:23.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:22 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:22.525+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:23.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:22 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:22.525+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:23.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:23 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:23.010+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:23.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:23 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:23.010+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:23.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:23 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:23.010+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:23.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:23 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:23.010+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:23.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:23 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:23.010+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:23.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:23 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:23.010+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:23.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:23 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:23.426+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:23.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:23 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:23.426+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:23.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:23 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:23.426+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:23.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:23 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:23.426+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:23.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:23 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:23.426+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:23.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:23 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:23.426+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:24.010 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:23 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:23.520+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:24.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:23 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:23.520+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:24.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:23 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:23.520+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:24.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:23 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:23.520+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:24.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:23 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:23.520+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:24.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:24 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:24.013+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:24.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:24 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:24.013+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:24.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:24 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:24.013+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:24.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:24 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:24.013+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:24.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:24 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:24.013+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:24.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:24 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:24.013+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:24.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:24 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:24.485+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:24.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:24 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:24.485+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:24.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:24 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:24.485+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:24.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:24 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:24.485+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:24.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:24 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:24.485+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:24.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:24 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:24.485+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:24.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:24 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:24.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:24.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:24 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:24.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:24.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:24 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:24.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:24.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:24 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:24.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:24.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:24 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:24.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:24.930 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:24 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:24.438+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:25.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:25 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:25.016+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:25.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:25 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:25.016+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:25.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:25 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:25.016+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:25.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:25 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:25.016+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:25.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:25 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:25.016+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:25.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:25 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:25.016+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:25.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:25 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:25.486+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:25.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:25 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:25.486+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:25.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:25 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:25.486+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:25.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:25 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:25.486+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:25.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:25 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:25.486+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:25.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:25 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:25.486+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:25.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:25 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:25.435+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.329376+0000 front 2024-04-25T02:12:08.329991+0000 (oldest deadline 2024-04-25T02:12:34.229626+0000) 2024-04-25T02:13:25.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:25 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:25.435+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:17.632910+0000 front 2024-04-25T02:12:17.633194+0000 (oldest deadline 2024-04-25T02:12:41.732801+0000) 2024-04-25T02:13:25.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:25 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:25.435+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:21.733886+0000 front 2024-04-25T02:12:21.734070+0000 (oldest deadline 2024-04-25T02:12:49.334810+0000) 2024-04-25T02:13:25.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:25 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:25.435+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:32.237303+0000 front 2024-04-25T02:12:32.237379+0000 (oldest deadline 2024-04-25T02:12:57.537569+0000) 2024-04-25T02:13:25.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:25 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:25.435+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:43.340874+0000 front 2024-04-25T02:12:43.340794+0000 (oldest deadline 2024-04-25T02:13:07.441030+0000) 2024-04-25T02:13:25.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:25 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9[63040]: 2024-04-25T02:13:25.435+0000 7f5d3e000700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:50.343298+0000 front 2024-04-25T02:12:50.343415+0000 (oldest deadline 2024-04-25T02:13:13.843281+0000) 2024-04-25T02:13:26.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:26 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:26.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:26.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:26 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:26.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:26.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:26 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:26.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:26.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:26 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:26.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:26.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:26 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:26.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:26.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:26 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:26.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:26.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:26 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:26.061+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:27.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:26 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:26.530+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:27.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:26 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:26.530+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:27.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:26 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:26.530+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:27.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:26 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:26.530+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:27.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:26 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:26.530+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:27.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:26 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:26.530+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:27.134 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:26 smithi102 podman[96675]: 2024-04-25 02:13:26.798133909 +0000 UTC m=+6.068832466 container died 87313dc8837790325d591448ff502e07f97c9f0a1904b6f38e50f204505ec7e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9) 2024-04-25T02:13:27.429 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:27 smithi102 podman[96675]: 2024-04-25 02:13:27.239806698 +0000 UTC m=+6.510505292 container cleanup 87313dc8837790325d591448ff502e07f97c9f0a1904b6f38e50f204505ec7e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T02:13:27.429 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:27 smithi102 bash[96675]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9 2024-04-25T02:13:27.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:27 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:27.095+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:27.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:27 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:27.095+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:27.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:27 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:27.095+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:27.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:27 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:27.095+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:27.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:27 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:27.095+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:27.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:27 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:27.095+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:27.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:27 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:27.095+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:27.928 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:27 smithi102 podman[96695]: 2024-04-25 02:13:27.491089064 +0000 UTC m=+0.670222712 container remove 87313dc8837790325d591448ff502e07f97c9f0a1904b6f38e50f204505ec7e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS) 2024-04-25T02:13:28.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:27 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:27.533+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:28.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:27 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:27.533+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:28.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:27 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:27.533+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:28.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:27 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:27.533+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:28.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:27 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:27.533+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:28.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:27 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:27.533+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:28.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:28 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:28.114+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:28.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:28 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:28.114+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:28.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:28 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:28.114+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:28.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:28 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:28.114+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:28.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:28 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:28.114+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:28.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:28 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:28.114+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:28.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:28 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:28.114+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:28.597 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:28 smithi102 podman[96782]: 2024-04-25 02:13:28.1693906 +0000 UTC m=+0.031969600 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:13:28.597 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:28 smithi102 podman[96782]: 2024-04-25 02:13:28.276893413 +0000 UTC m=+0.139472437 container create 07a395d87e3e0a5fb10e074cdbd8d41397c7340c80da7cbe2a4afce191e053e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T02:13:28.597 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:28 smithi102 podman[96782]: 2024-04-25 02:13:28.591088559 +0000 UTC m=+0.453667603 container init 07a395d87e3e0a5fb10e074cdbd8d41397c7340c80da7cbe2a4afce191e053e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T02:13:28.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:28 smithi102 podman[96782]: 2024-04-25 02:13:28.601708331 +0000 UTC m=+0.464287362 container start 07a395d87e3e0a5fb10e074cdbd8d41397c7340c80da7cbe2a4afce191e053e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:13:28.929 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:28 smithi102 podman[96782]: 2024-04-25 02:13:28.695826647 +0000 UTC m=+0.558405725 container attach 07a395d87e3e0a5fb10e074cdbd8d41397c7340c80da7cbe2a4afce191e053e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T02:13:29.010 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:28.574+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:29.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:28.574+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:29.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:28.574+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:29.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:28.574+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:29.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:28.574+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:29.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:28.574+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:29.410 INFO:journalctl@ceph.osd.9.smithi102.stdout:Apr 25 02:13:29 smithi102 podman[96782]: 2024-04-25 02:13:29.045845719 +0000 UTC m=+0.908424749 container died 07a395d87e3e0a5fb10e074cdbd8d41397c7340c80da7cbe2a4afce191e053e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T02:13:29.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:29 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:29.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:29.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:29 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:29.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:29.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:29 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:29.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:29.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:29 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:29.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:29.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:29 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:29.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:29.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:29 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:29.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:29.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:29 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:29.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:29.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T02:13:29.914 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:13:29.914 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T02:13:29.915 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T02:13:29.915 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.10 2024-04-25T02:13:30.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:29 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:29.542+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:30.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:29 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:29.542+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:30.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:29 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:29.542+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:30.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:29 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:29.542+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:30.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:29 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:29.542+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:30.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:29 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:29.542+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:30.224 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:29 smithi110 systemd[1]: Stopping Ceph osd.10 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.164+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.164+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.164+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.164+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.164+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.164+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.164+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.218+0000 7fdfe1526700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.218+0000 7fdfe1526700 -1 osd.10 111 *** Got signal Terminated *** 2024-04-25T02:13:30.225 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:30 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:30.218+0000 7fdfe1526700 -1 osd.10 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:13:30.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:30 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:30.495+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:30.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:30 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:30.495+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:30.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:30 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:30.495+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:30.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:30 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:30.495+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:30.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:30 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:30.495+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:30.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:30 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:30.495+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:31.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:31 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:31.137+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:31.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:31 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:31.137+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:31.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:31 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:31.137+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:31.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:31 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:31.137+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:31.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:31 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:31.137+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:31.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:31 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:31.137+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:31.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:31 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:31.137+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:32.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:31 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:31.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:32.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:31 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:31.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:32.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:31 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:31.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:32.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:31 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:31.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:32.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:31 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:31.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:32.011 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:31 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:31.539+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:32.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:32 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:32.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:32.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:32 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:32.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:32.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:32 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:32.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:32.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:32 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:32.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:32.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:32 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:32.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:32.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:32 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:32.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:32.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:32 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:32.134+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:32.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:32 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:32.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:32.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:32 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:32.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:32.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:32 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:32.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:32.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:32 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:32.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:32.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:32 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:32.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:32.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:32 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:32.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:33.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:33 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:33.127+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:33.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:33 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:33.127+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:33.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:33 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:33.127+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:33.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:33 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:33.127+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:33.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:33 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:33.127+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:33.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:33 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:33.127+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:33.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:33 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:33.127+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:33.760 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:33 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:33.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:33.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:33 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:33.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:33.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:33 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:33.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:33.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:33 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:33.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:33.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:33 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:33.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:33.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:33 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:33.500+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:34.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:34 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:34.139+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:34.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:34 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:34.139+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:34.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:34 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:34.139+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:34.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:34 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:34.139+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:34.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:34 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:34.139+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:34.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:34 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:34.139+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:34.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:34 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:34.139+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:34.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:34 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:34.491+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:34.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:34 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:34.491+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:34.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:34 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:34.491+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:34.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:34 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:34.491+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:34.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:34 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:34.491+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:34.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:34 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:34.491+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:34.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:34 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:34.491+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-25T02:13:09.484598+0000 front 2024-04-25T02:13:09.484668+0000 (oldest deadline 2024-04-25T02:13:33.584468+0000) 2024-04-25T02:13:35.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:35 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:35.107+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:08.531798+0000 front 2024-04-25T02:12:08.531533+0000 (oldest deadline 2024-04-25T02:12:32.631435+0000) 2024-04-25T02:13:35.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:35 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:35.107+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:15.433165+0000 front 2024-04-25T02:12:15.433205+0000 (oldest deadline 2024-04-25T02:12:38.933392+0000) 2024-04-25T02:13:35.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:35 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:35.107+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:24.834531+0000 front 2024-04-25T02:12:24.834753+0000 (oldest deadline 2024-04-25T02:12:48.334528+0000) 2024-04-25T02:13:35.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:35 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:35.107+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6814 osd.3 since back 2024-04-25T02:12:34.235885+0000 front 2024-04-25T02:12:34.235988+0000 (oldest deadline 2024-04-25T02:12:56.436246+0000) 2024-04-25T02:13:35.493 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:35 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:35.107+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.238130+0000 front 2024-04-25T02:12:42.238239+0000 (oldest deadline 2024-04-25T02:13:08.138179+0000) 2024-04-25T02:13:35.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:35 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:35.107+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:48.138780+0000 front 2024-04-25T02:12:48.138955+0000 (oldest deadline 2024-04-25T02:13:13.438661+0000) 2024-04-25T02:13:35.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:35 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10[58410]: 2024-04-25T02:13:35.107+0000 7fdfda313700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:13:00.340923+0000 front 2024-04-25T02:13:00.340942+0000 (oldest deadline 2024-04-25T02:13:25.040562+0000) 2024-04-25T02:13:35.760 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:35 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:35.503+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:35.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:35 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:35.503+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:35.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:35 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:35.503+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:35.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:35 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:35.503+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:35.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:35 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:35.503+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:35.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:35 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:35.503+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:35.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:35 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:35.503+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-25T02:13:09.484598+0000 front 2024-04-25T02:13:09.484668+0000 (oldest deadline 2024-04-25T02:13:33.584468+0000) 2024-04-25T02:13:36.243 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:35 smithi110 podman[71441]: 2024-04-25 02:13:35.91970069 +0000 UTC m=+5.810241242 container died 3b754627aeb2222b77031405d59ab4c2a69238e1b69d09fa3460d34ca037e641 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:13:36.743 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:36 smithi110 podman[71441]: 2024-04-25 02:13:36.250987504 +0000 UTC m=+6.141528065 container cleanup 3b754627aeb2222b77031405d59ab4c2a69238e1b69d09fa3460d34ca037e641 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-25T02:13:36.743 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:36 smithi110 bash[71441]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10 2024-04-25T02:13:36.743 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:36 smithi110 podman[71458]: 2024-04-25 02:13:36.418806678 +0000 UTC m=+0.498547543 container remove 3b754627aeb2222b77031405d59ab4c2a69238e1b69d09fa3460d34ca037e641 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-04-25T02:13:36.760 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:36.499+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:36.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:36.499+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:36.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:36.499+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:36.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:36.499+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:36.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:36.499+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:36.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:36.499+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:36.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:36.499+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-25T02:13:09.484598+0000 front 2024-04-25T02:13:09.484668+0000 (oldest deadline 2024-04-25T02:13:33.584468+0000) 2024-04-25T02:13:37.110 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:36 smithi110 podman[71549]: 2024-04-25 02:13:36.852899543 +0000 UTC m=+0.112818665 container create 16237a00d49106f7c8b7e5950c864e0c973171b9b6d74a4dcff85f0f093f424d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T02:13:37.110 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:36 smithi110 podman[71549]: 2024-04-25 02:13:36.758362317 +0000 UTC m=+0.018281447 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:13:37.111 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:37 smithi110 podman[71549]: 2024-04-25 02:13:37.030024684 +0000 UTC m=+0.289943820 container init 16237a00d49106f7c8b7e5950c864e0c973171b9b6d74a4dcff85f0f093f424d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T02:13:37.111 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:37 smithi110 podman[71549]: 2024-04-25 02:13:37.033075389 +0000 UTC m=+0.292994511 container start 16237a00d49106f7c8b7e5950c864e0c973171b9b6d74a4dcff85f0f093f424d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=squid-d2c45d9, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:13:37.111 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:37 smithi110 podman[71549]: 2024-04-25 02:13:37.104903756 +0000 UTC m=+0.364822880 container attach 16237a00d49106f7c8b7e5950c864e0c973171b9b6d74a4dcff85f0f093f424d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424) 2024-04-25T02:13:37.494 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:37 smithi110 podman[71549]: 2024-04-25 02:13:37.320554301 +0000 UTC m=+0.580473425 container died 16237a00d49106f7c8b7e5950c864e0c973171b9b6d74a4dcff85f0f093f424d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T02:13:37.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:37 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:37.479+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:37.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:37 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:37.479+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:37.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:37 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:37.479+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:37.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:37 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:37.479+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:37.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:37 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:37.479+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:37.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:37 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:37.479+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:37.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:37 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:37.479+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-25T02:13:09.484598+0000 front 2024-04-25T02:13:09.484668+0000 (oldest deadline 2024-04-25T02:13:33.584468+0000) 2024-04-25T02:13:37.862 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T02:13:37.884 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:37 smithi110 podman[71576]: 2024-04-25 02:13:37.841528345 +0000 UTC m=+0.515470582 container remove 16237a00d49106f7c8b7e5950c864e0c973171b9b6d74a4dcff85f0f093f424d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:13:37.884 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:37 smithi110 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.10.service: Deactivated successfully. 2024-04-25T02:13:37.884 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:37 smithi110 systemd[1]: Stopped Ceph osd.10 for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T02:13:37.884 INFO:journalctl@ceph.osd.10.smithi110.stdout:Apr 25 02:13:37 smithi110 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.10.service: Consumed 8min 50.974s CPU time. 2024-04-25T02:13:37.897 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:13:37.898 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T02:13:37.898 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T02:13:37.898 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.11 2024-04-25T02:13:38.261 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:37 smithi193 systemd[1]: Stopping Ceph osd.11 for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:13:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.311+0000 7f3c05924700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:13:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.311+0000 7f3c05924700 -1 osd.11 111 *** Got signal Terminated *** 2024-04-25T02:13:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.311+0000 7f3c05924700 -1 osd.11 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T02:13:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.435+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.435+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.435+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.435+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.435+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:38.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.435+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:38.762 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:38 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:38.435+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-25T02:13:09.484598+0000 front 2024-04-25T02:13:09.484668+0000 (oldest deadline 2024-04-25T02:13:33.584468+0000) 2024-04-25T02:13:39.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:39.455+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:39.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:39.455+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:39.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:39.455+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:39.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:39.455+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:39.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:39.455+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:39.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:39.455+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:39.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:39.455+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-25T02:13:09.484598+0000 front 2024-04-25T02:13:09.484668+0000 (oldest deadline 2024-04-25T02:13:33.584468+0000) 2024-04-25T02:13:39.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:39 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:39.455+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6820 osd.8 since back 2024-04-25T02:13:13.584970+0000 front 2024-04-25T02:13:13.584932+0000 (oldest deadline 2024-04-25T02:13:38.884946+0000) 2024-04-25T02:13:40.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:40.419+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:40.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:40.419+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:40.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:40.419+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:40.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:40.419+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:40.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:40.419+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:40.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:40.419+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:40.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:40.419+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-25T02:13:09.484598+0000 front 2024-04-25T02:13:09.484668+0000 (oldest deadline 2024-04-25T02:13:33.584468+0000) 2024-04-25T02:13:40.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:40 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:40.419+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6820 osd.8 since back 2024-04-25T02:13:13.584970+0000 front 2024-04-25T02:13:13.584932+0000 (oldest deadline 2024-04-25T02:13:38.884946+0000) 2024-04-25T02:13:41.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:41.403+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:41.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:41.403+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:41.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:41.403+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:41.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:41.403+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:41.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:41.403+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:41.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:41.403+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:41.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:41.403+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-25T02:13:09.484598+0000 front 2024-04-25T02:13:09.484668+0000 (oldest deadline 2024-04-25T02:13:33.584468+0000) 2024-04-25T02:13:41.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:41 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:41.403+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6820 osd.8 since back 2024-04-25T02:13:13.584970+0000 front 2024-04-25T02:13:13.584932+0000 (oldest deadline 2024-04-25T02:13:38.884946+0000) 2024-04-25T02:13:42.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:42.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-04-25T02:12:07.775206+0000 front 2024-04-25T02:12:07.775330+0000 (oldest deadline 2024-04-25T02:12:30.075215+0000) 2024-04-25T02:13:42.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:42.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6804 osd.1 since back 2024-04-25T02:12:16.377632+0000 front 2024-04-25T02:12:16.377608+0000 (oldest deadline 2024-04-25T02:12:41.578101+0000) 2024-04-25T02:13:42.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:42.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-04-25T02:12:23.179389+0000 front 2024-04-25T02:12:23.179588+0000 (oldest deadline 2024-04-25T02:12:48.479556+0000) 2024-04-25T02:13:42.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:42.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6812 osd.4 since back 2024-04-25T02:12:42.481880+0000 front 2024-04-25T02:12:42.481866+0000 (oldest deadline 2024-04-25T02:13:06.581826+0000) 2024-04-25T02:13:42.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:42.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-04-25T02:12:46.582272+0000 front 2024-04-25T02:12:46.582206+0000 (oldest deadline 2024-04-25T02:13:15.982890+0000) 2024-04-25T02:13:42.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:42.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.102:6822 osd.6 since back 2024-04-25T02:12:59.484029+0000 front 2024-04-25T02:12:59.483957+0000 (oldest deadline 2024-04-25T02:13:24.183678+0000) 2024-04-25T02:13:42.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:42.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.110:6820 osd.7 since back 2024-04-25T02:13:09.484598+0000 front 2024-04-25T02:13:09.484668+0000 (oldest deadline 2024-04-25T02:13:33.584468+0000) 2024-04-25T02:13:42.761 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:42 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11[58218]: 2024-04-25T02:13:42.439+0000 7f3bfe711700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6820 osd.8 since back 2024-04-25T02:13:13.584970+0000 front 2024-04-25T02:13:13.584932+0000 (oldest deadline 2024-04-25T02:13:38.884946+0000) 2024-04-25T02:13:44.092 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:43 smithi193 podman[71788]: 2024-04-25 02:13:43.81290576 +0000 UTC m=+5.723574133 container died 67fe9cfcbb319d8db1d3dac15d5c6ec9b0cf160a12f94a8c9f544c22f200246f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:13:44.459 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:44 smithi193 podman[71788]: 2024-04-25 02:13:44.186474891 +0000 UTC m=+6.097143263 container cleanup 67fe9cfcbb319d8db1d3dac15d5c6ec9b0cf160a12f94a8c9f544c22f200246f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-d2c45d9) 2024-04-25T02:13:44.459 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:44 smithi193 bash[71788]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11 2024-04-25T02:13:44.760 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:44 smithi193 podman[71806]: 2024-04-25 02:13:44.454218447 +0000 UTC m=+0.633524937 container remove 67fe9cfcbb319d8db1d3dac15d5c6ec9b0cf160a12f94a8c9f544c22f200246f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True) 2024-04-25T02:13:45.104 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:44 smithi193 podman[71898]: 2024-04-25 02:13:44.70021666 +0000 UTC m=+0.017890086 image pull f68e37b807ef3dfd57708ff7362bfa0edaabe0111536b8e8a4c26de22a72557e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901 2024-04-25T02:13:45.104 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:44 smithi193 podman[71898]: 2024-04-25 02:13:44.813262901 +0000 UTC m=+0.130936334 container create cad48f128f749209e337de5a99e473a1838a30ae3da04709eed8b7ed789ab3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T02:13:45.104 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:45 smithi193 podman[71898]: 2024-04-25 02:13:45.010245804 +0000 UTC m=+0.327919239 container init cad48f128f749209e337de5a99e473a1838a30ae3da04709eed8b7ed789ab3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T02:13:45.104 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:45 smithi193 podman[71898]: 2024-04-25 02:13:45.013544575 +0000 UTC m=+0.331218003 container start cad48f128f749209e337de5a99e473a1838a30ae3da04709eed8b7ed789ab3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:13:45.104 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:45 smithi193 podman[71898]: 2024-04-25 02:13:45.098737466 +0000 UTC m=+0.416410913 container attach cad48f128f749209e337de5a99e473a1838a30ae3da04709eed8b7ed789ab3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T02:13:45.510 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:45 smithi193 podman[71898]: 2024-04-25 02:13:45.298775197 +0000 UTC m=+0.616448622 container died cad48f128f749209e337de5a99e473a1838a30ae3da04709eed8b7ed789ab3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T02:13:45.836 INFO:journalctl@ceph.osd.11.smithi193.stdout:Apr 25 02:13:45 smithi193 podman[71924]: 2024-04-25 02:13:45.802432855 +0000 UTC m=+0.497632974 container remove cad48f128f749209e337de5a99e473a1838a30ae3da04709eed8b7ed789ab3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:13:45.841 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T02:13:45.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:13:45.882 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T02:13:45.883 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T02:13:45.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.a 2024-04-25T02:13:46.429 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 02:13:45 smithi102 systemd[1]: Stopping Ceph mds.a for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:13:46.929 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 02:13:46 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a[69715]: 2024-04-25T02:13:46.440+0000 7f7e6b800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:13:46.929 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 02:13:46 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a[69715]: 2024-04-25T02:13:46.440+0000 7f7e6b800700 -1 mds.a *** got signal Terminated *** 2024-04-25T02:13:50.544 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 02:13:50 smithi102 podman[96897]: 2024-04-25 02:13:50.232802138 +0000 UTC m=+3.920255308 container died dbf63e9e5b10d2923e70d793d29e111a626bf358382cc857b983bf79896f7317 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-25T02:13:50.908 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 02:13:50 smithi102 podman[96897]: 2024-04-25 02:13:50.659019824 +0000 UTC m=+4.346472996 container cleanup dbf63e9e5b10d2923e70d793d29e111a626bf358382cc857b983bf79896f7317 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=) 2024-04-25T02:13:50.908 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 02:13:50 smithi102 bash[96897]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a 2024-04-25T02:13:51.178 INFO:journalctl@ceph.mds.a.smithi102.stdout:Apr 25 02:13:50 smithi102 podman[96910]: 2024-04-25 02:13:50.901719738 +0000 UTC m=+0.653501998 container remove dbf63e9e5b10d2923e70d793d29e111a626bf358382cc857b983bf79896f7317 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T02:13:51.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T02:13:51.431 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:13:51.432 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T02:13:51.432 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T02:13:51.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.d 2024-04-25T02:13:51.929 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 02:13:51 smithi102 systemd[1]: Stopping Ceph mds.d for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:13:52.429 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 02:13:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d[68729]: 2024-04-25T02:13:52.027+0000 7f4b8e200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:13:52.430 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 02:13:52 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d[68729]: 2024-04-25T02:13:52.027+0000 7f4b8e200700 -1 mds.d *** got signal Terminated *** 2024-04-25T02:13:56.081 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 02:13:55 smithi102 podman[97018]: 2024-04-25 02:13:55.77636742 +0000 UTC m=+3.923587966 container died 404ae44b30b463e94cd20f23f322ab6977871fbf73f03ea755e7d5a48034ace4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T02:13:56.428 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 02:13:56 smithi102 podman[97018]: 2024-04-25 02:13:56.187564692 +0000 UTC m=+4.334785256 container cleanup 404ae44b30b463e94cd20f23f322ab6977871fbf73f03ea755e7d5a48034ace4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T02:13:56.429 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 02:13:56 smithi102 bash[97018]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d 2024-04-25T02:13:56.928 INFO:journalctl@ceph.mds.d.smithi102.stdout:Apr 25 02:13:56 smithi102 podman[97032]: 2024-04-25 02:13:56.480445262 +0000 UTC m=+0.687347605 container remove 404ae44b30b463e94cd20f23f322ab6977871fbf73f03ea755e7d5a48034ace4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:13:56.974 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T02:13:57.080 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:13:57.081 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T02:13:57.081 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T02:13:57.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.g 2024-04-25T02:13:57.620 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 25 02:13:57 smithi102 systemd[1]: Stopping Ceph mds.g for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:13:57.929 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 25 02:13:57 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-g[69351]: 2024-04-25T02:13:57.613+0000 7fe7b8c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:13:57.929 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 25 02:13:57 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-g[69351]: 2024-04-25T02:13:57.613+0000 7fe7b8c00700 -1 mds.g *** got signal Terminated *** 2024-04-25T02:14:01.928 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 25 02:14:01 smithi102 podman[97138]: 2024-04-25 02:14:01.636924084 +0000 UTC m=+4.143030231 container died 04cd9af40f240e5d859bda1c92200a99b9a3d13d3f9e53e2456c08288ff92baa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T02:14:02.307 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 25 02:14:02 smithi102 podman[97138]: 2024-04-25 02:14:02.092718807 +0000 UTC m=+4.598825011 container cleanup 04cd9af40f240e5d859bda1c92200a99b9a3d13d3f9e53e2456c08288ff92baa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T02:14:02.307 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 25 02:14:02 smithi102 bash[97138]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-g 2024-04-25T02:14:02.678 INFO:journalctl@ceph.mds.g.smithi102.stdout:Apr 25 02:14:02 smithi102 podman[97151]: 2024-04-25 02:14:02.300498 +0000 UTC m=+0.653137306 container remove 04cd9af40f240e5d859bda1c92200a99b9a3d13d3f9e53e2456c08288ff92baa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-d2c45d9, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:14:02.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T02:14:02.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:02.884 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T02:14:02.884 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T02:14:02.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.j 2024-04-25T02:14:03.178 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 02:14:02 smithi102 systemd[1]: Stopping Ceph mds.j for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:14:03.678 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 02:14:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j[68365]: 2024-04-25T02:14:03.399+0000 7fe3d9600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:14:03.678 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 02:14:03 smithi102 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j[68365]: 2024-04-25T02:14:03.399+0000 7fe3d9600700 -1 mds.j *** got signal Terminated *** 2024-04-25T02:14:07.764 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 02:14:07 smithi102 podman[97256]: 2024-04-25 02:14:07.426013045 +0000 UTC m=+4.153470382 container died fcb3f2944bffd85d138393f1f2abdaabf63de3a14e5b2a2ec7050e132f424ea1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-d2c45d9, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T02:14:08.076 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 02:14:07 smithi102 podman[97256]: 2024-04-25 02:14:07.869839674 +0000 UTC m=+4.597297027 container cleanup fcb3f2944bffd85d138393f1f2abdaabf63de3a14e5b2a2ec7050e132f424ea1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T02:14:08.077 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 02:14:07 smithi102 bash[97256]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j 2024-04-25T02:14:08.428 INFO:journalctl@ceph.mds.j.smithi102.stdout:Apr 25 02:14:08 smithi102 podman[97282]: 2024-04-25 02:14:08.070655892 +0000 UTC m=+0.631538353 container remove fcb3f2944bffd85d138393f1f2abdaabf63de3a14e5b2a2ec7050e132f424ea1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:14:08.583 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T02:14:08.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:08.692 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T02:14:08.692 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T02:14:08.692 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.b 2024-04-25T02:14:08.992 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:08 smithi110 systemd[1]: Stopping Ceph mds.b for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:14:09.492 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b[64106]: 2024-04-25T02:14:09.058+0000 7f627cbd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:14:09.493 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:09 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b[64106]: 2024-04-25T02:14:09.058+0000 7f627cbd8700 -1 mds.b *** got signal Terminated *** 2024-04-25T02:14:11.514 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:11 smithi110 podman[71658]: 2024-04-25 02:14:11.241955546 +0000 UTC m=+2.367359333 container died 0735d4cfb609f307069163e7dc3e02358a11b1f60f2d43ce3aa650ff24207ffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T02:14:11.772 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:11 smithi110 podman[71658]: 2024-04-25 02:14:11.583382858 +0000 UTC m=+2.708786634 container cleanup 0735d4cfb609f307069163e7dc3e02358a11b1f60f2d43ce3aa650ff24207ffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-25T02:14:11.773 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:11 smithi110 bash[71658]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b 2024-04-25T02:14:12.017 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T02:14:12.038 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:11 smithi110 podman[71684]: 2024-04-25 02:14:11.767744555 +0000 UTC m=+0.516788564 container remove 0735d4cfb609f307069163e7dc3e02358a11b1f60f2d43ce3aa650ff24207ffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T02:14:12.038 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:12 smithi110 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.b.service: Deactivated successfully. 2024-04-25T02:14:12.038 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:12 smithi110 systemd[1]: Stopped Ceph mds.b for 145ca8d0-02a5-11ef-bc93-c7b262605968. 2024-04-25T02:14:12.039 INFO:journalctl@ceph.mds.b.smithi110.stdout:Apr 25 02:14:12 smithi110 systemd[1]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.b.service: Consumed 23.708s CPU time. 2024-04-25T02:14:12.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:12.052 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T02:14:12.052 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T02:14:12.052 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.e 2024-04-25T02:14:12.368 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 02:14:12 smithi110 systemd[1]: Stopping Ceph mds.e for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:14:12.742 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 02:14:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e[63484]: 2024-04-25T02:14:12.362+0000 7f9b66e85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:14:12.742 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 02:14:12 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e[63484]: 2024-04-25T02:14:12.362+0000 7f9b66e85700 -1 mds.e *** got signal Terminated *** 2024-04-25T02:14:15.527 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 02:14:15 smithi110 podman[71792]: 2024-04-25 02:14:15.244436516 +0000 UTC m=+2.983788370 container died 29a3871a9cba1292d90092f0b4c57621415b5341fbf6dabba3e9673f81e66d8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-d2c45d9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T02:14:15.859 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 02:14:15 smithi110 podman[71792]: 2024-04-25 02:14:15.603823771 +0000 UTC m=+3.343175618 container cleanup 29a3871a9cba1292d90092f0b4c57621415b5341fbf6dabba3e9673f81e66d8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T02:14:15.860 INFO:journalctl@ceph.mds.e.smithi110.stdout:Apr 25 02:14:15 smithi110 bash[71792]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-e 2024-04-25T02:14:16.021 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T02:14:16.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:16.052 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T02:14:16.052 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T02:14:16.053 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.h 2024-04-25T02:14:16.379 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:14:16 smithi110 systemd[1]: Stopping Ceph mds.h for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:14:16.742 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:14:16 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h[63799]: 2024-04-25T02:14:16.373+0000 7f9b6fe4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:14:16.742 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:14:16 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h[63799]: 2024-04-25T02:14:16.373+0000 7f9b6fe4e700 -1 mds.h *** got signal Terminated *** 2024-04-25T02:14:19.565 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:14:19 smithi110 podman[71913]: 2024-04-25 02:14:19.241928809 +0000 UTC m=+2.980187891 container died 57e29242caf0576f53b9952ef158d3d1a934b390bf09de917d1add751325e3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T02:14:19.966 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:14:19 smithi110 podman[71913]: 2024-04-25 02:14:19.565239777 +0000 UTC m=+3.303498852 container cleanup 57e29242caf0576f53b9952ef158d3d1a934b390bf09de917d1add751325e3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T02:14:19.966 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:14:19 smithi110 bash[71913]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h 2024-04-25T02:14:19.966 INFO:journalctl@ceph.mds.h.smithi110.stdout:Apr 25 02:14:19 smithi110 podman[71927]: 2024-04-25 02:14:19.732963368 +0000 UTC m=+0.487141909 container remove 57e29242caf0576f53b9952ef158d3d1a934b390bf09de917d1add751325e3d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-h, RELEASE=squid-d2c45d9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-04-25T02:14:19.982 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T02:14:20.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:20.013 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T02:14:20.013 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T02:14:20.013 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.k 2024-04-25T02:14:20.242 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 02:14:20 smithi110 systemd[1]: Stopping Ceph mds.k for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:14:20.742 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 02:14:20 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-k[63163]: 2024-04-25T02:14:20.318+0000 7f943ab9e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:14:20.742 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 02:14:20 smithi110 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-k[63163]: 2024-04-25T02:14:20.318+0000 7f943ab9e700 -1 mds.k *** got signal Terminated *** 2024-04-25T02:14:23.992 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 02:14:23 smithi110 podman[72037]: 2024-04-25 02:14:23.689105427 +0000 UTC m=+3.459319594 container died 91adea0f106ba43564e994c8f2d56aa363c3f502958990436eeebbe1e16ea8f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T02:14:24.325 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 02:14:24 smithi110 podman[72037]: 2024-04-25 02:14:24.07825506 +0000 UTC m=+3.848469228 container cleanup 91adea0f106ba43564e994c8f2d56aa363c3f502958990436eeebbe1e16ea8f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T02:14:24.326 INFO:journalctl@ceph.mds.k.smithi110.stdout:Apr 25 02:14:24 smithi110 bash[72037]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-k 2024-04-25T02:14:24.458 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T02:14:24.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:24.489 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T02:14:24.489 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T02:14:24.489 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.c 2024-04-25T02:14:24.781 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:14:24 smithi193 systemd[1]: Stopping Ceph mds.c for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:14:25.039 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:14:24 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c[63666]: 2024-04-25T02:14:24.775+0000 7f5962178700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:14:25.039 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:14:24 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c[63666]: 2024-04-25T02:14:24.775+0000 7f5962178700 -1 mds.c *** got signal Terminated *** 2024-04-25T02:14:27.510 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:14:27 smithi193 podman[72003]: 2024-04-25 02:14:27.245080722 +0000 UTC m=+2.574279193 container died ae9e095acf8cfe39a80327b3da3abe35ec13ce3e33eb4a61849111e46a266410 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T02:14:27.897 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:14:27 smithi193 podman[72003]: 2024-04-25 02:14:27.607932451 +0000 UTC m=+2.937130912 container cleanup ae9e095acf8cfe39a80327b3da3abe35ec13ce3e33eb4a61849111e46a266410 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c, RELEASE=squid-d2c45d9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T02:14:27.897 INFO:journalctl@ceph.mds.c.smithi193.stdout:Apr 25 02:14:27 smithi193 bash[72003]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-c 2024-04-25T02:14:28.014 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T02:14:28.048 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:28.049 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T02:14:28.049 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T02:14:28.049 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.f 2024-04-25T02:14:28.260 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:14:28 smithi193 systemd[1]: Stopping Ceph mds.f for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:14:28.760 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:14:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f[63361]: 2024-04-25T02:14:28.362+0000 7f46a55b5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:14:28.760 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:14:28 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f[63361]: 2024-04-25T02:14:28.362+0000 7f46a55b5700 -1 mds.f *** got signal Terminated *** 2024-04-25T02:14:31.510 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:14:31 smithi193 podman[72136]: 2024-04-25 02:14:31.250012492 +0000 UTC m=+2.986618252 container died 899614771ba16e559701d6daf974ce753d2384acfe2a7ee478e771b7a6a9e658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T02:14:31.777 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:14:31 smithi193 podman[72136]: 2024-04-25 02:14:31.578701543 +0000 UTC m=+3.315307299 container cleanup 899614771ba16e559701d6daf974ce753d2384acfe2a7ee478e771b7a6a9e658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-04-25T02:14:31.777 INFO:journalctl@ceph.mds.f.smithi193.stdout:Apr 25 02:14:31 smithi193 bash[72136]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-f 2024-04-25T02:14:32.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T02:14:32.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:32.037 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T02:14:32.037 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T02:14:32.037 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.i 2024-04-25T02:14:32.371 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:14:32 smithi193 systemd[1]: Stopping Ceph mds.i for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:14:32.760 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:14:32 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: 2024-04-25T02:14:32.366+0000 7fbfa7b79700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:14:32.760 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:14:32 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i[63973]: 2024-04-25T02:14:32.366+0000 7fbfa7b79700 -1 mds.i *** got signal Terminated *** 2024-04-25T02:14:35.510 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:14:35 smithi193 podman[72259]: 2024-04-25 02:14:35.244015121 +0000 UTC m=+3.001670141 container died 8e755f7c76b79892d0a59a783c67a14cbaa5f4fb888aa4a60697d4949dd6c41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=squid-d2c45d9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T02:14:35.872 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:14:35 smithi193 podman[72259]: 2024-04-25 02:14:35.624527483 +0000 UTC m=+3.382182495 container cleanup 8e755f7c76b79892d0a59a783c67a14cbaa5f4fb888aa4a60697d4949dd6c41b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=squid-d2c45d9, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T02:14:35.872 INFO:journalctl@ceph.mds.i.smithi193.stdout:Apr 25 02:14:35 smithi193 bash[72259]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-i 2024-04-25T02:14:35.999 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T02:14:36.031 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:36.031 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T02:14:36.031 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T02:14:36.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.l 2024-04-25T02:14:36.260 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 02:14:36 smithi193 systemd[1]: Stopping Ceph mds.l for 145ca8d0-02a5-11ef-bc93-c7b262605968... 2024-04-25T02:14:36.760 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 02:14:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-l[63060]: 2024-04-25T02:14:36.345+0000 7f3890b45700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T02:14:36.760 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 02:14:36 smithi193 ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-l[63060]: 2024-04-25T02:14:36.345+0000 7f3890b45700 -1 mds.l *** got signal Terminated *** 2024-04-25T02:14:40.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 02:14:40 smithi193 podman[72380]: 2024-04-25 02:14:40.352335201 +0000 UTC m=+4.110054718 container died e5133fdb413beaf25aa2b8e090f8ba11f72ac8b13e58f136d6d22db49ad5d544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-d2c45d9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T02:14:41.010 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 02:14:40 smithi193 podman[72380]: 2024-04-25 02:14:40.803358928 +0000 UTC m=+4.561078493 container cleanup e5133fdb413beaf25aa2b8e090f8ba11f72ac8b13e58f136d6d22db49ad5d544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a7481a6d0dca93e16a29c8acf1c964b318bd33c455aa5b4e4025b7a00d008901, name=ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-l, RELEASE=squid-d2c45d9, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T02:14:41.010 INFO:journalctl@ceph.mds.l.smithi193.stdout:Apr 25 02:14:40 smithi193 bash[72380]: ceph-145ca8d0-02a5-11ef-bc93-c7b262605968-mds-l 2024-04-25T02:14:41.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-145ca8d0-02a5-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T02:14:41.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T02:14:41.268 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T02:14:41.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T02:14:41.585 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T02:14:46.514 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T02:14:46.704 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T02:14:49.088 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T02:14:49.277 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T02:14:51.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:14:51.698 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:14:51.729 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T02:14:51.755 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T02:14:51.757 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/remote/smithi102/crash 2024-04-25T02:14:51.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/crash -- . 2024-04-25T02:14:51.879 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T02:14:51.879 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2024-04-25T02:14:51.885 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/remote/smithi110/crash 2024-04-25T02:14:51.886 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/crash -- . 2024-04-25T02:14:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:tar: /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T02:14:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:tar: Error is not recoverable: exiting now 2024-04-25T02:14:51.929 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/remote/smithi193/crash 2024-04-25T02:14:51.930 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/crash -- . 2024-04-25T02:14:51.959 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T02:14:51.959 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-04-25T02:14:51.960 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T02:14:51.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T02:14:52.083 INFO:tasks.cephadm:Compressing logs... 2024-04-25T02:14:52.083 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T02:14:52.088 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T02:14:52.090 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T02:14:52.118 INFO:teuthology.orchestra.run.smithi193.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T02:14:52.118 INFO:teuthology.orchestra.run.smithi193.stderr:: No such file or directory 2024-04-25T02:14:52.118 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log 2024-04-25T02:14:52.118 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T02:14:52.118 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T02:14:52.118 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T02:14:52.118 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T02:14:52.118 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.f.log: /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T02:14:52.127 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- 92.4% /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T02:14:52.127 INFO:teuthology.orchestra.run.smithi193.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T02:14:52.127 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T02:14:52.127 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mon.c.log: -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T02:14:52.127 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mgr.z.log: 86.7% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T02:14:52.127 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose 90.3% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T02:14:52.127 INFO:teuthology.orchestra.run.smithi193.stderr: -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.5% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:: No such file or directory 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.7.log: -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T02:14:52.128 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T02:14:52.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T02:14:52.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T02:14:52.129 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.2% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T02:14:52.129 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T02:14:52.129 INFO:teuthology.orchestra.run.smithi110.stderr: 86.7% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log.gz/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.10.log: 2024-04-25T02:14:52.130 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.11.log: 92.3% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T02:14:52.130 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T02:14:52.131 INFO:teuthology.orchestra.run.smithi110.stderr: 92.2% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T02:14:52.132 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T02:14:52.132 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T02:14:52.132 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.e.log: 83.5% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T02:14:52.132 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T02:14:52.138 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T02:14:52.144 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T02:14:52.146 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T02:14:52.147 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T02:14:52.147 INFO:teuthology.orchestra.run.smithi102.stderr:: No such file or directory 2024-04-25T02:14:52.148 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log 2024-04-25T02:14:52.149 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T02:14:52.151 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T02:14:52.152 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T02:14:52.153 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T02:14:52.154 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T02:14:52.154 INFO:teuthology.orchestra.run.smithi102.stderr: 90.1% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T02:14:52.155 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T02:14:52.155 INFO:teuthology.orchestra.run.smithi102.stderr: 86.7% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T02:14:52.156 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T02:14:52.157 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T02:14:52.159 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T02:14:52.159 INFO:teuthology.orchestra.run.smithi102.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T02:14:52.159 INFO:teuthology.orchestra.run.smithi102.stderr: 94.3% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T02:14:52.162 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mon.a.log: /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.6.log: 94.4% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T02:14:52.163 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T02:14:52.165 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T02:14:52.165 INFO:teuthology.orchestra.run.smithi193.stderr: 94.4% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T02:14:52.165 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T02:14:52.166 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T02:14:52.166 INFO:teuthology.orchestra.run.smithi102.stderr: 83.7% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T02:14:52.168 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T02:14:52.171 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T02:14:52.174 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.3% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T02:14:52.185 INFO:teuthology.orchestra.run.smithi102.stderr: 94.4% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T02:14:52.185 INFO:teuthology.orchestra.run.smithi102.stderr: 90.9% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T02:14:52.537 INFO:teuthology.orchestra.run.smithi110.stderr: 93.5% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T02:14:52.639 INFO:teuthology.orchestra.run.smithi193.stderr: 93.8% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T02:14:52.646 INFO:teuthology.orchestra.run.smithi193.stderr: 93.5% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T02:14:52.652 INFO:teuthology.orchestra.run.smithi102.stderr: 90.3% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T02:14:52.779 INFO:teuthology.orchestra.run.smithi110.stderr: 91.7% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T02:14:52.852 INFO:teuthology.orchestra.run.smithi193.stderr: 91.6% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T02:14:53.398 INFO:teuthology.orchestra.run.smithi102.stderr: 90.3% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T02:14:55.530 INFO:teuthology.orchestra.run.smithi193.stderr: 93.8% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T02:14:55.790 INFO:teuthology.orchestra.run.smithi110.stderr: 94.0% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T02:14:56.540 INFO:teuthology.orchestra.run.smithi110.stderr: 93.7% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T02:15:07.489 INFO:teuthology.orchestra.run.smithi102.stderr: 94.0% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T02:15:07.730 INFO:teuthology.orchestra.run.smithi110.stderr: 94.0% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T02:15:08.036 INFO:teuthology.orchestra.run.smithi193.stderr: 94.0% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T02:15:09.133 INFO:teuthology.orchestra.run.smithi110.stderr: 93.9% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T02:15:10.122 INFO:teuthology.orchestra.run.smithi102.stderr: 93.8% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T02:15:11.935 INFO:teuthology.orchestra.run.smithi110.stderr: 94.0% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T02:15:12.084 INFO:teuthology.orchestra.run.smithi193.stderr: 94.0% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T02:15:12.350 INFO:teuthology.orchestra.run.smithi110.stderr: 94.1% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T02:15:12.353 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-04-25T02:15:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m20.248s 2024-04-25T02:15:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:user 1m18.821s 2024-04-25T02:15:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m2.893s 2024-04-25T02:15:15.073 INFO:teuthology.orchestra.run.smithi193.stderr: 93.7% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T02:15:15.484 INFO:teuthology.orchestra.run.smithi193.stderr: 93.8% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T02:15:15.485 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-04-25T02:15:15.486 INFO:teuthology.orchestra.run.smithi193.stderr:real 0m23.381s 2024-04-25T02:15:15.486 INFO:teuthology.orchestra.run.smithi193.stderr:user 1m23.821s 2024-04-25T02:15:15.486 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m3.165s 2024-04-25T02:15:18.817 INFO:teuthology.orchestra.run.smithi102.stderr: 93.9% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T02:15:21.000 INFO:teuthology.orchestra.run.smithi102.stderr: 94.0% -- replaced with /var/log/ceph/145ca8d0-02a5-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T02:15:21.005 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-04-25T02:15:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m28.884s 2024-04-25T02:15:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:user 1m23.539s 2024-04-25T02:15:21.005 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m7.105s 2024-04-25T02:15:21.006 INFO:tasks.cephadm:Archiving logs... 2024-04-25T02:15:21.006 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/remote/smithi102/log 2024-04-25T02:15:21.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T02:15:33.514 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/remote/smithi110/log 2024-04-25T02:15:33.514 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T02:15:44.143 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/remote/smithi193/log 2024-04-25T02:15:44.144 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T02:15:56.376 INFO:tasks.cephadm:Removing cluster... 2024-04-25T02:15:56.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 --force 2024-04-25T02:15:56.687 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T02:15:57.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 --force 2024-04-25T02:15:57.688 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T02:15:58.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 145ca8d0-02a5-11ef-bc93-c7b262605968 --force 2024-04-25T02:15:58.268 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: 145ca8d0-02a5-11ef-bc93-c7b262605968 2024-04-25T02:15:58.640 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T02:15:58.640 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T02:15:58.694 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T02:15:58.710 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T02:15:58.726 INFO:tasks.cephadm:Teardown complete 2024-04-25T02:15:58.726 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T02:15:58.738 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T02:15:58.738 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T02:15:58.745 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T02:15:58.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T02:15:58.887 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T02:15:58.898 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T02:15:58.898 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T02:15:58.906 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T02:15:58.908 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T02:15:58.921 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-04-25T02:15:58.922 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-04-25T02:15:58.925 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T02:15:58.925 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-04-25T02:15:58.925 INFO:teuthology.orchestra.run.smithi110.stdout:^- hv01.front.sepia.ceph.com 2 6 377 51 +1761us[+1761us] +/- 590ms 2024-04-25T02:15:58.925 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -272us[ -243us] +/- 62ms 2024-04-25T02:15:58.925 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +354us[ +354us] +/- 69ms 2024-04-25T02:15:58.925 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T02:15:58.926 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T02:15:58.927 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-04-25T02:15:58.927 INFO:teuthology.orchestra.run.smithi193.stdout:^- hv01.front.sepia.ceph.com 2 6 377 52 +1776us[+1782us] +/- 590ms 2024-04-25T02:15:58.927 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -264us[ -258us] +/- 62ms 2024-04-25T02:15:58.927 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +368us[ +368us] +/- 69ms 2024-04-25T02:15:58.927 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T02:15:58.947 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-04-25T02:15:58.956 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T02:15:58.956 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-04-25T02:15:58.956 INFO:teuthology.orchestra.run.smithi102.stdout:^- hv01.front.sepia.ceph.com 2 6 377 53 +1741us[+1769us] +/- 590ms 2024-04-25T02:15:58.956 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -287us[ -259us] +/- 62ms 2024-04-25T02:15:58.956 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +331us[ +358us] +/- 69ms 2024-04-25T02:15:58.956 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T02:15:58.957 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T02:15:58.968 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T02:15:58.970 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T02:15:58.981 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T02:15:59.188 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T02:15:59.256 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T02:15:59.321 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T02:15:59.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:15:59.417 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T02:15:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:15:59.447 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T02:15:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:15:59.474 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T02:15:59.486 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T02:15:59.502 INFO:teuthology.task.internal:Duration was 2429.977291 seconds 2024-04-25T02:15:59.502 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T02:15:59.513 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T02:15:59.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T02:15:59.519 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T02:15:59.522 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T02:15:59.558 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T02:15:59.559 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T02:15:59.656 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T02:16:00.047 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T02:16:00.047 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T02:16:00.047 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T02:16:00.122 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-04-25T02:16:00.123 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T02:16:00.149 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-04-25T02:16:00.149 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T02:16:00.176 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T02:16:00.177 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T02:16:00.184 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T02:16:00.192 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T02:16:00.267 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T02:16:00.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T02:16:00.273 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T02:16:00.275 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T02:16:00.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T02:16:00.784 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T02:16:00.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T02:16:00.867 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T02:16:00.891 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T02:16:00.915 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T02:16:00.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:16:00.940 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:16:00.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:16:00.961 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-04-25T02:16:00.978 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-04-25T02:16:01.030 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-04-25T02:16:01.070 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:16:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:16:01.116 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:16:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:16:01.130 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T02:16:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T02:16:01.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T02:16:01.157 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T02:16:01.158 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/remote/smithi102 2024-04-25T02:16:01.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T02:16:01.263 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/remote/smithi110 2024-04-25T02:16:01.263 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T02:16:01.298 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337/remote/smithi193 2024-04-25T02:16:01.299 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T02:16:01.334 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T02:16:01.334 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T02:16:01.342 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T02:16:01.344 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T02:16:01.393 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T02:16:01.408 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T02:16:01.408 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T02:16:01.424 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T02:16:01.425 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T02:16:01.431 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T02:16:01.434 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T02:16:01.446 INFO:teuthology.orchestra.run.smithi110.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 02:16 /home/ubuntu/cephtest 2024-04-25T02:16:01.448 INFO:teuthology.orchestra.run.smithi193.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 02:16 /home/ubuntu/cephtest 2024-04-25T02:16:01.479 INFO:teuthology.orchestra.run.smithi102.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 02:16 /home/ubuntu/cephtest 2024-04-25T02:16:01.482 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T02:16:01.497 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T02:16:01.526 INFO:teuthology.nuke:Checking targets against current locks 2024-04-25T02:16:01.552 DEBUG:teuthology.nuke:shortname: smithi102 2024-04-25T02:16:01.553 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T02:16:01.576 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:27:15.861733', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI80cQANYn6auBWie0SBScKiuFzz2PmEvH93W8Y3q9Y7kQRaX1BHdFKsP5WQRswML5W8CikniUufyDC6nWaa/+w='} 2024-04-25T02:16:01.579 DEBUG:teuthology.nuke:shortname: smithi110 2024-04-25T02:16:01.580 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T02:16:01.601 DEBUG:teuthology.nuke:shortname: smithi193 2024-04-25T02:16:01.602 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T02:16:01.607 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:27:15.864535', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJMoSnR1gKobgS4De0dX/OFqvNlR9inQTGyhB68FiOaRsGqt0GFifQg46k+8BsuFkoQzbZiEJHkLWaEyjXAwDo4='} 2024-04-25T02:16:01.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 01:27:15.863231', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBASZS/7rqPR1H3ItqxETJ0kvC81fX9Mm6etnf1mW6n1/+Z/IM03qWCnxnHPrRnRMF3A4xxKveguR+jT0D5SQQ8w='} 2024-04-25T02:16:01.639 INFO:teuthology.orchestra.console.smithi102:Power off 2024-04-25T02:16:01.639 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T02:16:01.664 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-04-25T02:16:01.664 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:16:01.680 INFO:teuthology.orchestra.console.smithi110:Power off 2024-04-25T02:16:01.681 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T02:16:01.694 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-04-25T02:16:01.703 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2024-04-25T02:16:01.703 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:16:01.720 INFO:teuthology.orchestra.console.smithi193:Power off 2024-04-25T02:16:01.720 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T02:16:01.729 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-04-25T02:16:01.742 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-04-25T02:16:01.743 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:16:01.766 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-04-25T02:16:05.695 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:16:05.731 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:16:05.767 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:16:05.806 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-04-25T02:16:05.841 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-04-25T02:16:05.878 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-04-25T02:16:09.811 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:16:09.842 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:16:09.878 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T02:16:09.921 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-04-25T02:16:09.953 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2024-04-25T02:16:09.989 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-04-25T02:16:10.023 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-04-25T02:16:10.054 INFO:teuthology.orchestra.console.smithi110:Power off completed 2024-04-25T02:16:10.090 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-04-25T02:16:10.195 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} duration: 2429.9772913455963 failure_reason: 'error during scrub thrashing: rank damage found: {''backtrace''}' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-04-25T02:16:10.195 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T02:16:10.276 INFO:teuthology.run:FAIL